Simple Packaging Tutorial: The Long Version
Last updated: Dec 23, 2020
This page is created to aid the attendees of the ഡെബിയന് പാക്കേജിങ്ങ് ബാലപാഠങ്ങള് പഠിയ്ക്കാം (Simple Packaging Tutorial with debmake) workshop held at the Malayalam MiniConf during DebConf 20 on 28th August 08:30 PM - 10:15 PM IST.
The workshop was taken by Pirate Praveen who is a Debian Developer and myself.
In the workshop we will be showing you guys how to make a simple Debian package. You could refer Simple Packaging Tutorial at the Debian Wiki. This page adheres to the structure of the workshop and provides addition information that is otherwise spread across multiple pages in the Debian Wiki.
Every command we execute in this tutorial will be done in the Unstable System.
Step 0 : Getting Organized
First Let’s get organized, let’s create a folder in our home directory dedicated for packaging. You can do so from the terminal itself by running the following command:
mkdir <directory name>
Where
Keeping things organized will help you later on when working with complex packages in the future. It will avoid confusion and help you stay on top of things without things getting overwhelming.
Now once we’ve created the said directory let’s let’s switch to it using the cd command
cd <directory name>
Packaging is mostly a straight forward ordeal. The process of packaging remains more or less the same regardless of whether it’s a JS or python or language x package. For the purpose of this tutorial we will be packaging a node module called pretty-ms.
Step 1: Acquire the source code.
To begin we need to first collect the source code of the piece of software that we intend to package which in our case is pretty-ms
. So let’s go to npmjs page for pretty-ms. Here in the right side of the page we can find the link for the repository and this is where we can find the source code. The source to pretty-ms
is hosted at Github here. What this package pretty, much does is convert time produced in milliseconds to more human understandable formats such as hours, minutes and seconds.
If we go through the files we can find the package.json
file which contains the metadata of the node module and README.md which contains documentation for the node nodule .The main files that is the actual module is the index.js file and we also have the test.js file which contains tests for the module to ensure that it works properly and the LICENSE file which contains the copyright information which will be useful for us later.
Now on Github we have Releases which contains various releases of the module. This can vary from one platform to another, most developers tag their releases so that is somewhere you can be sure to find releases if the platform that you encounter does not have a straight forward place to download releases from. Here the latest version of pretty-ms is 7.0.0. Let’s download the source code in tar.gz format. Apart from tar.gz, tar.bz2 and tar.xz are also used by developers. The difference between them is the algorithm used to compress the files. When you compare bz2 and gz you will find that bz2 has smaller file sizes but require more time to compress and decompress whereas gz has comparatively larger file sizes but is very fast in compressing and decompressing the files. It’s the developer’s choice in which one to use.
Now we can download from the browser itself but what’s the fun in that? Let’s copy the link to the file and use wget to download the tar file. So let’s first install wget in out schroot env and download the file using it.
sudo apt install wget
wget https://github.com/sindresorhus/pretty-ms/archive/v7.0.0.tar.gz
Package Name
Now we have the source code in our packaging folder, that is if we ran the wget command from that folder. Now Debian has specifications and formats for everything in a Debian package and there’s format for the source tarball of a Debian package too. This format is <packagename>_<version>.orig.<tarball format>
.
As far as the package name part goes if the piece of software we are packaging is something the users will run themselves for example Firefox or Rollup or Rails etc. we usually name the package the actual name of the package itself, if we take the above example their package names will remain firefox
, rollup
and rails
. On the otherhand if it is a library that the users don’t necessarily run themselves we use a different naming convention. The convention used is the system or programming language or framework the library was made for followed by a hyphen and then the package name sometimes the system or programming language or framework comes after the name of the library (whether it comes before or after the name of the package is a convention set by the team maintain the particular set of libraries.). If the module we intend to package is made for Node JS we add node
before the package name and if the package is relevant for JS in general we as libjs
before the package name. Again this is only applicable to libraries and not to be used for user facing executable programs. Since pretty-ms is a module made for Node JS, the name will be node-pretty-ms
. Another important thing to note is that we do not use underscores or uppercase characters in our package names, if your package contains an underscore convert those to hyphens and the upper case characters to lowercase characters when naming your package.
Now let’s complete the name of our tarball. So first comes the package name which is node-pretty-ms
next we add an underscore followed by the version of the package which is 7.0.0
and finally .orig.tar.gz
. So at the end the whole thing looks like node-pretty-ms_7.0.0.orig.tar.gz
.
Step 2: Debianization
Now let’s go ahead and extract the contents of our tarball with the following command:
tar -zxvf node-pretty-ms_7.0.0.orig.tar.gz
tar
is one of those commands that may seem usable at the beginning with -zxvf and everything but trust us in the long run using the command line will save you a lot of time once you get used to the command. If you guys have any doubts regarding any program in your distribution just type in man <package name>
to see the various options that package provides you.
Now let’s change the name of the directory according to the above mentioned naming convention except no underscore before the version number, instead of underscore for the directory we use a hyphen. The mv
command can be used to do this.
mv pretty-ms-7.0.0 node-pretty-ms-7.0.0
Now let’s change into that directory using the cd command. The contents of the folder should look something like this:
drwxr-xr-x 3 avronr avronr 4096 Apr 27 11:42 .
drwxr-xr-x 3 avronr avronr 4096 Aug 28 10:00 ..
-rw-r--r-- 1 avronr avronr 175 Apr 27 11:42 .editorconfig
-rw-r--r-- 1 avronr avronr 19 Apr 27 11:42 .gitattributes
drwxr-xr-x 2 avronr avronr 4096 Apr 27 11:42 .github
-rw-r--r-- 1 avronr avronr 23 Apr 27 11:42 .gitignore
-rw-r--r-- 1 avronr avronr 2948 Apr 27 11:42 index.d.ts
-rw-r--r-- 1 avronr avronr 3697 Apr 27 11:42 index.js
-rw-r--r-- 1 avronr avronr 833 Apr 27 11:42 index.test-d.ts
-rw-r--r-- 1 avronr avronr 1109 Apr 27 11:42 license
-rw-r--r-- 1 avronr avronr 19 Apr 27 11:42 .npmrc
-rw-r--r-- 1 avronr avronr 855 Apr 27 11:42 package.json
-rw-r--r-- 1 avronr avronr 3333 Apr 27 11:42 readme.md
-rw-r--r-- 1 avronr avronr 12608 Apr 27 11:42 test.js
-rw-r--r-- 1 avronr avronr 45 Apr 27 11:42 .travis.yml
If you’re file structure looks like this then you’re good. If not retrace your steps and find what went wrong. Next we begin the actual Debianization which is done using a tool called Debmake make the debian/
directory. so let’s install debmake
using sudo apt install debmake
and run debmake and see how the files structure looks after that.
- node-pretty-ms-7.0.0
- debian
- changelog
- compat
- control
- copyright
- patches
- README.Debian
- rules
- source
- watch
- .editorconfig
- .gitattributes
- .github
- funding.yml
- .gitignore
- index.d.ts
- index.js
- index.test-d.ts
- license
- .npmrc
- package.json
- readme.md
- test.js
- .travis.yml
- debian
Step 3: Creating a source package
Next let’s create a source package using dpkg-source -b .
. If you guys don’t have dpkg-source in your system you can get it by running sudo apt install build-essential
. The output of which should look like this:
❯ dpkg-source -b .
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building node-pretty-ms using existing ./node-pretty-ms_7.0.0.orig.tar.gz
dpkg-source: info: building node-pretty-ms in node-pretty-ms_7.0.0-1.debian.tar.xz
dpkg-source: info: building node-pretty-ms in node-pretty-ms_7.0.0-1.dsc
- If you look through this log you will see dpkg-source mentioning that it is using the quilt 3.0 source format.
- You will also notice that is using the .orig.tar.gz file that we created earlier
- It has build a tar of the debian directory too. But there is also something new here, the file is named
node-pretty-ms_7.0.0-1.debian.tar.xz
where you can see that there is a-1
after the version number. That number is called the debian revision number. Say we packaged a piece of software and uploaded it to the Debian archves and then someone reported a bug and we made and fix and want to release the package again in the archives with the fix. Now you see we have different ‘revisions’ of the same package with the same version number. So inorder to help keep a track of this we have debian revision number which is just a count of the number of times that particular version package was released into the Debian archives. So if we have made a bug fix in node-pretty-ms and release it again to the archives it will be releases asnode-pretty-ms_7.0.0-2
. - Lastly we have the
node-pretty-ms_7.0.0-1.dsc
file. This file is a security measure if you look into that file you file find information regarding the package extracted from the files in our debian directory and the check-sums of thenode-pretty-ms_7.0.0.orig.tar.gz
andnode-pretty-ms_7.0.0-1.debian.tar.xz
. This is so that others can check the integrity of these files once they have downloaded it from the archives after a Debian Developer uploads it there.
These three files together form the Debian Source Package.
The .dsc file
Let’s look at the contents of the .dsc file real quick.
Format: 3.0 (quilt)
Source: node-pretty-ms
Binary: node-pretty-ms
Architecture: any
Version: 7.0.0-1
Maintainer: Abraham Raji <[email protected]>
Homepage: <insert the upstream URL, if relevant>
Standards-Version: 4.1.4
Build-Depends: debhelper (>= 11~)
Package-List:
node-pretty-ms deb unknown optional arch=any
Checksums-Sha1:
28bb69ab4720c07dcdd73520b1305137ae354f90 6142 node-pretty-ms_7.0.0.orig.tar.gz
329139a95e273617ecfdb41466325cede4fa983d 2004 node-pretty-ms_7.0.0-1.debian.tar.xz
Checksums-Sha256:
256871d7b49dc76e33d841e46c5d36008858aceea9081d9e62c7f5760e65ea33 6142 node-pretty-ms_7.0.0.orig.tar.gz
3e28122e5bbcb1c771c7431b9af90bc74da45c0461be3a6bc8bcadd73e930707 2004 node-pretty-ms_7.0.0-1.debian.tar.xz
Files:
ab6e9b3155d0cd73b54d4f0ba8dd0774 6142 node-pretty-ms_7.0.0.orig.tar.gz
1d7cf58aef1718817cece400cb978ad9 2004 node-pretty-ms_7.0.0-1.debian.tar.xz
We can see various information such as the format used which as we saw before is quilt 3.0. The next two fields are interesting, we mentioned at the beginning of this workshop about names of user facing software vs libraries that comes into play here, the name of the source package vs the name of the binary which the users will use to install this package and such. Next we have Architecture which states what all architectures you can install this package on, here since it’s a JS package it can be installed on all architectures and that field should actually say all, but debmake hasn’t learnt to detect that yet so it put any there, we’ll fix that later. Then we have the version followed by the information about the person working on the package (which debmake got from my .zshrc which is where I’ve put it and if you use bash you can put it in your .bashrc and Debmake will pick that up). Next we have the Homepage feild where we can put the link to the homepage of the package. Next is Standards-Version which tells us which Debian policy the package follows and set to the version number of the package debian-policy
, this is also a bit outdated and we will fix it a little later. The next field is Build-Depends which specifies the packages needed to build executable files from the source. Next we have the checksums we mentioned above in three formats, Sha1, Sha256 and MD5. So that’s the DSC file.
Step 4: Building, Making Fixes and Satisfying Lintian
Building
Now that we have the source package we can build it. We need to be in the node-pretty-ms-7.0.0
folder to build it. Inside the folder we will execute the dpkg-buildpackage
command which will build the package for us. The log of that command will look something like this:
❯ dpkg-buildpackage
dpkg-buildpackage: info: source package node-pretty-ms
dpkg-buildpackage: info: source version 7.0.0-1
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by Abraham Raji <[email protected]>
dpkg-buildpackage: info: host architecture amd64
dpkg-source --before-build .
fakeroot debian/rules clean
dh clean
dh_clean
dpkg-source -b .
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building node-pretty-ms using existing ./node-pretty-ms_7.0.0.orig.tar.gz
dpkg-source: info: building node-pretty-ms in node-pretty-ms_7.0.0-1.debian.tar.xz
dpkg-source: info: building node-pretty-ms in node-pretty-ms_7.0.0-1.dsc
debian/rules build
dh build
dh_update_autotools_config
dh_autoreconf
create-stamp debian/debhelper-build-stamp
fakeroot debian/rules binary
dh binary
dh_testroot
dh_prep
dh_installdocs
dh_installchangelogs
dh_perl
dh_link
dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
dh_strip
dh_makeshlibs
dh_shlibdeps
dh_installdeb
dh_gencontrol
dpkg-gencontrol: warning: Depends field of package node-pretty-ms: substitution variable ${shlibs:Depends} used, but is not defined
dh_md5sums
dh_builddeb
dpkg-deb: building package 'node-pretty-ms' in '../node-pretty-ms_7.0.0-1_amd64.deb'.
dpkg-genbuildinfo
dpkg-genchanges >../node-pretty-ms_7.0.0-1_amd64.changes
dpkg-genchanges: info: including full source code in upload
dpkg-source --after-build .
dpkg-buildpackage: info: full upload (original source is included)
dpkg-buildpackage: warning: not signing UNRELEASED build; use --force-sign to override
Since it is a simple package it will build successfully. Also, if we look at the parent directory we have a few new files:
❯ ls ..
node-pretty-ms-7.0.0 node-pretty-ms_7.0.0-1.debian.tar.xz
node-pretty-ms_7.0.0-1_amd64.buildinfo node-pretty-ms_7.0.0-1.dsc
node-pretty-ms_7.0.0-1_amd64.changes node-pretty-ms_7.0.0.orig.tar.gz
node-pretty-ms_7.0.0-1_amd64.deb
Most notable of the new files are the .buildinfo, .changes and .deb files.
- .buildinfo gives us the info regarding the packages that were in our distribution at the time of building.
- .changes gives us a lot of information that you could also find in the .dsc file but also contents of the debian/changelog file.
- the .deb file is something we’re all familiar with and is what we use to install the package in our system.
Now if we install the generated .deb file and run the command dpkg -L node-pretty-ms
which essentially gives us all the files associated with that package in our system you will see that the important files such as the index.js or package.json was not installed.
So even though we have a source package that builds and installs properly we haven’t been able to install the package to our system yet. Let’s start fixing this package.
Making fixes
- First let’s fix the architecture which can be fixed by editing our debian/control file. In the control file has a section for Architecture which currently is set to any and we will set that to all. Let’s open a text editor of choice and make the necessary change. This is probably how it will look at first.
Source: node-pretty-ms
Section: unknown
Priority: optional
Maintainer: Abraham Raji <[email protected]>
Build-Depends: debhelper (>=11~)
Standards-Version: 4.1.4
Homepage: <insert the upstream URL, if relevant>
Package: node-pretty-ms
Architecture: any
Multi-Arch: foreign
Depends: ${misc:Depends}, ${shlibs:Depends}
Description: auto-generated package by debmake
This Debian binary package was auto-generated by the
debmake(1) command provided by the debmake package.
There are a few things we can fix here, namely:
- Section: this is a JS package so set it to javascript.
- Standards-Version: which is also using an old version, so set it the new one which is 4.5.0
- We could also add the homepage link
- The Architecture of course
- Add a description too. You can find this from the package’s homepage or README.md.
Once you’ve made these changes the file should look something like this:
Source: node-pretty-ms
Section: javascript
Priority: optional
Maintainer: Abraham Raji <[email protected]>
Build-Depends: debhelper (>=11~)
Standards-Version: 4.5.0
Homepage:https://github.com/sindresorhus/pretty-ms#readme
Package: node-pretty-ms
Architecture: all
Multi-Arch: foreign
Depends: ${misc:Depends}, ${shlibs:Depends}
Description: Convert milliseconds to a human readable string
- Next let’s tell Debian what all files to install. We can do so in a file inside the debian directory called install. In this case we want to install the
index.js
,index.d.ts
and thepackage.json
file and in debian we install it to `/usr/share/nodejs// (one thing to note here is that we don’t use the debian name but the actual name of the package for the folder). So open the debian/install file with your favorite text editor and add the required files. Once you’ve done that the debian/install file should look something like this:
❯ cat debian/install
index.js usr/share/nodejs/pretty-ms
index.d.ts usr/share/nodejs/pretty-ms
package.json usr/share/nodejs/pretty-ms
Let’s build again using dpkg-buildpackage
. If you installed the previous .deb file you should remove it before you install the new one we just generated.
Now if we install the new .deb file and run the command dpkg -L node-pretty-ms
, we can see that all the required files were installed properly.
- Next let’s fix the copyright file. We can find the missing information in that file from the upstream
license
file. Right now it will look something like this:
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: node-pretty-ms
Source: <url://example.com>
#
# Please double check copyright with the licensecheck(1) command.
Files: .editorconfig
.gitattributes
.github/funding.yml
.gitignore
.npmrc
.travis.yml
index.d.ts
index.js
index.test-d.ts
package.json
readme.md
test.js
Copyright: __NO_COPYRIGHT_NOR_LICENSE__
License: __NO_COPYRIGHT_NOR_LICENSE__
Files: license
Copyright: Sindre Sorhus <[email protected]> (sindresorhus.com)
License: Expat
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
.
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#----------------------------------------------------------------------------
# Files marked as NO_LICENSE_TEXT_FOUND may be covered by the following
# license/copyright files.
#----------------------------------------------------------------------------
# License file: license
MIT License
.
Copyright (c) Sindre Sorhus <[email protected]> (sindresorhus.com)
.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Softw
...
Make sensible changes and make a single line should not have more than 80 characters, you can achieve this using the macro functionality in your text editor. Once you’re done it should look something like this.
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: node-pretty-ms
Source: https://github.com/sindresorhus/pretty-ms/releases
Files: *
Copyright: Sindre Sorhus <[email protected]> (sindresorhus.com)
License: Expat
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/ or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions: .
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Satisfying Lintian
Now run the dpkg-buildpackage
again so that the changes we made takes effect. As of now we have resolved all the basic issues. Lintian will help us find the ones that we missed, lintian by default will give short messages but detailed descriptions will help us solve these issues. in order to do that let’s set an alias in our shell’s rc file (~/.bashr or ~/.zshrc). So pul the following lines in your .bashrc or .zshrc
alias lintian='lintian -iIEcv --pedantic --color auto'
Now run lintian
and it will throw a bunch of complaints on your face. We need to pay attention to the lines that start with E: or W:. Let’s fix an E which complains that changelog-is-dh_make-template
and if you read through the description you’ll see that the issue is in the debian/changelog. Let’s take a look at the file:
node-pretty-ms (7.0.0-1) UNRELEASED; urgency=low
* Initial release. Closes: #nnnn
<nnnn is the bug number of your ITP>
-- Abraham Raji <[email protected]> Fri, 28 Aug 2020 16:05:03 +0530
The issue is that when we wish to package a new package we need to send an Intend to Package mail which is registered as an ITP Bug. This helps co-ordinate packaging work and prevent duplicate work. The bug has a unique identification number. Which we put in the changelog after the hashtag in * Initial release. Closes: #nnnn
. So we need to check for existing bugs before we make a new ITP bug, we could search for existing bugs at wnpp.debian.net. Now we can solve this error by removing the line <nnnn is the bug number of your ITP>
and add the ITP bug number. For the sake of this workshop we’re going to add a random number there. Once we do that the file should look something like this:
node-pretty-ms (7.0.0-1) UNRELEASED; urgency=low
* Initial release. Closes: #982937
-- Abraham Raji <[email protected]> Fri, 28 Aug 2020 16:05:03 +0530
Now we still have a lot more complaints to solve but to keep the size of this page somewhat reasonable, I’m not going to include fixes to all the complaints here. Here’s an example from a package I updated a while ago. It is sort of a jackpot in the sense it has a bunch of fixes to common lintian complaints I’ve come across. Commit messages could be a little better and more creative though.
Now we did a lot of things manually. We actually have a tool that does most of what we already did called npm2deb.
Here’s a wiki page giving a tutorial on how to use npm2deb.
Here’s a wiki page I wrote regarding updating packages.