Edit

Introduction to packaging step-by-step example

Instaloader

Instaloader is a Python 3 application with a single dependency (Python’s requests). This makes it a relatively simple package, however not as straightforward as only packaging up a shell script would be. Because of the learning opportunities and simplicity, this makes it a good introduction package.

Instaloader Code Overview

The first thing we do is look at the application’s GitHub page. A few things stand out which we take a note of:

What we notice here is some information that will come in handy later:

  • The tool contains a setup.py script

  • It has a release

  • The license is MIT based

We’ll be digging into each of these more later, for now it is just information to know.

Setting Up The Environment

We will assume that we have already followed our documentation on setting up a packing environment.

Let’s set up our directories now for this package:

:~$ mkdir -p ~/kali/packages/instaloader/ ~/kali/upstream/
:~$

Everything that relates to us building a package will be using ~/kali/. In there will be two sub folders:

  • packages/ will be a source code of the package we are going to create

  • upstream/ will be a compressed file of the source code of the application (ideally from a tag version release which we saw before)

Downloading Tag Releases

Because we are making a new package from scratch, we’ll manually download the version of the tool we want to package up. If we were updating a package (and it was packaged correctly), there is a process to help speed it up. However, this will be covered in another guide.

Going to the GitHub’s release page, we can see the latest version (which at the time of writing is 4.4.4). Here is the option to download instaloader-v4.4.4-windows-standalone.zip, as well as Source Code (zip), and Source Code (tar.gz). We are interested in the tar.gz option.

We will use wget and make sure to format its name appropriately according to Debian’s standards for source packages (take note of .orig.tar.gz):

If there isn’t a tag release for the software (or it hasn’t had an release in some time), we can use the latest git commit. This is covered in another guide. However, it is preferred to use a tag release when available.

Creating Package Source Code

We need to switch paths to the working location of the package:

We are now going to create a new blank git repository:

If we wanted to, we can confirm this by looking at “status” and “log”:

Great. Everything is empty; we have a clean working area.

We can now import the upstream version into our packing source code by using the file downloaded from wget before. Because of the filename format, gbp is able to detect the values instaloader as the package name, and 4.4.4 as the version. We just press enter to accept the default values:

If we wanted to check everything is okay, once again, we can use git to do so:

So there is now an automatic commit created in the master branch (which is the current active branch, shown by the *), as well as two other branches:

  • pristine-tar which is metadata from the import

  • upstream which is the source code of the application, without any of our package modifications

We are creating a Kali package, and we don’t use the master branch, but rather kali/master. So let’s switch:

Now we can generate the necessary files required to build a Debain-based package and also remove any example files created. During the process, we will be asked if its:

  • Single binary

  • Arch-Independent

  • Library

  • Python

We are going to keep it simple, and go with “Single”. Then accept what’s on the screen with Y. If you would like more information about when to use what option, please see the manpage for dh_make.:

We use --file to say where the orig.tar.gz file is. If the file was one directory back (../), this would not be needed, however as we have created a separate location for the file it is.

If you would like to see what got generated when using dh_make, we can use git:

A quick overview of each of those files:

  • changelog - tracks when the package gets an update (including why and by who). This is responsible for the package version

  • control - is the metadata for the package (often seen with apt)

  • copyright - what is under what license. The package can be under something different to the work we have put in to create the package

  • rules - how to install the package

  • source/format - is the source package format

At this point, we have the base packaging files in place, and it feels like a good idea to commit before starting some real work:

We now need to edit most of these to make sure the information is accurate. We can use what we found on GitHub to supply the correct info into the debian/ files:

  • License

  • Dependencies

  • Maintainers

  • Description

Collecting InformationLicense/Maintainers

For this package, its straight forward. GitHub has given us a helping hand, and detected the license as MIT. We can also see there is a license file:

Reading the license, we can see there are two authors which are given credit too: Alexander Graf and André Koch-Kramer. However, we don’t have a method of contact for them. We continue to explore the rest of the git repository, looking for something which may give us more authors so we can give credit to them. There isn’t a fixed structure in place, however there are some things to check and look out for:

  • README* - authors may put contact information here

    • A few examples could be: README, README.txt, README.MD README.MKDOCS, or Readme.txt

  • AUTHOR* - They may have a dedicated file for author information

  • CREDIT* - They may have a dedicated file to who they give credit to

  • LICENSE* - Like mentioned above, the license file may give author information

  • docs/ - They may place all their documentation in a separate folder

  • The “main” starting point of the application may have comments at the top of the file - in this, instaloader.py

  • Git commits - git --no-pager log -s --format="%ae" | sort -u

For our package, we can see:

As it turns out, there is: AUTHORS.md, docs/, instaloader.py, and README.rst, so we have a few places to look at. Starting with AUTHORS.md, we can see the authors name and their method of contact:

So rather than an email address, it appears to be a username (could be just for GitHub, or a generic Internet handle). This is enough for us to go forward (even though its not ideal).

Another trick we could try is looking to see if they used a “legit” email address with git:

It doesn’t appear so. Was worth a try!

Dependencies/Maintainers

We need to see what is required to be installed on the machine in order for the application to work. Either pre-installed or will be installed using the application.

Some starting places to look at for this information:

  • README*

  • SETUP*

  • INSTALL*

  • docs/

There is a README for this application, but it just says how to install the application, rather than how to build it/compile from source:

Exploring the pip option is something we could do, but out of scope for this guide.

Next we spot setup.py, which contains a lot of useful information:

We managed to get the following information from this:

  • From the shebang, we can see its Python 3 (#!/usr/bin/env python3).

  • We can see it wants Python 3.5 or higher

  • We can see it wants requests and for it to be v2.4 or higher

  • We can see if its on Windows, it requires another dependency, but we are Linux, so not the case

  • We can see the program’s home URL

  • We can see the license (MIT)

  • We can see the authors and their email addresses

  • We can get a description of the program

Handy!

When packing, we are building a standalone package, which needs to be able to install offline. Something else which needs to be kept in mind, other systems package management systems, such as Python’s pip, or ruby’s gems. Any of these dependencies also need to be in the main OS package management. In our case, we need Python’s requests.

We have two ways of searching for it. We can use either:

But we also need to know what we are searching for. There is a naming convention, but if you are un-sure, doing multiple searches may help:

  • requests

  • python-requests

  • python3-requests

We will stick with the command line option for the time being.

Doing just requests gives a little too many results:

So we need to do better to shorten the list, by just searching the short version of the description (we will cover this more later, but its the visible part of the output):

After removing the documentation from the results, we don’t get any results. So on with the next search!:

The first result, python3-requests, looks exactly right! We can look closer:

And we can see its version is 2.23.0, which is higher than than 2.4, so we don’t need to update the package. This will be covered in another guide when required.

Maintainers

While doing the other parts, we have discovered the authors and maintainers of the software, so we don’t need to do anything extra for this.

Description

There are two descriptions that we need to supply, a long description and a short description. When we look at the GitHub page we can see an about section that we can use for the short description. For the long description, we can use the description in the README.

We also have a value from the setup.py.

Editing Package Source Code

Now that we have that information copied down, we can start to populate the files in the debian/ folder we created with dh_make.

More information on the subject can be found on the Debian documentation.

Changelog

If we followed the documentation on setting up a packaging environment, the only values we will need to alter would be distribution (from UNRELEASED to kali-dev), version (from 4.4.4-1 to 4.4.4-0kali1) and the log entry:

Control

This file is the metadata for the package, and contains a lot of information.

More information on the subject can be found on the Debian documentation.

Out of the box, it will look a little like this:

So we can see a few things that need updating:

  • Section - we set this to be misc, or if we know for sure it should be another section based off of the sections in Debian testing we can set it to that section

  • Maintainer - we switch to be the Kali team, rather than an individual

  • Uploaders - this is the individual(s) who are responsible for packaging up the application

  • Build-Depends - what packages are required to BUILD the package

  • Homepage - where is the tool located on the Internet

  • Vcs-Browser - package source code to view online

  • Vcs-Git - package source code location

  • Architecture - what machines can this work on

  • Depends - what other packages are required for this package to work

  • Description - short and long description

Most of this we have now figured out from before, so it should make it easier to fill in. We went ahead and created a remote empty git repository on our GitLab account. In our example, this is the end result:

NOTE: The Build-Depends & Depends are indented with one space (and end with commas). The Description is also indentend with one space.

There is a lot going on here, so lets point out a few things

Something to keep in mind with the formatting of the long descriptions, at about every 70 characters in (to the nearest whole word), we would put a new line, to help keep the formatting under control.

Now onto the dependencies, of which we have: Build & Package. For the build-dependencies of Python 3 we will have to have four things:

  • debhelper-compat

  • dh-python

  • python3-all

  • python3-setuptools

In a separate guide there will be an explanation as to why these are included, however only the first two are going to be a staple of Python 3 packaging as the latter two are for more specific cases.

In our application, we have another one, python3-requests, which we got from setup.py and that is a requirement from the application. Typically, if there was not a setup.py file, we would not need to include python3-requests in our “Build-Depends”. However, due to the setup.py file, we will need to include python3-requests in both the “Build-Depends” as well as the package “Depends”. This ensures these packages are always on the system when we install our package (especially handy when using “sbuild”).

The debhelper-compat level determines how the package will be built. The higher the compat level, the newer the version. Newer versions have certain menial tasks done automatically, so this should not be lowered.

The package dependencies are relatively straightforward. We get rid of the ${shlibs:Depends} as we are packaging up a Python tool, and instead replace it with the python3 depends version ${python3:Depends}. We also ensure that python3-requests is included as the tool requires this. No other dependencies are needed by this tool, so we are done.

The final thing we need to ensure we change is the architecture from any to all, as this tool can be installed on all architectures.

Copyright

Everything that gets created has an original author. They control what happens with it and it needs to be respected. We can call out this in the copyright file.

More information on the subject can be found on the Debian documentation and here.

Below is the skeleton template output (with comments removed):

The original tool’s author has ownership on their work, and the work we have put into creating the package belongs to us. After updating it, it looks like the following:

We altered the following:

  • We removed an optional parameter (Upstream-Contact), as that is touched on in the copyright file.

  • We put in the homepage of the application to Source

  • Put the two authors name and addresses from setup.py. The dates came from the LICENSE file

  • Rather than putting the whole block of MIT license text directly after, we placed it towards the end of the file, and gave a header to it.

  • We replaced the GPL-2+ used as default for the packaging section with the same MIT license, which is used in the application. This is the standard for Debian packages (packaging work should match the application’s license).

Rules

This file is a Makefile, for building the Debian package.

More information on the subject can be found on the Debian documentation.

The output of the template looks like the following:

So there are a lot of items which are pre-commented out, that may be handy for debugging & troubleshooting. Other than the shebang (#!/usr/bin/make -f), there is only two other lines which are currently in use:

Which is a wildcard (%), and feed in all the arguments into dh.

What needs to go here now starts to depend on the program and how complex it is. As our program is a python application we are going to have to tell it to build with python3. We also need to tell it to use pybuild to build, as we have a setup.py file included in the source of the application. If there was not a setup.py file, we would not add this flag. We also need to tell PyBuild the name of the application. This looks like:

NOTE: It uses TAB for indentation, as its a Makefile.

Watch

An additional file which we highly recommend to include is a watch file. This points to upstream, and is then used to detect if there is a more recent version of the application than what is packaged. This is useful when doing updates to packages.

For more information, and example formats, please see the Debian wiki. Using this wiki, we can see there is an example for GitHub which is where our project is stored - github.com/instaloader/instaloader/:

So lets now alter it to fit our needs:

NOTE: This has two spaces for any indentation

So let’s do a quick check to see if its working right:

Looks like its not! Its correctly detected all the versions, but its not sorted the order correctly (due to the release candidate). We know this by going to the release page:

Looking back at the Debian wiki, there is a section called Common mistakes:

Not mangling upstream versions that are alphas, betas or release candidates to make them sort before the final release. The solution is to use “uversionmangle” like this:

However, we need to edit it a bit to fit Instaloader. This can be figured out through trial and error using the above uversionmangle as the base.

Let’s see how it works:

Success!

.Install & Helper-Scripts

Everything we have done so far would just be for building the package, but we haven’t said how to install the application:

NOTE: There is no leading slash in the target directory

We can go forward with this, but it may not behave like we were expecting. This is because we don’t have anything in $PATH, so if we went to the command line and tried typing in instaloader.py its not going to work (also it has the file extension, .py). The solution is to create a helper-script, which is placed into $PATH (and we include in the .install file):

With that, all the necessary debian/ files are added. Time to build!

Packing Up

Time to bundle everything into a file. We are going to use sbuild to create the package. This has its pros and cons. One of the pros is that if it builds here, it will also build elsewhere as its meant for build daemons. The down side is, it will require access to a network repository as it will try and handle detecting and installing any dependencies missing in the chroot, making it slower to build.

If you don’t want to use sbuild, just drop it from the arguments (e.g. gbp buildpackage), However, you will be required then to install what’s in debian/control in the Build-Depends section (e..g sudo apt install -y dh-python python3-all python3-setuptools python3-requests).

So lets give sbuild a try:

Oops! We haven’t committed our changes to git. Shame on us.

If we wanted to, we could bypass this by doing gbp buildpackage --git-builder=sbuild --git-export=WC, which would allow us to test out our values in debian/ before committing to it, rather than cluttering up the git history with various debugging/troubleshooting commits. Then when we have our package in a working state, we can then commit to git, and try again, like so:

Let’s try and build again:

NOTE: You may not get the following error (as it depends on how “clean” your OS is):

If you see the above error, this is because dh-python is missing from our OS. We can quickly fix this by doing:

So, one more try and building:

The output here is very long, so we have truncated it, but we can see its being built successfully. Even with an error, warning, and information from lintian!

Let’s double check to see what got created:

We have output!

Making Lintian Happy

For more information, see Debian’s documentation.

Let’s try to understand the error E: instaloader source: source-is-missing [docs/_static/bootstrap-4.1.3.bundle.min.js]:

The issue with the file docs/_static/bootstrap-4.1.3.bundle.min.js is that it’s a minified Javascript. It’s not human readable, it can’t be modified, therefore it’s not considered as a source file. As Lintian suggests, we can provide the source for this file in debian/missing-sources, however as we do not have the source handy we should explore other options. For this guide we will focus on Lintian overrides. Because the offending file is in the docs directory, which, if we investigate the included files, is what Instaloader uses to host their documentation site pages, we can ignore this file and tell Lintian to as well.

To do this we will create the file instaloader.lintian-overrides located in the debian directory. From here we can copy and paste the error message from the : on. While we are here, we may as well ignore the warning about no-manual-page. Here is our resulting file:

We can now commit our changes and rebuild the package with the same command and see if it was successful with no error:

Uh oh! It looks like it still failed, and that it didn’t even use our override for the source-is-missing error! If we look closer, we can see a difference between the previous warning we were getting about no-manual-page and source-is-missing. The section before the :, telling us the level of error and the package name, includes source in our source-is-missing error. This is because the issue resides in the source package, or the imported package, rather than the output, or the binary package. To solve this we will need to create a new source directory in debian/ and a new lintian-overrides file. Lets do that now:

Don’t forget to remove the override from our previous file as well!

We can once more commit our changes and rebuild the package and see it was successful with no error:

Manual Install

Let’s now give our package a test drive:

Success!

This is looking good (not perfect), and eagle eye spotters may be able to spot why (in the output) - there is the file extension in the output (instaloader.py), yet the command used to call it doesn’t have it (instaloader). We are going to need to either patch the application or find a different way to call the application to address this. But this will be covered in another guide.

Let’s now make sure everything is in git locally, before we push it out to the remote repository (the one we defined in debian/control):

We don’t (yet) have a remote repository setup, so we go to GitLab and create a new project before continuing.

Afterwards:

Now we need to send our local work to the new remote repository (and don’t forget the tags):

At this point now, a ticket can be opened up on the Kali Linux bug tracker, with a suggestion of the tool & package which you have created, and our tool team will handle it from there.

PreviousIntermediate packaging step-by-step exampleNextGetting the best out of the Kali Bot

Last updated

Was this helpful?