Advanced Packaging Step-By-Step Example (FinalRecon & Python-icmplib)
This guide is accurate at the time of writing. As it references a lot of external resources out of our control, items may be different over time (as software gets updated).
FinalRecon is a Python 3 application with multiple Python dependencies. At the time of writing, one of the dependencies (python3-icmplib) is not in the Kali Linux repository. In this guide we will have to learn how to follow dependency chains, and fix anything required to ensure that the end package can be included. We will also create a patch, helper-script, as well as a runtime test for the package.
We will assume we have already followed our documentation on setting up a packaging environment as well as our previous other packaging guides #1 (Instaloader) & #2 (Photon) as this will explain their contents.
FinalRecon Code Overview
The first action we will take, will be to look at FinalRecon’s source code to see what information we can acquire. Using this, we notice the following:
It has no tag release
The MIT license file
There is no
setup.pyfile (which is used for setuptools)Various descriptions about the tool & usage guides
Various external links (if any additional research is required)
Missing Tag Releases
As FinalRecon does not have a tag release we will have to create our own upstream tar file. Looking to see what branches there are, we discover there is just one (there isn’t a stable/production one, or is there a beta/deployment/staging one). As a result, we will use whatever is the latest commit on the main branch until the author does a tag release. We can auto open up an issue request and/or email them seeing if they will response to such an act.
Note that having a “tagged” release is preferred when doing Debian packaging. End users often want something that is “stable”, and which has been fully tested. It’s also easier for the distribution to know when to update the package: we just wait for upstream to release something, which is a clear signal that the code is ready to be used. So when it’s possible, we favor packaging a tagged release over the latest Git commit.
License
This package has been detected as having a MIT license by GitHub. If we look at the specific license file we can see that there is not a lot to copy, so we will be copying this exactly as-is. Unfortunately, though we have found a maintainer we have not found any contact information yet. We will have to continue searching for contact information.
Dependencies
As there is a requirements.txt file (which is used for Python’s pip to install any Python dependencies that are required for this tool to work), we will need check to see what’s needed.
Description(s)
We will once again pull our description from FinalRecon’s GitHub.
For the short description we will use a modified version of the first line in the README, “A fast and simple python script for web reconnaissance.”
For the long description we will also use a modified version of the first line in the README, however we will expand this time, “A fast and simple python script for web reconnaissance that follows a modular structure and provides detailed information on various areas.”
Maintainer(s)
If we were to look all over the GitHub we would not find an email address. We could look in git log and view the email addresses associated, however these do not seem to be solid as there are multiple for “thewhiteh4t” (at least 3). Instead, we do more digging. We notice that there is a YouTube video demo linked in the README.md, if we go to the YouTube channel’s about page we can view an email address for business inquiries (which does not match to any in the git log). This will be a good choice to use as the contact information. With that said all said, not having a contact information is not an essential part , so if we were unable to find one we could still continue to package.
Setting Up The Environment
We will assume that we have already followed our documentation on setting up a packing environment.
Let’s set up our directories now for this package:
Downloading Git Snapshot
We’re going to download an archive of the upstream source code. Since upstream didn’t tag any release yet, we’ll package the latest Git commit on the main branch. There are different (many?) ways to do that, and in this example we will use uscan for the task. uscan is able to download a Git repository, pack it into a .tar.gz archive, and come up with a meaningful (and somewhat standard) version string.
This last point is important: a Debian package must have a version, however a Git commit doesn’t have a version per se. So we need to associate a version with a Git commit, and there are many ways to get that wrong. So rather than deciding by ourselves what the package version should be, we’ll let the tooling (uscan in this case) do that for us.
In order to use uscan, we need a watch file. This file is usually part of the packaging files, and located in debian/watch.
Let’s start by entering the working directory, and then create the debian dir:
And now let’s create the watch file. The purpose of the watch file is to provide instructions to find the latest upstream release online. In this particular case though, upstream didn’t provide any tagged release yet, so we’ll configure the watch file to track the latest Git commit on the main branch:
At this point, we have enough to run uscan to download and pack the latest Git commit from upstream:
This command warrants some explanations. Since at this point we run uscan from an almost empty directory, we need to be explicit about what we want to do. In particular:
--watchfiletells uscan where is the watch file that we want it to use.--packageis used to give the package name.--upstream-versionis actually the “current upstream version”. In general, uscan works by comparing the latest version found online with the version that is currently packaged, and it downloads the latest upstream version only if it’s newer than the current version. However here there’s no “current version” since we’re creating a new package, so we tell uscan that the current version is0~0, ie. the lowest version possible, so that whatever version found online is deemed higher than that.--destdirtells uscan where to save the download files.--force-downloadoverrides uscan’s guess of what it should do: we want it to download the latest upstream version.
To be sure, we can have a look in the ~/kali/upstream directory to check what files landed there:
uscan packed the code from Git in a .tar.xz file, and for some reason (see the line starting with uscan warn: above), it repacked in as a .tar.gz. We don’t really care about the compression, we’re fine with both .gz and .xz. What matters is that we’ll use the file which name ends with .orig.tar.*, so we’re going to use the .tar.gz.
uscan came up with a funny-looking (and rather complicated) version string: 0.0~git20201107.0d41eb6. Why is that?
0.0~is the lowest starting point for a version string. It’s handy to start from there, so that whenever upstream does a “tagged release”, whatever they choose, it will be greater than our version. So we’ll be able to use it for the package version “as is”.gitis informative, and it obviously refers to the VCS used by upstream (examples of other VCS:svnorbzr).20201107is the date (YYYYMMDDaka. ISO-8601 format) of the upstream commit that we package. Having the date part of the version string is needed so that whenever we’ll want to import a new Git snapshot, the date will be newer, and the new version string will be sorted above by the package manager (version strings must ALWAYS go ascending).0d41eb6is the Git commit. It’s informative, and it’s a non-ambiguous way to know exactly what upstream code is included in the package. Without it, a developer who wants to know what Git commit was packaged would rely on the date, and if there’s more than one commit on this date, it wouldn’t be clear what commit exactly was packaged. Additionally, this is an UTC date, while usual tools or web browser usuall show dates in local time: another source of error for those who rely on the date only. So having the Git commit part of the version string is really useful for developers (maybe not so much for users).
Alright, we hope that you appreciated this overwhelming amount of information. Let’s move on and keep working on the package.
Creating Package Source Code
We are now going to create a new empty Git repository:
We can now import the .tar.gz we previously downloaded into the empty Git repository we just created. When prompted, we remember to accept the default values (or use the flag --no-interactive):
We remember to change the default branch, from master to kali/master (as master is for upstream development), then delete the old branch. We also run a quick git branch -v to visually see the change:
We can now populate the debian/ folder with its related files. We will manually specify the upstream .tar.gz file (as it is not located in ../, but instead ~/kali/upstream/). We will also set the package name to use in the same naming convention as before (<packagename>_<version> as is Debian standards).
Note that we need to use the option --addmissing as there’s already a debian/ directory (we created it above for the only purpose of having a watch file).
Afterwards we will remove any example files that get automatically generated, as they are not used:
At this point, we have the base packaging files in place, and it feels like a good idea to commit before starting some real work:
We can now start to edit the files in the debian/ folder to make sure the information is accurate. We can use what we found from before on FinalRecon’s GitHub to supply the correct information. To recap, we need to make sure we got the following bits of information to locate:
Dependencies
Description
License
Maintainers
FinalRecon (Pip) Dependencies
As there is a requirements.txt file (which is used for Python’s pip to install any Python dependencies that are required for this tool to work), we will need check to see what’s needed.
For this tool to work, it requires additional software to be installed, aka dependencies. Depending on how the tool is coded, will depend on what is required (or only recommended) to be installed. FinalRecon is using various Python libraries and does not call any system commands.
In Python’s eco-system, there is pip. This is Python’s package manager, which can be used to download and install any Python libraries. However, we are trying to build a package for Debian package management instead. As a result, any Python libraries need to be ported over to Debian format, in order for our package to use them (so the OS can track any files, allowing for cleaner upgrades and un-installs of packages). Lets start out by looking to see what is needed outside of the standard values, for this tool to work:
We then try to search for each dependency from requirements.txt in apt-cache, to make sure that we have everything in Kali Linux’ repository:
We could search each one manually by repeating the above process for all items in requirements.txt, or we can make a quick loop to automate it.
During this process, we will notice one dependency which does not have an entry (icmplib):
We can try and broaden our search for icmplib, as we were limiting output last time (by using grep):
Unfortunately it appears that Kali Linux does not have this dependency (Python’s icmplib) in the repository at this point in time. This means we will need to extend our packaging process to accommodate for packaging up icmplib as well, to allow us to completely package up FinalRecon.
We will first look for icmplib in the pypi.org repository. We can easily find icmplib on PyPI along with the link to its GitHub page. If we do the same process with icmplib looking over the GitHub page as we did for FinalRecon, we can see that icmplib will not need additional dependencies (no requirements.txt file and setup.py does not list anything for install_requires) and therefore will be a relatively straightforward Python package.
We can now either:
Continue to package FinalRecon, before moving onto icmplib. We have to remember that we cannot successfully build a complete working package until we are done with
icmplib.Pause FinalRecon packaging, and switch our focus to icmplib. We have to make sure we took detailed notes with the work we have done so far and information gathered.
We will go with the former option, and continue as far as we can with FinalRecon.
Editing FinalRecon Package Source Code
We can now start to edit the files in the debian/ folder.
Changelog
We will now perform what are our standard changes (#1 (Instaloader) & #2 (Photon)) to the version, distribution and description. The resulting file should be similar to the following:
You could also use dch -r or gbp dch to edit the file, rather than vim.
You will need to update the version to make the same date as used previously.
Control
Using what we know from the information we have already gathered from GitHub and the source code, it is once again similar to our previous packaging guides (#1 (Instaloader) & #2 (Photon)). We should have a good understanding of what needs to be altered now.
As there is no code that needs to be compiled, we can set Architecture: all. This is true for most Python scripts, as they are not providing Python “extensions”. If they are, they would generate a compiled .so files (e.g. psycopg2).
We make sure to include the Python dependencies for building the package as well as the tool dependencies to run (the values from pip).
There is one thing to note, and that is python3-icmplib. This package does not exist yet. We are adding this in for the time being as we will be creating it soon, to prevent going back and adding it we will add it now. This does mean that we will be unable to build our package until we finish with icmplib:
Copyright
As we have already finished getting the copyright information (license, name, contact, year and source), we now just need to add it:
Rules
The start of the rules file will look very similar to #2 (Photon), however there is a new lower section. This part is to set the permissions on finalrecon.py, so when we call it using the symlinks (by debian/links), it will be executable:
Beware that the “dh” line needs to be indented by a single tabulation character, rather than spaces.
Watch
The watch file was already covered at the beginning of this example, and is configured to track the latest Git commit on the main branch.
You can also add the common configuration for GitHub, but leave it commented out, so that whenever upstream will issue a release, everything is ready in your watch file and you’ll just need to uncomment it:
Links
Whereas last time (#1 (Instaloader) & #2 (Photon)), we are not going to use a “helper-script” but instead create a symlink pointing to the main Python file, which will still be in $PATH:
One thing we notice during the testing process of the script, we have to cd into the directory before calling the Python script. This is because the tool attempts to drop files into the dumps folder, and if it is not in path FinalRecon will fail.
This could be discovered by auditing the source code, or trial and error when testing the package after its built.
.Install
We can now create the install file, which is required to say what files go where on the system during the unpacking of the package. We need to make sure to include everything from the root of the package directory:
There is not a leading slash on the destination directory
Patches
For this tool we will need to also implement a patch to disable the update and dependency checker. If the program self updates, the system will not be aware of any additional files outside of the package, so things then start to get messy. The dependency is also being handled by our package now instead. Knowing you need to do this, comes with either knowing the tool, or auditing the source code.
The patch process looks like the following (for more information see our previous guide, #2 (Photon)):
Runtime Test
The runtime test process looks like the following (for more information see our previous guide, #2 (Photon)). Just like last time, we will just create a minimal test to look for the help screen:
Completing dependencies
In theory, we should have a complete working package now with the exception of the missing icmplib dependency. So we now need to package up icmplib, before trying to finally build FinalRecong.
icmplibNaming Packages
Unlike our previous guides (#1 (Instaloader) & #2 (Photon)) where we use the same name for both source package and binary package, this time we will differ them.
The naming convention for a binary package is python3-<package>, which is important to follow as it has a impact at a technical level. However a source package can be python-<package> (or even just <package>). It does not matter if this is not followed as it will not break anything if its not followed. However, from a Kali team point of view we prefer and will use python-<package>. See this Debian resource for more information.
Cheat Sheet Packaging
This package is straightforward using python3-setuptools (like in our first guide (Instaloader)), so to prevent this guide from getting too long, we will not be going step by step for icmplib.
For more information on building Python libraries, see the Debian resource
Here is a quick overview of the commands needed to build the package:
Previewing the contents of the key filesin debian/:
Changelog
Straight forward, like all the other guides, #1 (Instaloader) & #2 (Photon), edit version, distribution and description.
Note, python-icmplib needs to match the source name in debian/control:
Control
This is a bit different to what we have seen previously with Section: python. This is because its a Python library. For more information see Debian’s write up as well as the different options.
We also need to name the package differently. The source package part of debian/control is the top part, which gets named with the Source: field, whereas the binary part the lower half and uses Package: to name. Were possible Kali Linux will always try and do both a source and binary package (See the Debian resource for more information).
Note, the source name python-icmplib needs to match in debian/changelog:
Copyright
As we renamed the orig.tar.gz, upstream name is incorrect, as it normally would not have a leading python3-. We can get this from the source URL:
Rules
We need to make sure to drop any leading python- when being defined in PYBUILD_NAME, even though the binary package which gets produced (as defined in debian/control) will be python3-icmplib. This is because of PyBuild, only wanting the Python module name:
Watch
Straight forward, like all the other guides, #1 (Instaloader) & #2 (Photon), using the Debian standard watch file for GitHub:
We have successfully managed to build a Python 3 library file, icmplib!
Final FinalRecon Build
As we may not have pushed out, had python3-icmplib being accepted yet into Kali Linux, or you may want to submit both at the same time, we can include the recently generated package in the chroot for sbuild to use, it is a listed requirement for FinalRecon.
We are also unsure about the status of the package, we may not want to commit the latest edits to Git. So we will add --git-export=WC when building the package:
Before we try to test our newly generated package, we remember that in debian/control we listed a few dependencies (not only to build the package but to run the package). Using dpkg, it will not satisfy these requirements, so we need to manually install them first. We can check what is missing from our operating system, by doing:
If you fail to do install the package, you may end up with the following mess:
You will then hit the issue of the next time you try and install or update a package, it will fail:
Following what it says, running sudo apt --fix-broken install often fixes the issue.
Our package has been built and dependencies have been installed. Its now time to finally install FinalRecon:
You can also install by doing:
We have successfully managed to build FinalRecon as a package!
Let’s test to make sure it works:
Saving Our Work
At this point, we can save the work we have put in:
We can now finish up the packaging by putting in a request on the Kali Linux bug tracker for these packages to be added!
Message From Kali Team
During the packaging process, we worked with the tool author (upstream), so the program would be better fit with FHS. An example of this is, not to use /usr/share/finalrecon/dumps as writing to /usr/share requires root privileges and we also do not want user files to be saved here.
Last updated