Advanced Packaging Step-By-Step Example (FinalRecon & Python-icmplib)
Last updated
Was this helpful?
Last updated
Was this helpful?
This guide is accurate at the time of writing. As it references a lot of external resources out of our control, items may be different over time (as software gets updated).
is a Python 3 application with multiple Python dependencies. At the time of writing, one of the dependencies () is not in the Kali Linux repository. In this guide we will have to learn how to follow dependency chains, and fix anything required to ensure that the end package can be included. We will also create a patch, helper-script, as well as a runtime test for the package.
We will assume we have already followed our documentation on setting up a packaging environment as well as our previous other packaging guides #1 (Instaloader) & #2 (Photon) as this will explain their contents.
FinalRecon Code Overview
The first action we will take, will be to look at to see what information we can acquire. Using this, we notice the following:
It has
The
There is no setup.py file (which is used for )
There is a requirements.txt (which is used for )
Various descriptions about the tool & usage guides
Various external links (if any additional research is required)
Missing Tag Releases
As FinalRecon does we will have to create our own upstream tar file. Looking to see what branches there are, we discover there is just one (there isnât a stable/production one, or is there a beta/deployment/staging one). As a result, we will use whatever is the latest commit on the main branch until the author does a tag release. We can auto open up an issue request and/or email them seeing if they will response to such an act.
Note that having a âtaggedâ release is preferred when doing Debian packaging. End users often want something that is âstableâ, and which has been fully tested. Itâs also easier for the distribution to know when to update the package: we just wait for upstream to release something, which is a clear signal that the code is ready to be used. So when itâs possible, we favor packaging a tagged release over the latest Git commit.
License
This package has been detected as having a license by . If we look at the specific we can see that there is not a lot to copy, so we will be copying this exactly as-is. Unfortunately, though we have found a maintainer we have not found any contact information yet. We will have to continue searching for contact information.
Dependencies
Description(s)
Maintainer(s)
Setting Up The Environment
We will assume that we have already followed our documentation on setting up a packing environment.
Letâs set up our directories now for this package:
Downloading Git Snapshot
Weâre going to download an archive of the upstream source code. Since upstream didnât tag any release yet, weâll package the latest Git commit on the main branch. There are different (many?) ways to do that, and in this example we will use uscan for the task. uscan is able to download a Git repository, pack it into a .tar.gz archive, and come up with a meaningful (and somewhat standard) version string.
This last point is important: a Debian package must have a version, however a Git commit doesnât have a version per se. So we need to associate a version with a Git commit, and there are many ways to get that wrong. So rather than deciding by ourselves what the package version should be, weâll let the tooling (uscan in this case) do that for us.
In order to use uscan, we need a watch file. This file is usually part of the packaging files, and located in debian/watch.
Letâs start by entering the working directory, and then create the debian dir:
And now letâs create the watch file. The purpose of the watch file is to provide instructions to find the latest upstream release online. In this particular case though, upstream didnât provide any tagged release yet, so weâll configure the watch file to track the latest Git commit on the main branch:
At this point, we have enough to run uscan to download and pack the latest Git commit from upstream:
This command warrants some explanations. Since at this point we run uscan from an almost empty directory, we need to be explicit about what we want to do. In particular:
--watchfile tells uscan where is the watch file that we want it to use.
--package is used to give the package name.
--upstream-version is actually the âcurrent upstream versionâ. In general, uscan works by comparing the latest version found online with the version that is currently packaged, and it downloads the latest upstream version only if itâs newer than the current version. However here thereâs no âcurrent versionâ since weâre creating a new package, so we tell uscan that the current version is 0~0, ie. the lowest version possible, so that whatever version found online is deemed higher than that.
--destdir tells uscan where to save the download files.
--force-download overrides uscanâs guess of what it should do: we want it to download the latest upstream version.
To be sure, we can have a look in the ~/kali/upstream directory to check what files landed there:
uscan packed the code from Git in a .tar.xz file, and for some reason (see the line starting with uscan warn: above), it repacked in as a .tar.gz. We donât really care about the compression, weâre fine with both .gz and .xz. What matters is that weâll use the file which name ends with .orig.tar.*, so weâre going to use the .tar.gz.
uscan came up with a funny-looking (and rather complicated) version string: 0.0~git20201107.0d41eb6. Why is that?
0.0~ is the lowest starting point for a version string. Itâs handy to start from there, so that whenever upstream does a âtagged releaseâ, whatever they choose, it will be greater than our version. So weâll be able to use it for the package version âas isâ.
git is informative, and it obviously refers to the VCS used by upstream (examples of other VCS: svn or bzr).
20201107 is the date (YYYYMMDD aka. ISO-8601 format) of the upstream commit that we package. Having the date part of the version string is needed so that whenever weâll want to import a new Git snapshot, the date will be newer, and the new version string will be sorted above by the package manager (version strings must ALWAYS go ascending).
0d41eb6 is the Git commit. Itâs informative, and itâs a non-ambiguous way to know exactly what upstream code is included in the package. Without it, a developer who wants to know what Git commit was packaged would rely on the date, and if thereâs more than one commit on this date, it wouldnât be clear what commit exactly was packaged. Additionally, this is an UTC date, while usual tools or web browser usuall show dates in local time: another source of error for those who rely on the date only. So having the Git commit part of the version string is really useful for developers (maybe not so much for users).
Alright, we hope that you appreciated this overwhelming amount of information. Letâs move on and keep working on the package.
Creating Package Source Code
We are now going to create a new empty Git repository:
We can now import the .tar.gz we previously downloaded into the empty Git repository we just created. When prompted, we remember to accept the default values (or use the flag --no-interactive):
We remember to change the default branch, from master to kali/master (as master is for upstream development), then delete the old branch. We also run a quick git branch -v to visually see the change:
We can now populate the debian/ folder with its related files. We will manually specify the upstream .tar.gz file (as it is not located in ../, but instead ~/kali/upstream/). We will also set the package name to use in the same naming convention as before (<packagename>_<version> as is Debian standards).
Note that we need to use the option --addmissing as thereâs already a debian/ directory (we created it above for the only purpose of having a watch file).
Afterwards we will remove any example files that get automatically generated, as they are not used:
At this point, we have the base packaging files in place, and it feels like a good idea to commit before starting some real work:
Dependencies
Description
License
Maintainers
FinalRecon (Pip) Dependencies
For this tool to work, it requires additional software to be installed, aka dependencies. Depending on how the tool is coded, will depend on what is required (or only recommended) to be installed. FinalRecon is using various Python libraries and does not call any system commands.
We then try to search for each dependency from requirements.txt in apt-cache, to make sure that we have everything in Kali Linuxâ repository:
We could search each one manually by repeating the above process for all items in requirements.txt, or we can make a quick loop to automate it.
During this process, we will notice one dependency which does not have an entry (icmplib):
We can try and broaden our search for icmplib, as we were limiting output last time (by using grep):
Unfortunately it appears that Kali Linux does not have this dependency (Pythonâs icmplib) in the repository at this point in time. This means we will need to extend our packaging process to accommodate for packaging up icmplib as well, to allow us to completely package up FinalRecon.
We can now either:
Continue to package FinalRecon, before moving onto icmplib. We have to remember that we cannot successfully build a complete working package until we are done with icmplib.
Pause FinalRecon packaging, and switch our focus to icmplib. We have to make sure we took detailed notes with the work we have done so far and information gathered.
We will go with the former option, and continue as far as we can with FinalRecon.
Editing FinalRecon Package Source Code
We can now start to edit the files in the debian/ folder.
Changelog
We will now perform what are our standard changes (#1 (Instaloader) & #2 (Photon)) to the version, distribution and description. The resulting file should be similar to the following:
You could also use dch -r or gbp dch to edit the file, rather than vim.
You will need to update the version to make the same date as used previously.
Control
Using what we know from the information we have already gathered from GitHub and the source code, it is once again similar to our previous packaging guides (#1 (Instaloader) & #2 (Photon)). We should have a good understanding of what needs to be altered now.
We make sure to include the Python dependencies for building the package as well as the tool dependencies to run (the values from pip).
There is one thing to note, and that is python3-icmplib. This package does not exist yet. We are adding this in for the time being as we will be creating it soon, to prevent going back and adding it we will add it now. This does mean that we will be unable to build our package until we finish with icmplib:
Copyright
As we have already finished getting the copyright information (license, name, contact, year and source), we now just need to add it:
Rules
The start of the rules file will look very similar to #2 (Photon), however there is a new lower section. This part is to set the permissions on finalrecon.py, so when we call it using the symlinks (by debian/links), it will be executable:
Beware that the âdhâ line needs to be indented by a single tabulation character, rather than spaces.
Watch
The watch file was already covered at the beginning of this example, and is configured to track the latest Git commit on the main branch.
Links
Whereas last time (#1 (Instaloader) & #2 (Photon)), we are not going to use a âhelper-scriptâ but instead create a symlink pointing to the main Python file, which will still be in $PATH:
One thing we notice during the testing process of the script, we have to cd into the directory before calling the Python script. This is because the tool attempts to drop files into the dumps folder, and if it is not in path FinalRecon will fail.
This could be discovered by auditing the source code, or trial and error when testing the package after its built.
.Install
We can now create the install file, which is required to say what files go where on the system during the unpacking of the package. We need to make sure to include everything from the root of the package directory:
There is not a leading slash on the destination directory
Patches
For this tool we will need to also implement a patch to disable the update and dependency checker. If the program self updates, the system will not be aware of any additional files outside of the package, so things then start to get messy. The dependency is also being handled by our package now instead. Knowing you need to do this, comes with either knowing the tool, or auditing the source code.
The patch process looks like the following (for more information see our previous guide, #2 (Photon)):
Runtime Test
The runtime test process looks like the following (for more information see our previous guide, #2 (Photon)). Just like last time, we will just create a minimal test to look for the help screen:
Completing dependencies
In theory, we should have a complete working package now with the exception of the missing icmplib dependency. So we now need to package up icmplib, before trying to finally build FinalRecong.
icmplibNaming Packages
Unlike our previous guides (#1 (Instaloader) & #2 (Photon)) where we use the same name for both source package and binary package, this time we will differ them.
Cheat Sheet Packaging
This package is straightforward using python3-setuptools (like in our first guide (Instaloader)), so to prevent this guide from getting too long, we will not be going step by step for icmplib.
Here is a quick overview of the commands needed to build the package:
Previewing the contents of the key filesin debian/:
Changelog
Straight forward, like all the other guides, #1 (Instaloader) & #2 (Photon), edit version, distribution and description.
Note, python-icmplib needs to match the source name in debian/control:
Control
Note, the source name python-icmplib needs to match in debian/changelog:
Copyright
As we renamed the orig.tar.gz, upstream name is incorrect, as it normally would not have a leading python3-. We can get this from the source URL:
Rules
Watch
We have successfully managed to build a Python 3 library file, icmplib!
Final FinalRecon Build
As we may not have pushed out, had python3-icmplib being accepted yet into Kali Linux, or you may want to submit both at the same time, we can include the recently generated package in the chroot for sbuild to use, it is a listed requirement for FinalRecon.
We are also unsure about the status of the package, we may not want to commit the latest edits to Git. So we will add --git-export=WC when building the package:
Before we try to test our newly generated package, we remember that in debian/control we listed a few dependencies (not only to build the package but to run the package). Using dpkg, it will not satisfy these requirements, so we need to manually install them first. We can check what is missing from our operating system, by doing:
If you fail to do install the package, you may end up with the following mess:
You will then hit the issue of the next time you try and install or update a package, it will fail:
Following what it says, running sudo apt --fix-broken install often fixes the issue.
Our package has been built and dependencies have been installed. Its now time to finally install FinalRecon:
You can also install by doing:
We have successfully managed to build FinalRecon as a package!
Letâs test to make sure it works:
Saving Our Work
At this point, we can save the work we have put in:
Message From Kali Team
As there is a requirements.txt (which is used for Pythonâs to install any Python dependencies that are required for this tool to work), we will need check to see whatâs needed.
We will once again pull our description from .
For the short description we will use a modified version of the first line in the , âA fast and simple python script for web reconnaissance.â
For the long description we will also use a modified version of the first line in the , however we will expand this time, âA fast and simple python script for web reconnaissance that follows a modular structure and provides detailed information on various areas.â
If we were to look all over the GitHub we would not find an email address. We could look in git log and view the email addresses associated, however these do not seem to be solid as there are multiple for âthewhiteh4tâ (at least 3). Instead, we do more digging. We notice that there is a YouTube video demo linked in the README.md, if we go to the YouTube channelâs we can view an email address for business inquiries (which does not match to any in the git log). This will be a good choice to use as the contact information. With that said all said, not having a contact information is not an essential part , so if we were unable to find one we could still continue to package.
We can now start to edit the files in the debian/ folder to make sure the information is accurate. We can use what we found from before on to supply the correct information. To recap, we need to make sure we got the following bits of information to locate:
As there is a requirements.txt (which is used for Pythonâs to install any Python dependencies that are required for this tool to work), we will need check to see whatâs needed.
In Pythonâs eco-system, there is . This is Pythonâs package manager, which can be used to download and install any Python libraries. However, we are trying to build a package for Debian package management instead. As a result, any Python libraries need to be ported over to Debian format, in order for our package to use them (so the OS can track any files, allowing for cleaner upgrades and un-installs of packages). Lets start out by looking to see what is needed outside of the standard values, for this tool to work:
We will first look for icmplib in the repository. We can easily find along with the link to its . If we do the same process with icmplib looking over the GitHub page as we did for FinalRecon, we can see that icmplib will not need additional dependencies (no requirements.txt file and setup.py install_requires) and therefore will be a relatively straightforward Python package.
As there is no code that needs to be compiled, we can set Architecture: all. This is true for most Python scripts, as they are not providing Python âextensionsâ. If they are, they would generate a compiled .so files (e.g. ).
You can also add the , but leave it commented out, so that whenever upstream will issue a release, everything is ready in your watch file and youâll just need to uncomment it:
The naming convention for a binary package is python3-<package>, which is important to follow as it has a impact at a technical level. However a source package can be python-<package> (or even just <package>). It does not matter if this is not followed as it will not break anything if its not followed. However, from a Kali team point of view we prefer and will use python-<package>. See this for more information.
For more information on building Python libraries, see the
This is a bit different to what we have seen previously with Section: python. This is because its a Python library. For more information see as well as the .
We also need to name the package differently. The source package part of debian/control is the top part, which gets named with the Source: field, whereas the binary part the lower half and uses Package: to name. Were possible Kali Linux will always try and do both a source and binary package (See the for more information).
We need to make sure to drop any leading python- when being defined in PYBUILD_NAME, even though the binary package which gets produced (as defined in debian/control) will be python3-icmplib. This is because of , only wanting the Python module name:
Straight forward, like all the other guides, #1 (Instaloader) & #2 (Photon), using the :
We can now finish up the packaging by putting in a request on the for these packages to be added!
During the packaging process, we worked with the tool author (upstream), so the program would be . An example of this is, not to use /usr/share/finalrecon/dumps as writing to /usr/share requires root privileges and we also do not want user files to be saved here.