Photon is a Python3 application with multiple dependencies. This makes it a more interesting package than Instaloader as potentially more work is involved.
Photon Code Overview
Like before with Instaloader we will first take a look at their GitHub page to see what information we can acquire. In this case we notice the following:
There is no setup.py, however there is a requirements.txt (file)
As this package has a tag release like Instaloader, we will follow the same process. We go to Photon’s GitHub’s release page and see the latest release is version 1.3.0. We then download it with the filename format of [name]_[version].orig.tar.gz:
We can now go to the working directory for the process now that the prerequisites are done:
:~$ cd ~/kali/packages/photon/
:~/kali/packages/photon$
We then can make it into an empty Git repository:
:~/kali/packages/photon$ git init
Initialized empty Git repository in /home/kali/packages/photon/.git/
:~/kali/packages/photon$
We can now import the .tar.gz we previously downloaded into the empty Git repository we just created. When prompted, we remember to accept the default values:
:~/kali/packages/photon$ gbp import-orig ~/kali/upstream/photon_1.3.0.orig.tar.gz
What will be the source package name? [photon]
What is the upstream version? [1.3.0]
gbp:info: Importing '/home/kali/kali/upstream/photon_1.3.0.orig.tar.gz' to branch 'upstream'...
gbp:info: Source package is photon
gbp:info: Upstream version is 1.3.0
gbp:info: Successfully imported version 1.3.0 of /home/kali/kali/upstream/photon_1.3.0.orig.tar.gz
:~/kali/packages/photon$
We remember to change the default branch, from master to kali/master (as master is for upstream development), then delete the old branch. We also run a quick git branch -v to visually see the change:
:~/kali/packages/photon$ git checkout -b kali/master
Switched to a new branch 'kali/master'
:~/kali/packages/photon$
:~/kali/packages/photon$ git branch -D master
Deleted branch master (was 95b196b).
:~/kali/packages/photon$
:~/kali/packages/photon$ git branch -v
* kali/master 95b196b New upstream version 4.4.4
pristine-tar 7e90962 pristine-tar data for photon_1.3.0.orig.tar.gz
upstream 95b196b New upstream version 4.4.4
:~/kali/packages/photon$
We can now generate the debian/ folder and related files. We will again go with Single for this package. We will manually specify the downloaded file (as it is not located in ../), and also the package name to use in the template:
:~/kali/packages/photon$ dh_make --file ~/kali/upstream/photon_1.3.0.orig.tar.gz -p photon_1.3.0
Type of package: (single, indep, library, python)
[s/i/l/p]?
Maintainer Name : Joseph O'Gorman
Email-Address :
Date : Mon, 13 Jul 2020 17:28:51 -0400
Package Name : photon
Version : 1.3.0
License : blank
Package Type : single
Are the details correct? [Y/n/q]
Skipping creating ../photon_1.3.0.orig.tar.gz because it already exists
Currently there is not top level Makefile. This may require additional tuning
Done. Please edit the files in the debian/ subdirectory now.
:~/kali/packages/photon$
:~/kali/packages/photon$ rm debian/*.docs debian/README* debian/*.ex debian/*.EX
:~/kali/packages/photon$
As a quick refresher of what each of these files are for:
debian/changelog - tracks when the package gets an update (including why and who by). This is responsible for the package version
debian/control - is the metadata for the package (often seen with apt)
debian/copyright - what is under what license. The package can be under something different to the work we have put in to create the package
debian/rules - how to build the software and turn it into a package
debian/source/format - is the source package format
At this point, we have the base packaging files in place, and it feels like a good idea to commit before starting some real work:
We now need to edit each one to make sure the information is accurate. We can use what we found on GitHub to supply the correct info into the debian/ files:
License
Dependencies
Maintainers
Description
Collecting InformationLicense/Maintainers
This package, like Instaloader, is very straightforward and GitHub has already detected the license as GPL-3.
For GPL-3 licenses we do not have to copy the entirety as-is in the upstream license file. If we look at /usr/share/common-licenses/ we can see that there are multiple licenses in their entirety already available locally. As we can see from GPL-3’s page from Debian we can use a shortened down version of the license that will be acceptable.
Unfortunately, when we look at the license file from upstream we do not see any contact information or names. We will have to look elsewhere for it.
If we look through the README.meon Photon’s GitHub page, we can see that s0md3v appears to be the sole maintainer as his Twitter is linked however no other maintainers are seen. If we look at s0md3v’s GitHub profile page, we notice that an email is shown (if we are logged in)! With this, we have the maintainer name and email address we can use to continue.
Dependencies/Maintainers
We now need to find out what dependencies are needed for this tool to work. Fortunately, we have a requirements.txt file that we spotted earlier. Looking at this we can see four dependencies needed:
We now need to find the proper name in apt to make sure that we have everything for when we edit the debian/control file later.
From Instaloader, we already know that requests will be available with python3-requests. However, requests[socks] will not so we have to find a way to install socks. If we search for python3-requests, we can see that there is no result that includes any mention of socks:
:~/kali/packages/photon$ apt-cache search python3-requests | grep -i socks
:~/kali/packages/photon$
:~/kali/packages/photon$ apt-cache search python3-requests
beancount - Double-entry accounting from text files
python-requests-toolbelt-doc - Utility belt for python3-requests (documentation)
python3-awsauth - AWS authentication for Amazon S3 for the python3-requests module
python3-beancount - Double-entry accounting from text files - Python module
python3-httmock - Mocking library for python3-requests
python3-proxmoxer - Python Wrapper for the Proxmox 2.x API (HTTP and SSH) (Python 3)
python3-requests - elegant and simple HTTP library for Python3, built for human beings
python3-requests-cache - persistent cache for requests library (Python 3)
python3-requests-file - File transport adapter for Requests - Python 3.X
python3-requests-futures - library for asynchronous HTTP requests (Python 3)
python3-requests-kerberos - Kerberos/GSSAPI authentication handler for python-requests - Python 3.x
python3-requests-mock - mock out responses from the requests package - Python 3.x
python3-requests-ntlm - Adds support for NTLM authentication to the requests library
python3-requests-oauthlib - module providing OAuthlib auth support for requests (Python 3)
python3-requests-toolbelt - Utility belt for advanced users of python3-requests
python3-requests-unixsocket - Use requests to talk HTTP via a UNIX domain socket - Python 3.x
python3-requestsexceptions - import exceptions from bundled packages in requests. - Python 3.x
units - converts between different systems of units
:~/kali/packages/photon$
If we broaden our search and utilize grep we are able to find what we need!
We can finally compare the upstream source from PySocks to the homepage provided with python3-socks and see that they match:
:~/kali/packages/photon$ apt-cache show python3-socks | grep Homepage
Homepage: https://github.com/Anorov/PySocks
:~/kali/packages/photon$
Keeping this in mind, lets find urllib3 and tld:
:~/kali/packages/photon$ apt-cache search python3 | grep urllib3
python3-urllib3 - HTTP library with thread-safe connection pooling for Python3
:~/kali/packages/photon$
:~/kali/packages/photon$ apt-cache search python3 | grep tld
python3-tld - Extract the top level domain (TLD) from a given URL (Python 3)
python3-tldextract - Python library for separating TLDs
:~/kali/packages/photon$
There are the other two! We now have all four dependencies figured out (python3-requests,python3-socks,python3-urllib3 & python3-tld), and can move forward.
Maintainers
Like Instaloader, while doing the other parts we have discovered the authors and maintainers of the software, so we don’t need to do anything extra for this.
Description
Similarly to Instaloader we will pull the descriptions from the GitHub page. Remember we need two description values, a short one and a long one.
The first description we create is the short description. For this one, we will take the summary from the about section of the GitHub and expand “OSINT” to be “Open Source INTelligence” for anyone who would not know what “OSINT” means.
For the long description we will repeat the about description, and then utilize the explanation under Data Extraction from the README. The reason we do this is because the long description cannot start with the name of the package, so we will repeat the short description and modify it to provide further context. However, for now this is something to remember until we edit the debian/control file.
Editing Package Source Code
Now that we have that information copied down, we can start to populate the files in the debian/ folder we created with dh_make.
Changelog
Here we will need to alter:
distribution from unstable to kali-dev
unstable is the development distribution for Debian, kali-dev is the one we use in Kali.
version (from 1.3.0-1 to 1.3.0-0kali1)
The format is [softwareversion]-0kali[release]. We put 0kali in case this package makes it into Debian, it avoids conflicting versions and ensures proper upgrade to the Debian version.
The log entry - We keep it simple with Initial release
Most of this we have now figured out from Instaloader, so it should make it easier to fill in. We supply the values that we found from our information gathering and get the following:
:~/kali/packages/photon$ vim debian/control
:~/kali/packages/photon$
:~/kali/packages/photon$ cat debian/control
Source: photon
Section: net
Priority: optional
Maintainer: Kali Developers <>
Uploaders: Joseph O'Gorman <>
Build-Depends: debhelper-compat (= 12),
dh-python,
python3-all,
python3-requests,
python3-socks,
python3-tld,
python3-urllib3,
Standards-Version: 4.5.0
Homepage: https://github.com/s0md3v/Photon
Vcs-Browser: https://gitlab.com/kalilinux/packages/photon
Vcs-Git: https://gitlab.com/kalilinux/packages/photon.git
Package: photon
Architecture: all
Depends: ${python3:Depends},
${misc:Depends},
python3-requests,
python3-socks,
python3-tld,
python3-urllib3,
Description: Incredibly fast crawler designed for open source intelligence
This package includes a fast and flexible crawler designed for open source
intelligence (OSINT).
.
Photon can extract the following data while crawling:
- URLs (in-scope & out-of-scope)
- URLs with parameters (example.com/gallery.php?id=2)
- Intel (emails, social media accounts, amazon buckets etc.)
- Files (pdf, png, xml etc.)
- Secret keys (auth/API keys & hashes)
- JavaScript files & Endpoints present in them
- Strings matching custom regex pattern
- Subdomains & DNS related data
.
The extracted information is saved in an organized manner or can be exported
as json.
:~/kali/packages/photon$
In Build-Depends now, this is a Python3 Package so we need to add in:
dh-python
python3-all
python3-requests
python3-socks
python3-tld
python3-urllib3
For this package however, as we do not have a setup.py file, we will make a small change compared to Instaloader. This change is that we do not include the build dependency of python3-setuptools. We will still include python3-all, and that is for a separate reason. We will also include all of the dependencies that are for the package. We do this to ensure that a test suite can be ran if there is one.
As stated previously, python3-all is still included. This is due to its reliance on a python module that requires compiled binary extensions. These can be found by doing the following:
This can be a bit overwhelming, and at first glance does not give too much helpful information. We can use this to learn if we are going to have any dependency that will eventually call in one of the previous files. To do this we do the following:
As we can see, python3-urllib3 suggests the python3 module cryptography, which we can tell from the previous list will have us require python3-all. This is sufficient to include python3-all.
In Depends we change ${shlibs:Depends} to ${python3:Depends} like before and add in the apt names that we found previously (python3-requests, python3-socks, python3-tld & python3-urllib3).
Copyright
We already have figured out the copyright license description we will use, and have the Upstream-Contact information, so we can easily populate this file:
:~/kali/packages/photon$ vim debian/copyright
:~/kali/packages/photon$
:~/kali/packages/photon$ cat debian/copyright
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: photon
Upstream-Contact: s0md3v <>
Source: https://gihub.com/s0md3v/Photon
Files: *
Copyright: 2020 s0md3v <>
License: GPL-3+
Files: debian/*
Copyright: 2020 Joseph O'Gorman <>
License: GPL-3+
License: GPL-3+
This package is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.
.
This package is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>
.
On Debian systems, the complete text of the GNU General
Public License version 3 can be found in "/usr/share/common-licenses/GPL-3".
:~/kali/packages/photon$
Rules
This rules file will look similar to Instaloader’s, with how to install, however there is one key difference that will change significantly how the package is built. As there is no setup.py, we do not need to set the pybuild build system:
Now we can leave this package here. It is a working package, and everything appears to be correct when we build it: gbp buildpackage --git-builder=sbuild --git-export=WC
However, we want this package to be the best possible package it can be! If we take a look at the photon.py file we can see that it has the ability to update built in. This is something that we should patch outusing gbp pq. Additionally, due to the helpfulness of autopkgtests we should contribute a minimal test.
Patches
Patches are occasionally used to fix an issue with the upstream tool or adhere to standards. For this tool we will be creating a patch that will remove the update functionality from photon.py, so users will update the tool from apt instead.
We will be using gbp pq to create a patch for this package:
:~/kali/packages/photon$ gbp pq import
gbp:info: Trying to apply patches at 'ac95fad43f0418dd05510a9647b9f8e08c24ce12'
gbp:info: 0 patches listed in 'debian/patches/series' imported on 'patch-queue/kali/master'
:~/kali/packages/photon$
We first will create a separate branch where we can work freely. With this command, any patches that are already created will be automatically imported and applied:
After we make our changes we will do a commit. The commit message will be the title of the patch:
:~/kali/packages/photon$ gbp pq export
gbp:info: On 'patch-queue/kali/master', switching to 'kali/master'
gbp:info: Generating patches from git (kali/master..patch-queue/kali/master)
:~/kali/packages/photon$
We export the commits into a patch with the titles of the commits we made. We can see there are some new files made up:
:~/kali/packages/photon$ ls debian/patches/
disable-update-option.patch series
:~/kali/packages/photon$
Autopkgtest
Autopkgtest is a huge help to detecting issues when updating packages. For this package, we will create a simple test that may not catch more advanced errors however it is better than nothing. We first will get started with creating the directory we will create the test in:
:~/kali/packages/photon$ mkdir -p debian/tests
We now have to create a control file. If we had multiple commands/tests, we would put different information into this file, however as we have only the one command/test we will be running we can work directly in this file.
For this test we will simply supply --help to photon and see if it works correctly. Because we only use photon, we only need what the tool is dependent on. For this we will use @. Finally, because it is a simple test that wont catch much, we make sure to note that this is a superficial test:
