Knowledgebase
  • Welcome!
  • Threats
    • Software
      • Malware
      • Ransomware
      • Macros
    • Hardware
      • Flipper Zero
        • Firmware
          • 🐬flipperzero
            • Getting Started
          • 🐬flipper-xtreme
            • Wiki
              • Key Combos
              • Generic Guides
              • iButton key file format
              • SubGhz
              • How to add new SubGHz frequencies
              • Sub-GHz Remote
              • LF RFID key file format
              • NFC Flipper File Formats
              • Infrared Flipper File Formats
              • BadKB
              • Asset Packs
              • Unit tests
              • OTA Updates
              • How To Build
              • Hardware Targets
              • Flipper Build Tool
              • FAP (Flipper Application Package)
              • Flipper Application Manifests (.fam)
          • 🐬roguemaster
          • 🐬unleashed
    • Human
      • Advanced Persistent Threats (APTs)
      • Social engineering
      • Phishing
      • Typosquatting
    • Disinformation
      • Black Propaganda
      • White Propaganda
      • Grey Propaganda
      • Info Warfare
      • Political Warfare
      • Astroturfing
      • Greenwashing
      • Bluewashing
      • Whisper Campaigns
      • Push Polling
      • "Joe Jobs"
      • False Flags
      • Deep Fakes
  • About
    • Ports
      • 20
      • 21
      • 22
      • 23
      • 25
      • 587
      • 2323
      • 53
      • 80
      • 194
  • Tools
    • Radio Frequency & SubGHZ
      • gnuradio
      • hackrf
    • Digital Forensics
      • afflib
    • Reverse Engineering
      • binwalk
      • radare2
    • Hardware & Virtualization
      • qemu
      • freerdp2
      • util-linux
      • lvm2
    • VPN Providers
      • ProtonVPN
      • NordVPN
      • ExpressVPN
      • Surfshark
      • CyberGhost
      • Private Internet Access
    • Database, Cloud, & Firewalls
      • sqlmap
      • cewl
      • gobuster
      • fwbuilder
      • clamav
    • Enumeration & Lists
      • crunch
      • aflplusplus
      • ffuf
      • maltego
        • maltego-teeth
      • getallurls
    • Penetration Testing
      • beef-xss
      • wifite
      • burpsuite
      • metasploit-framework
    • Passwords & Auth
      • john
      • hashcat
      • hydra
      • cryptsetup
    • Surface Intelligence
      • theharvester
      • subfinder
      • dsniff
      • dnsrecon
      • dirb
      • nikto
      • legion
      • spiderfoot
    • Networks & Wireless
      • nmap
      • impacket-scripts
      • tcpdump
      • traceroute
      • wireshark
      • responder
      • aircrack-ng
      • netcat
      • kismet
      • ubertooth
      • routersploit
      • apache2
      • ettercap
      • bettercap
      • bettercap-ui
      • freeradius
      • bind9
      • samba
      • net-snmp
      • tcpreplay
    • Social Media
      • sherlock
    • Miscellaneous
      • git
      • libnfc
      • llvm-defaults
  • Operating Systems
    • Ubuntu
      • Installation
        • Switching
          • From Windows
          • From macOS
          • From a different Linux
        • Applications
        • Ubuntu PreInstalled
    • Linux Mint
      • Installation Guide
        • Verify your ISO image
        • Choose the right edition
        • Boot Linux Mint
        • Create the bootable media
        • Install Linux Mint
        • Hardware drivers
        • Language support
        • EFI
        • Multimedia codecs
        • System snapshots
        • Pre-installing Linux Mint (OEM Installation)
        • Where to find help
        • Boot options
        • Partitioning
        • Multi-boot
      • User Guide
        • Grub Boot Menu
        • Snap Store
        • Chromium
        • Bluetooth
        • Windows ISOs and multiboot USB
        • How to upgrade to Linux Mint 20
        • Edge ISO Images
        • Lost Password
        • Upgrades
        • Printers and Scanners
        • How to upgrade to Linux Mint 21
      • Troubleshooting Guide
        • Expectation
        • Responsibility
        • Change
        • Reproducibility
        • Observation
        • Environment
        • What
        • When
        • Why
        • Errors
        • Where
        • How
      • Translation Guide
        • Using Launchpad
        • Verify your translations
        • Localization
      • Developer Guide
        • Getting Started
          • Setup
          • Technology
        • Mint Tools
        • Cinnamon
        • XApps
        • Development
          • Daily Builds
          • Coding Guidelines
          • Optimizing JS with Cinnamon
          • Building
    • Kali Linux
      • Installation
        • Installing Kali Linux
        • Bare-bones Kali
        • Installing Kali on Mac Hardware
        • Dual Booting Kali with Linux
        • Making a Kali Bootable USB Drive
        • Dual Booting Kali with macOS/OS X
        • Dual Booting Kali with Windows
        • BTRFS Install (Kali Unkaputtbar)
        • Deploying Kali over Network PXE/iPXE Install
      • Virtualization
        • Running Kali Linux as a Virtual Machine in Windows
        • Installing VMware on Apple Silicon (M1/M2) Macs (Host)
        • Customizing a Kali Vagrant Vagrantfile
        • Kali inside Proxmox (Guest VM)
        • Installing VMware on Kali (Host)
        • Installing VirtualBox on Kali (Host)
        • Import Pre-Made Kali VMware VM
        • Kali inside Parallels (Guest VM)
        • Kali inside Vagrant (Guest VM)
        • Kali inside VMware (Guest VM)
        • Kali inside VirtualBox (Guest VM)
        • Import Pre-Made Kali VirtualBox VM
        • Kali inside Hyper-V (Guest VM)
        • Kali inside UTM (Guest VM)
        • Kali inside QEMU/LibVirt with virt-manager (Guest VM)
        • Improving Virtual Machine Performance for VMware
        • Installing VMware Tools (Guest Tools)
        • Installing VirtualBox Guest Addition (Guest Tools)
        • Installing Hyper-V Enhanced Session Mode (Guest Tools)
        • Converting VMX to an OVA
      • USB
        • Making a Kali Bootable USB Drive (Linux)
        • Making a Kali Bootable USB Drive (macOS/OS X)
        • Updating Kali Linux on USB
        • Making a Kali Bootable USB Drive on Windows
        • Standalone Kali Linux 2021.4 Installation on a USB Drive, Fully Encrypted
        • Adding Persistence to a Kali Linux Live USB Drive
        • Adding Encrypted Persistence to a Kali Linux Live USB Drive
        • USB Boot in VirtualBox
        • USB Boot in VMware
      • Kali On ARM
        • BeagleBone Black
        • Acer Tegra Chromebook 13" (Nyan)
        • ASUS Chromebook Flip (Veyron)
        • Banana Pro
        • Banana Pi
        • CubieBoard2
        • CuBox-i4Pro
        • CubieTruck (CubieBoard3)
        • Gateworks Newport
        • CuBox
        • Gateworks Ventana
        • NanoPi NEO Plus2
        • NanoPi2
        • Mini-X
        • NanoPC-T3
        • ODROID-C0/C1/C1+
        • ODROID-XU3
        • ODROID-U2/U3
        • ODROID-C2
        • Pinebook
      • Containers
        • Kali Linux LXC/LXD Images
        • Official Kali Linux Docker Images
        • Installing Docker on Kali Linux
        • Using Kali Linux Docker Images
        • Using Kali Linux Podman Images
      • WSL
        • Win-KeX SL
        • Win-KeX ESM
        • Preparing a system for WSL
        • Win-KeX
        • Win-KeX Win
      • Cloud
        • Digital Ocean
        • AWS
        • Azure
        • Linode
      • Kali NetHunter
        • Installing NetHunter On the OnePlus 7
        • Installing NetHunter On the Gemini PDA
        • Installing NetHunter
        • Installing NetHunter On the TicWatch Pro 3
        • Installing NetHunter On the TicWatch Pro
        • NetHunter Application - Terminal
        • NetHunter BadUSB Attack
        • NetHunter Bluetooth-Arsenal
        • NetHunter Chroot Manager
        • NetHunter Components
        • NetHunter Custom Commands
        • NetHunter Home Screen
        • NetHunter DuckHunter Attacks
        • NetHunter HID Keyboard Attacks
        • NetHunter Exploit Database SearchSploit
        • NetHunter Kali Services
        • NetHunter MAC Changer
        • NetHunter MANA Evil Access Point
        • NetHunter Man In The Middle Framework
        • NetHunter KeX Manager
      • Tools
        • Installing Tor Browser on Kali Linux
        • Kali Tools
        • Installing snapd on Kali Linux
        • Metasploit Framework
        • Installing Flatpak on Kali Linux
        • Submitting tools to Kali
        • Removed Tools From Kali
      • Troubleshooting
        • Discovering Problems With Download Speed
        • Common Cloud Based Setup Information
        • The Basics of Troubleshooting
        • Troubleshooting Installations Failures
        • Troubleshooting Wireless Drivers
        • Minimum Install Setup Information
      • Kali Development
        • Contributing run-time tests with autopkgtest
        • Custom CuBox Image
        • Custom Beaglebone Black Image
        • Custom EfikaMX Image
        • Custom Chromebook Image
        • Custom MK/SS808 Image
        • Custom Raspberry Pi Image
        • Custom ODROID X2 U2 Image
        • Setting up a system for packaging
        • Intermediate packaging step-by-step example
        • Introduction to packaging step-by-step example
        • Getting the best out of the Kali Bot
        • Advanced Packaging Step-By-Step Example (FinalRecon & Python-icmplib)
        • Generate an Updated Kali ISO
        • Creating A Custom Kali ISO
        • Building Custom Kali ISOs
        • Rebuilding a Source Package
        • Recompiling the Kali Linux Kernel
        • ARM Build Scripts
        • Preparing a Kali Linux ARM chroot
    • Arch Linux
      • Installation Guide
      • Frequently Asked Questions
      • General Recommendations
      • Applications
        • Office & Docs
        • Internet
        • Multimedia
        • Science
        • Security
        • Utilities
        • Others
      • Arch compared to other distributions
    • NetBSD
      • Calls and Errors
      • Libraries
      • Lua Modules
      • Devices and Drivers
  • Law, Policy, and Ethics
    • Fair Use
    • DMCA
      • 🗄️Notable Cases
        • MGM Studios Inc. v. Grokster, Ltd.
        • Viacom International, Inc v YouTube, Inc
        • Capitol Records, Inc. v. Thomas-Rasset
        • Perfect 10, Inc. v. Amazon.com
        • Recording Industry Association of America (RIAA) v. Diamond Multimedia Systems, Inc.
        • A&M Records, Inc. v. Napster, Inc.
        • BMG Music v. Gonzalez
        • Sony Computer Entertainment America (SCEA) v. Connectix Corp.
        • Columbia Pictures Industries, Inc. v. Fung
        • Warner Bros. Entertainment Inc. v. RDR Books
        • BMG Music v. John Doe
        • Universal Music Group v. Veoh Networks, Inc.
        • Universal Music Group v. MySpace, Inc.
        • UMG Recordings, Inc. v. MP3.com, Inc.
        • Cartoon Network LP v. CSC Holdings, Inc.
        • Metro-Goldwyn-Mayer Studios Inc. v. Grokster, Ltd.
        • Viacom International Inc. v. Google Inc.
        • Tiffany (NJ) Inc. v. eBay Inc.
        • Perfect 10, Inc. v. Visa International Service Association
        • Universal City Studios Productions LLLP v. Reimerdes
        • Recording Industry Association of America (RIAA) v. Lime Group LLC
        • Sony BMG Music Entertainment v. Tenenbaum
        • Viacom International Inc. v. Time Warner Cable Inc.
        • UMG Recordings, Inc. v. Shelter Capital Partners LLC
        • Sony Computer Entertainment America Inc. v. Bleem LLC
        • Universal City Studios, Inc. v. Corley
        • Ticketmaster Corp. v. Tickets.com, Inc.
        • Authors Guild, Inc. v. Google, Inc.
        • Perfect 10, Inc. v. Cybernet Ventures, Inc.
        • Tiffany (NJ) Inc. v. Ningbo Beyond Home Textile Co., Ltd.
        • Google Inc. v. American Blind & Wallpaper Factory, Inc.
        • Columbia Pictures Industries, Inc. v. Redd Horne, Inc.
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub
  1. Operating Systems
  2. Kali Linux
  3. Kali Development

Contributing run-time tests with autopkgtest

PreviousKali DevelopmentNextCustom CuBox Image

Last updated 2 years ago

Was this helpful?

Why Kali could benefit from your help

With Kali Linux being a rolling distribution it will occasionally have packages that break due to changes in their dependencies. To best combat this, we have set up that have to be passed in order to allow the packages to migrate into . However, most packages at this time don’t have a comprehensive autopkgtest associated with it causing this process to not reach its full potential. This is where the community can help us, by contributing test for packages that lack them.

A bit of autopkgtest background

Autopkgtests were created as a way to test a package as accurately as possible on a real Debian system. This method can leverage virtualization and containers to create an accurate and re-usable environment that will be the testing area. There is a lot more information available on this topic. For a more in depth look than what will be covered here please see the following articles (which served as sources of information during writing):

We encourage users to spend some time looking through these after finishing reading the rest of this doc. We will only be really covering a small portion of what is a large topic to explore and learn.

Structuring Test Files

Adding autopkgtests to a source package is a matter of creating a debian/tests/control file: that file describes all the tests that have to be run. Just like debian/control, it uses a set of entries structured like mail headers (e.g. Field-Name: some value). In the simplest case, the test can be fully described in the control file, but for most non-trivial tests, you will just reference an external test script that you will just put alongside in debian/tests/.

Have a look a the to gain a good idea of what we are doing and what’s possible. A lot of the options are explained quite well, and through active experimentation and learning can be utilized in helpful ways, but for now we will stick to some of the basics. Lets take a look at their example control file:

Tests: fred, bill, bongo
Depends: pkg1, pkg2 [amd64] | pkg3 (>= 3)
Restrictions: needs-root, breaks-testbed

We can break this down pretty quickly. There are three test scripts: fred, bill, and bongo. These three test scripts exist in debian/tests/. While we don’t know what these tests do, we do know through the Depends fields that they will require two packages (the example voluntarily shows some advanced dependency syntax that you will usually not need, but for the sake of the explanation it means that you always want pkg1, and on an amd64 machine, you also need pkg2, while on other machines, you will want pkg3 with a version greater or equalt to 3). We also know that this test suite needs to use the root user, or have elevated privileges, and also has the potential to break the testbed (which means that autopkgtest will refuse to run the test if the testbed can’t be easily restored).

Looking at another example that they have:

Test-Command: foo-cli --list --verbose
Depends: foo
Restrictions: needs-root

Here we see that they are running a test through Test-Command. This is a way to prevent creating scripts to run tests, and can come in handy to prevent having a one-liner script that is being called. In this case, they run foo-cli which we can tell is part of the foo package, as shown by the Depends field. It seems like this test is superficial, as it is only one line, which should be noted in the Restrictions, so if this was a real package we may have wanted to investigate further. We can most commonly tell if a test should contain Restrictions: superficial if they only use one line of commands to test the tool.

Control files will include one of the methods of testing, whether it is a script or a one-liner, what the test depends on, and any potential restrictions or problems that the test might have. Keep in mind that if your test only needs the built binary packages, then you don’t need to specify any explicit Depends field as you can rely on the default value of Depends: @.

If you want to install a supplementary package foobar for your test, then you would use Depends: @, foobar as @ is as shorthand for the binary packages generated by the source package containing the tests.

A few other important restrictions that should be remembered (but do look through all that are possible!):

  • allow-stderr - Accepts output to stderr and will not fail when there it receives any.

  • rw-build-tree - The test will have access to the built source tree, with certain conditions applied.

  • isolation-container - Any services or open TCP ports that are a part of the test will utilize their own container or VM to prevent fails.

  • isolation-machine - The test will likely interact with the container/VM in a way that would typically result in an error, such as rebooting or interacting with the kernel.

  • needs-internet - The test will need unrestricted access to the internet.

Learning by examples

There are a few things to learn about with autopkgtests that we can see in action already. We will look at what tests can look like, what types are more beneficial, and a few tricks that we can see.

cloud_enum control file

Test-Command: cloud_enum --help
Depends: @
Restrictions: superficial

python-pip control file

Tests: pip3-root.sh
Restrictions: needs-root

Tests: pip3-user.sh
Restrictions: breaks-testbed

# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823358
Tests: pip3-editable.sh

With includes a test that looks like:

python-pip pip3-user.sh test file

#!/bin/sh

export HOME=$AUTOPKGTEST_TMP
export PATH=$PATH:$HOME/.local/bin
export PIP_DISABLE_PIP_VERSION_CHECK=1

if [ $(id -u) = 0 ]
then
    adduser --quiet --system --group --no-create-home testing
    user=testing
    mkdir $HOME/.cache
    chown -R $user $HOME
    runuser="runuser -p -u $user --"
else
    runuser=""
fi

$runuser python3 -m pip install world
$runuser python3 -m pip list --format=columns
$runuser python3 -m pip show world
ls -ld $HOME/.local/lib/python3.*/site-packages/world-*.dist-info
$runuser python3 -m pip uninstall -y world
$runuser python3 -m pip list --format=columns
# Temporarily disabled.  See #912379
#$runuser python3 -m pip list --outdated
if [ $(id -u) = 0 ]
then
    deluser --quiet testing
fi

It is apparent that there is a big difference between the two. Both will technically create a pass or a fail, and one might think that the first option, the cloud_enum control file, seems easier and better to use. However, this is a superficial test which means “The test does not provide significant test coverage, so if it passes, that does not necessarily mean that the package under test is actually functional”. Because of this, we would rather have the python-pip test environment. It is more detailed and includes tests of different types of end-user environments.

We did just throw a bunch of text and code at you all to look at without much explanation, as it was important to note the differences before getting too involved. But lets take a step back now and look more closely at this.

Looking more closely at the cloud_enum control file, we have three things going on:

Test-Command: cloud_enum --help
Depends: @
Restrictions: superficial

The first thing we do is tell what command we are running. Because this is just a superficial test, we do not tell it to run a script and instead just do one line to test if it outputs the help. We then tell it to depend on the same dependencies that the package depends on, by using ‘@’. Should the test need other dependencies, we can list them here. In another test we will look at, this will be seen in action. We finally tell it that yes, this test is just a superficial one. We could tell it other things, like that cloud_enum needs root, or that after running this test the testing environment will need to be scrapped.

Tests: test3-pytest-factoryboy
Depends: @, python3-pytest-pep8

As we can see, it depends on a python module. This python module will help us to test the tool, as we can see being done in the test:

#!/bin/sh
set -e
cp -r tests "$AUTOPKGTEST_TMP/" && cd "$AUTOPKGTEST_TMP"
for py in $(py3versions -i); do
    $py -Wd -m pytest -v -x tests 2>&1;
done

We can also see something else being done that is a good practice. $AUTOPKGTEST_TMP can be used during these tests, and will help to clean a system and start on a next test, assuming the breaks-testbed flag is not set.

What to do

So you have a system set up, have seen tests in practice, but now you need to actually write the tests. Once you find a package that has no test or a superficial test, you should learn the tool. Let us say that we have a tool that pings a server and then depending on the response will tell the user something. What does a fail look like? What does a success look like? Are there any edge cases that might mess up your test?

The next step after learning about a tool’s purpose is to figure out what you will need to tell autopkgtest. Using our tool from the previous example, will it need root to ping properly? What else might need to be done with the machine or environment to get the tool to work? What about after the test? Will there be an output file, and so therefore you should use $AUTOPKGTEST_TMP?

Once you learn and note what you can test about the tool, you should start to write the test using the best practices and tips given earlier. There are many available resources that stem from those resources, and lots more packages that have solid tests that you can learn from example with if need be.

Once you do get a completed green test, you should submit a merge request since your work is already on GitLab. The Save & Share section of contributing to packages gives a good example of this.

For more information on the environment itself, the autopkgtest-build-qemu and autopkgtest-virt-qemu man pages are beneficial. As are the docs located at /usr/share/doc/autopkgtest/.

Running tests locally

If it is needed to run the tests locally, it is luckily fairly straightforward but needs some preparation.

We first download the actual testing software and also vmdb2 which is to create the disk images to be used for the tests:

:~$ sudo apt install -y autopkgtest vmdb2

We then need to make a directory that can store our images and data:

:~$ sudo mkdir /srv/autopkgtest-images/

The following command creates a kali-rolling based image that is compatible with autopkgtests, we can now use this image for future tests:

:~$ sudo autopkgtest-build-qemu kali-rolling /srv/autopkgtest-images/kali-rolling.img http://http.kali.org/kali

Now to run your tests, all you have to do is run the following command: autopkgtest ~/packages/mytest/ -- qemu /srv/autopkgtest-images/kali-rolling.img and point to the directory that contains the package you want to test.

There are many other variations which will potentially be needed, and can be found at /usr/share/doc/autopkgtest/README.running-tests.html. It is encouraged to use the GitLab CI however, as many things have the potential to go wrong with local tests and it is easier to get help if needed when using GitLab.

The control file looks like the following:

Whereas the control file looks like the following:

The next control file we will learn from is :

Another package that can show a lot of info is . However, due to the size of the files and the breakdowns associated that will not be explained here.

Given that Kali uses GitLab, you should fork the where you want to add tests, and add your tests in a new branch. Our GitLab CI configuration should run the autopkgtest, so you can have a look at the output of the pipelines to see whether your tests have been correctly run and whether they succeeded. If they do not work, you can iterate (adding more commits) until you are satisfied with your test scripts.

automated runtime tests
kali-rolling
The official Debian CI doc page
A great talk covering autopkgtests
A paper going in-depth on writing As-Installed tests
The best practices Debian wiki page
official documentation of those control files
cloud_enum
python-pip
pytest-factoryboy’s
hyperion
source package