Following up on our previous post about yum repository internals, this blog post will dive into the internals of APT repositories. Similarly, we’ll cover what each index file does and take a look at how a user can inspect and verify the metadata themselves.
What is an APT repository?
An APT repository is a collection of deb packages with metadata that is readable by the
apt-* family of tools, namely,
apt-get. Having an APT repository allows you to perform package install, removal, upgrade, and other operations on individual packages or groups of packages.
APT repositories are essential for storing, managing, and delivering software to Debian and Ubuntu systems.
Create an APT repository with reprepro
Generate GPG key for signing an APT repository
(Feel free to skip this section if you already have a GPG key you wish to use)
We’ll start by creating a GPG key to sign our repository metadata for a fuller, more illustrative example:
$ gpg --gen-key
Since this is a throwaway blog example, we’ll just pick all the defaults without a passphrase.
NOTE: You’ll definitely want a passphrase for production deployments and
ask-passphrase in your
Now when we
gpg --list-keys, we should see:
$ gpg --list-keys pub 2048R/78AF53BD 2015-08-02 uid Test Repo User <email@example.com> sub 2048R/3F0EF11C 2015-08-02
Before taking a closer look at the repository metadata itself, let’s show how to get a repository set up by using the open source reprepro command line tool.
You can create an APT repository by using the command line tool
reprepro which you can install on Ubuntu and Debian systems by running:
$ sudo apt-get install reprepro
Once installed, let’s create a directory for our APT repository and reprepro configuration.
Let’s create the directory structure necessary for
reprepro and it’s configuration.
$ mkdir -p my_test_repo/apt/debian/conf
conf directory, create a file named
distributions, here we will list all the distributions and architectures our repository will support. For this example, we’ll only support one distribution, debian squeeze:
Origin: my_test_repo Label: my_test_repo Codename: squeeze Architectures: i386 amd64 Components: main Description: Apt repository for my_test_repo SignWith: 3F0EF11C Contents: percomponent
The value of
SignWith being the subkey ID from the GPG key we wish to use (the one we generated above, in this case)
This defines a
my_test_repo for debian squeeze.
Now let’s download an example package to add to this repository:
$ wget -q 'https://packagecloud.io/computology/packagecloud-test-packages/packages/ubuntu/precise/packagecloud-test_1.1-2_amd64.deb/download' -O /tmp/packagecloud-test_1.1-2_amd64.deb
Generate APT repository with
To add that package to this repository, we’lll call
reprepro with the desired distribution and path to debian package, from inside
my_test_repo/apt/debian directory, like so:
$ reprepro includedeb squeeze /tmp/packagecloud-test_1.1-2_amd64.deb
This will create and sign all the necessary repository metadata APT needs to install and verify this repository on another computer.
Hosting GPG key and APT repository over HTTP
Once the package has been added and repository indices have been generated, we’ll need to serve them up over HTTP so that APT can read it.
First, let’s export our GPG key and host it along with our repository, that way anyone installing this repository can also import our GPG public key.
$ gpg --export --armor 3F0EF11C > my_test_repo/gpg.key
Then, let’s serve up our repository using python’s built in HTTP server.
my_test_repo and run:
$ sudo python -m SimpleHTTPServer 80
(Do this in a screen session if you’d like use the same terminal session for the rest of the tutorial)
Installing Package from APT repository
Finally, we can install a package from our configured and hosted repository!
/etc/apt/sources.list.d/my_test_repo.list on the computer you wish you install our package on and add the following line, substituted with the IP of the computer hosting the repository:
deb http://my_test_repo.example/apt/debian squeeze main
Let’s also import our GPG key into APT:
$ wget -O - http://my_test_repo.example/gpg.key | sudo apt-key add -
Now we can run
apt-get update to fetch the repository metadata and install our package:
$ sudo apt-get update $ sudo apt-get install packagecloud-test=1.1-2
On the server, you should see requests coming in for various repository metadata as APT tries install this package:
[02/Aug/2015 17:54:59] "GET /apt/debian/dists/squeeze/Release.gpg HTTP/1.1" 200 - [02/Aug/2015 17:54:59] "GET /apt/debian/dists/squeeze/main/i18n/Translation-en.bz2 HTTP/1.1" 404 - [02/Aug/2015 17:54:59] "GET /apt/debian/dists/squeeze/Release HTTP/1.1" 200 - [02/Aug/2015 17:54:59] "GET /apt/debian/dists/squeeze/main/binary-amd64/Packages.bz2 HTTP/1.1" 404 - [02/Aug/2015 17:54:59] "GET /apt/debian/dists/squeeze/main/binary-amd64/Packages.lzma HTTP/1.1" 404 - [02/Aug/2015 17:54:59] "GET /apt/debian/dists/squeeze/main/binary-amd64/Packages.gz HTTP/1.1" 200 -
On the client, you can verify it successfully installed our package, by running:
Of course, using packagecloud is a much faster and simpler solution :) with SSL, GPG, authentication, collaboration, and everything else you need ready to go!
APT repository metadata
Now that we have verified our APT repository works by installing our package on another computer, let’s dive into the mechanics of how an APT repository works.
Let’s take a look at our
reprepro generated directory structure from the root of
my_test_repo/apt/debian and what each file does:
./dists/squeeze/Contents-<arch>.gz: A gzipped index containing a directory listing of all packages for this architecture, across all components.
./dists/squeeze/Release: The primary index which contains the locations and checksums of the architecture-specific Packages files.
repreprowill generate a detached signature for the above Release index.
./dists/squeeze/InRelease: Also only generated if
SignWithwas provided, it’s a version of Release with a clearsign signature, used by newer versions of APT.
./dists/squeeze/main/binary-<arch>/Packages: Lists metadata about each package available for a given architecture
./dists/squeeze/main/Contents-<arch>.gz: A gzipped index containing a directory listing of all packages for this architecture
<arch>, for the
./pool/main/*: Directory where the actual package files reside.
The examples below will show how you can view the index yourself to verify it contains what you expect.
GPG Verification of signed APT repository metadata
APT automatically verifies GPG signed metadata if it knows the public key (which we imported above), but we can also verify manually to be extra sure.
Let’s import the public key into our GPG keyring instead of APT’s:
wget -O - http://my_test_repo.example/gpg.key | gpg --import
Release file and its signature:
$ wget http://my_test_repo.example/apt/debian/dists/squeeze/Release.gpg $ wget http://my_test_repo.example/apt/debian/dists/squeeze/Release
Verify it with GPG:
$ gpg --verify Release.gpg Release
For more information on GPG signatures and APT packages, check out our post on gpg sign and verify deb packages.
Release metadata for APT repository
Let’s get the
Release for ubuntu precise for the packagecloud test packages repository:
$ curl -L https://packagecloud.io/computology/packagecloud-test-packages/ubuntu/dists/precise/Release
It will return a huge list of all the architectures and their relevant
Packages files and checksums, so for
binary-amd64, for our example we have:
94fb8d5d4355c0c891a8756453d0ac85cf34dff1fc461abfbd02ec7f85044e46 658 main/binary-amd64/Packages 6d54859644489197158046e0afd1bf5816eda8713d31227607df9fa23d52e1cb 486 main/binary-amd64/Packages.bz2 63db7747a633bb3f16cfa7996648f03b8d33aff027216302b02f0ac00aa54fb0 445 main/binary-amd64/Packages.gz f28c0ba65f233f4352ec3e33a7c6c5578d5ed4f862e4107706a82c22efbc8c38 524 main/binary-amd64/Packages.xz
Packages metadata for APT repository
Let’s get the
Packages for ubuntu precise and
binary-amd64 for the packagecloud test packages repository:
$ curl -L https://packagecloud.io/computology/packagecloud-test-packages/ubuntu/dists/precise/main/binary-amd64/Packages
We can also verify the checksum (SHA256 in this case) for this
Packages file contained in
$ curl -sL https://packagecloud.io/computology/packagecloud-test-packages/ubuntu/dists/precise/main/binary-amd64/Packages | shasum -a 256
This should return the expected checksum, which is:
Once the checksum is verified, we can look at the actual metadata, which looks like:
Package: packagecloud-test Priority: optional Section: misc Installed-Size: 39 Maintainer: Joe Damato <firstname.lastname@example.org> Architecture: amd64 Version: 1.1-2 Depends: libc6 (>= 2.2.5) Filename: pool/precise/main/p/packagecloud-test/packagecloud-test_1.1-2_amd64.deb Size: 2830 MD5sum: b709d6a5b2e6883ee97803e5fd1b204e SHA1: ba4500d3b385aced8c8ed3c48981eba2dd2005b4 SHA256: 8ed8bb73915088d422c26ce3e92c6477b3afdfcd7377713880db8ba68a697289 SHA512: afcde64a1bfc5fdce1fff00002a2007a7f05eef51e45eb223885b89e2deb45cefd51dd596d79dea1daf020312afb14967eaef2d1a4c45f7da28ec045bfc2efc6 Description: packagecloud-test Description-md5: 49d80a99b15b8081983c8a12416577a7
Contents metadata for APT repository
Lastly, let’s check out the
Contents file for the packagecloud test packages repository and
$ curl -sL https://packagecloud.io/computology/packagecloud-test-packages/ubuntu/dists/precise/Contents-amd64.gz | zless
This will list out all the files associated with each package:
usr/local/bin/packagecloud_hello misc/packagecloud-test=1.1-2 usr/share/doc/packagecloud-test/changelog.Debian.gz misc/packagecloud-test=1.1-2 usr/share/doc/packagecloud-test/copyright misc/packagecloud-test=1.1-2
APT repository metadata is comprised of a set of index files, checksums, and in some cases a GPG signature. The metadata describes which packages can be found in a repository, various attributes about each package, file and directory listings.
APT repository metadata can be manually examined and verified on the command line using a combination of
shasum. This is useful if you need to debug some sort of issue with your repository (missing package, missing dependency, incorrect version, etc) or are curious about the inner workings of one of the most important pieces to your operating system.