Using apt-cacher-ng with SSL/TLS

TL;DR

apt-cacher-ng is a write-through proxy that caches repository metadata and package indexes for other hosts, typically on the same network. It generally works out-of-the-box, except for when the repository is served over SSL/TLS, in which case special configuration is needed. This blog post will cover apt-cacher-ng basics, then how to set it up to work with SSL/TLS repositories (such as packgecloud.io).

Use Cases

The common use case for apt-cacher-ng is where you have a sizable network and would like to speed up all the apt-get update calls made by your nodes, especially if you are running chef or puppet periodically. It is also used for complying to security policies that restrict internet access to bulk of hosts except for a designated proxy server.

Installation & Configuration

On debian and ubuntu, it’s already part of the default repositories, to install it just do:

sudo apt-get install apt-cacher-ng

Then you can start it with with:

sudo /etc/init.d/apt-cacher-ng start

If everything went okay, then when visiting http://yourdomain.example:3142 you should see instructions on how to configure your clients to use the proxy.

If the repositories on your your systems do not make use of SSL/TLS then this setup will “just work”, no further server configuration required. However, if you wish to proxy SSL/TLS repositories (like packagecloud repositories), you’ll need to read the section below on “SSL/TLS Setup”.

The /etc/apt-cacher-ng/acng.conf file is pretty well commented, and the online documentation lists all the available options.

Client Proxy Configuration

There are two ways to tell apt-get to use your proxy:

Global /etc/apt/apt.conf method

Add this line to /etc/apt/apt.conf (or to a file named /etc/apt/apt.conf.d/00proxy):

Acquire::http { Proxy "http://yourdomain.example:3142"; };

This tells apt to proxy all requests to that address and port.

Per repository /etc/apt/sources.list method

You can edit the /etc/apt/sources.list file and append the proxy to the beginning like so:

Before:

deb http://ftp.debian.org/debian stable main contrib non-free
deb-src http://ftp.debian.org/debian stable main contrib non-free

After:

deb http://yourdomain.example:3142/ftp.debian.org/debian stable main contrib non-free
deb-src http://yourdomain.example:3142/ftp.debian.org/debian stable main contrib non-free

Note about Cache Visibility

It’s important to note that apt-get has it’s own internal cache, and is pretty good about not downloading the same indexes and packages over and over again. Unfortunately, this is at odds with apt-cacher-ng, as it cannot cache requests it never receives. You can force a download by clearing the contents of local caches (/var/cache/apt/* and /var/lib/apt/lists/*), then running apt-get update again to ensure that apt-cacher-ng sees all possible requests. Note: apt-get clean is not sufficient, as it only clears the local package cache, and not the index and list cache.

Administration & Statistics

You can verify your proxy is working by visiting yourdomain.example:3142/acng-report.html. Here you can see some statistics about your proxy and also tweak some settings from the web interface (more on this below). Clicking ‘Count Objects’ will show you your cache hit rate:

There is also a bundled script you can run to view (and purge) cache information, located at /usr/lib/apt-cacher-ng/distkill.pl, and its output looks like:

Scanning /var/cache/apt-cacher-ng, please wait...
Found distributions:
	1. testing (6 index files)
	2. sid (63 index files)
	3. etch-unikl (30 index files)
	4. etch (30 index files)
	5. experimental (505 index files)
	6. lenny (57 index files)
	7. unstable (918 index files)
	8. stable (10 index files)

Setting a password

Certain administrative features require a password, which can be set in /etc/apt-cacher-ng/security.conf with the AdminAuth: option:

AdminAuth: myuser:mypassword123

Make sure to restart apt-cacher-ng any time you edit /etc/apt-cacher-ng/acng.conf.

SSL/TLS (HTTPS) Setup

In order for SSL/TLS to work, apt-cacher-ng has to be told beforehand which domains it can CONNECT to via the PassThroughPattern: option in acng.conf. For instance, to allow apt-cacher-ng to proxy anything, you can do:

PassThroughPattern: .*

Or, you can be specific to your repository:

PassThroughPattern: apt\.repo\.example:443$

For packagecloud repositories, you’ll need to allow the packagecloud domain AND the S3 bucket subdomain with the following pattern:

PassThroughPattern: (packagecloud\.io|packagecloud-repositories\.s3\.dualstack\.us-west-1\.amazonaws\.com|packagecloud-prod\.global\.ssl\.fastly\.net):443$

Note: Once PassThroughPattern is set, apt-cacher-ng will proxy but not cache objects stored on SSL/TLS repositories. This is because it cannot understand the encrypted traffic it is proxying.

Caching objects

To cache packages and indexes retrieved via SSL/TLS repositories, we will have to configure apt-cacher-ng to remap a chosen non-https repository url to the actual https url.

Create a file named backends_packagecloud in /etc/apt-cacher-ng that looks like:

https://packagecloud.io/julio/test_repo/ubuntu

Now, in apt-cacher-ng /etc/apt-cacher-ng/acng.conf we can specify a mapping like:

Remap-pcloud: http://fakedomain.example ; file:backends_pcloud

Finally, we can now configure all our clients to use http://fakedomain.example instead of https://packagecloud.io/julio/test_repo/ubuntu, and apt-cacher-ng will transparently proxy, and more importantly, cache, objects from https://packagecloud.io/julio/test_repo/ubuntu.

Importing Packages

apt-cacher-ng supports importing packages beforehand, provided it has already cached the indexes referring to those packages.

Manually via a /var/cache/apt-cacher-ng/_import/

Create the directory /var/cache/apt-cacher-ng/_import/ on the apt-cacher-ng host and transfer all the packages you want proxied there..

Once that is complete, click ‘Start Import’ on the admin page, located at http://yourdomain.example:3142/acng-report.html.

Automatically from package indexes (Mirroring)

It’s also possible to have apt-cacher-ng automatically download and import packages given a pattern to look for packages, based on it’s previously cached indexes. For example, here we’re using the pcloud mapping we created earlier to download all binary-amd64 packages from our packagecloud repository.

PrecacheFor: pcloud/dists/trusty/*/binary-amd64/Packages*

Here are a few options worth setting in the admin page:

• Force the download of index files (yes)
• Calculate and display download size (yes)
• Download Package Files (yes)
• Restrict to packages related to previously cached files (no)

Then you can click ‘Start Mirroring’ to start the process. Once it finishes, you can see what packages it found it downloaded:

Conclusion

apt-cacher-ng is a great tool in the toolbox and can really speed things up, especially for larger, more automation-driven setups. Be sure to read the full documentation, especially the parts on repository url remapping and SSL/TLS.

Set up your own package repository.

Fast, reliable, and secure software starts here.

Try Packagecloud