Debugging SSL in Java using mitmproxy

TL;DR

In this post we’ll go over setting up the popular mitmproxy tool on an external host and configuring your Java programs to proxy traffic through it, allowing you to debug misbehaving HTTP clients and libraries.

Overview

Occasionally, you’ll be faced with a buggy Java program that is using HTTPS to communicate with an API. You could simply dump everything out using -Djavax.net.debug=all, but sometimes this approach is cumbersome, as it only lets you view traffic, not replay it or modify it. This is where mitmproxy comes in.

In this post, we’ll set up mitmproxy on an external host and use it to intercept some HTTPS traffic from our example client. Let’s get started!

What is mitmproxy?

mitmproxy is a python program that transparently proxies any traffic sent to it. Through its console interface, you can inspect, capture and modify HTTP/HTTPS traffic flows as they are happening. Think of it like a step-through debugger, but for HTTP requests and responses.

The example client

We’ve created a simple java program for this blog post that will serve as the target application we wish to capture traffic from. This program (shown below) simply does a GET via HTTPS and prints out the certificate issuers for the example_url system property, defaulting to "https://www.google.com".

Here is what our example client looks like (imports and exception handling omitted for brevity):

url = new URL(System.getProperty("example_url", "https://www.google.com/"));
connection = (HttpsURLConnection)url.openConnection();
connection.connect();

System.out.println(String.format("GET %s -> %s", connection.getURL(), connection.getResponseCode()));

Certificate[] serverCertificates = connection.getServerCertificates();
X509Certificate x509cert;
for (Certificate serverCertificate : serverCertificates) {
    x509cert = (X509Certificate)serverCertificate;
    System.out.println(x509cert.getIssuerX500Principal().getName());
}

If we run it with no arguments, the output looks like:

$ java -jar target/https-client-example-1.0.jar
GET https://www.google.com/ -> 200
CN=Google Internet Authority G2,O=Google Inc,C=US
CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US
OU=Equifax Secure Certificate Authority,O=Equifax,C=US

If you’d like to follow along, feel free to download or compile the example program.

Setting up the mitmproxy host

Now that we have verified that our example program works as intended, we can continue setting up mitmproxy on an external host.

Installing mitmproxy

Traditionally, most documentation and tutorials will have you install mitmproxy on the same host you wish you capture traffic from. However, we prefer using a cheap disposable VPS instead, this way you avoid any issues around setting up a working python environment on your local machine.

For this example, we’ll be using the $5 VPS from our friends over at DigitalOcean. Select the latest Ubuntu version from their dropdown and login to the machine once it’s ready.

$ ssh x.x.x.x

Since this package is available from Ubuntu’s repositories, we can just install it with apt-get install.

$ sudo apt-get update
$ sudo apt-get -y install mitmproxy

Starting mitmproxy

Since networks tend to block the default proxy port 8080, we’ll start our mitmproxy server on port 1080 instead.

$ mitmproxy --port 1080

If everything was installed correctly, it should bring up the mitmproxy console interface, letting us know the server is running and accepting connections on port 1080.

(Make sure to leave this session open and running)

Configuring the Client

In order for mitmproxy to successfully intercept and modify encrypted HTTPS traffic, Java needs to trust our proxy as a Certificate Authority. For the specifics on how this works in practice, see: Implementation Weakness of the Trusted Third Party Scheme and How mitmproxy works.

Viewing the generated certificates

When we started mitmproxy above, it should have generated a self-signed certificate bundle inside of ~/.mitmproxy. If we inspect that directory, we should see that it generated a few certificates in various different formats.

$ ls -1 ~/.mitmproxy
ls -1 ~/.mitmproxy/
mitmproxy-ca-cert.cer
mitmproxy-ca-cert.p12
mitmproxy-ca-cert.pem
mitmproxy-ca.pem
mitmproxy-dhparam.pem

Transferring the certificates to client

Since Java accepts PEM formatted certificates, we only need to copy the mitmproxy-ca-cert.pem file to our client computer.

$ scp root@x.x.x.x:/root/.mitmproxy/mitmproxy-ca-cert.pem .

Import certificate into Java KeyStore using the keytool command

Before we can import our certificate into the Java KeyStore, we need find the lib/security/cacerts location for our Java installation. If the JAVA_HOME environment variable is set, this is located at $JAVA_HOME/lib/security/cacerts. However, if this variable is not set, you’ll need to consult your system’s Java documentation for instructions.

The keytool command, located under $JAVA_HOME/bin/keytool, is a utility that manages keys and certificates for the Java keystore. To import our certificate we run the following command:

$ sudo $JAVA_HOME/bin/keytool -import -trustcacerts  \
-file /path/to/mitmproxy-ca-cert.pem                 \  
-alias mitmproxycert                                 \
-keystore $JAVA_HOME/jre/lib/security/cacerts  

You will be prompted for your sudo password first, then you will be prompted for the keystore password:

Password:
Enter keystore password:

The default password for the Java keystore is:

changeit

Upon successfully entering the password you will be shown the mitmproxy certificate and asked if you want to trust it:

Owner: O=mitmproxy, CN=mitmproxy
Issuer: O=mitmproxy, CN=mitmproxy
Serial number: d73540a161b
Valid from: Wed Nov 09 16:03:54 EST 2016 until: Mon Nov 11 16:03:54 EST 2019
Certificate fingerprints:
	 MD5:  5C:D2:07:B5:19:55:3A:E6:30:98:1D:54:FE:7D:90:A5
	 SHA1: 00:BE:EC:7E:71:7B:01:E0:A7:40:5F:30:A5:80:9A:47:96:85:94:9B
	 SHA256: 8C:F5:32:5D:D5:7D:DF:AB:32:6A:8E:89:AC:00:38:56:8E:6F:00:10:0A:64:A6:18:90:26:E9:C8:31:6E:65:33
	 Signature algorithm name: SHA256withRSA
	 Version: 3
   ...
   Trust this certificate? [no]: yes

Type yes and you should see:

Certificate was added to keystore

Running our example

We’ll call our example program again, this time passing in our proxy settings as arguments. You can also export a JAVA_OPTS environment variable containing the proxy configuration for situations where you can’t control the program’s arguments.

$ java -Dhttp.proxyHost=x.x.x.x          \
-Dhttp.proxyPort=1080                    \
-Dhttps.proxyHost=x.x.x.x                \
-Dhttps.proxyPort=1080                   \
-jar target/https-client-example-1.0.jar

Note: it is important that the proxy configuration come before the -jar argument.

If everything was setup correctly, you should see that the certificate for https://www.google.com is now issued by our own Certificate Authority, mitmproxy.

GET https://www.google.com/ -> 200
O=mitmproxy,CN=mitmproxy
O=mitmproxy,CN=mitmproxy

Subsequently, in our mitmproxy session, we should also see the intercepted flow on the UI:

Hitting enter on a selected flow lets you inspect it, hitting tab lets you cycle through Request, Response and TCP details.

Use q to go back to the list of flows.

Feel free to run the client a few more times so you can see how the mitmproxy UI handles multiple flows.

Hit ? for a list of things you can do to any selected flow.

A particularly useful one is r, which replays the selected request. Very handy for server development.

Conclusion

This minimal example should serve as a playground to explore the concepts around debugging Java HTTP behavior using mitmproxy. For bonus points, try intercepting some maven or sbt flows using JAVA_OPTS!

A full guide to using mitmproxy is out of the scope of this post but you should definitely check out their comprehensive documentation for a thorough guide on all of the things mitmproxy can do.

Set up your own package repository.

Fast, reliable, and secure software starts here.

Try Packagecloud