With the recent arrival of Let's Encrypt TLS/SSL certificate support on Compose's Elasticsearch and RabbitMQ, we've noticed some issues popping up from the rabbit hole that is this encryption support.

Top of that list is problems with Java connections erroring out with something like:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target  
...

This error effectively means that your Java runtime doesn't know about the Let's Encrypt certificates needed to verify the connection. But you may have also browsed sites with Let's Encrypt certificates, on the same system even, so how come Java doesn't know about them.

At it's simplest, a server may present a certificate saying "I am Bob, Alice can confirm that". That confirmation is performed through public key encryption which means that Alice will have signed the certificate digitally. We can be sure about that because we can use Alice's own public certificates to verify her signature. In our case, Bob is a Compose database host and Alice is Let's Encrypt.

Getting everyone to update their library of root certificates takes time, sometimes years. To get around that delay, Let's Encrypt got intermediate certificates that it signed cross-signed by IdenTrust, an existing certificate authority which was already trusted by major browsers. This meant that browsers worked pretty much immediately with Let's Encrypt certified sites.

This meant browsers would work for HTTPS connections, but the Let's Encrypt certificates weren't in all operating systems and language runtimes - that meant applications that relied on the operating system or runtime to handle SSL certificate checking still needed the Let's Encrypt certificates to be added to their list of trusted certificates. So over 2015 and 2016, those certificates were being steadily added to various operating systems. Java has its own operating system independent collection of trusted certificates and that meant it would have to be updated by Oracle; Let's Encrypt support in the operating system would make no difference to Java.

It took until August 2016 for Oracle to add Let's Encrypt's certificates to the Java distribution, and they did that addition as part of a scheduled Java update release. Practically, that means when an application on a Java earlier than Java 8U101, released in August 2016, tries to connect it can't completely check the certificate presented by the Let's Encrypt protected Compose deployments.

Upgrade your Java

The simplest solution, and it's recommended as it closes many other security vulnerabilities, is to upgrade to Java JDK/JRE 8U101 or later (and ideally later, as of writing, it's version 8U111). There is also Java 7 update, 7U111 which also has the certificates needed but that's only for Oracle clients on support contracts.

If your codebase is such that you can't upgrade or can't upgrade without significant downtime, we still suggest you at least start planning to upgrade anyway to ensure that you can keep being able to use the regular security updates to Java. In the interim, though, if you have a running application, you'll have some options:

or do nothing for now...

At Compose, we haven't turned off the support for the older self-signed certificates yet and haven't announced a retirement date for them yet, so you can continue to use the old connection strings and certificates you have already stored.

or manually add certificates

You can add the IdenTrust certificates to the version of Java you are using. Go to https://letsencrypt.org/certificates/ where you'll find "Various" der formatted certificates available there. You'll need to install them into the Java JRE.

There's plenty to look out for in this manual process. Make sure that your $JAVA_HOME environment variable is set correctly. The keystore store lives in $JAVA_HOME/jre/lib/security/cacerts and you'll want to make a copy of that before you begin with a command like:

sudo cp -a $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/cacerts.orig  

That will let you revert changes simply if anything goes amiss. The certificate we want will be 'Let’s Encrypt Authority X3 (IdenTrust cross-signed)", as of writing this article. That's the active cross-signed certificate. You can manually download it through the browser or wget it like so:

wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.der  

Installing this certificate involves using the Java keytool command.

sudo keytool -trustcacerts -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -noprompt -importcert -alias letsencryptauthorityx3 -file lets-encrypt-x3-cross-signed.der  

That should be enough to make SSL/TLS connections once you've restarted any Java using applications to reload the keystore. You will need to repeat this process on any system where you are using an older Java version to connect. Or, as we said, you can upgrade your Java installation and get the certificates included by default.