RELOAD: Test server

It took two very frustrating weeks for this, but I finally managed to install a public RELOAD Configuration and Enrollment test server. The frustration part is a consequence of my self-imposed list of requirements: Full implementation of the RELOAD spec, IPv4 and IPv6 support, JMX management over TLS with client certificates and the private key for the RELOAD CA stored in a PKCS#11 token. The code is still not perfect, but it at least fulfills all the requirements.

The libreload-java library contains everything to process the data returned by the server, but it is possible to use command-line tools to have a look to the configuration file and to generate RELOAD certificates. First step, finding the IP address of the configuration server for the “” overlay using the DNS:

$ host -t SRV has SRV record 40 0 443
$ host has address has IPv6 address 2604:3400:dc1:41:216:3eff:fe5b:8240

Next step, retrieving the current version of the configuration file:

$ curl --resolve

The configuration file contains the current URL of the enrollment server,

The next step is to generate an RSA key pair:

$ openssl genrsa -out cert.rsa
$ openssl pkcs8 -in cert.rsa -out cert.key -topk8 -nocrypt

Then we can generate a certificate request in DER form:

$ openssl req -new -key cert.key -outform der -out cert.req

The certificate request can be sent to the enrollment server, which will use it to generate a certificate:

$ wget "" --post-file=cert.req --header "Content-Type: application/pkcs10" --header "Accept: application/pkix-cert" -O cert.der

Note that the password is mandatory but any password can be used at this time, as there is no user management in the server yet. The certificate returned will contain one user name and one Node-ID, but a certificate with multiple Node-IDs can be requested with the nodeids= parameter, as specified in version -15 of the RELOAD I-D. The content of the certificate can be displayed with something like this:

$ openssl x509 -noout -text -inform DER -in cert.der

I also released a new version of the libreload-java package that contains some bug fixes and improvements related to version -15 of the I-D. For example the library can now generate a signed configuration file, even if the test server is not using it yet.

Update 6/2/2011: Use curl instead of wget so we can force the IP address and port to the result of the SRV query.

Update 8/13/2012: This this post for the updated version of the servers.

Application developers and DNS TTL

During the technical plenary of the 73th IETF meeting in Minneapolis MN, Dave Thaler made the interesting point that most DNS resolver API do not return the TTL of the resource resolved, e.g. an IP address. At the time he was proposing a modification of the existing APIs, and that made me thinking since.

The problem is that programmers generally think that resolving a domain name and then storing the IP address(es) resulting of this resolution is a good idea, as this technique could yield a better responsiveness for an application. But doing that without having a mechanism to know that the result of the resolution is invalid creates an even bigger problem. During my time at 8×8, I worked on the problem of designing a scalable Internet service, and my design was focused on having the scalability driven by the client (in other words: the whole load balancers idea is evil). Such techniques are very cheap and effective, but only if the client strictly obeys the TTL value received in the DNS response (I still have two pending patent applications about this techniques). Another well known problem is documented in RFC 4472 section 8.2, as keeping an IP address for too long prevents renumbering an IPv6 network, but there is plenty of other cases.

So the idea of passing the TTL together with the result of the DNS query seems like a good one, until you realize that in fact what developer have now to do is to implement a DNS cache in their application, and every evidence shows that this is not a simple task. As can be seen by the number of security vulnerabilities found during the years, even people who do read the RFC seem to have an hard time doing it right. Internet could probably do without another wave of DNS cache implemented incorrectly.

So in my opinion, adding the TTL to the API is not the solution – it will just exchange one problem with another. The correct solution is to do the resolution each time the resource is needed and do not store the result at all. If performances are too much impacted (after scientifically measuring them, we are between professionals here) then using an external DNS cache will fix the problem. The DNS cache can be in your network (for example having two DNS caches per data center), can be on each server (dnscache from the djbdns suite is easy to install and configure and has a good security track), or even directly in your application (for example dnsjava contains such a cache).