Tag: RELOAD

As people quickly discover, implementing RELOAD is not an easy task – the specification is complex, covers multiple network layers, is extensible and flexible and the fact that security is mandatory creates even more challenges at debug time. This is why new implementations generally focus on having the minimum set of features working between nodes using the same software.

Making two different RELOAD implementations interoperate requires a lot more work, mostly because connecting to a RELOAD overlay is not as simple as providing an IP address and port to connect to. Because of the extensibility of RELOAD, all the nodes in an overlay must use the same set of parameters, parameters that are collected and distributed in an XML document that need to be cryptographically signed. In addition to this, all nodes must communicate over (D)TLS links, using both client and server certificates signed by a CA that is local to the overlay. Configuration file and certificates must be distributed to each node and when two or more implementations wants to participate in the same overlay, adhoc methods to provision these elements are no longer adequate. The standard way to do that is through a configuration and enrollment server but unfortunately that is probably the part of the RELOAD specification that most implementers would assign the lowest priority, thus creating an higher barrier to interoperability testing than one would expect.

This is why during the last RELOAD interoperability testing event in Paris, I volunteered to provide configuration and enrollment servers as a service to RELOAD implementers, so they do not have to worry about this part. I already had my own configuration and enrollment servers, but I had to rewrite them from scratch because of two additional requirements: They had to work with any set of parameters, even some that my own implementation of RELOAD do not support yet, and it must be possible to host servers for multiple overlays on the same physical server (virtual server). A first set of servers are now deployed and in use by the participants of the last RELOAD interoperability event, so it is now time to open it to a larger set of participants.

First what this service is not: It is not to host commercial services, and it is not meant to showcase implementations. The service is free for RELOAD implementers (up to 5 overlays per implementer) for the explicit purpose of letting other implementers connect to your RELOAD implementation, which means that you are supposed to provision a username/password for any other implementer on request, on a reciprocity basis. Contact me directly if you are interested in an usage that does not fit this description.

The enrollment for the service is simple: send me an email containing the X500 names that will be used to provision your servers. Here’s an example to provision a fictional overlay named “my-overlay-reload.implementers.org”:

C=US, ST=California, L=Saratoga, O=Impedance Mismatch, LLC, OU=R&D,
CN=my-overlay-reload.implementers.org

The C=, ST=, L=, O= and OU= components should describe your organization (not you). The CN= component contains the name requested for your overlay. Note that the “-reload.implementers.org” part is mandatory, but you can choose to use whatever name before this suffix, as long as it is not already taken, that it follows the DNS label rules and that it does not contain a dot (wildcard certificates do not support sub-subdomains).

With these information I will provision the following:

• The DNS RR as described in the RELOAD draft
• A configuration server.
• An enrollment server, with its CA certificate
• A secure Operation, Administration and Management (OAM) server.

The DNS server will permit to retrieve the IP addresses and ports that can be used to connect to the configuration server. If we reuse our example above, the following command will retrieve the DNS name and port:

$ host -t SRV _reload-config._tcp.my-overlay-reload.implementers.org
_reload-config._tcp.my-overlay-reload.implementers.org has SRV record 40 0 443
my-overlay-reload.implementers.org.

Note that the example uses the new service and well-known URL name that were agreed on in the Vancouver meeting, but the current name (p2psip-enroll) will be supported until the updated specification is published.

The DNS name can then be resolved (the IPv6 address is functional):

$ host my-overlay-reload.implementers.org
my-overlay-reload.implementers.org has address 173.246.102.69
my-overlay-reload.implementers.org has IPv6 address
2604:3400:dc1:41:216:3eff:fe5b:8240

Then the configuration file can be retrieved by following the rules listed in the specification:

$ curl --resolve my-overlay-reload.implementers.org:443:173.246.102.69
https://my-overlay-reload.implementers.org/.well-known/reload-config

The returned configuration file will contain a root-cert element containing the CA certificate that was created for this overlay, and will be signed with a configuration signer that will be maintained by the configuration server. Basically the configuration server will automatically renew the configuration signer and resign the configuration file every 30 days, or sooner if you upload a new configuration file (more on this later). Note that to ensure that there is no lapse in the rollover of signer certificates, the configuration file must be retrieved periodically (the expiration attribute contains the expiration date of the signer certificate, so retrieving the configuration document one or two days before this date will guarantee that any configuration file can be used to validate the next one in sequence). This feature frees the implementers from developing its own signing tools (a future version will permit the implementers to maintain their own signer and to upload a signed configuration file).

The configuration file also contain an enrollment-server element, pointing to the enrollment server itself, that can be used to create certificates as described in the specification. The enrollment server requires a valid username/password to create a certificate and anyway the default configuration document returned is filled with the minimum parameters required, so they are useless as it to run a real overlay. Modifying the configuration document and managing the users that can request a certificate (and so join the overlay) is the responsibility of the OAM server.

Because the OAM server uses a client certificate for authentication, it uses a different domain name than the configuration and enrollment server. The domain name will use the “-oam-reload-implementers.org” suffix, and will use a separate CA to create the client certificate, so a user of the overlay cannot use its certificate to change the configuration (That would be a good idea to define a new X.509 extended key usage purpose for RELOAD to test for this).

The OAM server uses a RESTful API to manage the configuration and enrollment servers (well, as RESTful as possible, because the API is in fact auto-generated from a JMX API, and I did not find another solution that to map a JMX operation to a POST. But more on this in a future blog entry). Here’s the commands to add a new user, change a user password, list the users and remove a user:

$ curl --cert client.crt --key client.key --data "name=myname&password=mypassword"
https://my-overlay-oam-reload.implementers.org/type=Enrollment/addUser
$ curl --cert client.crt --key client.key --data "name=myname&password=mypassword"
https://my-overlay-oam-reload.implementers.org/type=Enrollment/modifyUser
$ curl --cert client.crt --key client.key https://notomele-reload.implementers.org/type=Enrollment/Users
$ curl --cert client.crt --key client.key --data "name=myname"
https://my-overlay-oam-reload.implementers.org/type=Enrollment/removeUser

The password is stored as a bcrypt hash, so it is safe as long as you do not use weak passwords.

The last step is to modify the configuration, probably to add a bootstrap element element. Currently the OAM server manages what is called a naked configuration, which is a configuration document stripped of all signatures. The current naked configuration can be retrieved with the following command:

$ curl --cert client.crt --key client.key https://my-overlay-oam-reload.implementers.org/type=Configuration/NakedConfiguration > config.relo

The file can then be freely modified with the following constraints:

• The file must be valid XML and must conform to the schema in the specification (including the use of namespaces).
• The sequence attribute value must be increased by exactly one, modulo 65535.
• The instance-name attribute must not be modified.
• The expiration attribute must not be modified.
• A node-id-length element must not be added.
• The root-cert element must not be removed or modified, and a new one must not be added.
• The enrollment-server element must not be removed or modified, and a new one must not be added.
• The configuration-signer element must not be removed or modified, and a new one must not be added.
• A shared-secret element must not be added.
• A self-signed-permitted element with a value of “true” must not be added.
• A kind-signer element must not be added.
• A kind-signature or signature element must not be added
• Additional configuration elements must not be added.

Then the file can be uploaded into the configuration server:

$ curl --cert client.crt --key client.key -T config.relo https://my-overlay-oam-reload.implementers.org/type=Configuration/NakedConfiguration

If there is a problem in the configuration, an HTTP error 4xx should be returned, hopefully with a text explaining the problem (please send me an email if you think that the text is not helpful, or if a 5xx error is returned).

In a previous post I said that one of my reasons to develop a RELOAD implementation was to implement VIPR. Another reason is to be able to develop software that is resistant to developer bugs.

The Internet is global, which means that a service should be available 24/7, and having a web page saying that a service is down for maintenance is just plain incompetence, and companies doing this should be publicly shamed for this. There is multiple ways to build systems that can still work during software update, operating system bugs and other disasters, and I think that Google did a lot of good in this domain by showing that it could be done in a way that does not require to spend millions of dollars in hardware – I advocated a Google-like architecture a long time before Google even existed and it was a relief when Google started to publicly disclose some of the things they were doing because then I could end my explanations by a “and BTW, this is exactly what Google is doing!”. Herd mentality always works.

Now imagine a service that is already doing all the right things – Amdahl’s law is the Law of the Land, sysadmin motto’s (“if it ain’t broke, don’t fix it”) is cause for immediate dismissal, programmers cannot commit without unit tests and peer reviews, and so on. The problem is that even when using the best practices (and by best I certainly do not mean the most expensive) a whole system can still go down because of a single programmer’s bug – and we certainly had some very public examples of that recently.

Bugs are not something programmers should be ashamed of – instead they should be thought as a probability, and this probability integrated into the design of a service. One of the best proof that bugs are not preventable mistakes but unavoidable limitations of software is the so-call “software bit rot”, i.e. the spontaneous degradation of software over time. Software does not really degrade over time; it’s just that the environment under which they are running is changing over time – faster processor, larger memory and so on. Basically this means that event if we can prove that a software is perfect now, unless it is possible to predict the future it is not possible to guarantee that a program will be without bugs in the future.

So, as always, if there is no solution to a problem then just change the problem. Bugs have a more or less constant probability to happen, i.e. X bugs for Y lines of code. I am not aware of any research in this domain (pointers welcome!) but intuitively I would say that two independent developers starting from the same set of specification will come with two different software and, more important to my point, with two different sets of bugs. If the probability of a major bug in any implementation is X, the probability that the same environment triggers the same bug in two completely different implementations should be X^2 (assuming that developers are not biased towards certain classes of bugs). In other words if we can run in parallel two or more implementations of the same specifications and be able to choose the one that is working correctly at a specific time, then we increase by multiple orders of magnitude the reliability of our service (one of my old posts used the same reasoning applied to ISP. Just replace “service down” by “someone will die because I cannot reach 911”).

Obviously running multiple implementations of the same specifications at the same time is really expensive and, as far as I know, is used only when a bug can threaten life, like in a nuclear electric plant. If we can solve the economics of this problem, then we should be able to offer far better services to end-users.

Now back to RELOAD. RELOAD is a standard API for peer-to-peer services. There is nothing really new in RELOAD – it uses technologies, like Chord, that are well-known and, for some people, that are probably close to obsolescence. The most important, at least in my opinion, is that for the first time we have a standard API to use these services on multiple implementations that can inter-operate with each other. Having a distributed database is great for reliability purpose, but having a distributed database that is running on different implementations, like RELOAD permits, is far better.

But RELOAD is not limited to a distributed database. With the help of ReDIR, we can have multiple implementations registering a common service, and so if one implementation fails, we still can count on the other implementations of the same service.

As for the economic side of having multiple implementations, instead of having only mine developed in-house I can swap it with similar code developed by another implementers. At the end if I am able to do this with 3 or 4 other implementers, all of us will have an incredibility more resilient service without spending a lot more money (this is the reason why I am so against having THE ONE open or free software implementation of a standard. The more implementations are developed, the better the users will be served. Implementation anarchy is a good thing, because it fosters evolution).

OK, I admit that was a long introduction for version 0.6.0 of the libreload-java package and the associated new Internet draft about access control policy scripts, especially because access control policy scripts paradoxically do not improve the reliability of a service based on RELOAD, but reduce it. The reason for this is that access control policy are ordinarily natively implemented so a bug is local to an implementation. This draft permits to distribute a new access control policy but as the same code will be shared between all the different implementations this creates a single point of failure. This is why access control policy scripts should be considered only as a temporary solution, until the policy can be implemented natively. The script copyright should prevent developers to copy it but we all know how that works, so to help developers do the right thing and not be tempted to look at the scripts when implementing the native version in their code, the source code in the XML configuration file is now obfuscated by compressing it and converting it to a base64 string.

The draft also introduce an idea that, in my opinion, could be worth developing, which is that the signer of a configuration file should play an active role in verifying that all implementations in the overlay follow the rules set up by the configuration file. For example the signer can send incorrectly signed configuration files to random peers, and add them to the blacklist in a future configuration file if they do not reject it.

One of my reasons for developing an implementation of RELOAD is to be able to develop an implementation of VIPR, a mechanism currently developed by the IETF to automatically and securely share SIP routes between VoIP providers. VIPR is using RELOAD as a giant distributed database where VoIP providers stores the phone numbers of their customers.

RELOAD in its last incarnation has all the features needed to implement VIPR, with two exceptions, the access control policy and the quota mechanism.

The access control policy in VIPR is similar to the standard USER-NODE-MATCH policy, but there is enough differences to mandate the implementation of a new policy. The preferred solution is to implement natively this policy, but a temporary solution could be to use the extension I designed to add new policies in the RELOAD configuration file. A future version of my implementation will implementation this policy natively, but meanwhile the following script can be used:

var equals = function(a, b) {
  if (a.length !== b.length) return false;
  for (var i = 0; i < a.length; i++) {
    if (a[i] !== b[i]) return false;
  }
  return true;
};
var length = configuration.node_id_length;
return equals(entry.key.slice(0, length),
  entry.value.slice(4, length + 4))
    && equals(entry.key.slice(0, length), signature.node_id);

The quota mechanism in VIPR is interesting. Basically it says that a VoIP provider must contribute a number of RELOAD servers for the distributed database that is proportional to the number of phone numbers it plans to register. Because this quota mechanism is useful for other usages than VIPR it now has its own Internet-Draft, separate from the VIPR drafts, with the goal of publishing it as an IETF standard. New quota mechanisms are not very frequently needed (AFAIK, this is the first quota mechanism created outside the RELOAD document) so it does not make sense to develop another API to write quota scripts. This means that this quota mechanism will have to be coded by RELOAD implementers (the VIPR configuration document should contain a <mandatory-extension> element to be sure that only servers implementing this extension will join the overlay).

Version 0.5.0 of the libreload-java package, that was released few minutes ago, not only permits to use the script listed above but also implement the new quota mechanism, making it suitable for implementing a VIPR server.

Update 07/05/2011: The VIPR access control policy is natively implemented in lib-reload-java 0.6.0.

IETF people do not live on the same planet than us – and I mean that literally. My theory is that they leave on a planet with a very strange orbit that put them in communication distance of Earth only 3 times per year, roughly around March, July and November, which could explain why it is mostly impossible to talk to them outside of these 3 months. During the rest of the time, speed of light limits our ability to communicate with them and it takes between one week and one month to receive a response to an email because of the huge distance (that, or their planet is so dense that time is running considerably slowly than ours).

Anyway, that is how it looks like to me and that’s very frustrating, especially as I got burned few years ago for the exact same reasons. Forking a specification is mostly prevented by the IETF license, so that was not an option, so I am trying something else: I published a document containing all the modifications I would have liked to see in the RELOAD specification. Here is the list of these modifications, with some explanations on how to try these modifications either with the source code or on the test server:

1. Client connected to bootstrap node

I am working on a major release of the libreload-java package that will contain the code for a RELOAD client, so I think that is it simpler to wait this release for the explanations on this extension.

2. Service Name and Path

The RELOAD document uses the “p2psip” name both in the SRV RRs and in the URL used to retrieve the configuration file. The problem is that RELOAD is no longer specific to P2PSIP, so this name should be “p2p” instead.

The DNS RR for the RELOAD test server can know be retrieved with the following command:

$ host -t SRV _p2p-enroll._tcp.implementers.org
_p2p-enroll._tcp.implementers.org has SRV record 40 0 443 implementers.org

Same thing, the configuration file can be retrieved with the following command:

$ curl --resolve 
implementers.org:443:2604:3400:dc1:41:216:3eff:fe5b:8240 
https://implementers.org/.well-known/p2p-enroll

Note that the standard names are still working.

3. Multiple Node-IDs for self-signed certificates

CA signed certificates can contain multiple Node-IDs, but self-signed certificates cannot. This extension fixes this problem.

Version 0.4.0 of the libreload-java package, that was released few minutes ago, now contains a static method in the SignerIdentity class to generate a self-signed certificate with one Node-IDs and a second static method to generate a self-signed certificate with multiple Node-IDs.

4. Re-sign certificates

With the current specification, it is not possible to extend the validity of a certificate because the Node-ID must be generated by the enrollment server. That mean basically that each time the certificate expires, the peer has to get a new Node-ID, which means leaving the overlay, joining with the new Node-ID and re-storing all the data that were stored with the previous Node-ID.

The extension permits to renew a certificate by sending it to the enrollment server instead of the certificate signing request, with something like this:

$ wget "https://implementers.org/enrollment?username=test&password=test" 
--post-file=cert.der --header "Content-Type: application/pkix-cert" 
--header "Accept: application/pkix-cert" -O cert2.der

You can even increase or decrease the number of Node-IDs in the resulting certificate, which can be useful.

Note that this works only if you keep the same key pair. If you want to change them then you need to send a certificate signing request, which will result in a certificate containing new Node-IDs. This is consistent with the way the self-certificates works.