Tag: development

In a previous post I said that one of my reasons to develop a RELOAD implementation was to implement VIPR. Another reason is to be able to develop software that is resistant to developer bugs.

The Internet is global, which means that a service should be available 24/7, and having a web page saying that a service is down for maintenance is just plain incompetence, and companies doing this should be publicly shamed for this. There is multiple ways to build systems that can still work during software update, operating system bugs and other disasters, and I think that Google did a lot of good in this domain by showing that it could be done in a way that does not require to spend millions of dollars in hardware – I advocated a Google-like architecture a long time before Google even existed and it was a relief when Google started to publicly disclose some of the things they were doing because then I could end my explanations by a “and BTW, this is exactly what Google is doing!”. Herd mentality always works.

Now imagine a service that is already doing all the right things – Amdahl’s law is the Law of the Land, sysadmin motto’s (“if it ain’t broke, don’t fix it”) is cause for immediate dismissal, programmers cannot commit without unit tests and peer reviews, and so on. The problem is that even when using the best practices (and by best I certainly do not mean the most expensive) a whole system can still go down because of a single programmer’s bug – and we certainly had some very public examples of that recently.

Bugs are not something programmers should be ashamed of – instead they should be thought as a probability, and this probability integrated into the design of a service. One of the best proof that bugs are not preventable mistakes but unavoidable limitations of software is the so-call “software bit rot”, i.e. the spontaneous degradation of software over time. Software does not really degrade over time; it’s just that the environment under which they are running is changing over time – faster processor, larger memory and so on. Basically this means that event if we can prove that a software is perfect now, unless it is possible to predict the future it is not possible to guarantee that a program will be without bugs in the future.

So, as always, if there is no solution to a problem then just change the problem. Bugs have a more or less constant probability to happen, i.e. X bugs for Y lines of code. I am not aware of any research in this domain (pointers welcome!) but intuitively I would say that two independent developers starting from the same set of specification will come with two different software and, more important to my point, with two different sets of bugs. If the probability of a major bug in any implementation is X, the probability that the same environment triggers the same bug in two completely different implementations should be X^2 (assuming that developers are not biased towards certain classes of bugs). In other words if we can run in parallel two or more implementations of the same specifications and be able to choose the one that is working correctly at a specific time, then we increase by multiple orders of magnitude the reliability of our service (one of my old posts used the same reasoning applied to ISP. Just replace “service down” by “someone will die because I cannot reach 911”).

Obviously running multiple implementations of the same specifications at the same time is really expensive and, as far as I know, is used only when a bug can threaten life, like in a nuclear electric plant. If we can solve the economics of this problem, then we should be able to offer far better services to end-users.

Now back to RELOAD. RELOAD is a standard API for peer-to-peer services. There is nothing really new in RELOAD – it uses technologies, like Chord, that are well-known and, for some people, that are probably close to obsolescence. The most important, at least in my opinion, is that for the first time we have a standard API to use these services on multiple implementations that can inter-operate with each other. Having a distributed database is great for reliability purpose, but having a distributed database that is running on different implementations, like RELOAD permits, is far better.

But RELOAD is not limited to a distributed database. With the help of ReDIR, we can have multiple implementations registering a common service, and so if one implementation fails, we still can count on the other implementations of the same service.

As for the economic side of having multiple implementations, instead of having only mine developed in-house I can swap it with similar code developed by another implementers. At the end if I am able to do this with 3 or 4 other implementers, all of us will have an incredibility more resilient service without spending a lot more money (this is the reason why I am so against having THE ONE open or free software implementation of a standard. The more implementations are developed, the better the users will be served. Implementation anarchy is a good thing, because it fosters evolution).

OK, I admit that was a long introduction for version 0.6.0 of the libreload-java package and the associated new Internet draft about access control policy scripts, especially because access control policy scripts paradoxically do not improve the reliability of a service based on RELOAD, but reduce it. The reason for this is that access control policy are ordinarily natively implemented so a bug is local to an implementation. This draft permits to distribute a new access control policy but as the same code will be shared between all the different implementations this creates a single point of failure. This is why access control policy scripts should be considered only as a temporary solution, until the policy can be implemented natively. The script copyright should prevent developers to copy it but we all know how that works, so to help developers do the right thing and not be tempted to look at the scripts when implementing the native version in their code, the source code in the XML configuration file is now obfuscated by compressing it and converting it to a base64 string.

The draft also introduce an idea that, in my opinion, could be worth developing, which is that the signer of a configuration file should play an active role in verifying that all implementations in the overlay follow the rules set up by the configuration file. For example the signer can send incorrectly signed configuration files to random peers, and add them to the blacklist in a future configuration file if they do not reject it.

One of my reasons for developing an implementation of RELOAD is to be able to develop an implementation of VIPR, a mechanism currently developed by the IETF to automatically and securely share SIP routes between VoIP providers. VIPR is using RELOAD as a giant distributed database where VoIP providers stores the phone numbers of their customers.

RELOAD in its last incarnation has all the features needed to implement VIPR, with two exceptions, the access control policy and the quota mechanism.

The access control policy in VIPR is similar to the standard USER-NODE-MATCH policy, but there is enough differences to mandate the implementation of a new policy. The preferred solution is to implement natively this policy, but a temporary solution could be to use the extension I designed to add new policies in the RELOAD configuration file. A future version of my implementation will implementation this policy natively, but meanwhile the following script can be used:

var equals = function(a, b) {
  if (a.length !== b.length) return false;
  for (var i = 0; i < a.length; i++) {
    if (a[i] !== b[i]) return false;
  }
  return true;
};
var length = configuration.node_id_length;
return equals(entry.key.slice(0, length),
  entry.value.slice(4, length + 4))
    && equals(entry.key.slice(0, length), signature.node_id);

The quota mechanism in VIPR is interesting. Basically it says that a VoIP provider must contribute a number of RELOAD servers for the distributed database that is proportional to the number of phone numbers it plans to register. Because this quota mechanism is useful for other usages than VIPR it now has its own Internet-Draft, separate from the VIPR drafts, with the goal of publishing it as an IETF standard. New quota mechanisms are not very frequently needed (AFAIK, this is the first quota mechanism created outside the RELOAD document) so it does not make sense to develop another API to write quota scripts. This means that this quota mechanism will have to be coded by RELOAD implementers (the VIPR configuration document should contain a <mandatory-extension> element to be sure that only servers implementing this extension will join the overlay).

Version 0.5.0 of the libreload-java package, that was released few minutes ago, not only permits to use the script listed above but also implement the new quota mechanism, making it suitable for implementing a VIPR server.

Update 07/05/2011: The VIPR access control policy is natively implemented in lib-reload-java 0.6.0.

IETF people do not live on the same planet than us – and I mean that literally. My theory is that they leave on a planet with a very strange orbit that put them in communication distance of Earth only 3 times per year, roughly around March, July and November, which could explain why it is mostly impossible to talk to them outside of these 3 months. During the rest of the time, speed of light limits our ability to communicate with them and it takes between one week and one month to receive a response to an email because of the huge distance (that, or their planet is so dense that time is running considerably slowly than ours).

Anyway, that is how it looks like to me and that’s very frustrating, especially as I got burned few years ago for the exact same reasons. Forking a specification is mostly prevented by the IETF license, so that was not an option, so I am trying something else: I published a document containing all the modifications I would have liked to see in the RELOAD specification. Here is the list of these modifications, with some explanations on how to try these modifications either with the source code or on the test server:

1. Client connected to bootstrap node

I am working on a major release of the libreload-java package that will contain the code for a RELOAD client, so I think that is it simpler to wait this release for the explanations on this extension.

2. Service Name and Path

The RELOAD document uses the “p2psip” name both in the SRV RRs and in the URL used to retrieve the configuration file. The problem is that RELOAD is no longer specific to P2PSIP, so this name should be “p2p” instead.

The DNS RR for the RELOAD test server can know be retrieved with the following command:

$ host -t SRV _p2p-enroll._tcp.implementers.org
_p2p-enroll._tcp.implementers.org has SRV record 40 0 443 implementers.org

Same thing, the configuration file can be retrieved with the following command:

$ curl --resolve 
implementers.org:443:2604:3400:dc1:41:216:3eff:fe5b:8240 
https://implementers.org/.well-known/p2p-enroll

Note that the standard names are still working.

3. Multiple Node-IDs for self-signed certificates

CA signed certificates can contain multiple Node-IDs, but self-signed certificates cannot. This extension fixes this problem.

Version 0.4.0 of the libreload-java package, that was released few minutes ago, now contains a static method in the SignerIdentity class to generate a self-signed certificate with one Node-IDs and a second static method to generate a self-signed certificate with multiple Node-IDs.

4. Re-sign certificates

With the current specification, it is not possible to extend the validity of a certificate because the Node-ID must be generated by the enrollment server. That mean basically that each time the certificate expires, the peer has to get a new Node-ID, which means leaving the overlay, joining with the new Node-ID and re-storing all the data that were stored with the previous Node-ID.

The extension permits to renew a certificate by sending it to the enrollment server instead of the certificate signing request, with something like this:

$ wget "https://implementers.org/enrollment?username=test&password=test" 
--post-file=cert.der --header "Content-Type: application/pkix-cert" 
--header "Accept: application/pkix-cert" -O cert2.der

You can even increase or decrease the number of Node-IDs in the resulting certificate, which can be useful.

Note that this works only if you keep the same key pair. If you want to change them then you need to send a certificate signing request, which will result in a certificate containing new Node-IDs. This is consistent with the way the self-certificates works.