Author: Marc Petit-Huguenin

Bad examples

Implementing Internet protocols is hard. It is hard for a variety of reasons – scalability, reliability, security are really important when anybody can write another implementation of the same protocol and use it to communicate on the Internet with your own implementation – but it is also hard because the specifications used to describe this protocols are not easy to read. It takes time to really understand a specification, which is why they generally contain one or more examples. Unfortunately some developers, instead of trying to understand how a protocol really works, try to implement them by imitating the examples they found. This creates two major problems. Firstly an example – by definition – can only show a limited subset of all the possibilities of a protocol and some subtle points will be missed if the normative text is not fully understood. The second problem is that it is not uncommon that examples contain mistakes, mistakes that will then been seen in careless implementations. I did a search in the RFC errata database, and found more than 70 errata on examples, so the problem is real.

In a perfect world, developers would understand that examples were not meant for them, or at least that they are not normative, but there is not much that can be done to improve this. Knowing that developers are lazy and will always take a shortcut if they can find one, the other solution is to improve the quality of examples in the specifications.

This is what I started to implement in version 0.1.5 of the rfc2629-tools. Normally an example is just an unstructured block of text in an artwork element, something like this (example copied from RFC 5245):

<figure>
  <artwork><![CDATA[
   v=0
   o=jdoe 2890844526 2890842807 IN IP4 10.0.1.1
   s=
   c=IN IP4 192.0.2.3
   t=0 0
   a=ice-pwd:asd88fgpdd777uzjYhagZg
   a=ice-ufrag:8hhY
   m=audio 45664 RTP/AVP 0
   b=RR:0
   b=RS:0
   a=rtpmap:0 PCMU/8000
   a=candidate:1 1 UDP 2130706431 10.0.1.1 8998 typ host
   a=candidate:2 1 UDP 1694498815 192.0.2.3 45664 typ srflx raddr
     10.0.1.1 rport 8998
]]></artwork>
</figure>

The problem with this example is that we do not know if it is correct. For example the specification can had changed between the time the example was inserted in the Internet-Draft and the time it was published as an RFC, and nobody checked that the example was still correct.

The solution I used for my own RFC was to generate the example from the reference code I wrote to check that my specification was implementable. I did this in three steps. First I wrote a reference implementation of the protocol. Then I wrote a set of unit tests to be sure that my implementation was correct. Then I added some code to automatically generate the examples. In RFC 5928, the figures 1, 2 and 3, as table 2 are all generated by a program, and so I am reasonably sure that they are correct, but also that they stayed correct during the long process of modifying the Internet-Draft.

The code I wrote for RFC 5928 was an ad-hoc program, but I wanted to integrate the possibility of easily generate examples in my rfc2629-tools, and so version 0.1.5 contains an example generator for SDP (RFC 4566), as SDP are often used with VoIP protocols. To generate the SDP example shown above, the following XML fragment can be inserted in the XML document:

<figure xmlns:ex="http://implementers.org/2010/examples">
  <artwork>
    <ex:sdp username="jdoe" session-id="2890844526" session-version="2890842807" session-address="10.0.1.1" address="192.0.2.3">
      <ex:attribute name="ice-pwd">asd88fgpdd777uzjYhagZg</ex:attribute>
      <ex:attribute name="ice-ufrag">8hhY</ex:attribute>
      <ex:media type="audio" port="45664" protocol="RTP/AVP" format="0">
        <ex:bandwidth type="RS">0</ex:bandwidth>
        <ex:bandwidth type="RR">0</ex:bandwidth>
        <ex:attribute name="rtpmap">0 PCMU/8000</ex:attribute>
        <ex:attribute name="candidate">1 1 UDP 2130706431 10.0.1.1 8998 typ host</ex:attribute>
        <ex:attribute name="candidate">2 1 UDP 1694498815 192.0.2.3 45664 typ srflx raddr 10.0.1.1 rport 8998</ex:attribute>
      </ex:media>
    </ex:sdp>
  </artwork>
</figure>

Note that the xmlns declaration should be in the rfc element, so it can be removed automatically after the generation (xml2rfc does not like namespaces, so there is a special process to remove them).

Currently only SDP examples can be generated and even for SDP there is still more work to do to guarantee that no invalid SDP can be generated (for example, the attributes are currently simple character strings. I will add the parsing of know attributes like candidates in a future version).

rfc2629-tools 0.1.4

I just released a new version of my RFC 2629 processing tools. The txt2kindle and xml2mobi commands are unchanged, but the rfc2629 command was rewritten from scratch. I needed to add some extensions in the XSLT processing that was not supported by xsltproc, so I rewrote it in Java. Because of this I was able to add a nice feature for including the references in a RFC 2629 document.

In the previous version of the tool, a reference can be added by using something like this:

<references xmsns:xi="http://www.w3.org/2001/XInclude" title="Normative References">
  <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.5054.xml" />
</references>

This is still supported, but now an IETF URN (RFC 2648) can be used instead:

<references xmsns:xi="http://www.w3.org/2001/XInclude" title="Normative References">
  <xi:include href="urn:ietf:rfc:2543" />
</references>

In addition to be less verbose, this notation has the advantage that I-D references can refer to a current version of an I-D instead of to the version that is publicly available. That was already in the previous version, but was cumbersome to use. With the current version, if you pass more than one RFC 2629 file to the rfc2629 command, it will use the version and data of the generated file in the cross references, instead of the public one.

In the future I will extend this feature to add a local cache. One of the most annoying problem with references (either when using Xinclude or <?rfc include ?>) is that you need to be connected to the Internet. It will be possible to store the references on the local disk and use them if there is no Internet connection available (or always use the cache for RFC references as they never change).

Note that only the rfc: and id: NSS are currently implemented.

Client certificates for a distributed development team

It’s that time of the decade again: I am building an environment for the development of a new Internet service. The previous time was at 8×8 and the environment was designed by trials and errors over a period of 6 years, between 2001 and 2006. Because I am now rebuilding an environment from scratch, I can spend a little bit of time fixing some of the issues that were not easy to fix in an existing environment.

One of the basic principle of the environment at 8×8 and of the new one is that they are designed to permit people to work remotely. My very first team at 8×8 had some developers in France, some in Canada and the remaining in the West coast of the USA so there was no other choice than build a distributed environment. I know that most startups try to keep developers as close as possible but having worked in both type of environment, I am now convinced that this setting gives a false impression of been efficient. For one this kind of setting increases the risk of micro-management, which I irrevocably dislike (that’s a story for another time, but escaping micro-management was one of the reasons I moved to the USA in the first place). But a distributed environment has one nice side effect which at the end makes all the difference between a good service and the usual crap that most people mistake for good engineering: documentation. Shouting explanations over the wall of a cubicle is probably appealing for lazy people, but it does not make a product or service better. Been forced to write down stuff (in an email, in a wiki, in an IRC channel…) is probably a little more work but it represents an immediate gain for the product. A recent Slashdot question was seeking an advice about seating arrangement for a team of developers. My advice would be simple: Be sure that there is at least a distance of 5 miles between each desk.

Working in a distributed environment probably means a VPN for most people. It happens that I do not like VPN very much, mostly for two reasons. Firstly VPN are generally proprietary products that work only on Windows and although my developers are free to use whatever OS and tools they prefer, Windows and MacOS are banned from my computers (or only running in a virtual machine, for testing purpose. VMWare is the condom of the Internet). The second reason is that having the developers working inside a secure environment does not make them good at handling security issues. If a developer is not capable of making her computers secure, how can she develop secure software? So the solution we used at 8×8 was SSH tunnels. That was working fine, but it is still a kind of VPN and it is not easy to deploy under Windows. The solution for this new environment is to use client certificates.

Client certificates have the advantage of working everywhere SSL/TLS is working and it is a great improvement from using passwords. Passwords are generally too easy to guess and when they are good, people reuse them on multiple websites, websites that can leak the good passwords. Client certificates does not have this issue as the key is not stored in the website and they are not guessable. The certificate itself must be protected by a password but as this password does not have to leave the computer it can be a good, unique password that never change so people do not have to write it down.

The first step to install a client certificate infrastructure is to install a server certificate. The easiest way is to buy one from the registrar of the domain (I use a domain name for development that is different from the corporate domain name. Code name does not have to change but product names and even company names can and will). The configuration in Apache is as follow:

SSLEngine on
SSLProtocol all -SSLv2
SSLCertificateFile /etc/ssl/private/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key

Using a real (i.e. not self-signed) server certificate is easier because we do not have to distribute the files for the CA for each developer.

The next step is to create a CA for the client certificates:

$ openssl genrsa -out ca.key 1024
$ openssl req -new -key ca.key -out ca.csr
$ openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt

and install it in Apache:

SSLCACertificateFile /etc/ssl/private/ca.crt
SSLVerifyClient require

After this, each developer can create a Certificate Signing Request and send it to the administrator of the server:

$ openssl genrsa -out client.key 1024
$ openssl req -new -key client.key -out client.csr

The administrator checks the request and then creates the client certificate and send it back to the developer:

$ openssl req -noout -text -in client.csr
$ openssl x509 -req -days 365 -CA /etc/ssl/private/ca.crt -CAkey /etc/ssl/private/ca.key -CAcreateserial -in client.csr -out client.crt

The developer can then use the certificate directly (e.g. with wget or Debian repositories) or can convert it to a PKCS#12 file that can be imported in a web browser:

$ openssl pkcs12 -export -clcerts -in client.crt -inkey client.key -out client.p12

JTAG and Serial Ports for WRT54GL

I have an insane number of bricked WRT54GL boxes lying around, and I needed to fix them for a project. The first step was to find a way to unbrick them, which means to use a JTAG cable. Nico (whom I probably do not credit enough for his help on all things geeky in this blog entries) pointed me to the Universal JTAG.

It was not easy to find the correct pins to use (the best page I found is in French, but the pictures are good enough to be used by everybody) but after soldering a connector and trying various combinations (only the buffered mode worked for me and, yes, the n_SRTST cable must be connected), I finally managed to unbrick the boxes.


As the boxes were already open, I also wanted to add a serial connection for the console, as my future project will require messing a lot with the network configuration of this boxes. There is various ways to add a serial port to this boxes, but all require to make some kind of circuit board to convert the TTL 3.3v to the RS-232 levels. I found an easiest way by using a USB RS-232 to TTL converter that I already used in a previous project. The perfect cable would have been the TTL-232R-3v3-AJ but unfortunately it was not available at the time, so I bought a TTL-232R-3v3-WE and soldered a 3.5mm male jack. The next step was to solder a female jack to the WRT54GL, as shown on the following picture.


Now I just have to connect the USB cable to the broken box to fix the network parameters.


Total cost for adding the serial port: $2.50 per box for the female jack, $20 for the USB converter.

Sirius Streaming Server

I am a Sirius subscriber since 2004. I listen to music mostly when working so I had an external satellite antenna hooked to a Kenwood DT-7000S receiver but last year I moved to a new house where this hook-up is not as easy because of the distance between the antenna and the audio system. Until I could find a solution I used the Sirius Internet Radio but this is not a really satisfactory system for various reasons – not been able to see the name of the song/artist; having to click on the renew button each hour; and, as always, mostly incompetent ISP.

I wanted to move back to a pure satellite solution, especially because there is already a DirectTV antenna which is oriented in the exact correct direction. It was easy to install the Sirius antenna in the same mast, without annoying the neighborhood association.

The next step was to reuse the cabling for the satellite antenna (I do not use DirectTV). and for this I just needed one adaptor to convert to RJ-6 cable and another to convert back to the satellite cable.

The next step was to find a way to send the audio over my home network – I have Netgear powerline adapters in most rooms of the house. The plan was to build an RTSP server with VLC, so I could listen to the radio on-demand, without having to continuously broadcast on the network. Multicast would have been a nice solution, but without managed switches that can direct the audio stream on the right Ethernet ports, the network would have been overwhelmed.

The satellite receiver has an optical output, so it does not make sense to convert the satellite signal to analog, to sample it back to digital in the streaming server. So I found at Fry’s this cheap USB sound device that has an optical S/PDIF input.

My first idea was to reuse a Mac mini which was already in the same room than the satellite receiver, but I then discovered that VLC on Mac OS cannot use the sound input of the computer. So plan B was to use a SheevaPlug. The Linux Unbuntu distribution used by the SheevaPlug was not working for some reason, so I installed Debian on an SD card. Alas, now the USB sound device was not working, probably because of a bug in the USB implementation. Inserting an USB hub between the device and the SheevaPlug solved the problem. Finding the correct parameters for VLC was not easy – for example I had to look at the source code to find that the URI to use the input port is “alsa://”. I also had to try multiple codecs (OGG was not supported by the RTSP implementation, MP3 was using more CPU than available) until finding that the A/52 (aka Dolby AC3) codec was perfect for this as VLC is using less than 8% of the CPU to stream a stereo channel at 256 Kb/s with a sampling rate of 48 Khz. Here’s the command line to start VLC:

cvlc --ttl 12 --rtsp-host 192.168.2.156:5554 --vlm-conf vlm.conf

and here’s the content of vlm.conf:

new sirius vod enabled
setup sirius output #transcode{acodec=a52,ab=256,channels=2,samplerate=48000}
setup sirius input alsa://

I would have preferred streaming over IPv6, but VLC does not seems to support this yet. I also use VLC on my desktop to control the audio stream, coupled with Jack to mix the audio from other audio sources. The URI I use is “rtsp://192.168.2.156:5554/sirius”, and I can pause and stop the stream.

So I now have an uninterrupted audio stream from my favorite satellite radio channels directly to any place in the home. One thing I lost is the possibility to change the channel remotely. The plan was to add a USB to serial port adaptor between the SheevaPlug and the satellite receiver (so the intermediate USB hub was necessary even if the USB sound device was not buggy) to drive the receiver, but unfortunately I was not able to find the RS-232 protocol for this. The only reference I found is in a Crestron module but $256 is expensive for a piece of software that I am not even sure to be able to reverse-engineer. Having the protocol would also have been nice to send the current song information in the audio stream. I would be interested by any information on this protocol as long as I do not have to sign a NDA. Perhaps there is someone in the Valley with a working RS-232 connection like this and who will be willing to let me record the signals exchanges. As always I have a plan B but that’s a painful one, to say the least.

RFID reader for IETF meeting

I attend the IETF meetings whenever I can and I follow the proceedings remotely when I am not able to travel. But following an IETF session remotely is not simple: You need one application to listen to the audio stream (I use XMMS), another application to connect to the Jabber room (Pidgin) and a PDF or Powerpoint viewer to look at the slides (Evince and OpenOffice.org). On top of this complex setup, there is a lot of annoying problems: The audio stream is not very good, the audio servers crash from time to time, and people forget to speak on the microphone; very often the slides are available few minutes before the beginning of a session – it even happened that the slides are not available at all. The Jabber room is in fact used to state the name of the person talking in the microphone, because people forget very often to do this. Another usage for the Jabber room is to state the current slide on the screen.

Obviously all of this does not make for a good experience. There was multiple attempts to improve this, and I did work at a time on one of this attempts. The project was canceled after I left my job at 8×8, but I still continued to work on it on my spare time. One of the first component I designed was to capture the name of an attendee speaking at a microphone. There was some similar experiences, one during IETF 74 by Columbia University, and another during IETF 76 by the host, but my goal was a little bit different: I wanted the system to protect privacy and to be as non-intrusive as possible, so people had just to stand in from of the microphone to have their RFID tag scanned. The following picture shows how the system works:


The RFID antenna needed to be big enough so the speaker just stand naturally in front of the microphone. The RW310 has an 11″ range but unfortunately only has a TTL interface which cannot be directly connected to most PC so I used a TTL to USB converter, with the whole circuit fitting inside the USB connector. The box itself was made from plexiglass – I used VariCAD to design the box and an Epilog laser cutter at Techshop to cut the plexiglass.

Finding a way to attach the RFID reader to the microphone stand was the most challenging part – I had to go through 3 prototypes until I found something that permit to change the distance between the reader and the stand and that does not unbalance the whole thing. I finally found a microphone holder that was perfect for this, with a microphone surface-mount attached on the box so the RFID reader can be easily removed at the end of the day.

Privacy is very important for the IETF meeting attendees, so there is some concerns with the introduction of RFID tags – and in my opinion, rightly so. Having an RFID tag without a way to shielding it is unwise, so I also designed a badge holder that would permit to shield the RFID tag also in a non-intrusive way, as shown in the following picture:


The idea is simple: Either you both show your name and give access to your RFID tag or you hide both. I think that this idea is key here – if you accept that people can read your name on your badge then there is no reason why you would not let an RFID reader do the same thing on your tag. The top half of the badge holder contains a metallic shield, but the tag itself is in the bottom half. When the holder is open your name is readable and your tag is not shielded; when closed both your name and your tag are invisible to the full electromagnetic spectrum.

Having the shield only on one side works because this RFID tag works at 13.56MHz (with 125 Khz tags, two layers of aluminum need to enclose the tag and this on both sides). Another reason for the 13.56Mhz tag is that it permit to store the user information directly in the tag, instead of simply retrieving a serial number and match it with a central database which, in my book, is a no-no as I want my personal data to be destroyed at the same time I destroy by badge.

Unfortunately the software is not ready at this time, but I’ll try to find some time to finish it if there is enough interest in this project.

Jarc: Junit integration

Unit testing is, in my opinion, one of the most important tool that a developer wants to use to write software. I am not fond of all the techniques proposed by the eXtreme Programming (to say the least) but having popularized unit testing is one of the things that the Agile methodologies should be credited for. Instead of having various tests scattered on multiple systems and prone to bit rotting, Junit permits to package all this tests together and to run them each time a product is built. I generally integrate the unit tests in the build, so a final package cannot be built without passing all the tests. The proprietary version of jarc had a switch to build and run the unit tests in one command line.

The version 0.2.5 of jarc continues in this spirit, but instead of having to build first a jar file and then compile and run the unit tests, it automatically build and run unit tests if they are available. Here’s an example of manifest file with unit tests:

Manifest-Version: 1.0
Main-Class: org.implementers.apps.turnuri.Main
X-Jarc-Target: 1.6

Name: org/implementers/apps/turnuri/Main.class

Name: org/implementers/apps/turnuri/TurnUriTest.class
X-Jarc-Test: true

With this manifest file, jarc will build a jar file with Main as root of the dependencies, then build a second, internal, jar file with TurnUriTest as root of the dependencies and use it as Junit test. The X-Jarc-Test attribute can also take the value “false” to deactivate a test temporarily.

Note that the jar file is written on the disk even if the unit tests fail.

RFC2629 tools

Now that I have a java build that is redistributable, I can finally release the code source of the two tools I wrote to convert I-Ds and RFCs to a format that can be used on a Kindle. The Debian package containing this two tools is named “rfc2629-tools” and can be found at the usual place. In addition to this two tools, I also put in this package an additional tool that I use to process my RFC2629 XML sources.

Some quick explanation for the people new to the IETF process. All IETF documents, standards and others, are published as RFC (which is no longer an acronym) which are immutable documents. Before been published as RFC, documents are published as Internet-Drafts (I-D). The canonical form of RFCs and I-Ds is a line printer-ready format that is often called text format. There is other formats possible like PDF or HTML, but only the text format must be used as reference. Writing a well-formed text I-D or RFC is difficult, so authors generally write it in another format, then use a conversion tool to generate the text, PDF or HTML file. Some people use Winword, other use nroff but the most interesting format is an XML application defined in RFC 2629. There is available tools to convert an RFC 2629 source to text, PDF, nroff and HTML, but nothing for the Kindle platform.

One of the tools in the rfc2629-tools package takes as input a RFC 2629 source and generate an intermediary file that can then be processed to create a .mobi file that can be loaded in a Kindle. Here’s the command lines to do this:

$ xml2mobi draft-ietf-behave-turn-uri-09.xml
$ kindlegen draft-ietf-behave-turn-uri-09.htm

Note that the xml2mobi program is very much a work in progress, as it is incomplete and really need more work to be able to process with any RFC2629 source. I released the code source because I do not have much time currently for this, so anybody interested can work on a fork. For example, the file generated is named from the content of the docName attribute in the rfc element in the input file.
The kindlegen program can be found on Amazon Web site.

The IETF Secretariat archives the RFC 2629 source if the author provides it when uploading the text file but few people do this so another solution is needed to be able to read I-Ds and RFCs that are only available in text form. This is what the text2kindle program do:

$ txt2kindle draft-ietf-behave-turn-uri-09.txt

This command creates a file named draft-ietf-behave-turn-uri-09.zip that can be installed directly in a directory named /pictures on the Kindle 2. On the Kindle 1, the file must be unzipped in the /pictures directory.
Note that the document is displayed as a series of pictures so you cannot annotate them.

Even if the RFC2629 source of an I-D is available on the IETF web site, that does not mean that you will be able to generate a mobi file for your Kindle that is identical in content to the canonical version. The first reason is that the full date is generally not set, which means that running a conversion tool on this source will generate a document with the current date instead of the date that is used in the canonical document. The second reason is that people generally uses the <?rfc include> statement to insert references. The problem with this is that the text included will change when a new version of the I-D referenced will released, so again it is not possible to generate a content identical to the canonical document. The solution to this problem is to generate an RFC2629 document that always contains a complete date and where all references are already included. With this what the third tool in the package provides:

$ rfc2629 draft-ietf-behave-turn-uri.xml

This command will copy the content of the file passed as input with the following transformations:

  • The destination file is named from the content of the docuName attribute (this helps when the document is stored in a source control system).

  • The date in the destination file is filled with the current date.

  • All the xinclude statements are replaced by their content

  • All comments are removed.

The resulting XML file can be uploaded to the IETF site with a guarantee that other formats can be generated from this file as identical as the files generated by the author.

The same script also implements a feature that permit to rename a reference. A reference to an I-D generally looks like this is the result text:

The TURN specification [I-D.ietf-behave-turn] defines a process for a

The map element can be used to change the name displayed:

<map anchor="TURN">
  <xi:include xi:href="http://xml.resource.org/public/rfc/bibxml3/reference.I-D.ietf-behave-turn.xml" />
</map>

Which, after been processed by the script and xml2rfc will generate this instead:

The TURN specification [TURN] defines a process for a TURN client to

The last feature of this script (for now) was added when I tried to release two I-Ds at the same time, with each I-D containing a reference to the other. Because the text that will be included is generated after the upload to the IETF, there was no way to automatically use the correct reference. What the script do is to generate a reference file automatically in /tmp for all the files processed. It is then easy to include the correctly generated reference is the document. For example draft-ietf-behave-turn-uri.xml and draft-petithuguenin-behave-turn-uri-bis.xml have a reference on each other. So the first step is to change the reference inside draft-ietf-behave-turn-uri.xml like this:

<xi:include xi:href="/tmp/reference.I-D.petithuguenin-behave-turn-uri-bis.xml" />

and to change the reference inside draft-petithuguenin-behave-turn-uri-bis.xml like this:

<xi:include xi:href="/tmp/reference.I-D.ietf-behave-turn-uri.xml" />

Then running the script on the two XML files at the same time generates the correct XML file, ready to be uploaded to the IETF:

$ rfc2629 ../src/share/docs/draft-ietf-behave-turn-uri.xml ../src/share/docs/draft-petithuguenin-behave-turn-uri-bis.xml

Jarc: Cross-compilation

The new version of jarc now supports cross-compilation, i.e. the possibility to build jar files that works on a version of the JRE lower than the the version of the JRE used to run the jarc program. The jarc program itself will always run under the most recent JRE and by default will build a jar file targeted to the same version. But adding an X-Jarc-Target option will request jarc to build for a lower version. For example this manifest file means that the resulting jar file will work with a 1.5 JRE:

Manifest-Version: 1.0
X-Jarc-Target: 1.5

Without this line, the jar file would have been created for the 1.6 JRE so executing it on a 1.5 JRE would fail.

It is also possible to request a specific JRE vendor:

Manifest-Version: 1.0
X-Jarc-Target: 1.6_openjdk

Note that there is no need to install the JDK for all versions of the JRE. Only the JDK for the latest version (i.e. the one that is used to run jarc) is needed. But at least one installed JRE is needed for each potential X-Jarc-Target value, because the rt.jar file is needed for the cross-compilation.

It is even possible to configure jarc to use the JDK 1.7 by modifying the configuration in /etc/jarc.conf (see comments at the end of the file). The jarc package itself is built by jarc running on 1.7 but targeted to 1.6.

The current version of jarc (0.2.2) can build for the JDKs 1.6 and 1.5 and supports sun and openjdk JVMs. A future version will also support the JVMs generated by java-package and perhaps the gcj JVM.