Tag: RFC2629

Bad examples

Implementing Internet protocols is hard. It is hard for a variety of reasons – scalability, reliability, security are really important when anybody can write another implementation of the same protocol and use it to communicate on the Internet with your own implementation – but it is also hard because the specifications used to describe this protocols are not easy to read. It takes time to really understand a specification, which is why they generally contain one or more examples. Unfortunately some developers, instead of trying to understand how a protocol really works, try to implement them by imitating the examples they found. This creates two major problems. Firstly an example – by definition – can only show a limited subset of all the possibilities of a protocol and some subtle points will be missed if the normative text is not fully understood. The second problem is that it is not uncommon that examples contain mistakes, mistakes that will then been seen in careless implementations. I did a search in the RFC errata database, and found more than 70 errata on examples, so the problem is real.

In a perfect world, developers would understand that examples were not meant for them, or at least that they are not normative, but there is not much that can be done to improve this. Knowing that developers are lazy and will always take a shortcut if they can find one, the other solution is to improve the quality of examples in the specifications.

This is what I started to implement in version 0.1.5 of the rfc2629-tools. Normally an example is just an unstructured block of text in an artwork element, something like this (example copied from RFC 5245):

<figure>
  <artwork><![CDATA[
   v=0
   o=jdoe 2890844526 2890842807 IN IP4 10.0.1.1
   s=
   c=IN IP4 192.0.2.3
   t=0 0
   a=ice-pwd:asd88fgpdd777uzjYhagZg
   a=ice-ufrag:8hhY
   m=audio 45664 RTP/AVP 0
   b=RR:0
   b=RS:0
   a=rtpmap:0 PCMU/8000
   a=candidate:1 1 UDP 2130706431 10.0.1.1 8998 typ host
   a=candidate:2 1 UDP 1694498815 192.0.2.3 45664 typ srflx raddr
     10.0.1.1 rport 8998
]]></artwork>
</figure>

The problem with this example is that we do not know if it is correct. For example the specification can had changed between the time the example was inserted in the Internet-Draft and the time it was published as an RFC, and nobody checked that the example was still correct.

The solution I used for my own RFC was to generate the example from the reference code I wrote to check that my specification was implementable. I did this in three steps. First I wrote a reference implementation of the protocol. Then I wrote a set of unit tests to be sure that my implementation was correct. Then I added some code to automatically generate the examples. In RFC 5928, the figures 1, 2 and 3, as table 2 are all generated by a program, and so I am reasonably sure that they are correct, but also that they stayed correct during the long process of modifying the Internet-Draft.

The code I wrote for RFC 5928 was an ad-hoc program, but I wanted to integrate the possibility of easily generate examples in my rfc2629-tools, and so version 0.1.5 contains an example generator for SDP (RFC 4566), as SDP are often used with VoIP protocols. To generate the SDP example shown above, the following XML fragment can be inserted in the XML document:

<figure xmlns:ex="http://implementers.org/2010/examples">
  <artwork>
    <ex:sdp username="jdoe" session-id="2890844526" session-version="2890842807" session-address="10.0.1.1" address="192.0.2.3">
      <ex:attribute name="ice-pwd">asd88fgpdd777uzjYhagZg</ex:attribute>
      <ex:attribute name="ice-ufrag">8hhY</ex:attribute>
      <ex:media type="audio" port="45664" protocol="RTP/AVP" format="0">
        <ex:bandwidth type="RS">0</ex:bandwidth>
        <ex:bandwidth type="RR">0</ex:bandwidth>
        <ex:attribute name="rtpmap">0 PCMU/8000</ex:attribute>
        <ex:attribute name="candidate">1 1 UDP 2130706431 10.0.1.1 8998 typ host</ex:attribute>
        <ex:attribute name="candidate">2 1 UDP 1694498815 192.0.2.3 45664 typ srflx raddr 10.0.1.1 rport 8998</ex:attribute>
      </ex:media>
    </ex:sdp>
  </artwork>
</figure>

Note that the xmlns declaration should be in the rfc element, so it can be removed automatically after the generation (xml2rfc does not like namespaces, so there is a special process to remove them).

Currently only SDP examples can be generated and even for SDP there is still more work to do to guarantee that no invalid SDP can be generated (for example, the attributes are currently simple character strings. I will add the parsing of know attributes like candidates in a future version).

rfc2629-tools 0.1.4

I just released a new version of my RFC 2629 processing tools. The txt2kindle and xml2mobi commands are unchanged, but the rfc2629 command was rewritten from scratch. I needed to add some extensions in the XSLT processing that was not supported by xsltproc, so I rewrote it in Java. Because of this I was able to add a nice feature for including the references in a RFC 2629 document.

In the previous version of the tool, a reference can be added by using something like this:

<references xmsns:xi="http://www.w3.org/2001/XInclude" title="Normative References">
  <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.5054.xml" />
</references>

This is still supported, but now an IETF URN (RFC 2648) can be used instead:

<references xmsns:xi="http://www.w3.org/2001/XInclude" title="Normative References">
  <xi:include href="urn:ietf:rfc:2543" />
</references>

In addition to be less verbose, this notation has the advantage that I-D references can refer to a current version of an I-D instead of to the version that is publicly available. That was already in the previous version, but was cumbersome to use. With the current version, if you pass more than one RFC 2629 file to the rfc2629 command, it will use the version and data of the generated file in the cross references, instead of the public one.

In the future I will extend this feature to add a local cache. One of the most annoying problem with references (either when using Xinclude or <?rfc include ?>) is that you need to be connected to the Internet. It will be possible to store the references on the local disk and use them if there is no Internet connection available (or always use the cache for RFC references as they never change).

Note that only the rfc: and id: NSS are currently implemented.

RFC2629 tools

Now that I have a java build that is redistributable, I can finally release the code source of the two tools I wrote to convert I-Ds and RFCs to a format that can be used on a Kindle. The Debian package containing this two tools is named “rfc2629-tools” and can be found at the usual place. In addition to this two tools, I also put in this package an additional tool that I use to process my RFC2629 XML sources.

Some quick explanation for the people new to the IETF process. All IETF documents, standards and others, are published as RFC (which is no longer an acronym) which are immutable documents. Before been published as RFC, documents are published as Internet-Drafts (I-D). The canonical form of RFCs and I-Ds is a line printer-ready format that is often called text format. There is other formats possible like PDF or HTML, but only the text format must be used as reference. Writing a well-formed text I-D or RFC is difficult, so authors generally write it in another format, then use a conversion tool to generate the text, PDF or HTML file. Some people use Winword, other use nroff but the most interesting format is an XML application defined in RFC 2629. There is available tools to convert an RFC 2629 source to text, PDF, nroff and HTML, but nothing for the Kindle platform.

One of the tools in the rfc2629-tools package takes as input a RFC 2629 source and generate an intermediary file that can then be processed to create a .mobi file that can be loaded in a Kindle. Here’s the command lines to do this:

$ xml2mobi draft-ietf-behave-turn-uri-09.xml
$ kindlegen draft-ietf-behave-turn-uri-09.htm

Note that the xml2mobi program is very much a work in progress, as it is incomplete and really need more work to be able to process with any RFC2629 source. I released the code source because I do not have much time currently for this, so anybody interested can work on a fork. For example, the file generated is named from the content of the docName attribute in the rfc element in the input file.
The kindlegen program can be found on Amazon Web site.

The IETF Secretariat archives the RFC 2629 source if the author provides it when uploading the text file but few people do this so another solution is needed to be able to read I-Ds and RFCs that are only available in text form. This is what the text2kindle program do:

$ txt2kindle draft-ietf-behave-turn-uri-09.txt

This command creates a file named draft-ietf-behave-turn-uri-09.zip that can be installed directly in a directory named /pictures on the Kindle 2. On the Kindle 1, the file must be unzipped in the /pictures directory.
Note that the document is displayed as a series of pictures so you cannot annotate them.

Even if the RFC2629 source of an I-D is available on the IETF web site, that does not mean that you will be able to generate a mobi file for your Kindle that is identical in content to the canonical version. The first reason is that the full date is generally not set, which means that running a conversion tool on this source will generate a document with the current date instead of the date that is used in the canonical document. The second reason is that people generally uses the <?rfc include> statement to insert references. The problem with this is that the text included will change when a new version of the I-D referenced will released, so again it is not possible to generate a content identical to the canonical document. The solution to this problem is to generate an RFC2629 document that always contains a complete date and where all references are already included. With this what the third tool in the package provides:

$ rfc2629 draft-ietf-behave-turn-uri.xml

This command will copy the content of the file passed as input with the following transformations:

  • The destination file is named from the content of the docuName attribute (this helps when the document is stored in a source control system).

  • The date in the destination file is filled with the current date.

  • All the xinclude statements are replaced by their content

  • All comments are removed.

The resulting XML file can be uploaded to the IETF site with a guarantee that other formats can be generated from this file as identical as the files generated by the author.

The same script also implements a feature that permit to rename a reference. A reference to an I-D generally looks like this is the result text:

The TURN specification [I-D.ietf-behave-turn] defines a process for a

The map element can be used to change the name displayed:

<map anchor="TURN">
  <xi:include xi:href="http://xml.resource.org/public/rfc/bibxml3/reference.I-D.ietf-behave-turn.xml" />
</map>

Which, after been processed by the script and xml2rfc will generate this instead:

The TURN specification [TURN] defines a process for a TURN client to

The last feature of this script (for now) was added when I tried to release two I-Ds at the same time, with each I-D containing a reference to the other. Because the text that will be included is generated after the upload to the IETF, there was no way to automatically use the correct reference. What the script do is to generate a reference file automatically in /tmp for all the files processed. It is then easy to include the correctly generated reference is the document. For example draft-ietf-behave-turn-uri.xml and draft-petithuguenin-behave-turn-uri-bis.xml have a reference on each other. So the first step is to change the reference inside draft-ietf-behave-turn-uri.xml like this:

<xi:include xi:href="/tmp/reference.I-D.petithuguenin-behave-turn-uri-bis.xml" />

and to change the reference inside draft-petithuguenin-behave-turn-uri-bis.xml like this:

<xi:include xi:href="/tmp/reference.I-D.ietf-behave-turn-uri.xml" />

Then running the script on the two XML files at the same time generates the correct XML file, ready to be uploaded to the IETF:

$ rfc2629 ../src/share/docs/draft-ietf-behave-turn-uri.xml ../src/share/docs/draft-petithuguenin-behave-turn-uri-bis.xml