Author: Marc Petit-Huguenin

Jarc: Now running Java 1.7

At last, installable packages for OpenJDK 7 are available in the experimental Debian repository. So I released a new version of jarc that is now directly supporting Java 7 code but, because the openjdk-7 packages are in an experimental repository, I uploaded jarc 0.2.21 in an experimental repository so people would not be hit with dependency issues. I updated my Ubuntu/Debian repository page to explain how to add my experimental repository to the repository configuration file.

The reason to release this experimental version of jarc is that I need two the new features of the new JVM for my RELOAD implementation. The first one is the support of TLS 1.2, aka RFC 5246, which is used by RELOAD to authenticate and encrypt all the TCP connections. The only thing missing to completely implement RELOAD will be an implementation of DTLS 1.0 (which is to datagrams what TLS is to streams). I guess I will have to bite the bullet and write this one myself.

The second feature that I needed is an implementation of the SCTP protocol. TCP is great for transferring files but it has one major flaw when used with multiplexed client/server transactions, which is called the Head-Of-Line problem – a message lost will block all the subsequent messages, even if they belong to a different transaction, until the lost message is retransmitted and received. UDP does not have this issue but comes with its own set of problems. SCTP is somewhere between UDP and TCP, and this is why, in my opinion, it is a good transport protocol for something like RELOAD (well, between RELOAD nodes with a public IP address, as most NATs – one exception been the one I supervised the development at 8×8 – do not NAT SCTP).

So the plan will be to add SCTP to RELOAD, using either TLS (RFC 3436) or DTLS (RFC 6083) depending on the DTLS implementation described above, and to write an I-D, perhaps in time for IETF 81 in Quebec City.

Fireproof backup: another update

A little bit more than one year after replacing it, my backup disk was starting to show signs of weakness, so it was time for another replacement. This time, I was able to find a 1.5 Tb disk that does not require an external power supply. With this disk size I will be able to keep 2 weeks of backup and fully automatize the removal of old backups (it’s currently a manual process).

But this disk is so new that it is not really well supported by Linux. The first step was to update the /usr/share/smartmontools/drivedb.h file with the following entry, so smartmontools can recognize it:

{ "USB: Seagate FreeAgent Go Flex USB 3.0; ",
"0x0bc2:0x5031",
"",
"",
"-d sat"
},

Unfortunately hdparm does not seem to recognize it either, but it found a way to test the standby status by using smartclt instead (I need this to be sure that the disk is really in standby mode before opening the safe – see my previous posts on this subject for the reason):

$ sudo smartctl -n standby /dev/sde
smartctl 5.41 2011-03-16 r3296 [x86_64-unknown-linux-gnu-2.6.38-2-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Device is in STANDBY mode, exit(2)

It seems that this disk is a little bit slower than the previous one – a full backup takes ~9 hours now, where with the previous disk it took a little bit more than 7 hours. The new disk is supporting USB 3.0, it’s a shame I cannot use it.

Last minor issue, there is a bug in the kernel, so each time the disk is used a spurious log entry appears in syslog:

[171412.410641] sd 16:0:0:0: [sde] Sense Key : Recovered Error [current] [descriptor]
[171412.418494] Descriptor sense data with sense descriptors (in hex):
[171412.425309] 72 01 04 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
[171412.432089] 00 00 00 00 40 50
[171412.436059] sd 16:0:0:0: [sde] ASC=0x4 ASCQ=0x1d

It seems that there is a patch available for this problem, but I do not see it applied in the latest kernel yet.

Update 05/22/2011:

A full backup with USB 3.0 takes only 13% less time than with USB 2.0. So more or less the same speed that I had before changing the disk. It also seems that the USB 3.0 driver is not complete as I had to force -d sat,12 to use smartctl, and the disk never goes to standby mode. And as expected, Linux 2.6.39 does not fix the sense data bug.

Update 11/19/2011:

Problem finally fixed in Linux 3.1.

Update 07/15/2013:

Fixed link to previous post on this subject.

Parallax: The Race to Measure the Cosmos

I had a long discussion the other day with my sister during which I told her how one-fifth of North-American believe that the the Sun is orbiting the Earth. I kept thinking about this after the end of the discussion, and I was wondering if there was a way to demonstrate to people, by a sort of experience that everybody can do on their home, that it is in fact the Earth that is moving around the Sun. I was thinking of something like the experience were you put a non-moving stick of butter in a microwave oven, measure the distance between the melting point and by multiplying by the frequency of the microwave, demonstrate the speed of light. Obviously if it took until two hundred years ago to find this proof, that was probably more complex that I expected. But perhaps that, like the measurement of the speed of light that everybody can do at home thanks to the availability of stuff that scientists did not had access to in the past, there was a way to also reduce the complexity to a combination of available equipment (I am an incorrigible optimist who believe that any problem can be solved by time and a better education).

Certainly the most understandable proof that we have today is that we are capable to send missions to Mars and that this would be impossible if our knowledge of how the solar system works was wrong. But that would be forgetting that most people who believe that the Sun is orbiting the Earth probably also believe that the travels to the Moon were an hoax, so we are back to the same problem: Until people can do the experience themselves, there is no way they will believe it. This is the reason why those people say that evolution is just a theory when they have no problem with the gravitation theory.

So I started to search what was the proof that the Earth is orbiting the Sun (I had no idea myself what was this proof, even if I do not remember ever having any doubt about this). The most fascinating one was called the aberration of light and was soon searching what kind of hardware James Bradley used to make this discovery. I knew it was some sort of meridian telescope and I was thinking that hooking up a Celestron telescope with a webcam connected on the Internet would permit to show to everybody the movement of the star that indicates that the Earth is moving on an ellipse – hoping that the progress in optics and computer would permit to replicate the experience. This is how I went to read on Google one or two pages of “Parallax: The Race to Measure the Cosmos”, immediately order it on Amazon, and read it back to back on two evenings. The book is a fascinating reading and I strongly recommend but it was a disappointment to discover that the kind telescope used by Bradley was still out of reach for an amateur (Here’s a picture that shows exactly what I mean).

I have some ideas on how to perhaps solve these problems but by lack of available time this project will unfortunately have to go on the “things to do when I am retired” folder. But that was a great book to read.

RELOAD: Access Control Policy distribution

This blog had been quiet for the last month because I was working on an implementation of RELOAD, the peer-to-peer protocol developed at the IETF as a component of peer-to-peer SIP (I do not plan to develop a peer-to-peer SIP implementation, but I needed a RELOAD implementation for another project). The plan is to release a complete implementation of RELOAD as a Java library under an Affero GPL 3 license and to let the company I co-founded, Stonyfish Inc, sell commercial licenses to people in need of a less restrictive license. The code will be released progressively between now and July, so keep an eye on this blog for progress.

Anyway during the development I came to see some limitations in the RELOAD specification (which is still not an RFC). Most of them were solved by the P2PSIP Working Group and there is still few that are waiting for discussions and decisions by the WG. Hopefully all will be solved before the final release. But there was one specific problem that required a somewhat different treatment, and this is the subject of this blog entry.

RELOAD is, among other things, a Storage protocol. Any user of a network of RELOAD servers (an overlay) can store data in it, and these data are automatically replicated and available to any other user. Because the overlay is designed to be secure even in presence of users with less than nice goals, only the user that stored a piece of data can modify it. The rules that are used to decide who can or cannot write or modify a piece of data in an overlay are grouped into what is called an Access Control Policy. There is four different Access Control Policies defined in the RELOAD specification, and the intent was that these four policies would cover most of the future needs. And even if there is a way to add new Access Control Policies, only a limited number would be defined.

Unfortunately, it turns out that there is a need for more than the four policies already existing. After a survey of all the existing proposals for new types of data to be stored in an overlay (this is called an Usage), I discovered that more than 50% of the new Usage are requiring a new Access Control Policy. In my opinion that creates a problem that could kill the usefulness of RELOAD before it even start to be deployed.

Let’s say that I start a new overlay and that I distribute my software to hundred of thousand of users, each of them using the overlay to store their data in the way that was defined in this first version. Everything works fine until I decide to introduce a new feature that require a new Access Control Policy. The problem is that it is not only the users that will use this new feature that will have to upgrade their copy of the software. No, to be able to even start deploying this new feature, I will have to wait that ALL the users upgrade the software. If the story of IE6 teaches us anything, it is that it will never happen. And the problem is even worse if the software used comes from different vendors.

So the proposal I made to the P2PSIP Working Group is to automatically distribute the code of a new Access Control Policy, without having to upgrade the software. This way instead of waiting months or years to deploy a new feature, it will take only 24 hours or so to have all the users ready to store the new data.

Obviously this code has to be portable so it can be executed on any language used to write the RELOAD implementations in an overlay. So I chose JavaScript (or more precisely ECMAScript) to do that – not because I like the language (JavaScript is the rubber band that hold the Web together, and I do not mean that in a nice way) but because, thanks to the current Web Browsers war, there is very good implementations available.

I am presenting this concept at IETF 80 in Prague on March 31th. You can read the Internet-Draft or the slides for the presentation if you cannot attend.

Jarc: Junit errors as compilation errors

The jarc tool supports running Junit tests as part of a jar build since version 0.2.5. Since then when a unit test fails during a jar build the output of the jarc tool looked like this:

$ jarc reload
testInit(org.implementers.nio.channels.reload.ReloadNodeTest)
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.fail(Assert.java:98)
at org.implementers.nio.channels.reload.ReloadNodeTest.testInit(ReloadNodeTest.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)

The problem with this output is that it is not well integrated with other tools. Gvim for example cannot parse this output to display the current junit test file and line and worst, the output is lost so I had to remember the line number to be able to find where the test failed.
One way to fix this could have been to change the error format string in Gvim to parse the output, but that would not helped for other tools. So instead of this, jarc v0.2.20 is now formatting Junit errors with the same format than Java compilation errors, so the error above looks like this now:

$ jarc reload
../src/share/classes/org/implementers/nio/channels/reload/ReloadNodeTest.java:150: java.lang.AssertionError:

Now Gvim and other tools can parse the output and display directly the line that failed.

Sharpening the axe

“If I had six hours to chop down a tree, I’d spend the first four hours sharpening the axe – Abraham Lincoln”

Developing good software requires to have the right tools and a good part of a job of a developer is to find what are the best tools for the job. It takes years of research and experimentation to assemble the right toolset. Obviously it starts with the operating system – my quest started with DOS, then OS/2 (the first OS I used that had multithreading) then Windows NT until daily BSOD made it unbearable. I switched for a few months to MacOS (when it was still non preemptive and so daily BSOD were simply replaced by daily freezes) to finally settle with Linux. That was back in 1999 and I never looked back – the number of available tools for developers was simply overwhelming.

In a way writing software on Windows is like working on your motorcycle in your living room. You certainly can, but it’s probably a better idea to do it in a workshop equipped with the right tools, most of them freely modifiable to fit your needs, and that exactly what Linux is. To push the analogy a little bit further, developing on MacOS would be like working on your motorcycle in an hotel suite.

Sure it was not an easy switch – I was still addicted to using IDE, which were simply nonexistent on Linux at the time, so I tried Emacs (I still have the GNU Emacs book) but then definitively switched to gvim. I also tried various Window managers – Gnome at the beginning, Enlightenment for a long time until I settled with Awesome, which give me a fast and minimal window system that can be totally controlled from the keyboard, like gvim (the mouse, like IDEs, are fine when learning. After some time you outgrew then and just want to be more productive, and they simply get on the way).

One of the most fantastic tool offered by the Debian GNU/Linux distribution was the packaging system. I still think that this is one of the most important tool I use today. Packaging software is a concern since a long time for me, and I started doing packaging back when I was developing for the AS/400, in the beginning of the 90s. The reason to do it at the time is exactly the same than now: Controlling the environment that is used to run the software I develop. I would say that half of the bugs found in a software comes from an environment that is different from the environment used by the developer during the testing (The famous “but it works on my computer”). Forcing the deployment of the software in a controlled environment cut down on the quantity of testing that need to be done and at the end on the number of bugs. All the teams I managed since 2000 had to follow this rule: Software does not exist until it is packaged.

Source control is another very important tool – after some painful experience with SourceSafe (a special hell should be reserved to whoever thought that locking source was a good idea), I used CVS, then Subversion, and for the last two years Git – which alone would justify switching to Linux – and with Gerrit as my current favorite peer review tool.

But all of this is merely choosing the right tool, not sharpening it.

Sometimes who cannot find the tool that you need. One solution is to write it from scratch because there is nothing like it (like I did for jarc), but that’s possible with any operating system. Often there is an existing tool that is close to what you need, but not enough to be useful. This is where using FOSS pays off, because now what a developer can do is to modify an existing tool so it fits its own needs – and this is what I really call sharpening the axe. Few years ago I worked intensively on multiple implementation of the TURN protocol, so my team wrote a Wireshark dissector for TURN. Now at Stonyfish were are working on an implementation of RELOAD, so we wrote a Wireshark RELOAD dissector, so we are distributing it as a plugin, so it can help other RELOAD implementers:

Wireshark dissector for RELOAD

Jarc: Support for servlets

I have been busy during all 2010 with Stonyfish (a start-up I founded with a former colleague from 8×8), but I did not stop improving the jarc tool. The most important new feature is the ability of generating war files (packaged servlets):

A war file can be created by adding an X-Jarc-Web-App attribute in the manifest header that point to the servlet descriptor file. e.g.:

Manifest-Version: 1.0
X-Jarc-Web-App: org/implementers/servlet/http/web.xml

The syntax of the web.xml file has also been extended to permit to list file and jar files that need to be copied into the WEB-INF/classes directory:

<web-app>
  <lib>file1.jar</lib>
  <lib copy="true">file2.jar</lib>
  <data>lang/project.properties</data>
</web-app>

The latest version (0.2.19) also run the junit tests with the assertions enabled and fixes a lot of bugs found during 2010.

See the previous blog entries on this subject to find how to install the jarc tool. as always the source code is available under an Affero GPL 3 license.

Fireproof backup: an update

One year ago I posted a blog entry about my fireproof backup. The backup is still in place today, and prevented the loss of multiple days of work, but there was some occasional problems at the beginning that could be interesting to share.

The first disk I used died after something like 3 months, probably because I opened the safe during a backup (the disk is located in the safe’s door). But one’s could have expected more from a disk supposed to be “ultra portable”. Anyway, I bough a second disk, but there was bad sectors which created problems for the first few weeks, until I reformatted it with “mkfs.ext3 -c”. I am still removing the the old backups file manually each week before the full backup, and for good measure I do also a “fsck -f” before removing the files on the disk (even if it takes more time, I learned the hard way that the fsck should be done *before* removing the files, in case fsck decide that the most recent files are corrupted…)

As for the problem that opening the safe door during the backup is not a good idea, I tried to find a way to have a visual clue on the safe itself that the disk is currently used, but without success. Something as simple as a male/female dongle that could be inserted in the USB cable, with an LED that is on when the current drained on the +5V line is above some threshold would be perfect. Another project to put on the stack…

I also would like to use a larger disk – 1.5 Tb would be perfect, as I could keep two full backups, plus two weeks of incremental backup. Currently I have to be careful and, for example, synchronize and rebuild the complete Android tree only on the day before the full backup. Unfortunately I was not able yet to find a disk powered by the USB connector that can have this capacity (but removing all the ogg files from my desktop computer, as explained in the previous blog entry, helped keeping this under control for now).

I also tried to convert my setting to USB 3.0 for accelerate the backup (it currently takes between 8 and 15 hours) – going as far as installing a USB 3.0 card in my desktop – until I realized that I cannot change the USB cable that goes inside the safe without compromising the waterproof seal.

And talking about waterproof seal, one of the problem of the safe itself is that humidity quickly build up inside the safe (probably exacerbated by the heat of the disk), so after some time the documents stored feel kind of damp. I bought a dehumidifier that did solve the problem. I just have to plug it overnight each three months or so to renew the crystals.

Android in the car

I am working on an experiment on how I can reduce to the minimum the things I have to carry with me when I am out of the house. I successfully reduced the size of my wallet to 90mm x 66mm x 17mm. Keys are one area I still have to improve, but I focused mainly on the smartphone, which is the third item I have with me when traveling. My current smartphone is a Nexus One, and my first goal was to integrate it as much as possible with my car.

For a long time, I used a Motorola T505 speakerphone, which, in addition to the normal speakerphone feature, also contain an FM emitter, which gave me the possibility to use the Nexus One as an MP3 player (well, I use only OGG files – another advantage of the Android phone – so that would be an OGG player). This was working fine, and I will still keep my T505 to use in rental cars when I am traveling, but there was some limitations: I had to remember to charge the T505 from time to time (although I used it for the whole nine hours of driving between Saratoga and Las Vegas), I had to remember to connect it and disconnecting it, but the worst, especially on long travel, was the low quality and the need to periodically change the FM station.

So the goal was to have a better sound and a better integration in the car. After some research I decided to buy a Clarion BLT370. This is also a speakerphone but instead of having an integrated speaker, it has audio connectors to connect to the auxiliary input of the car’s audio system. It is also powered directly by the car battery, so there is no need to remember to charge it. Also the Bluetooth automatically disconnect when stopping the car engine and, more important, automatically tries to reconnect to the Nexus One when the engine starts, which is perfect, as I do not even have to remove the phone from my pocket to start playing music or receiving phone calls.

One of the potential problem was that now a cable had to go down the windshield, but fortunately I was able to hide it by running it inside the piece of plastic on the left of the windshield. Excepted that I need to be a bit more careful when opening and closing the softop, the integration as perfect as possible.

Now the next problem was that even if the Clarion had connectors to connect to the car’s audio system, the audio system in a 2004 MX5 does not have auxiliary audio input (that was before the iPod, I guess).

Luckily, other people had the same issue, and I found this terrific product, the Sylfex AuxMod. This card can be installed in the slot that is normally used for the tape player. The picture on the left shows the card installed, with the ground loop isolator required to eliminate a clicking noise. With this the sound is exceptionally good, so good that I decided to buy a 32Gb SD card for my Nexus One, rip again all my CDs in 256 Kb/s OGG files, and keep them only on my Nexus One. Now instead of playing music from my desktop computer, I play it from my Nexus One, comfortably installed in the Nexus One Desk Dock, at least until I configure a direct stream between the Nexus One and my desktop computer.

The last step was to connect the Clarion to the car battery. I did not want to do any irreversible modification in this car, so I used a Add-A-Fuse thingy, and some Velcro to keep the cables from falling on the driver’s feet.

I also installed a Nexus One Card Dock, but I do not use the integrated speakerphone, just the charging function and the ability to keep the Nexus One keep in a position easily accessible when using the GPS (the GPS turn by turn instructions are send to the audio system through Bluetooth and the Clarion transceiver).

The next step will be to find a good replacement for the cable used to charge the Nexus which, as can be seen in the picture on the left, is not really well integrated in the car. The difficulty is that although is is easy to find a cable that connect to the battery for mini-USB connector, it is not for micro-USB connectors like the one on this car dock. The only one I could find was on a web site in England.

Application developers and DNS TTL

During the technical plenary of the 73th IETF meeting in Minneapolis MN, Dave Thaler made the interesting point that most DNS resolver API do not return the TTL of the resource resolved, e.g. an IP address. At the time he was proposing a modification of the existing APIs, and that made me thinking since.

The problem is that programmers generally think that resolving a domain name and then storing the IP address(es) resulting of this resolution is a good idea, as this technique could yield a better responsiveness for an application. But doing that without having a mechanism to know that the result of the resolution is invalid creates an even bigger problem. During my time at 8×8, I worked on the problem of designing a scalable Internet service, and my design was focused on having the scalability driven by the client (in other words: the whole load balancers idea is evil). Such techniques are very cheap and effective, but only if the client strictly obeys the TTL value received in the DNS response (I still have two pending patent applications about this techniques). Another well known problem is documented in RFC 4472 section 8.2, as keeping an IP address for too long prevents renumbering an IPv6 network, but there is plenty of other cases.

So the idea of passing the TTL together with the result of the DNS query seems like a good one, until you realize that in fact what developer have now to do is to implement a DNS cache in their application, and every evidence shows that this is not a simple task. As can be seen by the number of security vulnerabilities found during the years, even people who do read the RFC seem to have an hard time doing it right. Internet could probably do without another wave of DNS cache implemented incorrectly.

So in my opinion, adding the TTL to the API is not the solution – it will just exchange one problem with another. The correct solution is to do the resolution each time the resource is needed and do not store the result at all. If performances are too much impacted (after scientifically measuring them, we are between professionals here) then using an external DNS cache will fix the problem. The DNS cache can be in your network (for example having two DNS caches per data center), can be on each server (dnscache from the djbdns suite is easy to install and configure and has a good security track), or even directly in your application (for example dnsjava contains such a cache).