Jarc: Now running Java 1.7

At last, installable packages for OpenJDK 7 are available in the experimental Debian repository. So I released a new version of jarc that is now directly supporting Java 7 code but, because the openjdk-7 packages are in an experimental repository, I uploaded jarc 0.2.21 in an experimental repository so people would not be hit with dependency issues. I updated my Ubuntu/Debian repository page to explain how to add my experimental repository to the repository configuration file.

The reason to release this experimental version of jarc is that I need two the new features of the new JVM for my RELOAD implementation. The first one is the support of TLS 1.2, aka RFC 5246, which is used by RELOAD to authenticate and encrypt all the TCP connections. The only thing missing to completely implement RELOAD will be an implementation of DTLS 1.0 (which is to datagrams what TLS is to streams). I guess I will have to bite the bullet and write this one myself.

The second feature that I needed is an implementation of the SCTP protocol. TCP is great for transferring files but it has one major flaw when used with multiplexed client/server transactions, which is called the Head-Of-Line problem – a message lost will block all the subsequent messages, even if they belong to a different transaction, until the lost message is retransmitted and received. UDP does not have this issue but comes with its own set of problems. SCTP is somewhere between UDP and TCP, and this is why, in my opinion, it is a good transport protocol for something like RELOAD (well, between RELOAD nodes with a public IP address, as most NATs – one exception been the one I supervised the development at 8×8 – do not NAT SCTP).

So the plan will be to add SCTP to RELOAD, using either TLS (RFC 3436) or DTLS (RFC 6083) depending on the DTLS implementation described above, and to write an I-D, perhaps in time for IETF 81 in Quebec City.

Fireproof backup: another update

A little bit more than one year after replacing it, my backup disk was starting to show signs of weakness, so it was time for another replacement. This time, I was able to find a 1.5 Tb disk that does not require an external power supply. With this disk size I will be able to keep 2 weeks of backup and fully automatize the removal of old backups (it’s currently a manual process).

But this disk is so new that it is not really well supported by Linux. The first step was to update the /usr/share/smartmontools/drivedb.h file with the following entry, so smartmontools can recognize it:

{ "USB: Seagate FreeAgent Go Flex USB 3.0; ",
"0x0bc2:0x5031",
"",
"",
"-d sat"
},

Unfortunately hdparm does not seem to recognize it either, but it found a way to test the standby status by using smartclt instead (I need this to be sure that the disk is really in standby mode before opening the safe – see my previous posts on this subject for the reason):

$ sudo smartctl -n standby /dev/sde
smartctl 5.41 2011-03-16 r3296 [x86_64-unknown-linux-gnu-2.6.38-2-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Device is in STANDBY mode, exit(2)

It seems that this disk is a little bit slower than the previous one – a full backup takes ~9 hours now, where with the previous disk it took a little bit more than 7 hours. The new disk is supporting USB 3.0, it’s a shame I cannot use it.

Last minor issue, there is a bug in the kernel, so each time the disk is used a spurious log entry appears in syslog:

[171412.410641] sd 16:0:0:0: [sde] Sense Key : Recovered Error [current] [descriptor]
[171412.418494] Descriptor sense data with sense descriptors (in hex):
[171412.425309] 72 01 04 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
[171412.432089] 00 00 00 00 40 50
[171412.436059] sd 16:0:0:0: [sde] ASC=0x4 ASCQ=0x1d

It seems that there is a patch available for this problem, but I do not see it applied in the latest kernel yet.

Update 05/22/2011:

A full backup with USB 3.0 takes only 13% less time than with USB 2.0. So more or less the same speed that I had before changing the disk. It also seems that the USB 3.0 driver is not complete as I had to force -d sat,12 to use smartctl, and the disk never goes to standby mode. And as expected, Linux 2.6.39 does not fix the sense data bug.

Update 11/19/2011:

Problem finally fixed in Linux 3.1.

Update 07/15/2013:

Fixed link to previous post on this subject.

Sharpening the axe

“If I had six hours to chop down a tree, I’d spend the first four hours sharpening the axe – Abraham Lincoln”

Developing good software requires to have the right tools and a good part of a job of a developer is to find what are the best tools for the job. It takes years of research and experimentation to assemble the right toolset. Obviously it starts with the operating system – my quest started with DOS, then OS/2 (the first OS I used that had multithreading) then Windows NT until daily BSOD made it unbearable. I switched for a few months to MacOS (when it was still non preemptive and so daily BSOD were simply replaced by daily freezes) to finally settle with Linux. That was back in 1999 and I never looked back – the number of available tools for developers was simply overwhelming.

In a way writing software on Windows is like working on your motorcycle in your living room. You certainly can, but it’s probably a better idea to do it in a workshop equipped with the right tools, most of them freely modifiable to fit your needs, and that exactly what Linux is. To push the analogy a little bit further, developing on MacOS would be like working on your motorcycle in an hotel suite.

Sure it was not an easy switch – I was still addicted to using IDE, which were simply nonexistent on Linux at the time, so I tried Emacs (I still have the GNU Emacs book) but then definitively switched to gvim. I also tried various Window managers – Gnome at the beginning, Enlightenment for a long time until I settled with Awesome, which give me a fast and minimal window system that can be totally controlled from the keyboard, like gvim (the mouse, like IDEs, are fine when learning. After some time you outgrew then and just want to be more productive, and they simply get on the way).

One of the most fantastic tool offered by the Debian GNU/Linux distribution was the packaging system. I still think that this is one of the most important tool I use today. Packaging software is a concern since a long time for me, and I started doing packaging back when I was developing for the AS/400, in the beginning of the 90s. The reason to do it at the time is exactly the same than now: Controlling the environment that is used to run the software I develop. I would say that half of the bugs found in a software comes from an environment that is different from the environment used by the developer during the testing (The famous “but it works on my computer”). Forcing the deployment of the software in a controlled environment cut down on the quantity of testing that need to be done and at the end on the number of bugs. All the teams I managed since 2000 had to follow this rule: Software does not exist until it is packaged.

Source control is another very important tool – after some painful experience with SourceSafe (a special hell should be reserved to whoever thought that locking source was a good idea), I used CVS, then Subversion, and for the last two years Git – which alone would justify switching to Linux – and with Gerrit as my current favorite peer review tool.

But all of this is merely choosing the right tool, not sharpening it.

Sometimes who cannot find the tool that you need. One solution is to write it from scratch because there is nothing like it (like I did for jarc), but that’s possible with any operating system. Often there is an existing tool that is close to what you need, but not enough to be useful. This is where using FOSS pays off, because now what a developer can do is to modify an existing tool so it fits its own needs – and this is what I really call sharpening the axe. Few years ago I worked intensively on multiple implementation of the TURN protocol, so my team wrote a Wireshark dissector for TURN. Now at Stonyfish were are working on an implementation of RELOAD, so we wrote a Wireshark RELOAD dissector, so we are distributing it as a plugin, so it can help other RELOAD implementers:

Wireshark dissector for RELOAD

Jarc: Support for servlets

I have been busy during all 2010 with Stonyfish (a start-up I founded with a former colleague from 8×8), but I did not stop improving the jarc tool. The most important new feature is the ability of generating war files (packaged servlets):

A war file can be created by adding an X-Jarc-Web-App attribute in the manifest header that point to the servlet descriptor file. e.g.:

Manifest-Version: 1.0
X-Jarc-Web-App: org/implementers/servlet/http/web.xml

The syntax of the web.xml file has also been extended to permit to list file and jar files that need to be copied into the WEB-INF/classes directory:

<web-app>
  <lib>file1.jar</lib>
  <lib copy="true">file2.jar</lib>
  <data>lang/project.properties</data>
</web-app>

The latest version (0.2.19) also run the junit tests with the assertions enabled and fixes a lot of bugs found during 2010.

See the previous blog entries on this subject to find how to install the jarc tool. as always the source code is available under an Affero GPL 3 license.

Sirius Streaming Server

I am a Sirius subscriber since 2004. I listen to music mostly when working so I had an external satellite antenna hooked to a Kenwood DT-7000S receiver but last year I moved to a new house where this hook-up is not as easy because of the distance between the antenna and the audio system. Until I could find a solution I used the Sirius Internet Radio but this is not a really satisfactory system for various reasons – not been able to see the name of the song/artist; having to click on the renew button each hour; and, as always, mostly incompetent ISP.

I wanted to move back to a pure satellite solution, especially because there is already a DirectTV antenna which is oriented in the exact correct direction. It was easy to install the Sirius antenna in the same mast, without annoying the neighborhood association.

The next step was to reuse the cabling for the satellite antenna (I do not use DirectTV). and for this I just needed one adaptor to convert to RJ-6 cable and another to convert back to the satellite cable.

The next step was to find a way to send the audio over my home network – I have Netgear powerline adapters in most rooms of the house. The plan was to build an RTSP server with VLC, so I could listen to the radio on-demand, without having to continuously broadcast on the network. Multicast would have been a nice solution, but without managed switches that can direct the audio stream on the right Ethernet ports, the network would have been overwhelmed.

The satellite receiver has an optical output, so it does not make sense to convert the satellite signal to analog, to sample it back to digital in the streaming server. So I found at Fry’s this cheap USB sound device that has an optical S/PDIF input.

My first idea was to reuse a Mac mini which was already in the same room than the satellite receiver, but I then discovered that VLC on Mac OS cannot use the sound input of the computer. So plan B was to use a SheevaPlug. The Linux Unbuntu distribution used by the SheevaPlug was not working for some reason, so I installed Debian on an SD card. Alas, now the USB sound device was not working, probably because of a bug in the USB implementation. Inserting an USB hub between the device and the SheevaPlug solved the problem. Finding the correct parameters for VLC was not easy – for example I had to look at the source code to find that the URI to use the input port is “alsa://”. I also had to try multiple codecs (OGG was not supported by the RTSP implementation, MP3 was using more CPU than available) until finding that the A/52 (aka Dolby AC3) codec was perfect for this as VLC is using less than 8% of the CPU to stream a stereo channel at 256 Kb/s with a sampling rate of 48 Khz. Here’s the command line to start VLC:

cvlc --ttl 12 --rtsp-host 192.168.2.156:5554 --vlm-conf vlm.conf

and here’s the content of vlm.conf:

new sirius vod enabled
setup sirius output #transcode{acodec=a52,ab=256,channels=2,samplerate=48000}
setup sirius input alsa://

I would have preferred streaming over IPv6, but VLC does not seems to support this yet. I also use VLC on my desktop to control the audio stream, coupled with Jack to mix the audio from other audio sources. The URI I use is “rtsp://192.168.2.156:5554/sirius”, and I can pause and stop the stream.

So I now have an uninterrupted audio stream from my favorite satellite radio channels directly to any place in the home. One thing I lost is the possibility to change the channel remotely. The plan was to add a USB to serial port adaptor between the SheevaPlug and the satellite receiver (so the intermediate USB hub was necessary even if the USB sound device was not buggy) to drive the receiver, but unfortunately I was not able to find the RS-232 protocol for this. The only reference I found is in a Crestron module but $256 is expensive for a piece of software that I am not even sure to be able to reverse-engineer. Having the protocol would also have been nice to send the current song information in the audio stream. I would be interested by any information on this protocol as long as I do not have to sign a NDA. Perhaps there is someone in the Valley with a working RS-232 connection like this and who will be willing to let me record the signals exchanges. As always I have a plan B but that’s a painful one, to say the least.

Debian package for shadok

Shadok is a tool written by Stéphane Bortzmeyer to verify and create graphs for state machines written in a very simple language named Cosmogol. Everything is explained at the following URL:

http://www.cosmogol.fr/shadok.html

The language was proposed some time ago to the IETF as a way to describe a state machine inside a text document like an RFC. I still think that this is a great idea and in the hope that more people use it I packaged it for Debian and Ubuntu and uploaded it on a public Debian repository.

To access this repository, add the following lines in your /etc/apt/sources.list file:

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/

You will also need to install a GPG key to verify the repository:

$ wget http://debian.implementers.org/keys/key.asc
$ sudo apt-key add key.asc

The fingerprint of the key is as follow:

CDB3 BC52 DFDC 8A58 35F1 714D 0309 943D 9AD0 8F3F

You can then run the following commands to install the shadok package:

$ sudo apt-get update
$ sudo install shadok

The repository contains only the binary packages for the i386 and amd64 architectures. You can download the source and recompile it for other architectures with the following commands:

$ sudo apt-get build-dep shadok
$ apt-get source shadok
$ cd shadok-0.0.0
$ dpkg-buildpackage -D -b -us -uc

Enjoy!