Fireproof backups

After the last three earthquakes in the Bay Area, I started thinking that developing code without backup in an house made entirely of wood was probably not the best idea in the world. I work from home since a long time but until last year I always committed my code in the central repository of my employer, so it was not really an issue. I use RAID10 for my development computer so losing a disk is not a problem, but I would still loose all the code I wrote during the last year if the house burns down.

I basically never delete anything – when my disks are full I just buy bigger disks and copy everything on them. The consequence is that I have currently 572Gb to backup, so that excludes doing remote backup – I used rsync.net in the past and they have a terrific service, but uploading 300Gb of compressed data is not an option here – and I do not want to save only a subset of the disks.

Doing a backup on an external disk was the only solution remaining, but the problem is still that if the house burn then the backup disk will also burn. I could have rented a safe box at my bank but having to exchange the disks each week was too much of a burden. The solution I choose was to put the backup disk in a fireproof safe installed in the house. Now the next problem is that the house can burn when I am doing the backup – which will be done each night, so that’s more or less eight hours each day with an unprotected backup. One neat solution I found was a disk that was directly encased in a fireproof safe. The big advantage of this disk is that it is protected even during the backup. The problem is that I also wanted to be able to store my passport and other documents in it. So I found the perfect solution: A USB fireproof safe. I can now put a 1TB disk inside the safe, and use it without having to open and close the safe.

The next step was the backup itself. I wanted to use good old dump/restore but with only one partition containing everything I was not able to unmount it to do the backup and doing a backup on a live partition – even during the night – is not a good idea. And anyway with a full backup taking ten hours to complete that was not an option. The best solution I found was to use a LVM snapshot – you just create a snapshot of the partition to backup, then you backup the snapshot and can continue to use the main partition. Unfortunately I did not had LVM installed so I had to copy the whole partition on an external disk, change the partitions to support LVM and then copy back the whole external disk – it took twenty-four hours to do this, but it was worth it.

The last step was to install dump and configure it to do a full backup each Sunday and an incremental backup each other night. I used some of the scripts delivered in the dump package, with some modifications to adjust them to my needs.

Now I receive an email each morning with the result of the backup. The last thing remaining to do will be to try a full restore.

Updated 01/10/2011

Jarc: a jar file compiler

Using the “make” tool to build a jar file was never a good option, mostly because Java files are different from C/C++ files. Today most people use Ant but I never liked it so around 2002 I started writing a tool that would permit to build a jar file in one step. I had two requirements in mind: First the tool had to be faster than running javac and jar – at the time the JVM was really slow, so I used zip and jikes, at least until IBM stopped updating jikes.

The second requirement was to compile only the Java files that were necessary to build the jar file, the idea been to give to the jar tool only the name of the “root” Java classes, and let it figure out the dependencies (I am not an Ant specialist but it seems to me that Ant is still not capable of doing this correctly). What was missing was a convenient way of passing the list of “root” classes to the tools, and this is when I thought of using the manifest file for this.

A manifest file is a text file stored inside the META-INF directory of the jar file. This file contains the meta-information about the jar file and is generated by the jar tool. The idea was to invert the process – instead of having the jar tool generating this manifest file from the command line parameters, why not use this file as the source of the information needed to build a jar file?

The jarc tool (jar compiler) takes as input one or more manifest files, parses the content and tries to build a jar file of same name. The tool will use the class names found in the manifest file as the “root” classes to use as dependencies. A manifest file looks like this:

Manifest-Version: 1.0
Class-Path: /usr/share/java/jspeex.jar
Main-Class: org.implementers.apps.Main

Name: org/implementers/apps/Main.class

Name: org/implementers/apps/resources/schema.xsd

If this text is stored in a file named apps.mf, then running “jarc apps” will build a jar file named apps.jar by executing the following steps:

  • The org.implementers.apps.Main class will be compiled together with all the classes that depends on it. The content of the Class-Path attribute is passed to the compiler as if a -classpath option was used.

  • The org/implementers/apps/resources/schema.xsd file will be copied from the disk to the jar file.

  • The manifest file itself will be filtered and copied inside the jar file.
  • A short bash program will be generated to call the class described in the Main-Class attribute

There is also some attributes that can be added in the manifest to drive the build. One example is the X-Jarc-Service that can be added to an entry to declare it as a service as shown in the following example:

Name: org/implementers/nio/channels/spi/CompositeProvider.class
X-Jarc-Service: java.nio.channels.spi.SelectorProvider

When jarc reads this line, it generates in the jar file a service file in META-INF/services.

The tools is accessible on the following Debian repository (see a previous post on this blog for the explanation on how to configure Ubuntu or Debian):

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/

The tools is contained in the “jarc” package and is distributed under the GPL v3 license.

I wrote the previous version of jarc when I was working for my former employer, so to be sure to not infringe on their rights, I rewrote the tool from scratch in Java (the original version was a bash script). Actually the new version implements only the minimum necessary to build itself, so expect updates in the future.

Update 01/31/2010:
jarc was put in the unstable repository by mistake. The new versions (starting with 0.2.3) will be in testing. The best is to have both testing and stable in your /etc/apt/sources.list file and remove unstable completely:

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/
deb http://debian.implementers.org/ stable/$(ARCH)/
deb http://debian.implementers.org/ stable/all/
deb-src http://debian.implementers.org/ stable/source/

Debian package for shadok

Shadok is a tool written by Stéphane Bortzmeyer to verify and create graphs for state machines written in a very simple language named Cosmogol. Everything is explained at the following URL:

http://www.cosmogol.fr/shadok.html

The language was proposed some time ago to the IETF as a way to describe a state machine inside a text document like an RFC. I still think that this is a great idea and in the hope that more people use it I packaged it for Debian and Ubuntu and uploaded it on a public Debian repository.

To access this repository, add the following lines in your /etc/apt/sources.list file:

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/

You will also need to install a GPG key to verify the repository:

$ wget http://debian.implementers.org/keys/key.asc
$ sudo apt-key add key.asc

The fingerprint of the key is as follow:

CDB3 BC52 DFDC 8A58 35F1 714D 0309 943D 9AD0 8F3F

You can then run the following commands to install the shadok package:

$ sudo apt-get update
$ sudo install shadok

The repository contains only the binary packages for the i386 and amd64 architectures. You can download the source and recompile it for other architectures with the following commands:

$ sudo apt-get build-dep shadok
$ apt-get source shadok
$ cd shadok-0.0.0
$ dpkg-buildpackage -D -b -us -uc

Enjoy!

Empty synchronized statement

The selectors API in Java is a paradigm that I spent much time understanding during this last months. I developed a small but, I think, powerful library related to this topic, and this post will be followed by several other posts that will introduce this library. But before starting on the gory details I wanted to talk a bit about a pattern that I am using with selectors.

Selectors are best used in a more or less infinite loop, something like this:

Selector selector = Selector.open();
while (true) {
  if (selector.select() != 0) {
    for (Iterator<SelectionKey> iterator = selector.selectedKeys();
iterator.hasNext(); ) {
      SelectionKey key = iterator.next();
      // Do something with the key
      iterator.remove();
      }
    }
  }

The problem is that the selector is synchronized on itself and on the keys set when blocked in a select. This means that registering a new channel will simply be stuck waiting on a monitor until the select exits. If the selector is used with other channels, and there is some activity on this channels then the register will execute quickly and this bug will not be seen. But if this is the first channel to be registered then there will be a deadlock. One step in the right direction is to force the selector to wakeup just before registering:

selector.wakeup();
channel.register(channel, OP_READ);

Unfortunately, this still does not work. The reason is that nothing prevents the loop to execute the select a second time after the wakeup and before the call to register. It is easy to write a test program to verify this. The solution is to use a synchronization to prevent the loop to reenter the select before the register is executed. The register is inside the synchronized:

Object lock = new Object();
synchronized (lock) {
  selector.wakeup();
  channel.register(channel, OP_READ);
}

and the select loop now needs an empty synchronized:

Selector selector = Selector.open();
while (true) {
  if (selector.select() != 0) {
    for (Iterator<SelectionKey> iterator = selector.selectedKeys();
iterator.hasNext(); ) {
      SelectionKey key = iterator.next();
      // Do something with the key
      iterator.remove();
      }
    }
  synchronized (lock) {}
  }

There is other cases that would need to use the same pattern, like reading the selectedKeys set or the keys set in a different thread, as both need to be synchronized but the select method already synchronizes on them.

On the other hand, the selector close method does not need to use this pattern.

The interest set of selection keys is another interesting case. In the current implementation, there is no need to use the synchronization pattern shown above, so calling wakeup after changing the interest set is working fine. But the javadoc is clear that an implementation could behave differently:

“In a naive implementation, reading or writing the interest set may block indefinitely if a selection operation is already in progress;”

So always using the pattern above guarantees that the code will work on different Java implementations.

Now the problem could be that in the future a very optimized compiler or runtime decides that the empty synchronized statement is in fact useless, and decide to remove it.

Vacation readings

Vacation is more or less the only time I can read books. My Kindle 2 made it easy to carry them around and read them on planes, trains and metro during my last vacations in France:

  • Absinthe & Flamethrowers: Projects and Ruminations on the Art of Living Dangerously (William Gurstelle):

    A short and interesting book, but I would not put drinking absinthe in the list of dangerous things – well at least in my opinion it is way less dangerous than eating fast food regularly. I will probably write a bit more about the ideas on this book, as there is some that are related to something I will spend most of my time for the next year.

  • Daemon (Daniel Suarez):

    An excellent novel that is, a the difference of most novels about technology, completely realistic. I cannot wait for the sequel.

  • Thomas Jefferson (R. B. Bernstein):

    Thomas Jefferson is probably, with Benjamin Franklin, the most interesting of the Founder Fathers. This biography was so interesting that I found it way too short and I will try to find something that would go deeper in the life of this extraordinary person.

  • Declare (Tim Powers):

    There is no such thing as a bad Tim Powers book and there is no such thing as an easy to read Tim Powers book. My favorite is still Last Call but Declare was really captivating after I managed to read pass the first one hundred pages.

  • Fugitives and Refugees: A Walk in Portland, Oregon (Chuck Palahniuk):

    Well, I read everything from Palahniuk (*cough* Haunted *cough*), so I had to read this one, even if it is not a novel but a travel guide. As always, extremely entertaining.

WWRD?


This is in fact inspired by DC’s comics The Question #17 “A Dream of Rorschach” (page 19). I still didn’t see the movie adaptation of Watchmen.

Spreadshirt instructions:

Text: Flex Print, "WWRD?", Arial normal, 146pt, white; "What Would Rorschach Do?", Arial normal, 67pt, white

A Farewell to SIP

The IETF SIPPING meeting today in San Francisco was probably my last SIP meeting. There is nearly 10 years since I started my first implementation of SIP – a lot more followed and two of them still exist today (thankfully vastly improved by other developers since) , in Virtual Office and in Packet8.

I was enthusiastic about SIP from the very beginning, which was not surprising after having to work with H.323. I understood very quickly how SIP offered way more possibilities than the PSTN, and I anticipated the same revolution for real time communication that happened at the same time with the Web. So I waited, and waited and… nothing came. Where was my universal communicator? All I can see was the same old telecoms model transposed, as if the Internet had the same constraints than a bunch of relays, jacks and copper cables. My forte is on the server side but with nothing coming I decided to have a shot at it, and this is how the Zap project started. Unfortunately the Zap project is now without sponsor so I will probably never see it finished as I envisioned it.

SIP always had some flaws in its design, but for a long time I thought that simply ignoring them was enough. The problem is that this flaws were what helped the telecoms to implement their crap in the exact the same way than on copper – my latent paranoia makes me suspecting that this flaws were introduced for this very reason but, in application of Hanlon’s Razor, this is probably just plain stupidity. The net result is that the end-users lost, and the telecoms won.

I think that there is some lessons to learn from this failure. Don’t let the IETF design a protocol, there is too much big money influence to have a protocol that serves the end-users. Instead design the best protocol possible, write some FOSS code for it, give free access to servers running it, grow the end-user base and then, and only then, go to the IETF to standardize it. The small but powerful academic population of the IETF will probably be on your side and the big money population will have very little possibility to fuck up the protocol, as the IETF is a pragmatic organization.

Multiple Interfaces

I attended the MIF (Multiple Interfaces) BOF today at the IETF meeting in San Francisco. One hour was definitively too short to discuss this, but there was a strong consensus that more work should be done on this subject at the IETF.

When a host (as defined by the IETF – a network node that does not route packets) is connected to the Internet by multiple interfaces, there is a set of unique challenges to solve. There is more and more hosts with multiple interfaces – a cellphone with Wifi and 3G connection, or a desktop with broadband and a VPN falls in this category. Choosing the right connection to use for a specific destination can be difficult.

What is the most interesting, in my opinion, is that MIF can solve some problems that are basically impossible to solve with one interface. One example is E911, a protocol based on SIP to make emergency calls. With one interface an emergency call using the Internet is less reliable than a 911 call made over the PSTN. There is the obvious reason that the DSL/Cable modem, the eventual NAT and the VoIP device or computer need to be powered, at the difference of a POTS. This can be solved by connecting all this network elements on a UPS but even in this case there is still the problem that the Internet is designed as an unreliable network. There is nothing that can be done about this – this is engraved in the Internet DNA. Your ISP will probably try in the future to sell you a premium connection that “guarantee” you a reliable connection but the bottom line is that a reliable Internet connection is an oxymoron. This is addition to the fact that the acronym ISP should really stand for Intermittent Service Provider, not Internet Service Provider (yes Comcast, I am talking about you – one hour of downtime per month for maintenance between 2am and 3am is not OK).

If the availability of your Internet connection is 99.9% (one hour of downtime per month), then having a second Internet connection with the same availability will boost your global Internet availability to 99.9999 (2 seconds of downtime per month). Obviously you need to choose Internet connections that use very different mediums – if the two connections share the same wire or router somewhere between your home and the PSAP then the availability will not be as good, so one DSL connection and one Cable connection should be a good, albeit expensive choices. Perhaps there is a business model for a company that buys wholesale connections to ISP and resales bundle like this to end users.

Shared connection for TURN TCP

I presented this morning in the BEHAVE session my proposal to multiplex the data between a TURN client and a TURN server for TCP allocation:

http://ietf.implementers.org/IETF20090325.pdf

The proposal was rejected but none of the arguments presented convinced me that there will be no problems in the future with the multiple connections proposal. But without some hard numbers I do not think that this can be reversed and I do not feel like implementing TURN TCP right now, so until then the proposal goes to the Pending Told Ya stack.