Tag: Java

Jarc: Now running Java 1.7

At last, installable packages for OpenJDK 7 are available in the experimental Debian repository. So I released a new version of jarc that is now directly supporting Java 7 code but, because the openjdk-7 packages are in an experimental repository, I uploaded jarc 0.2.21 in an experimental repository so people would not be hit with dependency issues. I updated my Ubuntu/Debian repository page to explain how to add my experimental repository to the repository configuration file.

The reason to release this experimental version of jarc is that I need two the new features of the new JVM for my RELOAD implementation. The first one is the support of TLS 1.2, aka RFC 5246, which is used by RELOAD to authenticate and encrypt all the TCP connections. The only thing missing to completely implement RELOAD will be an implementation of DTLS 1.0 (which is to datagrams what TLS is to streams). I guess I will have to bite the bullet and write this one myself.

The second feature that I needed is an implementation of the SCTP protocol. TCP is great for transferring files but it has one major flaw when used with multiplexed client/server transactions, which is called the Head-Of-Line problem – a message lost will block all the subsequent messages, even if they belong to a different transaction, until the lost message is retransmitted and received. UDP does not have this issue but comes with its own set of problems. SCTP is somewhere between UDP and TCP, and this is why, in my opinion, it is a good transport protocol for something like RELOAD (well, between RELOAD nodes with a public IP address, as most NATs – one exception been the one I supervised the development at 8×8 – do not NAT SCTP).

So the plan will be to add SCTP to RELOAD, using either TLS (RFC 3436) or DTLS (RFC 6083) depending on the DTLS implementation described above, and to write an I-D, perhaps in time for IETF 81 in Quebec City.

Jarc: Junit errors as compilation errors

The jarc tool supports running Junit tests as part of a jar build since version 0.2.5. Since then when a unit test fails during a jar build the output of the jarc tool looked like this:

$ jarc reload
testInit(org.implementers.nio.channels.reload.ReloadNodeTest)
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.fail(Assert.java:98)
at org.implementers.nio.channels.reload.ReloadNodeTest.testInit(ReloadNodeTest.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)

The problem with this output is that it is not well integrated with other tools. Gvim for example cannot parse this output to display the current junit test file and line and worst, the output is lost so I had to remember the line number to be able to find where the test failed.
One way to fix this could have been to change the error format string in Gvim to parse the output, but that would not helped for other tools. So instead of this, jarc v0.2.20 is now formatting Junit errors with the same format than Java compilation errors, so the error above looks like this now:

$ jarc reload
../src/share/classes/org/implementers/nio/channels/reload/ReloadNodeTest.java:150: java.lang.AssertionError:

Now Gvim and other tools can parse the output and display directly the line that failed.

Jarc: Support for servlets

I have been busy during all 2010 with Stonyfish (a start-up I founded with a former colleague from 8×8), but I did not stop improving the jarc tool. The most important new feature is the ability of generating war files (packaged servlets):

A war file can be created by adding an X-Jarc-Web-App attribute in the manifest header that point to the servlet descriptor file. e.g.:

Manifest-Version: 1.0
X-Jarc-Web-App: org/implementers/servlet/http/web.xml

The syntax of the web.xml file has also been extended to permit to list file and jar files that need to be copied into the WEB-INF/classes directory:

<web-app>
  <lib>file1.jar</lib>
  <lib copy="true">file2.jar</lib>
  <data>lang/project.properties</data>
</web-app>

The latest version (0.2.19) also run the junit tests with the assertions enabled and fixes a lot of bugs found during 2010.

See the previous blog entries on this subject to find how to install the jarc tool. as always the source code is available under an Affero GPL 3 license.

rfc2629-tools 0.1.4

I just released a new version of my RFC 2629 processing tools. The txt2kindle and xml2mobi commands are unchanged, but the rfc2629 command was rewritten from scratch. I needed to add some extensions in the XSLT processing that was not supported by xsltproc, so I rewrote it in Java. Because of this I was able to add a nice feature for including the references in a RFC 2629 document.

In the previous version of the tool, a reference can be added by using something like this:

<references xmsns:xi="http://www.w3.org/2001/XInclude" title="Normative References">
  <xi:include href="http://xml.resource.org/public/rfc/bibxml/reference.RFC.5054.xml" />
</references>

This is still supported, but now an IETF URN (RFC 2648) can be used instead:

<references xmsns:xi="http://www.w3.org/2001/XInclude" title="Normative References">
  <xi:include href="urn:ietf:rfc:2543" />
</references>

In addition to be less verbose, this notation has the advantage that I-D references can refer to a current version of an I-D instead of to the version that is publicly available. That was already in the previous version, but was cumbersome to use. With the current version, if you pass more than one RFC 2629 file to the rfc2629 command, it will use the version and data of the generated file in the cross references, instead of the public one.

In the future I will extend this feature to add a local cache. One of the most annoying problem with references (either when using Xinclude or <?rfc include ?>) is that you need to be connected to the Internet. It will be possible to store the references on the local disk and use them if there is no Internet connection available (or always use the cache for RFC references as they never change).

Note that only the rfc: and id: NSS are currently implemented.

Jarc: Junit integration

Unit testing is, in my opinion, one of the most important tool that a developer wants to use to write software. I am not fond of all the techniques proposed by the eXtreme Programming (to say the least) but having popularized unit testing is one of the things that the Agile methodologies should be credited for. Instead of having various tests scattered on multiple systems and prone to bit rotting, Junit permits to package all this tests together and to run them each time a product is built. I generally integrate the unit tests in the build, so a final package cannot be built without passing all the tests. The proprietary version of jarc had a switch to build and run the unit tests in one command line.

The version 0.2.5 of jarc continues in this spirit, but instead of having to build first a jar file and then compile and run the unit tests, it automatically build and run unit tests if they are available. Here’s an example of manifest file with unit tests:

Manifest-Version: 1.0
Main-Class: org.implementers.apps.turnuri.Main
X-Jarc-Target: 1.6

Name: org/implementers/apps/turnuri/Main.class

Name: org/implementers/apps/turnuri/TurnUriTest.class
X-Jarc-Test: true

With this manifest file, jarc will build a jar file with Main as root of the dependencies, then build a second, internal, jar file with TurnUriTest as root of the dependencies and use it as Junit test. The X-Jarc-Test attribute can also take the value “false” to deactivate a test temporarily.

Note that the jar file is written on the disk even if the unit tests fail.

RFC2629 tools

Now that I have a java build that is redistributable, I can finally release the code source of the two tools I wrote to convert I-Ds and RFCs to a format that can be used on a Kindle. The Debian package containing this two tools is named “rfc2629-tools” and can be found at the usual place. In addition to this two tools, I also put in this package an additional tool that I use to process my RFC2629 XML sources.

Some quick explanation for the people new to the IETF process. All IETF documents, standards and others, are published as RFC (which is no longer an acronym) which are immutable documents. Before been published as RFC, documents are published as Internet-Drafts (I-D). The canonical form of RFCs and I-Ds is a line printer-ready format that is often called text format. There is other formats possible like PDF or HTML, but only the text format must be used as reference. Writing a well-formed text I-D or RFC is difficult, so authors generally write it in another format, then use a conversion tool to generate the text, PDF or HTML file. Some people use Winword, other use nroff but the most interesting format is an XML application defined in RFC 2629. There is available tools to convert an RFC 2629 source to text, PDF, nroff and HTML, but nothing for the Kindle platform.

One of the tools in the rfc2629-tools package takes as input a RFC 2629 source and generate an intermediary file that can then be processed to create a .mobi file that can be loaded in a Kindle. Here’s the command lines to do this:

$ xml2mobi draft-ietf-behave-turn-uri-09.xml
$ kindlegen draft-ietf-behave-turn-uri-09.htm

Note that the xml2mobi program is very much a work in progress, as it is incomplete and really need more work to be able to process with any RFC2629 source. I released the code source because I do not have much time currently for this, so anybody interested can work on a fork. For example, the file generated is named from the content of the docName attribute in the rfc element in the input file.
The kindlegen program can be found on Amazon Web site.

The IETF Secretariat archives the RFC 2629 source if the author provides it when uploading the text file but few people do this so another solution is needed to be able to read I-Ds and RFCs that are only available in text form. This is what the text2kindle program do:

$ txt2kindle draft-ietf-behave-turn-uri-09.txt

This command creates a file named draft-ietf-behave-turn-uri-09.zip that can be installed directly in a directory named /pictures on the Kindle 2. On the Kindle 1, the file must be unzipped in the /pictures directory.
Note that the document is displayed as a series of pictures so you cannot annotate them.

Even if the RFC2629 source of an I-D is available on the IETF web site, that does not mean that you will be able to generate a mobi file for your Kindle that is identical in content to the canonical version. The first reason is that the full date is generally not set, which means that running a conversion tool on this source will generate a document with the current date instead of the date that is used in the canonical document. The second reason is that people generally uses the <?rfc include> statement to insert references. The problem with this is that the text included will change when a new version of the I-D referenced will released, so again it is not possible to generate a content identical to the canonical document. The solution to this problem is to generate an RFC2629 document that always contains a complete date and where all references are already included. With this what the third tool in the package provides:

$ rfc2629 draft-ietf-behave-turn-uri.xml

This command will copy the content of the file passed as input with the following transformations:

  • The destination file is named from the content of the docuName attribute (this helps when the document is stored in a source control system).

  • The date in the destination file is filled with the current date.

  • All the xinclude statements are replaced by their content

  • All comments are removed.

The resulting XML file can be uploaded to the IETF site with a guarantee that other formats can be generated from this file as identical as the files generated by the author.

The same script also implements a feature that permit to rename a reference. A reference to an I-D generally looks like this is the result text:

The TURN specification [I-D.ietf-behave-turn] defines a process for a

The map element can be used to change the name displayed:

<map anchor="TURN">
  <xi:include xi:href="http://xml.resource.org/public/rfc/bibxml3/reference.I-D.ietf-behave-turn.xml" />
</map>

Which, after been processed by the script and xml2rfc will generate this instead:

The TURN specification [TURN] defines a process for a TURN client to

The last feature of this script (for now) was added when I tried to release two I-Ds at the same time, with each I-D containing a reference to the other. Because the text that will be included is generated after the upload to the IETF, there was no way to automatically use the correct reference. What the script do is to generate a reference file automatically in /tmp for all the files processed. It is then easy to include the correctly generated reference is the document. For example draft-ietf-behave-turn-uri.xml and draft-petithuguenin-behave-turn-uri-bis.xml have a reference on each other. So the first step is to change the reference inside draft-ietf-behave-turn-uri.xml like this:

<xi:include xi:href="/tmp/reference.I-D.petithuguenin-behave-turn-uri-bis.xml" />

and to change the reference inside draft-petithuguenin-behave-turn-uri-bis.xml like this:

<xi:include xi:href="/tmp/reference.I-D.ietf-behave-turn-uri.xml" />

Then running the script on the two XML files at the same time generates the correct XML file, ready to be uploaded to the IETF:

$ rfc2629 ../src/share/docs/draft-ietf-behave-turn-uri.xml ../src/share/docs/draft-petithuguenin-behave-turn-uri-bis.xml

Jarc: Cross-compilation

The new version of jarc now supports cross-compilation, i.e. the possibility to build jar files that works on a version of the JRE lower than the the version of the JRE used to run the jarc program. The jarc program itself will always run under the most recent JRE and by default will build a jar file targeted to the same version. But adding an X-Jarc-Target option will request jarc to build for a lower version. For example this manifest file means that the resulting jar file will work with a 1.5 JRE:

Manifest-Version: 1.0
X-Jarc-Target: 1.5

Without this line, the jar file would have been created for the 1.6 JRE so executing it on a 1.5 JRE would fail.

It is also possible to request a specific JRE vendor:

Manifest-Version: 1.0
X-Jarc-Target: 1.6_openjdk

Note that there is no need to install the JDK for all versions of the JRE. Only the JDK for the latest version (i.e. the one that is used to run jarc) is needed. But at least one installed JRE is needed for each potential X-Jarc-Target value, because the rt.jar file is needed for the cross-compilation.

It is even possible to configure jarc to use the JDK 1.7 by modifying the configuration in /etc/jarc.conf (see comments at the end of the file). The jarc package itself is built by jarc running on 1.7 but targeted to 1.6.

The current version of jarc (0.2.2) can build for the JDKs 1.6 and 1.5 and supports sun and openjdk JVMs. A future version will also support the JVMs generated by java-package and perhaps the gcj JVM.

Jarc: a jar file compiler

Using the “make” tool to build a jar file was never a good option, mostly because Java files are different from C/C++ files. Today most people use Ant but I never liked it so around 2002 I started writing a tool that would permit to build a jar file in one step. I had two requirements in mind: First the tool had to be faster than running javac and jar – at the time the JVM was really slow, so I used zip and jikes, at least until IBM stopped updating jikes.

The second requirement was to compile only the Java files that were necessary to build the jar file, the idea been to give to the jar tool only the name of the “root” Java classes, and let it figure out the dependencies (I am not an Ant specialist but it seems to me that Ant is still not capable of doing this correctly). What was missing was a convenient way of passing the list of “root” classes to the tools, and this is when I thought of using the manifest file for this.

A manifest file is a text file stored inside the META-INF directory of the jar file. This file contains the meta-information about the jar file and is generated by the jar tool. The idea was to invert the process – instead of having the jar tool generating this manifest file from the command line parameters, why not use this file as the source of the information needed to build a jar file?

The jarc tool (jar compiler) takes as input one or more manifest files, parses the content and tries to build a jar file of same name. The tool will use the class names found in the manifest file as the “root” classes to use as dependencies. A manifest file looks like this:

Manifest-Version: 1.0
Class-Path: /usr/share/java/jspeex.jar
Main-Class: org.implementers.apps.Main

Name: org/implementers/apps/Main.class

Name: org/implementers/apps/resources/schema.xsd

If this text is stored in a file named apps.mf, then running “jarc apps” will build a jar file named apps.jar by executing the following steps:

  • The org.implementers.apps.Main class will be compiled together with all the classes that depends on it. The content of the Class-Path attribute is passed to the compiler as if a -classpath option was used.

  • The org/implementers/apps/resources/schema.xsd file will be copied from the disk to the jar file.

  • The manifest file itself will be filtered and copied inside the jar file.
  • A short bash program will be generated to call the class described in the Main-Class attribute

There is also some attributes that can be added in the manifest to drive the build. One example is the X-Jarc-Service that can be added to an entry to declare it as a service as shown in the following example:

Name: org/implementers/nio/channels/spi/CompositeProvider.class
X-Jarc-Service: java.nio.channels.spi.SelectorProvider

When jarc reads this line, it generates in the jar file a service file in META-INF/services.

The tools is accessible on the following Debian repository (see a previous post on this blog for the explanation on how to configure Ubuntu or Debian):

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/

The tools is contained in the “jarc” package and is distributed under the GPL v3 license.

I wrote the previous version of jarc when I was working for my former employer, so to be sure to not infringe on their rights, I rewrote the tool from scratch in Java (the original version was a bash script). Actually the new version implements only the minimum necessary to build itself, so expect updates in the future.

Update 01/31/2010:
jarc was put in the unstable repository by mistake. The new versions (starting with 0.2.3) will be in testing. The best is to have both testing and stable in your /etc/apt/sources.list file and remove unstable completely:

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/
deb http://debian.implementers.org/ stable/$(ARCH)/
deb http://debian.implementers.org/ stable/all/
deb-src http://debian.implementers.org/ stable/source/

Empty synchronized statement

The selectors API in Java is a paradigm that I spent much time understanding during this last months. I developed a small but, I think, powerful library related to this topic, and this post will be followed by several other posts that will introduce this library. But before starting on the gory details I wanted to talk a bit about a pattern that I am using with selectors.

Selectors are best used in a more or less infinite loop, something like this:

Selector selector = Selector.open();
while (true) {
  if (selector.select() != 0) {
    for (Iterator<SelectionKey> iterator = selector.selectedKeys();
iterator.hasNext(); ) {
      SelectionKey key = iterator.next();
      // Do something with the key
      iterator.remove();
      }
    }
  }

The problem is that the selector is synchronized on itself and on the keys set when blocked in a select. This means that registering a new channel will simply be stuck waiting on a monitor until the select exits. If the selector is used with other channels, and there is some activity on this channels then the register will execute quickly and this bug will not be seen. But if this is the first channel to be registered then there will be a deadlock. One step in the right direction is to force the selector to wakeup just before registering:

selector.wakeup();
channel.register(channel, OP_READ);

Unfortunately, this still does not work. The reason is that nothing prevents the loop to execute the select a second time after the wakeup and before the call to register. It is easy to write a test program to verify this. The solution is to use a synchronization to prevent the loop to reenter the select before the register is executed. The register is inside the synchronized:

Object lock = new Object();
synchronized (lock) {
  selector.wakeup();
  channel.register(channel, OP_READ);
}

and the select loop now needs an empty synchronized:

Selector selector = Selector.open();
while (true) {
  if (selector.select() != 0) {
    for (Iterator<SelectionKey> iterator = selector.selectedKeys();
iterator.hasNext(); ) {
      SelectionKey key = iterator.next();
      // Do something with the key
      iterator.remove();
      }
    }
  synchronized (lock) {}
  }

There is other cases that would need to use the same pattern, like reading the selectedKeys set or the keys set in a different thread, as both need to be synchronized but the select method already synchronizes on them.

On the other hand, the selector close method does not need to use this pattern.

The interest set of selection keys is another interesting case. In the current implementation, there is no need to use the synchronization pattern shown above, so calling wakeup after changing the interest set is working fine. But the javadoc is clear that an implementation could behave differently:

“In a naive implementation, reading or writing the interest set may block indefinitely if a selection operation is already in progress;”

So always using the pattern above guarantees that the code will work on different Java implementations.

Now the problem could be that in the future a very optimized compiler or runtime decides that the empty synchronized statement is in fact useless, and decide to remove it.