Jarc: Generated service

It is not permitted for an annotation processor to modify a Java source file, so a processor willing to add code to an existing class is left with only two solutions (if we exclude method instrumentation): Generating a superclass or generating a subclass.

Generating a superclass has the advantage that the constructors of the annotated class can be used directly. Let say that we have a annotation processor that is designed to help implement class composition, as described in Effective Java, item #16. Instead of writing the whole ForwardingSet class, an annotation processor could generate it automatically from this code fragment:

@Forwarding(ForwardingSet.class)
public class InstrumentedSet<E> extends ForwardingSet<E> {
  InstrumentedSet(Set<E> s) {
    super(new HashSet<>());
  }

But generating a superclass is not always possible. For example let’s imagine an annotation processor that generates the JMX boilerplate necessary to export attributes. An existing class with such annotation could look something like this:

public class MyData {
  @Attribute int counter;
  }

In this case the processor for the @Attribute annotation will generate a JMX interface (Let’s call it MyDataMXBean) that declares the getcounter and setcounter methods, and a class extending MyData and implementing the JMX interface (let’s call it MyDataImpl).

The code generated would take care of the boring stuff, like synchronization and so on, which is certainly an improvement over writing and maintaining it. But the problem with subclasses is that we do not know the name of the class that was generated. Note that we have no other choice than to know the name of the superclass because we have to inherit from it. For subclasses it is better to let the processor choose the name, but now we need a way to be able to instantiate the generated class without knowing this name (in our example to register it in the MBean server).

The obvious way of doing this is to use a ServiceLoader. We can add a factory method in the MyData class to instantiate the generated class, something like this:

static MyData newInstance() {
  return ServiceLoader.load(MyData.class).next();
  }

But for this technique to work, we need to describe the service in the jar file. Using the method explained in a previous post does not help in this case, because we still do not know what will be the name of the generated class.

The version 0.2.30 of jarc provides a solution to this problem. This new version contains a new annotation, @Service, that can be used to annotate a generated class. A processor integrated in jarc will read this annotation at compile time, and automatically generates the service entry in the built jar file, as if an X-Jarc-Service attribute has been added to the manifest file. This works because this processor will be invoked after the @Attribute processor, and so knows the name of the class that has been generated. Here is for example the code fragment that the code generator would have generated for MyDataImpl:

@Service(MyData.class)
@MXBean
class MyDataImpl extends MyData implements MyDataMXBean {

Note that classes used as services require an empty constructor and that can be a problem if the class it extend does not have an empty constructor itself. The solution in this case is to define an additional factory class as the service.

First we define our factory as an abstract class:

abstract class Factory {
  MyData newInstance(int init);
  }

We adjust our factory method accordingly:

static MyData newInstance(int init) {
  return ServiceLoader.load(Factory.class).next().newInstance(init);
  }

The @Attribute processor must generate an additional class that extends the factory class and this is the class which is declared as a service:

@MXBean
class MyDataImpl extends MyData implements MyDataMXBean {
  @Service(Factory.class)
  static class FactoryImpl extends Factory {
  MyDataImpl newInstance(int init) {
    return new MyDataImpl(init);
    }
  }

  MyDataImpl(init init) {
    super(init);
  }

Jarc: Annotation processors

Version 0.2.27 of jarc now supports to run annotations processors when a jar file is built. The syntax is simple, just add a X-Jarc-Processors attribute in the manifest file header with a list of jar files, and jarc will automatically run the processors found by querying the META-INF/services/javax.annotation.processing.Processor file inside the jar files:

Manifest-Version: 1.0
Main-Class: org.implementers.apps.turnuri.Main
X-Jarc-Target: 1.7
X-Jarc-Processors: /usr/share/lib/processor.jar

Name: org/implementers/apps/turnuri/Main.class

In turn, it is easy to build an annotation processor with jarc, here’s the manifest file for a processor I am currently developing:

Manifest-Version: 1.0
Class-Path: /usr/share/java/jcip.jar
X-Jarc-Target: 1.6

Name: org/implementers/management/annotations/processing/Main.class
X-Jarc-Service: javax.annotation.processing.Processor

Note that the Java compiler does not run processors on dependent files, so you need to add a “Name:” attribute for all Java files that need to be processed.

Also starting with this version, the JDK 1.7 is required to run jarc – but jarc can still cross-compile for all the versions of Java starting with 1.5.

Jarc: Now running Java 1.8

I like to learn new Java features before they are officially released, and that requires using unstable builds. The difficulty is to integrate the new compiler into a build – for the JDK 1.7 I released jarc as an experimental package, but that was not a very good solution.

Since version 0.2.26, jarc can use an experimental compiler, like the one supporting lambda. If you installed the new JDK at /home/petithug/jdk8, you will only need to add the following lines to the /etc/jarc.conf file to be able to build jar files that use closures:

jdk-java_8-openjdk=/home/petithug/jdk8/bin/java
jdk-tools_8-openjdk=/home/petithug/jdk8/lib/tools.jar
canonical_8=1.8-openjdk
canonical_1.8=1.8-openjdk
canonical_8-openjdk=1.8-openjdk
canonical_1.8-openjdk=1.8-openjdk
jre-check_1.8-openjdk=/home/petithug/jdk8/jre/bin/java
jre-bootclasspath_1.8-openjdk=/home/petithug/jdk8/jre/lib/rt.jar:/home/petithug/jdk8/jre/lib/jce.jar
jre-source_1.8-openjdk=1.8
jre-exec_1.8-openjdk=/home/petithug/jdk8/jre/bin/java

Jarc always use by default the most recent compiler, but you can override this with the -Jjdk=7 or -Jjdk=6 option.

The new version of jarc also support passing parameters to the JVM – either at build time or at run time – by using the -J option.

Finally it is now possible to add an X-Jarc-Debug parameter at the manifest level. This option works just like the -g option in javac. I added this option to be able to build programs for aparapi – more about this in a future post.

Jarc: Javadoc generation

During the preparation of the libreload-java package, I discovered that the javadoc tool – the JDK tool used to generate an HTML documentation of a Java API – does not generate the javadoc based on Java dependency. It can either generate the javadoc for whole packages, or for a subset of Java files (e.g. File*.java). But we cannot ask it to generate the documentation for one specific class and all the classes depending on this class.

I guess that my way of doing things is different: I do not have one source tree per project, but one source tree that contains the code for all of them (and much, much more code that is not released). A particular Debian package contains only a subset of the code available in my source tree – the code needed for this particular package. One way to see this is that a package is a view on my source repository. Ant or javadoc cannot easily extract only the sources files needed for a particular view (unless I list all the files individually, which would be unmaintainable), so the jarc tool relies on Java dependency.

When I tried to build the javadoc for the libreload-java package, I noticed that the test units files (used by Junit) were part of the javadoc, although they clearly are not part of the API. This is because the javadoc tool had to process the whole directory. I cannot move these two files in another directory, because they need to have access to constructors and methods that are not public.

So what I did is that I integrated the javadoc tool inside jarc. Because jarc knows the list of dependent Java files during a jar file build, I just have to feed it to the javadoc tool to have the exact API documentation corresponding to the jar file built. The new version of jarc (0.2.22) supports a new parameter (-javadoc ) that can is used to request the generation of the javadoc at the same time the package is build and unit tested.

One can notice that the Debian source packages that are distributed do not contain the whole tree, but only the subset of the files needed to rebuild this package. To do this (I obviously did not want to release my whole source tree), I had to rely on a trick. My hard disk is mounted with the strictatime options, which means that each time a file is accessed, the data/time of this access is kept. When I build a Debian package I start to build it from my whole source repository:

$ touch run-stamp
$ SRC=../src dpkg-buildpackage -D -b -us -uc

Then I run this command to copy only the files that where used for this build into a new directory:

$ find ../src -anewer run-stamp -type f -exec cp --parents -t src {} ;

I can then build again the packages, but this time with a source package that contains only these files (and that can be used to eventually rebuild the binary packages):

$ SRC=src dpkg-buildpackage -I

Note the difference in the $SRC variable content: ../src is my complete source repository and src is just the subset for this package.

Jarc: Now running Java 1.7

At last, installable packages for OpenJDK 7 are available in the experimental Debian repository. So I released a new version of jarc that is now directly supporting Java 7 code but, because the openjdk-7 packages are in an experimental repository, I uploaded jarc 0.2.21 in an experimental repository so people would not be hit with dependency issues. I updated my Ubuntu/Debian repository page to explain how to add my experimental repository to the repository configuration file.

The reason to release this experimental version of jarc is that I need two the new features of the new JVM for my RELOAD implementation. The first one is the support of TLS 1.2, aka RFC 5246, which is used by RELOAD to authenticate and encrypt all the TCP connections. The only thing missing to completely implement RELOAD will be an implementation of DTLS 1.0 (which is to datagrams what TLS is to streams). I guess I will have to bite the bullet and write this one myself.

The second feature that I needed is an implementation of the SCTP protocol. TCP is great for transferring files but it has one major flaw when used with multiplexed client/server transactions, which is called the Head-Of-Line problem – a message lost will block all the subsequent messages, even if they belong to a different transaction, until the lost message is retransmitted and received. UDP does not have this issue but comes with its own set of problems. SCTP is somewhere between UDP and TCP, and this is why, in my opinion, it is a good transport protocol for something like RELOAD (well, between RELOAD nodes with a public IP address, as most NATs – one exception been the one I supervised the development at 8×8 – do not NAT SCTP).

So the plan will be to add SCTP to RELOAD, using either TLS (RFC 3436) or DTLS (RFC 6083) depending on the DTLS implementation described above, and to write an I-D, perhaps in time for IETF 81 in Quebec City.

Jarc: Junit errors as compilation errors

The jarc tool supports running Junit tests as part of a jar build since version 0.2.5. Since then when a unit test fails during a jar build the output of the jarc tool looked like this:

$ jarc reload
testInit(org.implementers.nio.channels.reload.ReloadNodeTest)
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.fail(Assert.java:98)
at org.implementers.nio.channels.reload.ReloadNodeTest.testInit(ReloadNodeTest.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)

The problem with this output is that it is not well integrated with other tools. Gvim for example cannot parse this output to display the current junit test file and line and worst, the output is lost so I had to remember the line number to be able to find where the test failed.
One way to fix this could have been to change the error format string in Gvim to parse the output, but that would not helped for other tools. So instead of this, jarc v0.2.20 is now formatting Junit errors with the same format than Java compilation errors, so the error above looks like this now:

$ jarc reload
../src/share/classes/org/implementers/nio/channels/reload/ReloadNodeTest.java:150: java.lang.AssertionError:

Now Gvim and other tools can parse the output and display directly the line that failed.

Jarc: Support for servlets

I have been busy during all 2010 with Stonyfish (a start-up I founded with a former colleague from 8×8), but I did not stop improving the jarc tool. The most important new feature is the ability of generating war files (packaged servlets):

A war file can be created by adding an X-Jarc-Web-App attribute in the manifest header that point to the servlet descriptor file. e.g.:

Manifest-Version: 1.0
X-Jarc-Web-App: org/implementers/servlet/http/web.xml

The syntax of the web.xml file has also been extended to permit to list file and jar files that need to be copied into the WEB-INF/classes directory:

<web-app>
  <lib>file1.jar</lib>
  <lib copy="true">file2.jar</lib>
  <data>lang/project.properties</data>
</web-app>

The latest version (0.2.19) also run the junit tests with the assertions enabled and fixes a lot of bugs found during 2010.

See the previous blog entries on this subject to find how to install the jarc tool. as always the source code is available under an Affero GPL 3 license.

Jarc: Junit integration

Unit testing is, in my opinion, one of the most important tool that a developer wants to use to write software. I am not fond of all the techniques proposed by the eXtreme Programming (to say the least) but having popularized unit testing is one of the things that the Agile methodologies should be credited for. Instead of having various tests scattered on multiple systems and prone to bit rotting, Junit permits to package all this tests together and to run them each time a product is built. I generally integrate the unit tests in the build, so a final package cannot be built without passing all the tests. The proprietary version of jarc had a switch to build and run the unit tests in one command line.

The version 0.2.5 of jarc continues in this spirit, but instead of having to build first a jar file and then compile and run the unit tests, it automatically build and run unit tests if they are available. Here’s an example of manifest file with unit tests:

Manifest-Version: 1.0
Main-Class: org.implementers.apps.turnuri.Main
X-Jarc-Target: 1.6

Name: org/implementers/apps/turnuri/Main.class

Name: org/implementers/apps/turnuri/TurnUriTest.class
X-Jarc-Test: true

With this manifest file, jarc will build a jar file with Main as root of the dependencies, then build a second, internal, jar file with TurnUriTest as root of the dependencies and use it as Junit test. The X-Jarc-Test attribute can also take the value “false” to deactivate a test temporarily.

Note that the jar file is written on the disk even if the unit tests fail.

Jarc: Cross-compilation

The new version of jarc now supports cross-compilation, i.e. the possibility to build jar files that works on a version of the JRE lower than the the version of the JRE used to run the jarc program. The jarc program itself will always run under the most recent JRE and by default will build a jar file targeted to the same version. But adding an X-Jarc-Target option will request jarc to build for a lower version. For example this manifest file means that the resulting jar file will work with a 1.5 JRE:

Manifest-Version: 1.0
X-Jarc-Target: 1.5

Without this line, the jar file would have been created for the 1.6 JRE so executing it on a 1.5 JRE would fail.

It is also possible to request a specific JRE vendor:

Manifest-Version: 1.0
X-Jarc-Target: 1.6_openjdk

Note that there is no need to install the JDK for all versions of the JRE. Only the JDK for the latest version (i.e. the one that is used to run jarc) is needed. But at least one installed JRE is needed for each potential X-Jarc-Target value, because the rt.jar file is needed for the cross-compilation.

It is even possible to configure jarc to use the JDK 1.7 by modifying the configuration in /etc/jarc.conf (see comments at the end of the file). The jarc package itself is built by jarc running on 1.7 but targeted to 1.6.

The current version of jarc (0.2.2) can build for the JDKs 1.6 and 1.5 and supports sun and openjdk JVMs. A future version will also support the JVMs generated by java-package and perhaps the gcj JVM.

Jarc: a jar file compiler

Using the “make” tool to build a jar file was never a good option, mostly because Java files are different from C/C++ files. Today most people use Ant but I never liked it so around 2002 I started writing a tool that would permit to build a jar file in one step. I had two requirements in mind: First the tool had to be faster than running javac and jar – at the time the JVM was really slow, so I used zip and jikes, at least until IBM stopped updating jikes.

The second requirement was to compile only the Java files that were necessary to build the jar file, the idea been to give to the jar tool only the name of the “root” Java classes, and let it figure out the dependencies (I am not an Ant specialist but it seems to me that Ant is still not capable of doing this correctly). What was missing was a convenient way of passing the list of “root” classes to the tools, and this is when I thought of using the manifest file for this.

A manifest file is a text file stored inside the META-INF directory of the jar file. This file contains the meta-information about the jar file and is generated by the jar tool. The idea was to invert the process – instead of having the jar tool generating this manifest file from the command line parameters, why not use this file as the source of the information needed to build a jar file?

The jarc tool (jar compiler) takes as input one or more manifest files, parses the content and tries to build a jar file of same name. The tool will use the class names found in the manifest file as the “root” classes to use as dependencies. A manifest file looks like this:

Manifest-Version: 1.0
Class-Path: /usr/share/java/jspeex.jar
Main-Class: org.implementers.apps.Main

Name: org/implementers/apps/Main.class

Name: org/implementers/apps/resources/schema.xsd

If this text is stored in a file named apps.mf, then running “jarc apps” will build a jar file named apps.jar by executing the following steps:

  • The org.implementers.apps.Main class will be compiled together with all the classes that depends on it. The content of the Class-Path attribute is passed to the compiler as if a -classpath option was used.
  • The org/implementers/apps/resources/schema.xsd file will be copied from the disk to the jar file.
  • The manifest file itself will be filtered and copied inside the jar file.
  • A short bash program will be generated to call the class described in the Main-Class attribute

There is also some attributes that can be added in the manifest to drive the build. One example is the X-Jarc-Service that can be added to an entry to declare it as a service as shown in the following example:

Name: org/implementers/nio/channels/spi/CompositeProvider.class
X-Jarc-Service: java.nio.channels.spi.SelectorProvider

When jarc reads this line, it generates in the jar file a service file in META-INF/services.

The tools is accessible on the following Debian repository (see a previous post on this blog for the explanation on how to configure Ubuntu or Debian):

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/

The tools is contained in the “jarc” package and is distributed under the GPL v3 license.

I wrote the previous version of jarc when I was working for my former employer, so to be sure to not infringe on their rights, I rewrote the tool from scratch in Java (the original version was a bash script). Actually the new version implements only the minimum necessary to build itself, so expect updates in the future.

Update 01/31/2010:
jarc was put in the unstable repository by mistake. The new versions (starting with 0.2.3) will be in testing. The best is to have both testing and stable in your /etc/apt/sources.list file and remove unstable completely:

deb http://debian.implementers.org/ testing/$(ARCH)/
deb http://debian.implementers.org/ testing/all/
deb-src http://debian.implementers.org/ testing/source/
deb http://debian.implementers.org/ stable/$(ARCH)/
deb http://debian.implementers.org/ stable/all/
deb-src http://debian.implementers.org/ stable/source/