fitzsim's development log

My FOSDEM talk was on IcedTea and LiveConnect. I’ve been working on a gcjwebplugin rewrite called IcedTeaPlugin on-and-off for the past five months. Whereas gcjwebplugin is a Netscape 4.x plugin, IcedTeaPlugin is an XPCOM plugin that implements OJI — the Open JVM Interface — for LiveConnect support. I showed two demos at FOSDEM, one demo for each of the two test applets I’ve used for development. Here are the screenshots, taken just after I got each applet working:

JavaScript-to-Java direction

The text field and setText button are HTML form widgets. The “hello” string is passed from the text field to the Java applet on the right and displayed in bold. You can see the code in the page source of the original example.

Because of LiveConnect’s extensive use of Java reflection upon initialization, making any applet work, even the trivial “Hello, World” applet shown here, involves implementing most of OJI.

Java-to-JavaScript direction

The Java applet on this page calls into JavaScript to retrieve the page’s background color, then sets its own background color to match. Here’s the example source code.

I put my presentation slides online.

I’m pleased that I was able to retain gcjwebplugin’s out-of-process-JVM design in IcedTeaPlugin, despite the fact that OJI was obviously designed with libjvm.so embedding in mind. Basic applets work but there’s still lots to be done before IcedTeaPlugin can be declared production-ready. It is hosted in the IcedTea 7 Mercurial repository enabled by the –enable-liveconnect configure option.

I learned at FOSDEM that Sun is also working on a new plugin framework, in a private repository. LiveConnect support is important to Fedora plugin users, and that’s what motivates my work on IcedTeaPlugin. If Sun’s new plugin framework were released and could be deployed in Fedora faster than IcedTeaPlugin then I’d be willing to work on it instead.

On Monday, I found Conkeror through the EmacsWiki. It’s a Firefox extension that provides an Emacs-like interface to the browser. It includes all the essential Emacs features (minibuffer, incremental search, bookmark tab completion, buffer management) and an additional killer feature: numbered links. I’ve always wanted an efficient way to follow links and this is it. You can’t get much faster than <number>-<Return>!

I integrated gcjwebplugin into the IcedTea repository today. With Sun’s class library I’m able to install a proper applet security manager so that gcjwebplugin can run untrusted applets safely.

Here are some testing strategies that I’d like to see implemented for IcedTea. I’ve asked one of our interns to investigate these points.

Reduce Mauve-on-IcedTea failures to 0
Provide reproducible way of running Mauve on IcedTea which should result in 0 failures
“make dist” target for Mauve

Once these are complete we can a) set up nightly regression runs on http://icedtea.classpath.org, b) insist on having no regressions in package releases by running Mauve as part of the RPM build.

Make IcedTea demos build standalone

The IcedTea demos include libraries built during the IcedTea build, but these libraries cannot be built using the provided sample Makefiles. Installing the java-1.7.0-icedtea-demo RPM should a) install the demo source code in /usr/share/java-1.7.0-icedtea/demo, b) install the demo code’s build requirements as dependencies, c) provide working sample Makefiles. That way, trying the demos can be reduced to something like:

yum install java-1.7.0-icedtea-demo
cp -r /usr/share/java-1.7.0-icedtea/demo ~
cd ~/demo
make -f /Makefile.sample
make -f /Makefile.sample run

Ideally we’d test building and running the demos as part of the RPM build.

Run DaCapo benchmark reproducibly on IcedTea

Once DaCapo runs reliably we can set up nightly benchmark runs on http://icedtea.classpath.org.

Run graphical demos with java.awt.Robot and Xvfb

I’d like to include such graphical tests in the RPM build if the setup isn’t too fragile.

Longer term if Sun releases the jtreg harness (and some tests), we won’t hate it. More likely we’ll want to integrate it too into our RPM builds.

Here are some thoughts about how to make OpenJDK easily maintainable on Fedora. Achieving this goal will require changes to OpenJDK, updates to the Fedora implementation of the JPackage conventions and an rpm patch.

Here is a list of problems and a strawman solution for each. Each item needs careful investigation to determine the most appropriate solution.

OpenJDK Changes

Read OpenSSL certs directly

Now: We generate a cacerts file from certificates shipped with the Fedora OpenSSL package.

Future: OpenJDK should parse the OpenSSL certificates directly. That would give us one point of maintenance for the certificates and allow OpenJDK to take advantage of new certificates when OpenSSL is updated.

Replace font.properties with fontconfig

Now: fontconfig.Fedora.properties hard-codes Fedora fonts that OpenJDK uses.

Future: OpenJDK should use fontconfig to determine which fonts to use, to match desktop font preferences.

Read glibc timezone and locale data

Now: OpenJDK bundles its own set of timezone and locale data.

Future: OpenJDK should read glibc’s timezone and locale data. This provides a single point of maintenance for these data and eliminates the need to reinstall OpenJDK, or run a separate tool like tzupdater, whenever timezone or locale data updates are required.

Security policy and provider configuration under /etc/java

Now: OpenJDK security policies are installed and configured under /usr/lib/jvm/java-$version-$vendor/jre/lib/security.

Future: OpenJDK security policies and providers should be configured from /etc/java, since /etc is the standard location for administrator-edited configuration files. If configuration were under /etc/java/java-$version, the settings could be shared between same-versioned JDKs. There should also be a .d directory so that new security policies and providers can be easily installed by separate packages.

Endorsed directories: /usr/lib*/java-$version/endorsed, /usr/share/java-$version/endorsed

Now: OpenJDK loads endorsed jars from /usr/lib/jvm/java-$version-$vendor/jre/lib/ext.

Future: OpenJDK should load endorsed jars from locations like /usr/lib/java-$version/endorsed and /usr/share/java-$version/endorsed so that separate packages can easily install and uninstall additional endorsed jars.

Properties files in /etc/java

Now: OpenJDK loads .properties files from /usr/lib/jvm/java-$version-$vendor/jre/lib.

Future: OpenJDK should load .properties files from /etc/java, since /etc is the standard location for config files.

Split out bundled code, push Sun patches upstream

Now: OpenJDK bundles source code packages from many other projects.

Future: Using the new module system or some other mechanism, OpenJDK should use the system-installed versions of these packages. Any modifications to these projects that Sun has been maintaining in their tree should go upstream.

Default JNI load path, to support loading 32-bit and 64-bit JNI libraries from the same jar

Now: System.loadLibrary relies on the default library paths.

Future: System.loadLibrary should also look in standard paths where packages can install JNI libraries: /usr/lib/jni on 32-bit JVMs and /usr/lib64/jni on 64-bit JVMs.

Default /usr/share/java “CLASSPATH” directory

Now: JPackage provides a build-classpath script to build CLASSPATH settings from jars found in system-default locations such as /usr/share/java.

Future: OpenJDK’s module system should resolve dependencies from these system-default locations, obviating the build-classpath script. Related: Red Hat bug 170128.

OpenJDK Package Changes

Multiarch support in jpackage-utils

Now: the jpackage-utils package is built as a noarch package but contains references to /usr/lib.

Future: jpackage-utils should also support JDK installations in /usr/lib64. Doing so will raise questions about how JDK alternatives are handled, since they are based in /usr/lib. It may be best to change the alternatives tool itself to special-case /usr/lib and /usr/lib64.

Simplify package split to base, javadoc

Now: JDK packages are split up into base (JRE), devel, demo, src and javadoc subpackages.

Future: We could reduce this to just a base “java” package, which includes the JRE, development tools, demos and src.zip. javadoc will still need to remain a separate subpackage because of its installed size.

Simplify alternatives to just “java”

Now: There are many JDK primary alternatives: java, javac, jre_$vendor, $jre_$version, java_sdk_$vendor, java_sdk_$version, libjavaplugin.so

Future: With the simplified package split we could reduce this list to two primary alternatives: java and javadoc. But simplification may not be required if everyone just uses system-switch-java instead of dealing with alternatives directly.

RPM Changes

Fix rpm $1 arch-specific bug

Now: Red Hat bug 127359 prevents correct upgrading of alternatives symlinks in multilib installations.

Future: The bug must be fixed.

I’m experimenting with GNOME’s accessibility support; specifically, how it interacts with Swing and AWT applications. I’m impressed with the framework’s implementation. I had trouble saving GOK‘s alternative pointer device setting, and there seems to be an issue with having the alternative pointer device control the core pointer, but otherwise the interface is stable and usable. As an experiment, I wrote this blog post using only a single-button mouse we had lying around the office The process was slow but smooth. My other experiments were also successful: GOK works with Swing and AWT applications over the Java Access Bridge.

Our team at Red Hat has been doing some planning now that OpenJDK has been released. First, a summary of OpenJDK’s status:

What Sun has released:

• most of the 1.6-level JDK, including:

• the Hotspot virtual machine with x86 and x86_64 JITs
• the Java compiler
• most of the class libraries

What Sun hasn’t released:

• 4% of the class library code — some of the class library code is “encumbered” meaning that Sun can’t release it under the GPL. The encumbrances are in the sound system, font rasterizer, graphics rasterizer, SNMP and crypto areas.
• plugin and javaws
• a 1.5 version

The second list points the way to some of our immediate goals. Here is a list of some of the goals we decided on:

1. Packaging OpenJDK as IcedTea

   We can’t call what we ship “OpenJDK” because that’s a Sun trademark. We’ve decided on the temporary name “IcedTea”, and we can say that it’s based on code from openjdk.java.net. We’ve posted an SRPM containing the free OpenJDK sources. But it still requires non-free stuff to build, which brings us to the next goal.

2. Building with free software

   Currently OpenJDK requires openmotif, Sun’s proprietary JDK 1.6 and a bundle of proprietary “plugs” (to replace the encumbrances) to build. Our group’s first goal is to make OpenJDK buildable against a set of fully-free packages.

3. Setting up icedtea.classpath.org

   With help from some GNU Classpath community members we’re setting up icedtea.classpath.org with a Wiki, mailing list and a Subversion repository, to host the work we’ll be doing on OpenJDK. This is a temporary site where we can experiment with the OpenJDK code while Sun works out its contribution processes. The site is not complete yet.

4. Testing without the encumbrances

   Because we haven’t built OpenJDK without the binary plugs for the encumbrances yet, we don’t know exactly how it fails if they’re not present. We need to see if the result can be proposed for inclusion in Fedora.

5. Replacing encumbered bits with parts from GNU Classpath

   Once we’ve tested running without the encumbrances we’ll know where to focus our replacement efforts.

6. Testing OpenJDK interpreter on other architectures

   Since at this point we only have x86 and x86_64 JITs, platform coverage will be a problem. In the short term we should test the OpenJDK interpreter portability since it may be possible to use it as a stop-gap on the architectures for which we don’t have JITs.

7. Working with Sun on contributor license agreements, use of trademarks, self-certification and governance.

   Eventually, we’d like to ship in Fedora a certified Java implementation that we can call OpenJDK and “Java compatible”. Sun intends to make that possible through a self-certification process. Eventually we should be able to contribute code directly to the OpenJDK project — Sun has worked out the governance model it will use and is implementing a Mercurial repository.

8. Working with other Red Hat groups on OpenJDK

   We expect to work closely with other Red Hat groups on both system integration (kernel, tools, desktop, packaging) and on requests from internal “consumers” of the JDK (JBoss, Eclipse).

GCJ

We will continue to support GCJ where it is deployed, but any new work we do will be on the OpenJDK codebase. GCJ still has better architecture coverage than OpenJDK, supporting all 7 RHEL architectures where OpenJDK supports 2, but we’ve decided that the benefits of OpenJDK outweigh this advantage of GCJ, and that effort is better spent trying to get (or write, if need be) more JITs for OpenJDK.

Timelines

We don’t have definite timelines yet. We’re in an investigation stage right now, so precise timelines will have to wait until we’ve had more time to discuss the OpenJDK codebase and Sun’s timelines.

I packaged the new code drop from the OpenJDK project using the JPackage conventions. Because naming rules are still being worked out for derivatives of OpenJDK code, I’ve decided to call the result “java-1.7.0-icedtea”. This is just a preliminary development package; I hope Sun will approve renaming it “java-1.7.0-openjdk” or “java-1.6.0-openjdk” when it officially lands in Fedora.

http://people.redhat.com/fitzsim/java-1.7.0-icedtea-1.7.0.0-0.1.nonfree.src.rpm

The OpenJDK build still depends on some non-free software so I’m providing an SRPM containing only the GPL’d code. People interested in building the complete not-fully-free OpenJDK can fill the gaps with binary plugs.

I’m going to investigate eliminating the non-free requirements. I haven’t studied the code enough to know if it’s worth producing a fully free build before starting encumbrance replacement efforts — it depends how much functionality is lost by excluding the binary blobs. Another interesting experiment would be bootstrapping OpenJDK with a free runtime.

Andrew Overholt recently pointed me to a blog entry on Java module systems. I can’t comment in detail on the JSRs or how they should interact because they’re all behind click-through licenses. My instinct tells me the ideal answers are: “Sun, drop JSR 277 and adopt OSGi; IBM, develop OSGi within the Java tree, on the same schedule as Java”, though this may be politically impossible.

Beyond politics, the post got me thinking about how a Java module system should integrate with Fedora. Here is a sketch of how I think it should work.

OSGi Modules in Fedora

OSGi modules distributed with Fedora will need to be wrapped by RPMs, so that they can be built, released and managed using existing tools and processes.

rpmbuild Integration

OSGi and RPM metadata provide similar information, so the adoption of OSGi will simplify the creation of Java RPMs. The packager should only be required to list the proper build dependencies in a spec file; Requires and Provides should be auto-calculated by rpmbuild. This is how C module (DSO) RPMs are currently produced.

The reverse operation should also be supported, for transforming non-modularized upstream projects into OSGi modules at rpmbuild time, using packager-provided metadata (the Name, Version, Requires and Provides fields). Then we could easily convert our currently-shipping Java packages to OSGi modules. At the same time, we should encourage upstream projects to properly modularize their jars, including any build dependency jars they bundle.

rpmbuild should automatically namespace packager-built modules with a Fedora release prefix, to distinguish them from the equivalent upstream module.

Module Loader Customizations

The Fedora OSGi loader should prioritize these Fedora-namespaced modules over non-Fedora modules with the same version.

yum Integration

Likewise, the OSGi updater should compare upstream modules to those available in Fedora, and where versions match, pull the Fedora module. There will need to be an OSGi-yum bridge so that the OSGi updater can request module metadata from yum, and instruct it which package to install. The required yum interfaces already exist to support update programs like pirut and pup. For dependencies not resolved in the yum repositories, the OSGi updater should fall back to its list of jar repositories.

Upstream Integration

The biggest improvement to the current Java packaging situation may well come from upstream projects. Once a module system becomes part of the Java standard, I hope to see upstream projects adopt it in a disciplined way. If upstream projects started bundling modules instead of just jars, packaging these projects would become simpler. The problem has been that it’s unclear which version of a particular jar is bundled with a project’s source — a module answers that question, and helps the packager set up BuildRequires. If this versioning convention becomes common, the BuildRequires generation step could even be automated by a “make new-sources-java” target in Makefile.common.

Projects also bundle dependencies in binary jars, for deployment on systems lacking native package management. With upstream module versioning, Fedora namespacing, and the Fedora module loader customizations, system-installed modules, where available, would be used in favour of the equivalent bundled modules. This should be the desired behaviour, since system administrators should trust Fedora Java packages more than the upstream binaires.

rpm Integration

Along these lines, system administrators may be interested in performing operations on the set of all system-installed OSGi packages. There is talk of adding tagging support to rpm and yum; once it is in place, we should apply a common tag to all packages that contain OSGi modules.

OSGi versus JSR 277

I used OSGi as the example module system because I’m familiar with it, but the above sketch would probably apply equally well to JSR 277. However, to complete the usefulness of a module system, the system Java environment must recognize it. If Fedora wanted to use OSGi system-wide (not just within Eclipse) then the system Java’s behaviour would have to deviate (irresponsibly) from the Java standard, to recognize and resolve OSGi modules. So while our packages can all be made “OSGi-ready” now, only Eclipse would see the benefits.

Since not deviating from the Java standard is more important than the benefits of modules, whichever module system is integrated into Java itself will become the Fedora-wide Java module system. Once a decision is made, we can start implementing the integration improvements I’ve sketched out.

Tom Marble is a wizard at getting the right people in a room together. In the Distro DevJam we had Java packagers from Debian, Ubuntu, Fedora, Red Hat, OpenSUSE, SUSE, Mandriva, Gentoo and FreeBSD, together with all the right people from Sun. This was the perfect group for discussing packaging issues surrounding OpenJDK (and a great audience for my talk).

My feeling is that there is now a general consensus among the distros about how to solve the technical issues of packaging OpenJDK. The remaining tricky issues are procedural:

1) creating a good OpenJDK contributor license agreement that works for free software developers
2) giving distros the ability to self-certify
3) making the JCP open to free software contributors (e.g., no click-throughs on specs)

We discussed 2 and 3 in the DevJam but it was too soon to talk about solutions. I hinted at 1 in my slides with “accepts most patches upstream”, but in retrospect I feel I should have emphasized the importance of a good CLA when seeking copyright assignment from external free software contributors. Anyway, given the discussion in the room, I feel that these problems will eventually be solved but that some patience will be required.

Sun’s showing at FOSDEM was very impressive. The presentations were helpful in that they provided a preview of Sun’s development processes. Because I’ve worked on GNU Classpath’s AWT, I was especially glad to meet Igor Nekrestyanov, who is working on the Sun Java2D team to replace the class library encumbrances.

Already as a result of our FOSDEM meetings we’re seeing opportunities for collaboration. External contributors are still in an awkward position until the class library code drop actually happens, but in the meantime there are opportunities to contribute to third-party libraries that Sun is using to replace the encumbrances. Along these lines, I learned at FOSDEM that the Java plugin will not be released with the initial code drop. Maybe gcjwebplugin can be adapted to act as a temporary stop-gap.

One non-Java FOSDEM observation: the Fedora booth was excellent, complete with Fedora flip-flop/boating-key-chain giveaways.