fitzsim's development log

CommunityOne 2008

Monday, May 5

I thought my CommunityOne talk, The IcedTea Project: Developing OpenJDK for Deployment on GNU/Linux, went pretty well. The audience was small but full of important people 🙂 Martin Buchholz introduced himself after my presentation and we chatted a little about Google’s plans to contribute to OpenJDK. After the talk I was interviewed briefly by a reporter from InfoWorld.

I arrived at the presentation room halfway through the Lightning Talks session that preceded mine. One demo really jumped out at me: an astronomy viewer that the speaker was porting from C++ to Java. I can’t remember its name and Google didn’t turn it up readily but it was a very impressive demo.

After my talk I dropped in for the last half of Karsten Wade’s Insight into the Fedora Distro and Community. I thought it was a nice blend of high-level observations on community building and real world examples of how the Fedora project engages contributors. Later that day, Patrick Macdonald, Karsten and I recorded a podcast with Sun’s Barton George.

JavaOne 2008

Tuesday, May 6

The keynote was fun. Both the Kindle and JavaFX demos were problematic, apparently because Moscone’s wireless network was unreliable. Rich Green covered with a good line: “This shows that the JavaOne demos are live, not pre-recorded”. I had heard Neil Young would be there explaining “what Java means to him” which I thought was pretty funny. I was expecting a contrived marketing spiel but of course Neil Young kept it real with a convincing story about his use of Java (he’s producing digital archives of his life’s work). Best quote of the keynote: Neil Young: “We’ve got some demos to show you. Now these are fake, so we know they’re gonna work.”

Later on Tuesday I attended Is There a Place For Applets in Web 2.0?. There I learned about the upcoming JRE u10 plugin features, including JRE version selection and the new one-JVM-per-applet model (which gcjwebplugin uses for the sake of simplicity). I was surprised to learn that >= 1.5 JREs provide DOM APIs through LiveConnect. The speakers promoted a “cool applets” blog which I’ve been following.

Wednesday, May 7

I co-presented Growing Open Source Developer Communities with Dalibor Topic. I really appreciated Dalibor inviting me to speak with him. This talk was well-attended; about 130 people showed up. I had never co-presented before but I enjoyed the experience — preparing over tea at the amazing Samovar Tea Lounge was great fun. There were some good questions from the audience both during Q&A and after the talk so I think it was well-received.

Next, I chatted with David Herron about OpenJDK testing strategies for Red Hat’s packages. David gave me some suggestions about how best to apply the newly-freed jtreg harness in our processes. We also briefly discussed how to share TCK test results between Red Hat and Sun.

Mark Reinhold’s OpenJDK Community Update covered a story that was mostly familiar to me (he attended my talk, which I’m sure was familiar to him :-)), but also broke news about community governance. The deadline for writing the OpenJDK project’s constitution was this year’s JavaOne, but it’s been extended to next year’s JavaOne. And since Dalibor had joined Sun, the governance board seated three Sun employees and two non-Sun members. To address this Sun added two more seats to the board.

Later that night I dropped in on the tail end of the modules BOF, Modularity in the Java Platform: Demos and Q&A. Unfortunately it was the last session and security personnel ushered us out before I had a chance to chat with Alex Buckley.

On Tuesday and Wednesday I spent some time wandering through the JavaOne pavilion. Highlights:

• Intel’s Itanium port of Sun’s Java: This is almost complete and is due out later this year. Unfortunately it seems this won’t go into OpenJDK.
• Second Life: Sun had set up a Second Life replica of the JavaOne pavilion — recursively creepy!
• Wonderland preview: Wonderland is a Sun project similar to Second Life but with tricked-out 3D audio, shared whiteboards and in-world shared X windows.
• Java realtime demo: This was a series of tubes that sorted falling coloured balls according to selections made in a Swing GUI. I was able to temporarily jam the sorter by exploiting a GUI/sorter race condition, hehehe.

Thursday, May 8

The Open Source Development Tools for the Java Platform, Enterprise Edition (Java EE Platform), Web 2.0, and SOA talk introduced me to Drools. The presenters demoed modifying the canonical “online store” Java EE application from the point of view of developers, store employees and store customers. The graphical tools seemed pretty impressive — they allowed effortless connections to be established between pre-made components resulting in automatic updates to the employee/customer “workflows”.

Later I recorded another podcast, this time with Dalibor Topic for Sun’s OpenJDK blog.

Thursday night was the OpenJDK Porters’ BOF where I described Gary Benson’s work on IcedTea porting: his libffi port and Shark, his LLVM port. Jonathan Springer described his 64-bit MIPS port, and Volker Simonis described SAP’s porting efforts: 64-bit server JITs for PowerPC, Itanium and zSeries.

Friday, May 9

On Friday afternoon I attended the lab, “OpenOffice.org Extensions with NetBeans”. The tutorials were very well-organized. Once you’re familiar with the terminology — that is, once you know which wizards to invoke — it’s fairly straightforward to create first-class OO.o UI elements with NetBeans and the Java-to-UNO bridge.

Wrap-up

I talked to Jaroslav Tulach about progress getting NetBeans packaged and into Fedora. I also chatted with Chok Poh and Marc Schoenefeld (great to finally meet Marc in person!) about handling JDK security updates. I wanted to meet with Kenneth Russell, team lead for the new Java plugin, but our paths didn’t cross.

Dalibor was very helpful throughout the week facilitating various meetings and I appreciated his efforts to get good exposure for IcedTea, Fedora and Red Hat’s efforts in general. Plus he was a great drinking buddy 🙂 On that note, it was great to meet up with friends throughout the week: Casey Marshall, Erinn Clark, Alex Gravely, John McCutchan, Tom Marble and all the familiar members of Sun’s OpenJDK team.

My FOSDEM talk was on IcedTea and LiveConnect. I’ve been working on a gcjwebplugin rewrite called IcedTeaPlugin on-and-off for the past five months. Whereas gcjwebplugin is a Netscape 4.x plugin, IcedTeaPlugin is an XPCOM plugin that implements OJI — the Open JVM Interface — for LiveConnect support. I showed two demos at FOSDEM, one demo for each of the two test applets I’ve used for development. Here are the screenshots, taken just after I got each applet working:

JavaScript-to-Java direction

The text field and setText button are HTML form widgets. The “hello” string is passed from the text field to the Java applet on the right and displayed in bold. You can see the code in the page source of the original example.

Because of LiveConnect’s extensive use of Java reflection upon initialization, making any applet work, even the trivial “Hello, World” applet shown here, involves implementing most of OJI.

Java-to-JavaScript direction

The Java applet on this page calls into JavaScript to retrieve the page’s background color, then sets its own background color to match. Here’s the example source code.

I put my presentation slides online.

I’m pleased that I was able to retain gcjwebplugin’s out-of-process-JVM design in IcedTeaPlugin, despite the fact that OJI was obviously designed with libjvm.so embedding in mind. Basic applets work but there’s still lots to be done before IcedTeaPlugin can be declared production-ready. It is hosted in the IcedTea 7 Mercurial repository enabled by the –enable-liveconnect configure option.

I learned at FOSDEM that Sun is also working on a new plugin framework, in a private repository. LiveConnect support is important to Fedora plugin users, and that’s what motivates my work on IcedTeaPlugin. If Sun’s new plugin framework were released and could be deployed in Fedora faster than IcedTeaPlugin then I’d be willing to work on it instead.

On Monday, I found Conkeror through the EmacsWiki. It’s a Firefox extension that provides an Emacs-like interface to the browser. It includes all the essential Emacs features (minibuffer, incremental search, bookmark tab completion, buffer management) and an additional killer feature: numbered links. I’ve always wanted an efficient way to follow links and this is it. You can’t get much faster than <number>-<Return>!

I integrated gcjwebplugin into the IcedTea repository today. With Sun’s class library I’m able to install a proper applet security manager so that gcjwebplugin can run untrusted applets safely.

Here are some testing strategies that I’d like to see implemented for IcedTea. I’ve asked one of our interns to investigate these points.

Reduce Mauve-on-IcedTea failures to 0
Provide reproducible way of running Mauve on IcedTea which should result in 0 failures
“make dist” target for Mauve

Once these are complete we can a) set up nightly regression runs on http://icedtea.classpath.org, b) insist on having no regressions in package releases by running Mauve as part of the RPM build.

Make IcedTea demos build standalone

The IcedTea demos include libraries built during the IcedTea build, but these libraries cannot be built using the provided sample Makefiles. Installing the java-1.7.0-icedtea-demo RPM should a) install the demo source code in /usr/share/java-1.7.0-icedtea/demo, b) install the demo code’s build requirements as dependencies, c) provide working sample Makefiles. That way, trying the demos can be reduced to something like:

yum install java-1.7.0-icedtea-demo
cp -r /usr/share/java-1.7.0-icedtea/demo ~
cd ~/demo
make -f /Makefile.sample
make -f /Makefile.sample run

Ideally we’d test building and running the demos as part of the RPM build.

Run DaCapo benchmark reproducibly on IcedTea

Once DaCapo runs reliably we can set up nightly benchmark runs on http://icedtea.classpath.org.

Run graphical demos with java.awt.Robot and Xvfb

I’d like to include such graphical tests in the RPM build if the setup isn’t too fragile.

Longer term if Sun releases the jtreg harness (and some tests), we won’t hate it. 😉 More likely we’ll want to integrate it too into our RPM builds.

Here are some thoughts about how to make OpenJDK easily maintainable on Fedora. Achieving this goal will require changes to OpenJDK, updates to the Fedora implementation of the JPackage conventions and an rpm patch.

Here is a list of problems and a strawman solution for each. Each item needs careful investigation to determine the most appropriate solution.

OpenJDK Changes

Read OpenSSL certs directly

Now: We generate a cacerts file from certificates shipped with the Fedora OpenSSL package.

Future: OpenJDK should parse the OpenSSL certificates directly. That would give us one point of maintenance for the certificates and allow OpenJDK to take advantage of new certificates when OpenSSL is updated.

Replace font.properties with fontconfig

Now: fontconfig.Fedora.properties hard-codes Fedora fonts that OpenJDK uses.

Future: OpenJDK should use fontconfig to determine which fonts to use, to match desktop font preferences.

Read glibc timezone and locale data

Now: OpenJDK bundles its own set of timezone and locale data.

Future: OpenJDK should read glibc’s timezone and locale data. This provides a single point of maintenance for these data and eliminates the need to reinstall OpenJDK, or run a separate tool like tzupdater, whenever timezone or locale data updates are required.

Security policy and provider configuration under /etc/java

Now: OpenJDK security policies are installed and configured under /usr/lib/jvm/java-$version-$vendor/jre/lib/security.

Future: OpenJDK security policies and providers should be configured from /etc/java, since /etc is the standard location for administrator-edited configuration files. If configuration were under /etc/java/java-$version, the settings could be shared between same-versioned JDKs. There should also be a .d directory so that new security policies and providers can be easily installed by separate packages.

Endorsed directories: /usr/lib*/java-$version/endorsed, /usr/share/java-$version/endorsed

Now: OpenJDK loads endorsed jars from /usr/lib/jvm/java-$version-$vendor/jre/lib/ext.

Future: OpenJDK should load endorsed jars from locations like /usr/lib/java-$version/endorsed and /usr/share/java-$version/endorsed so that separate packages can easily install and uninstall additional endorsed jars.

Properties files in /etc/java

Now: OpenJDK loads .properties files from /usr/lib/jvm/java-$version-$vendor/jre/lib.

Future: OpenJDK should load .properties files from /etc/java, since /etc is the standard location for config files.

Split out bundled code, push Sun patches upstream

Now: OpenJDK bundles source code packages from many other projects.

Future: Using the new module system or some other mechanism, OpenJDK should use the system-installed versions of these packages. Any modifications to these projects that Sun has been maintaining in their tree should go upstream.

Default JNI load path, to support loading 32-bit and 64-bit JNI libraries from the same jar

Now: System.loadLibrary relies on the default library paths.

Future: System.loadLibrary should also look in standard paths where packages can install JNI libraries: /usr/lib/jni on 32-bit JVMs and /usr/lib64/jni on 64-bit JVMs.

Default /usr/share/java “CLASSPATH” directory

Now: JPackage provides a build-classpath script to build CLASSPATH settings from jars found in system-default locations such as /usr/share/java.

Future: OpenJDK’s module system should resolve dependencies from these system-default locations, obviating the build-classpath script. Related: Red Hat bug 170128.

OpenJDK Package Changes

Multiarch support in jpackage-utils

Now: the jpackage-utils package is built as a noarch package but contains references to /usr/lib.

Future: jpackage-utils should also support JDK installations in /usr/lib64. Doing so will raise questions about how JDK alternatives are handled, since they are based in /usr/lib. It may be best to change the alternatives tool itself to special-case /usr/lib and /usr/lib64.

Simplify package split to base, javadoc

Now: JDK packages are split up into base (JRE), devel, demo, src and javadoc subpackages.

Future: We could reduce this to just a base “java” package, which includes the JRE, development tools, demos and src.zip. javadoc will still need to remain a separate subpackage because of its installed size.

Simplify alternatives to just “java”

Now: There are many JDK primary alternatives: java, javac, jre_$vendor, $jre_$version, java_sdk_$vendor, java_sdk_$version, libjavaplugin.so

Future: With the simplified package split we could reduce this list to two primary alternatives: java and javadoc. But simplification may not be required if everyone just uses system-switch-java instead of dealing with alternatives directly.

RPM Changes

Fix rpm $1 arch-specific bug

Now: Red Hat bug 127359 prevents correct upgrading of alternatives symlinks in multilib installations.

Future: The bug must be fixed.

I’m experimenting with GNOME’s accessibility support; specifically, how it interacts with Swing and AWT applications. I’m impressed with the framework’s implementation. I had trouble saving GOK‘s alternative pointer device setting, and there seems to be an issue with having the alternative pointer device control the core pointer, but otherwise the interface is stable and usable. As an experiment, I wrote this blog post using only a single-button mouse we had lying around the office 🙂 The process was slow but smooth. My other experiments were also successful: GOK works with Swing and AWT applications over the Java Access Bridge.

Our team at Red Hat has been doing some planning now that OpenJDK has been released. First, a summary of OpenJDK’s status:

What Sun has released:

• most of the 1.6-level JDK, including:

• the Hotspot virtual machine with x86 and x86_64 JITs
• the Java compiler
• most of the class libraries

What Sun hasn’t released:

• 4% of the class library code — some of the class library code is “encumbered” meaning that Sun can’t release it under the GPL. The encumbrances are in the sound system, font rasterizer, graphics rasterizer, SNMP and crypto areas.
• plugin and javaws
• a 1.5 version

The second list points the way to some of our immediate goals. Here is a list of some of the goals we decided on:

1. Packaging OpenJDK as IcedTea

   We can’t call what we ship “OpenJDK” because that’s a Sun trademark. We’ve decided on the temporary name “IcedTea”, and we can say that it’s based on code from openjdk.java.net. We’ve posted an SRPM containing the free OpenJDK sources. But it still requires non-free stuff to build, which brings us to the next goal.

2. Building with free software

   Currently OpenJDK requires openmotif, Sun’s proprietary JDK 1.6 and a bundle of proprietary “plugs” (to replace the encumbrances) to build. Our group’s first goal is to make OpenJDK buildable against a set of fully-free packages.

3. Setting up icedtea.classpath.org

   With help from some GNU Classpath community members we’re setting up icedtea.classpath.org with a Wiki, mailing list and a Subversion repository, to host the work we’ll be doing on OpenJDK. This is a temporary site where we can experiment with the OpenJDK code while Sun works out its contribution processes. The site is not complete yet.

4. Testing without the encumbrances

   Because we haven’t built OpenJDK without the binary plugs for the encumbrances yet, we don’t know exactly how it fails if they’re not present. We need to see if the result can be proposed for inclusion in Fedora.

5. Replacing encumbered bits with parts from GNU Classpath

   Once we’ve tested running without the encumbrances we’ll know where to focus our replacement efforts.

6. Testing OpenJDK interpreter on other architectures

   Since at this point we only have x86 and x86_64 JITs, platform coverage will be a problem. In the short term we should test the OpenJDK interpreter portability since it may be possible to use it as a stop-gap on the architectures for which we don’t have JITs.

7. Working with Sun on contributor license agreements, use of trademarks, self-certification and governance.

   Eventually, we’d like to ship in Fedora a certified Java implementation that we can call OpenJDK and “Java compatible”. Sun intends to make that possible through a self-certification process. Eventually we should be able to contribute code directly to the OpenJDK project — Sun has worked out the governance model it will use and is implementing a Mercurial repository.

8. Working with other Red Hat groups on OpenJDK

   We expect to work closely with other Red Hat groups on both system integration (kernel, tools, desktop, packaging) and on requests from internal “consumers” of the JDK (JBoss, Eclipse).

GCJ

We will continue to support GCJ where it is deployed, but any new work we do will be on the OpenJDK codebase. GCJ still has better architecture coverage than OpenJDK, supporting all 7 RHEL architectures where OpenJDK supports 2, but we’ve decided that the benefits of OpenJDK outweigh this advantage of GCJ, and that effort is better spent trying to get (or write, if need be) more JITs for OpenJDK.

Timelines

We don’t have definite timelines yet. We’re in an investigation stage right now, so precise timelines will have to wait until we’ve had more time to discuss the OpenJDK codebase and Sun’s timelines.

I packaged the new code drop from the OpenJDK project using the JPackage conventions. Because naming rules are still being worked out for derivatives of OpenJDK code, I’ve decided to call the result “java-1.7.0-icedtea”. This is just a preliminary development package; I hope Sun will approve renaming it “java-1.7.0-openjdk” or “java-1.6.0-openjdk” when it officially lands in Fedora.

http://people.redhat.com/fitzsim/java-1.7.0-icedtea-1.7.0.0-0.1.nonfree.src.rpm

The OpenJDK build still depends on some non-free software so I’m providing an SRPM containing only the GPL’d code. People interested in building the complete not-fully-free OpenJDK can fill the gaps with binary plugs.

I’m going to investigate eliminating the non-free requirements. I haven’t studied the code enough to know if it’s worth producing a fully free build before starting encumbrance replacement efforts — it depends how much functionality is lost by excluding the binary blobs. Another interesting experiment would be bootstrapping OpenJDK with a free runtime.

Andrew Overholt recently pointed me to a blog entry on Java module systems. I can’t comment in detail on the JSRs or how they should interact because they’re all behind click-through licenses. My instinct tells me the ideal answers are: “Sun, drop JSR 277 and adopt OSGi; IBM, develop OSGi within the Java tree, on the same schedule as Java”, though this may be politically impossible.

Beyond politics, the post got me thinking about how a Java module system should integrate with Fedora. Here is a sketch of how I think it should work.

OSGi Modules in Fedora

OSGi modules distributed with Fedora will need to be wrapped by RPMs, so that they can be built, released and managed using existing tools and processes.

rpmbuild Integration

OSGi and RPM metadata provide similar information, so the adoption of OSGi will simplify the creation of Java RPMs. The packager should only be required to list the proper build dependencies in a spec file; Requires and Provides should be auto-calculated by rpmbuild. This is how C module (DSO) RPMs are currently produced.

The reverse operation should also be supported, for transforming non-modularized upstream projects into OSGi modules at rpmbuild time, using packager-provided metadata (the Name, Version, Requires and Provides fields). Then we could easily convert our currently-shipping Java packages to OSGi modules. At the same time, we should encourage upstream projects to properly modularize their jars, including any build dependency jars they bundle.

rpmbuild should automatically namespace packager-built modules with a Fedora release prefix, to distinguish them from the equivalent upstream module.

Module Loader Customizations

The Fedora OSGi loader should prioritize these Fedora-namespaced modules over non-Fedora modules with the same version.

yum Integration

Likewise, the OSGi updater should compare upstream modules to those available in Fedora, and where versions match, pull the Fedora module. There will need to be an OSGi-yum bridge so that the OSGi updater can request module metadata from yum, and instruct it which package to install. The required yum interfaces already exist to support update programs like pirut and pup. For dependencies not resolved in the yum repositories, the OSGi updater should fall back to its list of jar repositories.

Upstream Integration

The biggest improvement to the current Java packaging situation may well come from upstream projects. Once a module system becomes part of the Java standard, I hope to see upstream projects adopt it in a disciplined way. If upstream projects started bundling modules instead of just jars, packaging these projects would become simpler. The problem has been that it’s unclear which version of a particular jar is bundled with a project’s source — a module answers that question, and helps the packager set up BuildRequires. If this versioning convention becomes common, the BuildRequires generation step could even be automated by a “make new-sources-java” target in Makefile.common.

Projects also bundle dependencies in binary jars, for deployment on systems lacking native package management. With upstream module versioning, Fedora namespacing, and the Fedora module loader customizations, system-installed modules, where available, would be used in favour of the equivalent bundled modules. This should be the desired behaviour, since system administrators should trust Fedora Java packages more than the upstream binaires.

rpm Integration

Along these lines, system administrators may be interested in performing operations on the set of all system-installed OSGi packages. There is talk of adding tagging support to rpm and yum; once it is in place, we should apply a common tag to all packages that contain OSGi modules.

OSGi versus JSR 277

I used OSGi as the example module system because I’m familiar with it, but the above sketch would probably apply equally well to JSR 277. However, to complete the usefulness of a module system, the system Java environment must recognize it. If Fedora wanted to use OSGi system-wide (not just within Eclipse) then the system Java’s behaviour would have to deviate (irresponsibly) from the Java standard, to recognize and resolve OSGi modules. So while our packages can all be made “OSGi-ready” now, only Eclipse would see the benefits.

Since not deviating from the Java standard is more important than the benefits of modules, whichever module system is integrated into Java itself will become the Fedora-wide Java module system. Once a decision is made, we can start implementing the integration improvements I’ve sketched out.