Improved Multilib Support for JDK Packages

The first feature is improved multilib support for Java packages. This will involve fixing the darn persistent rpm scriptlet argument bug and adding multilib support to jpackage-utils and java-1.6.0-openjdk. I already have patches for these latter two but they depend on the rpm bug being fixed. We’ll have to carefully test upgrade paths when introducing these changes.

I’m hoping we can make java-gcj-compat multilib-compatible as well. My current plan involves using wrapper binaries for the tools instead of the direct symlinks we use now.

If we complete this feature Fedora 10 will allow parallel installation of 32-bit and 64-bit OpenJDK packages. Likewise for subsequent RHEL releases.

Java Packaging Improvements

The second feature deals with miscellaneous messiness in the current packages.

• Merging the rest of java-gcj-compat into libgcj. We’ve made steady progress in this direction with each Fedora release. Hopefully Fedora 10 will complete the process. Josh Sumali is already making headway on this.
• Security provider and extensions packaging improvements. This will allow us to package extensions like BouncyCastle such that they’re automatically loaded by the JRE. It will also allow us to move the GNOME Java Access Bridge currently bundled in the OpenJDK packages into its own separate package.
• Move %config files under /usr to /etc and replace them with symlinks.
• Rename java-1.6.0-openjdk* to openjdk*? Now that OpenJDK is becoming a standard component of Fedora we should revisit the JPackage naming conventions for it. The idea would be to use the python naming approach. The latest OpenJDK packages would be called openjdk and older releases would be renamed e.g., compat-openjdk6. (I’m a little conflicted about this. There’s some merit to the shared library approach — that once a name’s been used you can’t reuse it for an incompatible update because you’ll break dependents — but retaining one name across major versions forces dependent packages to update to the new version (a good thing for Fedora) and also simplifies dependency specifications.)
• Eliminate embedded full version strings from the OpenJDK directory names. We’d collapse directories like /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0 to just /usr/lib/jvm/java-1.6.0-openjdk. Our internal rpmdiff tool has trouble with versioned directories — this change would reduce rpmdiff output noise. It would mean that you couldn’t install OpenJDK 1.6.0.0 and OpenJDK 1.6.0.1 in parallel, but people should never need to do that. If they did, it would be to work around a bug that should just be fixed instead.
• Eliminate alternatives use from the OpenJDK packages, making them the exclusive owners of /usr/bin/* symlinks. I wish! It’s probably too soon to relegate other JDK alternatives to /opt though.

The primary goal of this feature is to make our OpenJDK packages easier to maintain. A secondary goal is to make packages that depend on OpenJDK easier to maintain. And the larger goal is smooth long-term maintenance of these packages in RHEL.

Plugin and Web Start

A shout-out to Planet JDK readers: Wouldn’t it be awesome if Sun’s cool plugin and Web Start rewrites were released in time to be shipped in Fedora 10?

Unfortunately I missed the Toronto Fedora 9 release party since I was on vacation. From the pictures it looks like it was a fun time!

Monday, May 5

I thought my CommunityOne talk, The IcedTea Project: Developing OpenJDK for Deployment on GNU/Linux, went pretty well. The audience was small but full of important people Martin Buchholz introduced himself after my presentation and we chatted a little about Google’s plans to contribute to OpenJDK. After the talk I was interviewed briefly by a reporter from InfoWorld.

I arrived at the presentation room halfway through the Lightning Talks session that preceded mine. One demo really jumped out at me: an astronomy viewer that the speaker was porting from C++ to Java. I can’t remember its name and Google didn’t turn it up readily but it was a very impressive demo.

After my talk I dropped in for the last half of Karsten Wade’s Insight into the Fedora Distro and Community. I thought it was a nice blend of high-level observations on community building and real world examples of how the Fedora project engages contributors. Later that day, Patrick Macdonald, Karsten and I recorded a podcast with Sun’s Barton George.

JavaOne 2008

Tuesday, May 6

The keynote was fun. Both the Kindle and JavaFX demos were problematic, apparently because Moscone’s wireless network was unreliable. Rich Green covered with a good line: “This shows that the JavaOne demos are live, not pre-recorded”. I had heard Neil Young would be there explaining “what Java means to him” which I thought was pretty funny. I was expecting a contrived marketing spiel but of course Neil Young kept it real with a convincing story about his use of Java (he’s producing digital archives of his life’s work). Best quote of the keynote: Neil Young: “We’ve got some demos to show you. Now these are fake, so we know they’re gonna work.”

Later on Tuesday I attended Is There a Place For Applets in Web 2.0?. There I learned about the upcoming JRE u10 plugin features, including JRE version selection and the new one-JVM-per-applet model (which gcjwebplugin uses for the sake of simplicity). I was surprised to learn that >= 1.5 JREs provide DOM APIs through LiveConnect. The speakers promoted a “cool applets” blog which I’ve been following.

Wednesday, May 7

I co-presented Growing Open Source Developer Communities with Dalibor Topic. I really appreciated Dalibor inviting me to speak with him. This talk was well-attended; about 130 people showed up. I had never co-presented before but I enjoyed the experience — preparing over tea at the amazing Samovar Tea Lounge was great fun. There were some good questions from the audience both during Q&A and after the talk so I think it was well-received.

Next, I chatted with David Herron about OpenJDK testing strategies for Red Hat’s packages. David gave me some suggestions about how best to apply the newly-freed jtreg harness in our processes. We also briefly discussed how to share TCK test results between Red Hat and Sun.

Mark Reinhold’s OpenJDK Community Update covered a story that was mostly familiar to me (he attended my talk, which I’m sure was familiar to him ), but also broke news about community governance. The deadline for writing the OpenJDK project’s constitution was this year’s JavaOne, but it’s been extended to next year’s JavaOne. And since Dalibor had joined Sun, the governance board seated three Sun employees and two non-Sun members. To address this Sun added two more seats to the board.

Later that night I dropped in on the tail end of the modules BOF, Modularity in the Java Platform: Demos and Q&A. Unfortunately it was the last session and security personnel ushered us out before I had a chance to chat with Alex Buckley.

On Tuesday and Wednesday I spent some time wandering through the JavaOne pavilion. Highlights:

• Intel’s Itanium port of Sun’s Java: This is almost complete and is due out later this year. Unfortunately it seems this won’t go into OpenJDK.
• Second Life: Sun had set up a Second Life replica of the JavaOne pavilion — recursively creepy!
• Wonderland preview: Wonderland is a Sun project similar to Second Life but with tricked-out 3D audio, shared whiteboards and in-world shared X windows.
• Java realtime demo: This was a series of tubes that sorted falling coloured balls according to selections made in a Swing GUI. I was able to temporarily jam the sorter by exploiting a GUI/sorter race condition, hehehe.

Thursday, May 8

The Open Source Development Tools for the Java Platform, Enterprise Edition (Java EE Platform), Web 2.0, and SOA talk introduced me to Drools. The presenters demoed modifying the canonical “online store” Java EE application from the point of view of developers, store employees and store customers. The graphical tools seemed pretty impressive — they allowed effortless connections to be established between pre-made components resulting in automatic updates to the employee/customer “workflows”.

Later I recorded another podcast, this time with Dalibor Topic for Sun’s OpenJDK blog.

Thursday night was the OpenJDK Porters’ BOF where I described Gary Benson’s work on IcedTea porting: his libffi port and Shark, his LLVM port. Jonathan Springer described his 64-bit MIPS port, and Volker Simonis described SAP’s porting efforts: 64-bit server JITs for PowerPC, Itanium and zSeries.

Friday, May 9

On Friday afternoon I attended the lab, “OpenOffice.org Extensions with NetBeans”. The tutorials were very well-organized. Once you’re familiar with the terminology — that is, once you know which wizards to invoke — it’s fairly straightforward to create first-class OO.o UI elements with NetBeans and the Java-to-UNO bridge.

Wrap-up

I talked to Jaroslav Tulach about progress getting NetBeans packaged and into Fedora. I also chatted with Chok Poh and Marc Schoenefeld (great to finally meet Marc in person!) about handling JDK security updates. I wanted to meet with Kenneth Russell, team lead for the new Java plugin, but our paths didn’t cross.

Dalibor was very helpful throughout the week facilitating various meetings and I appreciated his efforts to get good exposure for IcedTea, Fedora and Red Hat’s efforts in general. Plus he was a great drinking buddy On that note, it was great to meet up with friends throughout the week: Casey Marshall, Erinn Clark, Alex Gravely, John McCutchan, Tom Marble and all the familiar members of Sun’s OpenJDK team.

JavaScript-to-Java direction

The text field and setText button are HTML form widgets. The “hello” string is passed from the text field to the Java applet on the right and displayed in bold. You can see the code in the page source of the original example.

Because of LiveConnect’s extensive use of Java reflection upon initialization, making any applet work, even the trivial “Hello, World” applet shown here, involves implementing most of OJI.

Java-to-JavaScript direction

The Java applet on this page calls into JavaScript to retrieve the page’s background color, then sets its own background color to match. Here’s the example source code.

I put my presentation slides online.

I’m pleased that I was able to retain gcjwebplugin’s out-of-process-JVM design in IcedTeaPlugin, despite the fact that OJI was obviously designed with libjvm.so embedding in mind. Basic applets work but there’s still lots to be done before IcedTeaPlugin can be declared production-ready. It is hosted in the IcedTea 7 Mercurial repository enabled by the –enable-liveconnect configure option.

I learned at FOSDEM that Sun is also working on a new plugin framework, in a private repository. LiveConnect support is important to Fedora plugin users, and that’s what motivates my work on IcedTeaPlugin. If Sun’s new plugin framework were released and could be deployed in Fedora faster than IcedTeaPlugin then I’d be willing to work on it instead.

Reduce Mauve-on-IcedTea failures to 0
Provide reproducible way of running Mauve on IcedTea which should result in 0 failures
“make dist” target for Mauve

Once these are complete we can a) set up nightly regression runs on http://icedtea.classpath.org, b) insist on having no regressions in package releases by running Mauve as part of the RPM build.

Make IcedTea demos build standalone

The IcedTea demos include libraries built during the IcedTea build, but these libraries cannot be built using the provided sample Makefiles. Installing the java-1.7.0-icedtea-demo RPM should a) install the demo source code in /usr/share/java-1.7.0-icedtea/demo, b) install the demo code’s build requirements as dependencies, c) provide working sample Makefiles. That way, trying the demos can be reduced to something like:

yum install java-1.7.0-icedtea-demo
cp -r /usr/share/java-1.7.0-icedtea/demo ~
cd ~/demo
make -f /Makefile.sample
make -f /Makefile.sample run

Ideally we’d test building and running the demos as part of the RPM build.

Run DaCapo benchmark reproducibly on IcedTea

Once DaCapo runs reliably we can set up nightly benchmark runs on http://icedtea.classpath.org.

Run graphical demos with java.awt.Robot and Xvfb

I’d like to include such graphical tests in the RPM build if the setup isn’t too fragile.

Longer term if Sun releases the jtreg harness (and some tests), we won’t hate it. More likely we’ll want to integrate it too into our RPM builds.

Here is a list of problems and a strawman solution for each. Each item needs careful investigation to determine the most appropriate solution.

OpenJDK Changes

Read OpenSSL certs directly

Now: We generate a cacerts file from certificates shipped with the Fedora OpenSSL package.

Future: OpenJDK should parse the OpenSSL certificates directly. That would give us one point of maintenance for the certificates and allow OpenJDK to take advantage of new certificates when OpenSSL is updated.

Replace font.properties with fontconfig

Now: fontconfig.Fedora.properties hard-codes Fedora fonts that OpenJDK uses.

Future: OpenJDK should use fontconfig to determine which fonts to use, to match desktop font preferences.

Read glibc timezone and locale data

Now: OpenJDK bundles its own set of timezone and locale data.

Future: OpenJDK should read glibc’s timezone and locale data. This provides a single point of maintenance for these data and eliminates the need to reinstall OpenJDK, or run a separate tool like tzupdater, whenever timezone or locale data updates are required.

Security policy and provider configuration under /etc/java

Now: OpenJDK security policies are installed and configured under /usr/lib/jvm/java-$version-$vendor/jre/lib/security.

Future: OpenJDK security policies and providers should be configured from /etc/java, since /etc is the standard location for administrator-edited configuration files. If configuration were under /etc/java/java-$version, the settings could be shared between same-versioned JDKs. There should also be a .d directory so that new security policies and providers can be easily installed by separate packages.

Endorsed directories: /usr/lib*/java-$version/endorsed, /usr/share/java-$version/endorsed

Now: OpenJDK loads endorsed jars from /usr/lib/jvm/java-$version-$vendor/jre/lib/ext.

Future: OpenJDK should load endorsed jars from locations like /usr/lib/java-$version/endorsed and /usr/share/java-$version/endorsed so that separate packages can easily install and uninstall additional endorsed jars.

Properties files in /etc/java

Now: OpenJDK loads .properties files from /usr/lib/jvm/java-$version-$vendor/jre/lib.

Future: OpenJDK should load .properties files from /etc/java, since /etc is the standard location for config files.

Split out bundled code, push Sun patches upstream

Now: OpenJDK bundles source code packages from many other projects.

Future: Using the new module system or some other mechanism, OpenJDK should use the system-installed versions of these packages. Any modifications to these projects that Sun has been maintaining in their tree should go upstream.

Default JNI load path, to support loading 32-bit and 64-bit JNI libraries from the same jar

Now: System.loadLibrary relies on the default library paths.

Future: System.loadLibrary should also look in standard paths where packages can install JNI libraries: /usr/lib/jni on 32-bit JVMs and /usr/lib64/jni on 64-bit JVMs.

Default /usr/share/java “CLASSPATH” directory

Now: JPackage provides a build-classpath script to build CLASSPATH settings from jars found in system-default locations such as /usr/share/java.

Future: OpenJDK’s module system should resolve dependencies from these system-default locations, obviating the build-classpath script. Related: Red Hat bug 170128.

OpenJDK Package Changes

Multiarch support in jpackage-utils

Now: the jpackage-utils package is built as a noarch package but contains references to /usr/lib.

Future: jpackage-utils should also support JDK installations in /usr/lib64. Doing so will raise questions about how JDK alternatives are handled, since they are based in /usr/lib. It may be best to change the alternatives tool itself to special-case /usr/lib and /usr/lib64.

Simplify package split to base, javadoc

Now: JDK packages are split up into base (JRE), devel, demo, src and javadoc subpackages.

Future: We could reduce this to just a base “java” package, which includes the JRE, development tools, demos and src.zip. javadoc will still need to remain a separate subpackage because of its installed size.

Simplify alternatives to just “java”

Now: There are many JDK primary alternatives: java, javac, jre_$vendor, $jre_$version, java_sdk_$vendor, java_sdk_$version, libjavaplugin.so

Future: With the simplified package split we could reduce this list to two primary alternatives: java and javadoc. But simplification may not be required if everyone just uses system-switch-java instead of dealing with alternatives directly.

RPM Changes

Fix rpm $1 arch-specific bug

Now: Red Hat bug 127359 prevents correct upgrading of alternatives symlinks in multilib installations.

Future: The bug must be fixed.

What Sun has released:

• most of the 1.6-level JDK, including:

• the Hotspot virtual machine with x86 and x86_64 JITs
• the Java compiler
• most of the class libraries

What Sun hasn’t released:

• 4% of the class library code — some of the class library code is “encumbered” meaning that Sun can’t release it under the GPL. The encumbrances are in the sound system, font rasterizer, graphics rasterizer, SNMP and crypto areas.
• plugin and javaws
• a 1.5 version

The second list points the way to some of our immediate goals. Here is a list of some of the goals we decided on:

1. Packaging OpenJDK as IcedTea

   We can’t call what we ship “OpenJDK” because that’s a Sun trademark. We’ve decided on the temporary name “IcedTea”, and we can say that it’s based on code from openjdk.java.net. We’ve posted an SRPM containing the free OpenJDK sources. But it still requires non-free stuff to build, which brings us to the next goal.

2. Building with free software

   Currently OpenJDK requires openmotif, Sun’s proprietary JDK 1.6 and a bundle of proprietary “plugs” (to replace the encumbrances) to build. Our group’s first goal is to make OpenJDK buildable against a set of fully-free packages.

3. Setting up icedtea.classpath.org

   With help from some GNU Classpath community members we’re setting up icedtea.classpath.org with a Wiki, mailing list and a Subversion repository, to host the work we’ll be doing on OpenJDK. This is a temporary site where we can experiment with the OpenJDK code while Sun works out its contribution processes. The site is not complete yet.

4. Testing without the encumbrances

   Because we haven’t built OpenJDK without the binary plugs for the encumbrances yet, we don’t know exactly how it fails if they’re not present. We need to see if the result can be proposed for inclusion in Fedora.

5. Replacing encumbered bits with parts from GNU Classpath

   Once we’ve tested running without the encumbrances we’ll know where to focus our replacement efforts.

6. Testing OpenJDK interpreter on other architectures

   Since at this point we only have x86 and x86_64 JITs, platform coverage will be a problem. In the short term we should test the OpenJDK interpreter portability since it may be possible to use it as a stop-gap on the architectures for which we don’t have JITs.

7. Working with Sun on contributor license agreements, use of trademarks, self-certification and governance.

   Eventually, we’d like to ship in Fedora a certified Java implementation that we can call OpenJDK and “Java compatible”. Sun intends to make that possible through a self-certification process. Eventually we should be able to contribute code directly to the OpenJDK project — Sun has worked out the governance model it will use and is implementing a Mercurial repository.

8. Working with other Red Hat groups on OpenJDK

   We expect to work closely with other Red Hat groups on both system integration (kernel, tools, desktop, packaging) and on requests from internal “consumers” of the JDK (JBoss, Eclipse).

GCJ

We will continue to support GCJ where it is deployed, but any new work we do will be on the OpenJDK codebase. GCJ still has better architecture coverage than OpenJDK, supporting all 7 RHEL architectures where OpenJDK supports 2, but we’ve decided that the benefits of OpenJDK outweigh this advantage of GCJ, and that effort is better spent trying to get (or write, if need be) more JITs for OpenJDK.

Timelines

We don’t have definite timelines yet. We’re in an investigation stage right now, so precise timelines will have to wait until we’ve had more time to discuss the OpenJDK codebase and Sun’s timelines.