ActiveRecord 100%, Performance Doubling, Java Support Improving

I haven't blogged a solid JRuby update recently...and this one isn't going to be as solid as I'd like, but it should let you all know that the wheels are still moving and the train is picking up speed. It's astounding how much progress is being made.

ActiveRecord Now Fully Supported

It's good to have such incredible community members and great core team members.

Ola Bini, who recently accepted an offer from ThoughtWorks, has been hitting Rails unit tests hard. And the module with the most failures has always been ActiveRecord. It's the most complicated and the weakest point in the JRuby on Rails story.

Until now.

1031 tests, 3839 assertions, 0 failures, 0 errors

That headline should read "ActiveRecord Fully Supported on JRuby", for you press affiliates out there.

Of course there's a few caveats, but they're pretty minor in this case. That's zero failures and zero errors when running against MySQL, but it's certainly a de-facto standard in the Rails world. The next step is to get as many other databases to 0/0 as possible. And those are much smaller steps than this first one.

So what does this mean for JRuby on Rails users? Well in general, it means that the heart of your DB-driven JRoR apps--ActiveRecord--is now the best-supported piece of the puzzle. It also means that overall Rails support is well into the 99% range and still climbing.

Performance Continuing to Improve

In addition to the focus on getting Rails working well (which is really a focus on improving overall Ruby compatibility) we've been hitting performance much harder the past couple weeks. I especially have been splitting my time between interpreter and compiler work. As a result of all our efforts, the current trunk code is almost another 2X the performance of 0.9.8, released less than a month ago. The compiler also supports almost 2X the number of syntactic constructs it did in 0.9.8, and it's getting faster and more stable every day.

There are actually a number of benchmarks running faster than MRI (the C version of Ruby) even when interpreted, showing that we're finally making good on our promise to have interpreted JRuby run as fast as MRI. So it truly does seem that the compiler will be "bonus" performance once we work out the remaining bottlenecks in the system.

Java Support Looking and Feeling Better

There have been two areas of focus for Java support the past few months: rounding out features and syntax and improving performance. Both had some great jumps in JRuby 0.9.8, with a couple small fixes improving performance many times and with concrete/abstract class extension finally making it into a release.

Since 0.9.8 we've continued to make progress on both fronts. I've made a few additional small changes to improve overall performance, and Ola's been moving some especially critical pieces of code into Java, since they represented a lot of back-and-forth across that barrier. Where calling into Java code was previously as bad as twenty times slower than calling Ruby code, it's now a modest two times slower. We expect to improve that further, getting as close to Ruby invocation speed as possible.

We've also just added what should hopefully be the final revision to our various "import" syntaxes. Oddly enough, we've come full circle to "import":

# create a new constant "System" pointing at it
import java.lang.System

# use Java::syntax for less-common packages
import Java::my.weird.package.MyClass

And we're considering some improvements to how multiple classes are imported and how to specify an alternate constant name (for classes like String, which would be a bad idea to overwrite).

You can expect to see this all solidifying as we run these last miles toward JRuby 1.0.

--

I will blog more on all this as we get into April, but it's truly shaping up to be Springtime for JRuby in JavaLand.

Can Your Ruby Do This?

# scripting.rb

include Scriptable

x = 'Hello'
y = 'world'

groovy <<G
def z = 'multilanguage'
println("${x}, ${z} ${y}")
G

javascript <<JS
z = 'multilanguage'
println(x + ', ' + z + ' ' + y)
JS

# both => "Hello, multilanguage world!"

Ruby on Grails? Why the hell not?

I've spent some time this weekend looking over the Grails codebase. After a truly gigantic checkout (Grails bundles production JARs for all the libraries it integrates) and after swimming through a substantial, single-project sea of source files, I think I'm starting to follow it (though a tour by someone familiar with Grails internals would be nice). But there's something else I've learned in the process...

There's really no reason this couldn't use Ruby just as easily as Groovy.

Let's look at some really dumb numbers. Grails, as it turns out, is actually over 2/3 Java code, unlike Rails which is 100% Ruby. Grails contains around 436 Java source files for about 54kloc of code and 273 Groovy source files for about 23kloc of code. So Java outnumbers Groovy by a 2:1 ratio. And yes, Java's probably more verbose than Groovy in most cases. But my point stands, and I'll back it up with more...

Then there's the libraries. The Grails binary release weighs in at a whopping 26MB compressed and 49MB uncompressed (*wow*). But 31MB of that is third-party libraries completely independent of Groovy, things like Hibernate, XFire, Quartz, Apache Commons, and so on. Almost all of which (or perhaps all of which) are pure Java code. So the ratio is even more heavily toward Java.

These numbers and my exploration of the code base lead me to a surprising conclusion: Grails is not a "Groovy on Rails" by any means. It's a large number of popular, well-established Java libraries stitched together mostly by Java code with a thin layer at the front (20kloc is not much) of Groovy code for end-users to see and work with. And I guarantee you it's not the Groovy part that's most interesting here. Hell, you could write that part in Java if you followed enough "convention over configuration", and you certainly could write it in Ruby.

I decided to do this exploration after realizing there's nothing about Ruby or JRuby that couldn't be used to wire a bunch of Java libraries together--indeed, that's one of the biggest selling points of JRuby, the fact that you can write plain old Ruby code but call Java libraries almost seamlessly (with day-by-day improving integration points and performance numbers). So why not take Grails and do a port to Ruby? Groovy code isn't far off from Ruby in many ways, and everything you can do in Groovy you can do in Ruby (plus more), so the port should be pretty straightforward, right?

Then I find out that Grails is mostly Java code. And instead of being annoyed or repulsed by this revelation, I was intrigued. Why not make Grails work with Ruby as well? And really, why not make it work with any scripting language? Why limit that nicely-integrated back end by forcing everyone to use Groovy, regardless of preference? Have we learned nothing from a decade of "100% Pure Java"?

So I propose the following.

• I believe it would be possible to completely isolate Grails' tastier bits from Groovy. I don't claim it would be easy, but Grails is fairly well interfacified, injectificated, and abstractilicious, so making that dividing line clear and bold is certainly possible (and heck, that's the point of dependency injection and component frameworks anyway, isn't it?). Really, I could start implementing a few of those interfaces right now with Ruby code, but that's not the right approach.
• Grails' value is not in Groovy, but in the clean integration of established libraries behind the "Convention Over Configuration" mantra. There's nothing specifically Groovy (or Ruby) about that religion, and there's nothing Groovy does in Grails that another JVM language couldn't do just as easily. This is *truth*. And Grails would benefit tremendously by expanding to other languages. Being tied to Groovy alone will hold it back (just as a framework tied to any one language will be held back, Ruby included).
• I also believe that if the Grails project doesn't make an official effort in this direction, others are likely to take up that mantle themselves. Though the debates will forever rage about which JVM dynlang should carry us into the future, the same facts that have guided programming for half a century still apply today: No one language will be enough to carry us forward; no languages will survive indefinitely; and the language you'll use in ten years you've probably not even heard of yet. Did anyone expect Java to be the most widely-used, widely-deployed language ten years ago? Well today, it is. It won't always be.
• There's also a very interesting angle here for language enthusiasts. Nobody doubts that the Java platform is officially becoming multi-lingual, possibly a complete reversal from years past. But there are going to be growing pains here, and we're already feeling them on some projects at Sun. We have this wonderful platform with all these wonderful languages...and they all do things differently. We have a first stab at official "Scripting" support in JSR-223, but it only defines the integration point from Java to the language in question...only partially defining the integration points for those languages calling back to Java (a tricky problem) and defining *nothing* related to making those languages interoperate with each other. Again, Java becomes the integration point, and the bottleneck (as in "how do you pass a dynamic-typed object from one language to another by sending it through a statically typed language?"). The very exercise of making Grails (or any other framework) work equally well across multiple user-facing language interfaces will require better interoperability and more clearly-defined integration points for dynamic and static languages on the JVM.
• The Phobos guys at Sun are already tackling many of these same problems while expanding their JavaScript-mostly Rails-inspired web framework to other JSR-223 languages (notably, Ruby). So anyone taking on the Grails effort would certainly not be alone, and there are some damn smart engineers at Sun interested in tackling these very same problems.

And I guess in the end it's a question of how visionary we all can be. Can the Grails team envision a future for Grails that doesn't require the use of Groovy? Are you (and am I) as a non-Groovy developer willing to help them get there? And in the end, isn't this what thought leaders on a multi-lingual platform are supposed to be doing?

--

So, enough of the inflammatory bull-baiting. I dare you to tell me why I'm wrong here. Even better, I dare you to look into it yourself. 75kloc of code, only a third of it Groovy. Established libraries you're sure to have used at some point. Clean integration, following the principals inspired and proven by Rails.

But fronted with your language of choice.

Make it so.

NetBeans 6 Ruby Support Even Better Than Expected

See, here's the deal. I'm not a very advanced IDE user. I used Eclipse for years and JDE before that, and I wouldn't ever have considered myself a "pro" with either. I learned what I needed to get the job done and not a whole lot else. Beyond that I never knew some of the coolest features, and was never so good at convincing people to make a switch. This actually has helped my migration to NetBeans, since it basically does all the things I was used to in Eclipse. But it has been problematic when I try to convey how cool the new NetBeans Ruby support really is.

It's super cool. It does things I've not seen any other IDE or editor do.

However, it's best not to take my advice, and Roumen Strobl has recorded two seriously excellent demonstrations of Rails and Ruby support in NetBeans. They make the sale way better than I could (though I now have many good ideas for how to sell it better in the future).

If you have any interest in Ruby editors or IDEs, you should take the time and watch these, especially the Ruby support.

Two Demos: JRuby on Rails and Advanced Ruby Editing in NetBeans!

Behind The Scenes: JRuby 0.9.8 Released!

Things have been moving quickly in JRuby-land, and we've just kicked out a major release in JRuby 0.9.8. Among the big features:

• Ruby classes can extend concrete/abstract Java classes and override methods
• New Java primitive array syntax
• Reimplementation of String, Numeric classes, and Array to be more correct and performant
• Significant bottlenecks have been identified. In some cases IO is 6.5x faster than previous releases. Java included classes are significantly faster than in the past.
• 220 Jira issues resolved since last release

Now these are certainly delicious. Performance improvements are always great, and we're been improving Java integration little by little. But there's another bullet I omitted that is worth a little more attention:

• Ruby on Rails support

Now you may feel free to read that as you wish. I read it as "we expect pure-Ruby Rails apps to work." By "pure Ruby", I mean no unsupported C-language extensions required. By "work" I mean "work."

It was our original plan over two months ago that by Februrary's end we'd be passing enough of Rails' own test cases to call it "supported". And in a relative rarity for an open source project, we hit that milestone. It was actually early last week we started saying we officially support Rails, which at the time meant better than 95% of Rails test cases passing. By the time we cleaned up the remaining release items, we had passed 98%. And we're still climbing.

So what does this mean for a practical Rails user? Well, it means that the dream of deploying Rails apps on any Java-based server in any Java-based organization is another big step closer to coming true. I've actually been demoing Rails apps running in a WAR file on GlassFish for all my recent talks, and it's pretty solid. I don't imagine it's perfect, of course, but it's looking really very good. It will be "ready" by the time we hit 1.0.

A few folks have been wondering about the jump from JRuby 0.9.2 to JRuby 0.9.8. The simple answer is this: we're two releases away from 1.0. We wanted to make it obvious how much work has gone into 0.9.8 (as many SVN revisions in this release as the last three combined) and show our commitment to getting a 1.0 release out very soon. The 0.9.8 release will be followed by 0.9.9, which will be followed by...well, you get the picture. We're in the home stretch now.

The other core team members have reported on the release, so I won't cover all that. Here's a list of links:

Tom Enebo
Ola Bini
Nick Sieger

In related news, there's also a new release of ActiveRecord-JDBC, the gem-installable module enabling JDBC databases to be used by ActiveRecord. This is also very welcome, since there have been a number of bug fixes in trunk that weren't easily installable the past couple months. Now you should be able to run both Rails 1.2.2 and 1.1.6 well with JRuby, just using released code. It helps complete the puzzle.

So what's next for JRuby? Well, with the 0.9.8 release we've reached a really big milestone as far as Ruby compatibility, so we're likely to turn our attentions toward areas not directly related to Ruby 1.8. Specifically:

• Work will continue on the bytecode compiler, and I'm hoping to have the JIT permanently enabled within the next week
• Java integration will get a heavier focus, to start bringing us more in line with languages like Groovy for ease-of-integration. In general things are looking pretty solid right now, but we know there's a lot of work to do here. I'd like for us to come as close as possible to Groovy's level of tight integration as possible.
• We will also start working on native Unicode support, by exposing a Chars class like the one provided in Rails' MultiByte library. This will allow us to maintain compatibility with Ruby and provide solid Unicode support without introducing an incompatible set of features.
• Both Tom and Ola have expressed an interest in continuing work on our Ruby 2.0 bytecode engine. Ola would like to continue improving and expanding the bytecode support, while Tom would like to play with a second parser that goes straight to bytecode. Both of those will help feed into a potentially faster interpreted mode in the future, giving us many of the perf gains shown in Ruby 1.9 benchmarks.

For me personally, performance is going to be the big item. The recent Ruby Shootout showed that we have some work to do to bring out performance up to MRI standards, but it didn't tell the whole story. The truth of the matter is more like this:

• JRuby interpreted mode is roughly 3-4x as slow as Ruby 1.8
• JRuby compiled mode is roughly 1.5-2x faster than Ruby 1.8
• KRI (Ruby 1.9, "Koichi's Ruby Implementation") is many times faster than MRI for the posted benchmarks
• KRI is 2x or less faster than MRI for the general case

And this is without us doing any optimization of the compiler or adding in any special-case optimizations like KRI provides for fixnums. We're still looking great for future performance, and I want to start tackling this area heavily over the next couple months.

As always, we welcome contributors, bug reports, patches, and user anecdotes for JRuby. We love to hear about bugs, we want to know your opinion, and we respect and honor our community whenever we get the chance. Remember, JRuby is your project, and can only be successful with your help.

Visit www.jruby.org to learn more about the project, download the release, and get involved!

On to the next release!

JRuby World Tour: Bangalore

After my adventures in Hyderabad, the next stop on the tour took me to the heart of India's IT industry, where I would present my TechDays talk again for Sun Microsystems' Indian Engineering Center. But I scheduled in a little free time as well, to do some exploring.

I arrived on Saturday the 24th after a short flight on Sahara Air. For once, my travel arrangements worked out, and I was able to buy my ticket upon arriving at the Hyderabad airport. A few hours later, I had landed in Bangalore. A hotel car was waiting for me (this is now only the second time I've had a car waiting with my name on a sign, and I think I like it), so I headed straightaway to the Richmond Hotel in central Bangalore.

The Richmond has a fairly old-world feel, apparently having been converted from an apartment or office building some time in the past. It had a musty smell, but was otherwise pretty clean and well-kept. I'm generally more comfortable in "clean and comfortable" hotels than in "pristine" hotels like the Novotel, so the Richmond was just fine. That night I made arrangements for the next day's tour, ordered in some room service, and caught up on sleep.

It still amazes me how everything has changed since blogging became mainstream. Even as short as five years ago, I had a devil of a time finding local tour guides for my travels. Generally the best bet was to fork over too much money to a paid guide or to travel with a tour group. Neither of those are really conducive to free-form exploration, so I've always been forced to strike out on my own. But those days are certainly behind us.

In response to my "World Tour" blog posts, I had no less than 5 different people offer to show me around Bangalore. They ranged from JRuby enthusiasts to casual readers, and for once I had more guide offers than I could use. So I settled on the first two to contact me: K "Venkat" Venkatsubramaniyan and Håkan Raberg.

Venkat was up first on Sunday the 25th. He had actually stopped into the JRuby IRC channel (#jruby on FreeNode) as well as emailing me. His plan for the day included a trip to a cultural/folk center outside the city, lunch and dinner at a couple places to be determined, and shopping stops for a few things I wanted to purchase. So we started out bright and early at 9AM.

We made the following stops, in order:

• Rajarajeshwari Temple in western Bangalore, where I paid my respects and was gifted with a wreath of flowers and a tilak on my forehead. It was a moving experience, even though I felt a bit out of place in my western clothes. And the classical architecture of this modern shrine was quite impressive.
• Janapada Loka (Folk-Cultural World), a collection of folk art museums and exihibits on the road from Bangalore to Mysore.
• Kamath Yathrinivas Restaurant, where I had my first traditional Indian meal, complete with banana-leaf plate and eating with my hands. It was very tasty.
• Nalli Silk Arcade, to purchase sarees for my wife. I think I was the only guy in there shopping, other than a few poor saps who were obviously dragged along by their wives.
• Cauvery Crafts Emporium on Mahatma Ghandi Road, a nice collection of handicrafts, clothes, and nick-nacks, but very much tailored to western tourists. I looked at pearls for my wife briefly here, but decided not to buy.
• Navaratna Jeweleries, up the road from Cauvery, where I found the pearls I wanted. I went back on Tuesday before flying home to have the string extended and a clasp put on.
• Garuda Mall and Bangalore Central Mall...we just kinda ended up here looking for a place to buy tea and wasting time before our evening's restaurant opened.
• Amaravathi Restaurant, another more traditional place (more banana leaves and eating with hands), but this time with some delicious spicy curries and a crab dish as tasty as it was difficult to eat.

Venkat was an excellent guide, and I can certainly claim a new friend in Bangalore.

Monday the 26th was my day on the spot at Sun Microsystems. Conveniently, Sun's office in Bangalore was only about two blocks from the Richmond, so I headed over at around 10AM to prepare for my 11AM talk.

For some reason, every Sun office I visit is a terrific maze of doors, offices, and cubicals. The Menlo Park office has all L-shaped buildings with multiple intersecting hallways. Santa Clara's offices are plus-shaped (I think) which is often even more confusing. The Bangalore office put them to shame. It is hosted on the 4th, 5th, 6th, and 7th floors of a circular building with a central unsecured walkway and multiple secure entries and exits as you progress around the outer ring of offices. What complicated things most was that hallways extended out from the center of the ring like spokes of a wheel, but the perpendicular hallways did not always curve to match the circle. This meant that hallways often met at bizarre angles, and it was impossible to maintain any sense of direction. I got lost more than once wandering around, and unlike the California offices, I had no guarantee that following a given hallway would eventually lead me to an outside wall. Bizarre.

Anyway, I did the same talk from TechDays for Sun folks from 11AM to 12PM. Unfortunately, I had lost favor with the demo gods. My NetBeans demonstration froze up and refused to come back. My Rails demo failed with an error I still haven't isolated. And because of a peculiar scheduling snafu, the last 15 minutes of my talk had to be condensed into five minutes, resulting in more errors and a very flustered presentation. What had gone perfectly a few days earlier at TechDays went very poorly at Sun. Luckily, it seems to have still been well-received, and the classic JIRB demo was enough to wow the crowd.

The rest of the day was spent meeting with various folks at the IEC, talking about their projects and how JRuby could help. There's some cool stuff coming out of Sun India, so keep your eyes and ears open over the next few months.

Monday night my new friend Hakan Raberg from ThoughtWorks arranged to meet with me and two other ThoughtWorkers for dinner. It turns out Hakan is an old friend of Anders Bengsston, one of the premier JRuby contributors in the old days before I joined the project. Such a small world.

As a nice change of pace, the ThoughtWorkers and I had dinner at Barbeque Nation, a trendy, modern-feeling restaurant featuring fairly traditional northern Indian cuisine. In other words, lots of meat. Barbeque Nation is one of those "cook it yourself" places, where the center of the table is a small charcoal stove and the meats are delivered mostly-cooked on metal skewers. After marinating with a choice of marinades and allowing the food to cook the last 10% of the way, a delicious entree results. To make it even more acceptable to a western eater: it's all-you-can-eat, with skewers continually delivered until you tell them to stop. For the first time in a week, I was stuffed. It's very hard to turn down delicious seasoned and grilled meats when they keep arriving in front of you.

The ThoughtWorkers also had a number of interesting projects going on, many related to Ruby and a couple likely to use JRuby in the near future. Hakan had even worked on a code-generated implementation of the JRuby selector table's switch statements, which he promised to tidy up and send me. It could be very useful for us to optimize user-created classes in the same way we're starting to optimize core classes, and it was actually a technique we considered doing ourselves. More on that as it develops.

Tuesday, I packed up and checked out of the hotel. I was a little sad to be leaving Bangalore, but since I still had an entire day before my flight I walked up to M.G. Road to take care of some last minute souvenir shopping. My first stop was at Navaratna Jewelers for the modifications to my wife's new pearl necklace I mentioned earlier. While I waited for the work to be completed, I visited a few shops off the main boulevard and purchased the following items to bring home:

• A small wooden Ganesha (about 4" tall), to bring success and prosperity to my office.
• A small wooden Emblem of India, atop a 4" wooden column with a felt base. It is a perfect reminder of my Indian adventure (and probably the coolest national emblem I know). This is my favorite souvenir by far.
• A small wooden puzzle box for my son.
• Two larger versions of the puzzle-box with hand-painted images of Indian women on the tops.

I had also previously purchased the following items:

• Two silk ties with different images of dancing elephants on them.
• Two painted clay hippopotamuses, as gifts for whoever.
• A comic/cartoon book of stories of "Heroes of the Mahabharata", for my boy to read.
  

For lunch, I was not feeling particularly adventurous, and also decided something familiar might be safest before 24 hours of plane flights, so I ate at Pizza Hut. Yeah, I know. It wasn't even that good.

Comfortably full from lunch and loaded down with all my new possessions, I returned to the hotel to catch a cab to the airport. Goodbye, Bangalore...or so I thought.

Bangalore Airport was to be a baffling ordeal. People were lined up for a solid block to get into the terminal, mostly encumbered with a half-dozen bags each. To confuse even more, there was an option whereby non-flying visitors could pay for admission to the terminal. Luckily, I figured out I didn't have to pay, and after muscling my way to the entrance, I moved on to the second challenge in this travel gauntlet: Thai Airways had no reservation for me.

My original tickets to India were incorrectly issued to "Charles Onutter", an unfortunate combination of my middle initial and last name. Normally, this wouldn't have been a big deal, but apparently international flights are extremely strict about the name for which tickets are issued. If it doesn't match your passport exactly, you're out of luck and can't check in. In a flurry of activity the day before I left, I managed to arrange new tickets. But because it was such short notice, I would have to buy two of my tickets at the originating airports: Sahara Air from Hyderabad to Bangalore and Thai Airways from Bangalore to Bangkok. Sahara air went fine...either because they found my reservation or because they had plenty of room on the flight. But Thai Airways had cancelled my reservation because they overbooked the flight.

Luckily, after talking to the ticket agents and American Express's emergency travel service, I was able to purchase an unused seat for the flight. I would fly first-class to Bangkok. I rushed through security, ran to the gate, and finally settled into my seat. Onward to the next stop in the tour: Bangkok, Thailand!