Rails Support Status Update

Things are moving along well...so well I've found time to reimplement String with byte[], work on closures in the compiler, and, well those are posts for another day. Today I update you on the progress of supporting Rails in JRuby.

Rails is quite an interesting beast. I've learned more about Ruby looking through Rails code than from any other source...out of sheer necessity. They say the best way to learn a language is through immersion, right? Is debugging 50-deep stack traces on a questionable interpreter, digging for a reducible test case immersive enough for you? Yeah, I thought so.

We actually hit Rails support hard right at the beginning of this month, or the end of January. The early results were pretty solid, so our efforts have been distracted onto other large JRuby issues not necessarily Rails-related (but all still critical for an eventual 1.0). We also made the bold move of jumping to Rails 1.2.x for all our testing, to show we can keep up with the Rails development process. And to show that we've made great progress, I give you the following results.

ActiveSupport

ActiveSupport is the base module for Rails. It monkey-patches a number of core classes, provides a multibyte String wrapper for UTF-8 encoded text, and handles most of the details of running and configuring an application. It "supports" the other libraries, and is the first crucial leg needed to run Rails. And so we must "support" it well.

Here's the current results of a full test run:

498 tests, 1809 assertions, 16 failures, 6 errors

That's already in the 95% range, so we're in darn good shape. But it turns out a number of these errors are caused by a glitch in our parser related to KCODE. So if we work around that known issue, the results improve to:

498 tests, 1845 assertions, 12 failures, 1 errors

So more like 97% passing once we fix the KCODE parser problem. The remaining issues are almost all related to time-formatting bugs, with a couple multibyte and exception-handling issues tossed in. I haven't attacked the formatting bugs because Printf code is bloody painful, the multibyte issues are waiting on the KCODE parser fix, and the others...well, they're boring.

ActionPack

ActionPack is the brains of a Rails app, housing the mechanisms for controllers, views, and any code to support them. So it's leg two of the crucial three-legged support necessary to run a Rails application.

I hit ActionPack hard in the past, and a bit this month. Ola took off with it and completed most of the remaining failures. Running with the same KCODE workaround, we have the following results today:

1157 tests, 4811 assertions, 7 failures, 14 errors

That's above 98% passing. The remaining failures include a number of dupes (a single failure that breaks a number of tests), some additional string-formatting failures, and a couple that run ok outside of Rake. So it's damn close to perfect.

ActiveRecord

You should all know ActiveRecord by now, right? It's Rails' DB layer, based on the ActiveRecord pattern. It is the third core leg of the Rails platform, though you can certainly have apps that don't use AR for database support.

ActiveRecord is an extensive piece of code and it's the only part of Rails that usually requires a native library to run properly (though there is a pure Ruby impl of its MySQL support that nobody uses). In order to support AR, a number of the JRuby community members have cooperated over the past 9 months to build ActiveRecord-JDBC, a gem-installable module that provides ActiveRecord DB support via JDBC. It's a great bit of hackery, and runs surprisingly well considering our less-than-beautiful Java integration performance (under repair).

To limit the scope of this month's Rails work, we're targetting MySQL, since it's the de-facto standard for Rails apps in most quarters. But most of the work we're doing will apply equally well to the other databases, since JDBC is generally very consistent. Tom has been spending lots of time on Derby, for example, to the point that our ActiveRecord-on-Derby failures are almost entirely limited to SQL features it doesn't support yet.

So then ActiveRecord test results, minus the KCODE workaround (since Tom ran these for me):

1012 tests, 3417 assertions, 41 failures, 35 errors

Here our results dip to around 92% passing, but it's really the extended features of AR that have failures here. We've come a long way on this; results on the months-outdated JRuby wiki show Rails 1.1.6's ActiveRecord only passing about 60% of that release's tests, so there's been a ton of improvement since November. And we generally understand how to fix the remaining failures, so it's only a matter of (short) time.

ActionMailer

Can you guess what ActionMailer does? ActionMailer provides support for composing, formatting, and sending email from a Rails app. And that's about it. It's not a huge library, but it's essential for many apps.

64 tests, 142 assertions, 9 failures, 6 errors

That's about 75% passing, with a grand total of six test scripts. We haven't focused on this much, other than Tom's initial KCODE work a few months back. The current failures are almost all mail-formatting or SMTP-wrangling issues. I don't expect them to be hard to repair.

ActionWebservice

Because ActionWebservice's tests require some database setup, we haven't tackled them yet. But I would lay even money that they'll be comparable to ActiveRecord at their worst. For now, they all just fail because MySQL isn't set up correctly for them.

96 tests, 0 assertions, 0 failures, 96 errors

Any community member that wants to dive into ActionWebservice or ActionMailer would fast become a JRuby Hero.

Railties

Railties is the final piece of the Rails puzzle, and it...well..."ties Rails" together. I show the results here mostly for completeness; many of the libraries in Railties we'll never support (fcgi, for example) and most JRuby-on-Rails deployments will use alternative mechanisms for hitting the web.

5 tests, 19 assertions, 2 failures, 0 errors

Official Rails Support?

Because things are looking pretty solid, we've been looking for an answer to this question: What does Rails Support in JRuby mean? Do we have to pass all test cases 100% to "officially" support Rails? That might never happen, since there's POSIX and external library stuff we won't ever handle (nor will we need to for JRuby on Rails apps). So then is it a certain percentage? 95%? 98%?

I think the truth is that we could really announce support for Rails now. Almost all the visible, outstanding issues with actually *running* Rails apps have been resolved, and most apps and scripts work fine. There's ongoing work to improve ActiveRecord-JDBC's support for other databases, but that's an endless quest. And of course there's more work needed to support Grizzly, Mongrel, and WAR-based deployment of JRuby on Rails, but those are peripheral to the official announcement. Even when we do make an official announcement, it will be for a pre-1.0 version of JRuby, since we know there's another few months left on 1.0 fixes and features.

So what do you think, dear reader? At what point would you feel safe saying "Let's have our non-JRuby hackers try using JRuby on Rails"? You probably would *be* safe right now, since even if you found issues we've got a busy community ready to help solve them. And we'll probably tiptoe closer to "perfect" Rails support in JRuby over the next couple months, chasing the long tail of Ruby compatibility. But how do these numbers and this update make you feel about JRuby on Rails today?

And as always, we love to have additional contributors, so we'll bend over backwards to make it easy for you to help. Join the lists, join #jruby on freenode IRC, or toss us email privately. JRuby is an amazing community-driven success story, and the only thing missing is you.

California Schemin'

Last week Tim Bray and I were at Menlo Park to meet with the Open Source Software Society Shimane, a delegation of developers, managers, and company heads from the Shimane prefecture of Japan. They were visiting Sun to talk with us about opportunities for cooperation, Sun hardware and software, and most importantly: Ruby. For you see Shimane is the home of Yukihiro "Matz" Matsumoto, creator of Ruby, and he accompanied the group to California.

The evening before the event, we went to Fuki Sushi in Palo Alto, a short two blocks from my hotel. I don't believe I've ever eaten such quantity or variety of Japanese cuisine, and I think our guests felt the same way. They marveled at the size of most dishes, especially the ice-cream-scoop-sized lump of wasabi and the two-foot-long sushi tray. They also photographed almost everything...I think I posed for a couple dozen snaps.

During the following day, Thursday, they sat through numerous presentations on Sun hardware and software. Tim and I also discussed Sun's position on Ruby and JRuby for an hour before and about forty minutes after lunch. Tim hit the high-level points about where Ruby will likely fit into the Java ecosystem in the future, and I supplied details and demos of JRuby. I also threw in a demo of JRuby's compiler beating Ruby 1.8 in the standard fib algorithm, which elicited a smile and laugh from Matz himself (whew! I was worried how he'd react!).

Most interesting to me, however, was my discussion with Matz that night.

I was invited to join the delegation for a crab dinner in San Francisco. We went to Crustacean, a moderately upscale joint near California and 101. And after the attendant tied my plastic bib on, we were ready to go.

Since Matz and I ended up sitting together, and since very few others at the table spoke English, we managed to get in some time discussing a couple Ruby 2.0 design issues. Here's a quick summary:

• Matz seems to have come around to my visibility proposal for "private" in Ruby 2.0, which is largely the same as how Java handles private visibility. I believe this model is a good simplification over the original proposal. See ruby-core:9996 and related for the original discussion. The basic facts of private then would be:
• You must dispatch to private methods using a functional call, as in foo() versus xyz.foo(). I didn't like this at first, but I've come around to using call syntax to force certain aspects of visibility.
  
• Dispatches to private methods will only look in the same class for the method definition.
• Methods that are public in superclasses can't be made private in subclasses.
• Methods that are private in superclasses are not visible to subclasses, and so new methods of the same name and any visibility can exist in subclasses.
• Protected methods in Ruby 2.0 could potentially act like private methods now, though Matz is worried it would be too much of a change. I think it's appropriate; current private method behavior is very similar to Java's model for protected methods, where the methods can't be seen from outside the hierarchy, but can be called and overridden within the hierarchy as normal. I voiced my opinion, so we'll see where Matz goes from here.
  
• Matz is still comfortable with removing set_trace_func if a better mechanism for profiling and debugging can replace it. I had a few suggestions for alternate mechanisms, but I also promised to look into Java's model, since it seems to work quite well. I also suggested there may be something to learn from DTrace.
  
• Matz has come around to the idea that encoded character sequences are a different type than unencoded byte arrays, though he still wants them to have the same outward interface.

This last item warrants a bit more discussion.

The topic of encoded character strings came up a few times during Matz's visit, usually with him asking how we're doing things in JRuby. I explained that we mostly just follow Ruby 1.8, with our String now being backed by a byte[], but that we're also providing out-of-the-box native support for the new Rails ActiveSupport::MultiByte Chars class, a wrapper around string that enforces character boundaries and encodings.

At dinner, we continued the discussion. I made my case for a separate type with the following points:

• A separate type would not require String's interface to change, and it could remain a byte array
• By having separate types, we can use polymorphic behavior to avoid checking and re-checking encodings for every operation

The first item was mostly a non-issue...Matz is fairly intent on changing the String interface in 2.0, and much of that work is already complete. But he had an interesting response to the second item: he's already planning to have separate types internally for encoded character strings. This was very good news to me, since it meant that JRuby could easily support M17N in the future by simply providing different String types that handle the other encodings, where our UTF-16 String implementation could simply be backed up by java.lang.String/StringBuffer/Builder.

So the result of the String discussion can be summarized in a few points:

• String's interface will change from 1.8 to work with characters rather than bytes, both in the encoded and unencoded forms of String. The plan for String methods' behaviors does not change from current Ruby 1.9.
• String will have subtypes that represent encoded character data, though in most cases you won't need to know about those types. If you do need to go after a UTF8String (my name), you can, but there will also be some sort of factory model for generating encoded strings and Ruby 2's encoding pragma will handle literals.

All told, I think it was a very productive trip, and it was great to help Matz work through a few Ruby 2.0 design questions.