Alioth Numbers for JRuby 1.0

Someone pointed out to me the other day that the Alioth "Compuer Language Benchmarks Game" (as they call it now) had started to include JRuby 1.0 in the results. And how do we fare? We're slower than Ruby 1.8.6! Hooray!

But not by much. Depending on your definition of "on par", I'd say we're safely in the range of being on par with Ruby 1.8.6 performance, at least for these too-short, too-small measurements.

Alioth benchmarks are regularly panned by language enthusiasts...or at least enthusiasts on the "slow" end of the benchmarks. In the case of JVM-based language implementations, the problem is the same old excuse: not enough time for the JVM JIT to work its magic. This case isn't all that different, but it's fair to say that Alioth isn't testing straight-line performance in a hot application, but getting from point A to point B without a running start. And in that domain, it provides a reasonable window into implementation performance.

So then, the JRuby 1.0 numbers. You can click the link to see them, but they break down about like this:

Performance:

• Four tests are equal or faster in JRuby--usually not more than 2x faster, but "4.8x" faster in one case. That fastest one involves "concurrency", and I haven't studied it enough to know whether it's meaningful or not.
• Eight tests are less than 2x slower in JRuby.
• The remaining four tests are greater than 2x slower in JRuby.
  
• Startup time is considerably worse; it's listed as 305x slower for JRuby (!!!) but it's not a particularly useful ratio. We take a couple seconds to get going compared to Ruby's hundredths. That's life.

Memory:

• All except one worse in JRuby
• Most more than 2x worse in JRuby
• Several more then 10x worse in JRuby
• Does anyone care?

(Update: Commenters have made it clear that they do care...but of course I was being a little bit glib here :) We continue every release to do what we can to improve performance AND memory usage, and other than some additional overhead from the JVM itself our memory usage is pretty reasonable.)

So I guess that's all I've got to point out. We've been working on performance since 1.0, and there's a number of major improvements planned for the 1.1 release. And considering that "beating Ruby performance" wasn't a primary goal for JRuby 1.0, I think our "roughly on par" numbers here are pretty damn good. Granted, Ruby 1.8.x isn't the fastest implementation in the world, but we're pretty happy to have improved performance by an order of magnitude in the past year.

Now, onward to the future!