Monday, November 05, 2007

Updated Alioth Numbers for JRuby 1.1b1

Oh yes, you all know you love the Alioth Shootout.

Isaac Gouy has updated the JRuby numbers, and modified the default comparison to be with Ruby 1.8.6 rather than with Groovy as it was before. And true to form, JRuby is faster than Ruby on 14 out of 18 benchmarks.

There are reasons for all four benchmarks that are slower:

  • pidigits is simply too short for JRuby to hit its full stride. Alioth runs it with n = 2500, which on my system doing a simple "time" results in JRuby taking 11 seconds and Ruby taking 5. If I bump that up to 5000, JRuby takes 27 seconds to Ruby's 31.
  • regex-dna and recursive-complement are both hitting the Regexp performance problem we have in JRuby 1.0.x and in the 1.1 beta. We expect to have that resolved for 1.1 final, and Ola Bini and Marcin Mielczynski are each developing separate Regexp engines to that end.
  • startup, beyond being a touch unfair for a JVM-based language right now, is actually about half our fault and half the JVM's. The JVM half is the unpleasantly high cost of classloading, and specifically the cost of generating many small temporary classes into their own classloaders, as we have to do in JRuby to avoid leaking classes as we JIT and bind methods at runtime. The JRuby half is the fact that we're loading and generating so many classes, most of them too far ahead of time or that will never be used. So there's blame to go around, but we'll never have Ruby's time for this.
Standard disclaimer applies about reading too much into these kinds of benchmarks, but it's certainly a nice set of numbers to wake up to.

5 comments:

Isaac Gouy said...

"startup, beyond being a touch unfair for a JVM-based language right now ..."

I have to disagree with that ;-)

It might be irrelevant if we only ever invoke the programs on a long running JRuby server, but that might not be what we're doing.

We're talking about JRuby startup ~25x slower than Java startup, ~10x slower than Rhino startup.

If you think startup will still be this slow beyond the beta then I think you need to loudly set expectations.


"pidigits is simply too short for JRuby to hit its full stride"

Which is a most euphemistic way of saying JRuby startup is so slow that it swamps the calculation time :-)

Long startup time is an explanation why a program is slow - it doesn't make the program any faster.

Stuart Holden said...

That's great news, but I have to agree with Isaac. Startup times shouldn't be ignored - especially given the scripting language aspect of Ruby. If you want to use JRuby for short scripts, you'd want it to startup pretty much immediately.

Putting Ruby on the JVM brings some amazing benefits, but there's no getting away from the fact it's Java. That's the price of progress I suppose :-/ Is there any way that JRuby could be launched in a 'light' mode that sacrifices long term performance for a quicker start up through dynamic loading, or is this entirely down to JVM startup times?

Good Work!

Stu

Isaac Gouy said...

Wierdly, the JRuby thread-ring program doesn't seem to show a result when run inside our bash/perl measurement programs, but seems fine when run in a terminal window.

Any idea what's going on?

Isaac Gouy said...

mea culpa I failed to notice that there really is a problem!

With JRuby the program fails at 495 messages, any idea what's going wrong?

~/tmp $ /opt/jruby-1.1b1/bin/jruby -J-server -J-Xbatch -O threadring.rb 494
495
~/tmp $ /opt/jruby-1.1b1/bin/jruby -J-server -J-Xbatch -O threadring.rb 495
~/tmp $

thread-ring (new) Ruby JRuby program

Charles Oliver Nutter said...

Isaac: It should be fixed now in trunk; Kernel#sleep was implemented incorrectly.

http://jira.codehaus.org/browse/JRUBY-1575