Friday, October 12, 2007

Performance Update

As some of you may know, I've been busily migrating all method binding to use Java annotations. The main reasons for this are to simplify binding and to provide end-to-end metadata that can be used for optimizing methods. It has enabled using a single binding generator for 90% of methods in the system (and increasing). And today that has enabled making some impressive perf improvements.

The method binding ends up looking like this:

@JRubyMethod(name = "[]", name2 = "slice",
required = 1, optional = 1)
public IRubyObject aref(IRubyObject[] args) {
...

This binds the aref Java method to the two Ruby method names [] and slice and enforces a minimum of one argument and a maximum of two. And it does this all automatically; no manual arity checking or method binding is necessary. Neat. But that's not the coolest result of the migration.

The first big step I took today was migrating all annotation-based binding to directly generate unique DynamicMethod subclasses rather than unique Callback subclasses that would then be wrapped in a generic DynamicMethod implementation. This moves generated code closer to the actual calls.

The second step was to completely disable STI dispatch. STI, we shall miss you.

So, benchmarks. Of course fibonacci numbers are indicative of only a very narrow range of performance, but I think they're a good indicator of where general performance will go in the future, as we're able to expand these optimizations to a wider range of methods.

JRuby before the changes:
$ jruby -J-server -O bench_fib_recursive.rb
1.039000 0.000000 1.039000 ( 1.039000)
1.182000 0.000000 1.182000 ( 1.182000)
1.201000 0.000000 1.201000 ( 1.201000)
1.197000 0.000000 1.197000 ( 1.197000)
1.208000 0.000000 1.208000 ( 1.208000)
1.202000 0.000000 1.202000 ( 1.202000)
1.187000 0.000000 1.187000 ( 1.187000)
1.188000 0.000000 1.188000 ( 1.188000)

JRuby after:
$ jruby -J-server -O bench_fib_recursive.rb
0.864000 0.000000 0.864000 ( 0.863000)
0.640000 0.000000 0.640000 ( 0.640000)
0.637000 0.000000 0.637000 ( 0.637000)
0.637000 0.000000 0.637000 ( 0.637000)
0.642000 0.000000 0.642000 ( 0.642000)
0.643000 0.000000 0.643000 ( 0.643000)
0.652000 0.000000 0.652000 ( 0.652000)
0.637000 0.000000 0.637000 ( 0.637000)

This is probably the largest performance boost since the early days of the compiler, and it's by far the fastest fib has ever run. Here's MRI (Ruby 1.8) and YARV (Ruby 1.9) numbers for comparison:

MRI:
$ ruby bench_fib_recursive.rb
1.760000 0.010000 1.770000 ( 1.813867)
1.750000 0.010000 1.760000 ( 1.827066)
1.760000 0.000000 1.760000 ( 1.796172)
1.760000 0.010000 1.770000 ( 1.822739)
1.740000 0.000000 1.740000 ( 1.800645)
1.750000 0.010000 1.760000 ( 1.751270)
1.750000 0.000000 1.750000 ( 1.778388)
1.740000 0.000000 1.740000 ( 1.755024)

And YARV:
$ ./ruby -I lib bench_fib_recursive.rb
0.390000 0.000000 0.390000 ( 0.398399)
0.390000 0.000000 0.390000 ( 0.412120)
0.400000 0.010000 0.410000 ( 0.424013)
0.400000 0.000000 0.400000 ( 0.415217)
0.400000 0.000000 0.400000 ( 0.409039)
0.390000 0.000000 0.390000 ( 0.415853)
0.400000 0.000000 0.400000 ( 0.415201)
0.400000 0.000000 0.400000 ( 0.504051)

What I think is really awesome is that I'm comfortable showing YARV's numbers, since we're getting so close--and YARV has a bunch of additional integer math optimizations we don't currently support and thought we'd never be able to compete with. Well, I guess we can.

However a more reasonable benchmark is the "pentomino" benchmark in the YARV suite. We've always been slower than MRI...much slower some time ago when nothing compiled. But times they are a-changin'. Here's JRuby before the changes:
$ time jruby -J-server -O sbench/bm_app_pentomino.rb

real 1m50.463s
user 1m49.990s
sys 0m1.131s

And after:
$ time jruby -J-server -O bench/bm_app_pentomino.rb

real 1m25.906s
user 1m26.393s
sys 0m0.946s

MRI:
$ time ruby test/bench/yarv/bm_app_pentomino.rb

real 1m47.635s
user 1m47.287s
sys 0m0.138s

And YARV:
$ time ./ruby -I lib bench/bm_app_pentomino.rb

real 0m49.733s
user 0m49.543s
sys 0m0.104s

Again, keep in mind that YARV is optimized around these benchmarks, so it's not surprising it would still be faster. But with these recent changes--general-purpose changes that are not targeted at any specific benchmark--we're now less than 2x slower.

My confidence has been wholly restored.

14 comments:

Anonymous said...

What's the bottleneck with JRuby on Rails in development mode. With today's trunk build of JRuby, production mode seems to be reasonably snappy, but development mode seems to lag far behind MRI, to the point where it's painful to use. How far are we from catching up in this aspect. Obviously reaching par with MRI in this regard would be much beneficial as it would greatly enhance development of Rails appplications on JRuby.

Ola Bini said...

Anonymous: see my recent post on the matter at http://ola-bini.blogspot.com. There are many numbers, but the Rails performance is awful. We don't know the reason yet, but we're working on it.

Christian said...

@anonymous: try turning off the JRuby JIT (-J-Djruby.jit.enabled=false) and see if it makes any difference. Maybe the overhead of creating the bytecode in development mode over and over again (when changing classes) causes the slowdown!?

Strange enough even in production mode Rails is faster without JIT in my case.

Brad C. said...

Quick comment on the use of annotations (hopefully not insulting your intelligence) but instead of having "name" and "name2" you could instead just have "name" and make it an array. A usage example would be:

@JRubyMethod(name = { "[]", "slice", "third_name", "etc." }, required = 1, optional = 1)

and if users only wanted to specify one name, they would just leave off the {}

@JRubyMethod(name = "[]", required = 1, optional = 1)

Charles Oliver Nutter said...

Brad C: That would be great, wouldn't it? Unfortunately annotations don't allow arrays. The only values you can put into an annotation must be constant literal values, classes, or enums. I'd love to have a better option for binding multiple names, but this will have to do for now.

Brad C. said...

Actually, they do allow arrays: it's really cool! Check out the definition of the @Target annotation (the meta-annotation used whenever you're creating your own annotation). It takes an ElementType[] and you can use @Target with one parameter:

@Target(ElementType.FIELD)

or multiple parameters:

@Target({ ElementType.FIELD, ElementType.METHOD })

So I gather your @JRubyMethod definition would look something like:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface JRubyMethod {
String[] name();
int required();
int optional();
}
plus any default values that you have set.

Neat stuff!

Anonymous said...

Unfortnately, attempting to disable jit at the command line brings light to another problem.

environment is Windows XP

starting webrick

>jruby -O -J-server -J-Djruby.jit.enabled=false script/server
Error opening script file: false (The system cannot find the file specified)

same deal for irb

>jruby -O -J-server -J-Djruby.jit.enabled=false -S irb
Error opening script file: false (The system cannot find the file specified)

Charles Oliver Nutter said...

anonymous: Can you report that as a bug? I can't reproduce it here on OS X.

Brad C: We'll I'll be darned. Looks like I'll be migrating the annotations to arrays!

Brad C. said...

glad to be of some small help :-)

Jacob Hookom said...

We didn't see you at Minnedemo last night-- were you there?

Andres Almiray said...

This is so great Charles, congratulations!

apohllo said...

Have you tried to contact the creator of the shootout page (http://shootout.alioth.debian.org/gp4/benchmark.php) to include JRuby?
It would be nice to have it compared with other languages/implementations by an independent "expert" :)

Mike McKinney said...

Is there a graph somewhere showing the performance numbers over time? I would love to see that STEEP climb!

Anonymous said...

For JRuby and Groovy "shootout" comparisons, check the "Extra languages timed on:" at http://shootout.alioth.debian.org/

Here's the direct link for instance:

http://shootout.alioth.debian.org/gp4sandbox/groovy.php