Friday, January 08, 2010

Busy Week: JRuby with Android, Maven, Rake, C exts, and More!

(How long has it been since I last blogged? Answer: a long time. I'll try to blog more frequently now that the (TOTALLY INSANE) fall conference season is over. Remind me to never have three conferences in a month ever again.)

Hello friends! It's been a busy week! I thought I'd show some of what's happening with JRuby, so you know we're still here plugging away.

Android Update

Earlier this week I helped Jan Berkel get the Ruboto IRB application working on a more recent JRuby build. There were a few minor tweaks needed, and Jan was interested in helping out, so I made the tweaks and added him as a committer.

The Ruboto IRB project repository is here: http://github.com/headius/ruboto-irb

Lately my interest in JRuby on Android has increased. I realized very recently that JRuby is just about the only mainstream JVM languge that can create *new* code while running on the device, which opens up a whole host of possibilities. It is not possible to implement an interactive shell in Groovy or Scala or Clojure, for example, since they all must first compile code to JVM bytecode, and JVM bytecode can't run on the Dalvik VM directly.

As I played with Ruboto IRB this week, I discovered something even more exciting: almost all the bugs that prevented JRuby from working well in early Android releases have been solved! Specifically, the inability to reflect any android.* classes seems to be fixed in both Android 1.6 and Android 2.0.x. Why is this so cool? It's cool because with Ruboto IRB you can interactively play with almost any Android API:



This example accesses the Activity (the IRB class in Ruboto IRB), constructs a new WebView, loads some HTML into it, and (not shown) replaces the content WebView. Interactively. On the device. Awesome.



I am trusting you to not go mad with power, and to use Ruboto only for good.

RubyGems/Maven Integration

JRuby has always been a Ruby implementation first, and as a result we've often neglected Java platform integration. But with Ruby 1.8.7 compatibility very solid and Ruby 1.9 work under way, we've started to turn our attentions toward Java again.

One of the key areas for language integration is tool support. And for Java developers, tool support invariably involves Maven.

About a year ago, I started a little project to turn Maven artifacts into RubyGems. The mapping was straightforward: both have dependencies, a name, a description, a unique identifier, version numbers, and a standard file format for describing a given package. The maven_gem project was my proof-of-concept that it's possible to merge the two worlds.

The maven_gem repository is here: http://github.com/jruby/maven_gem

The project sat mostly dormant until I circled back to it this fall. But once I got the guys from Sonatype involved (purveyors of the Nexus Maven server) things really got interesting.

Thanks to Tamas Cservenak from Sonatype, we now have something once thought impossible: full RubyGems integration of all the Maven artifacts in the world!

The Nexus RubyGems support repository is here: http://github.com/cstamas/nexus-ruby-support/

~/projects/jruby ➔ gem install com.lowagie.itext-rtf
Successfully installed bouncycastle.bcmail-jdk14-138-java
Successfully installed bouncycastle.bcprov-jdk14-138-java
Successfully installed bouncycastle.bctsp-jdk14-138-java
Successfully installed com.lowagie.itext-rtf-2.1.7-java
4 gems installed
Installing ri documentation for bouncycastle.bcmail-jdk14-138-java...
Installing ri documentation for bouncycastle.bcprov-jdk14-138-java...
Installing ri documentation for bouncycastle.bctsp-jdk14-138-java...
Installing ri documentation for com.lowagie.itext-rtf-2.1.7-java...
Installing RDoc documentation for bouncycastle.bcmail-jdk14-138-java...
Installing RDoc documentation for bouncycastle.bcprov-jdk14-138-java...
Installing RDoc documentation for bouncycastle.bctsp-jdk14-138-java...
Installing RDoc documentation for com.lowagie.itext-rtf-2.1.7-java...


Here's a an example of a full session, where I additionally install Rhino and then use it from IRB: http://gist.github.com/271764

I should reiterate what this means for JRuby users: as of JRuby 1.5, you'll be able to install or use in dependencies any Java library ever published to the public Maven repository. In short, you now have an additional 60000-some libraries at your fingertips. Awesome, no?

There are some remaining issues to work through, like the fact that RubyGems itself chokes on that many gems when generating indexes, but there's a test server up and working. We'll get all the issues resolved by the time we release JRuby 1.5 RC1. Jump on the JRuby mailing list if you're like to discuss this new capability.

Rake? Ant? Why not Both?

Another item on the integration front is Tom Enebo's work on providing seamless two-way integration of Rake (Ruby's build tool) and Ant. There's three aspects to Rake/Ant integration:
  • Using Rake tasks from Ant and Ant tasks from Rake
  • Calling Rake targets from Ant and Ant targets from Rake
  • Mixed build systems with part in Rake and part in Ant
Tom's work so far has focused on the first bullet, but the other two will come along as well. You'll be able to translate your Ant script to Rake and have it work without modification, call out to Rake from Ant, include a Rakefile in Ant and use its targets, and so on.

Here's an example of pulling in a build.xml file, depending on its targets, and calling Ant tasks from Rake:
require 'ant'

ant.load 'build.xml' # defines tasks :mkdir + :setup in ant!

task :compile => [:mkdir, :setup] do
ant.javac(:srcdir => src_dir, :destdir => "./build/classes") do
classpath :refid => "project.class.path"
end
end

Ideally we'll cover all possible integration scenarios, and finally blur the lines between Rake and Ant. And we'll be able to move JRuby's build into Rake, which will make all of us very happy. Look forward to this in JRuby 1.5 as well.

The C Extension Question

One aspect of Ruby itself that we've punted on is support for Ruby's C extension API. We haven't done that to spite C extension users or developers--far from it...we'd love to flip a switch and have C extensions magically work. The problem is that Ruby's C API provides too-invasive access to the internals of objects, and there's just no way we can support that sort of access without incurring a massive overhead (because we'd have to copy stuff back and forth for every operation).

But there's another possibility we've started to explore: supporting only a "safe" subset of the Ruby C API, and providing a few additional functions to replace the "unsafe" invasive bits. To that end, we (as in Wayne Meissner, creator of the FFI implementations for JRuby and Ruby) have cleaned up and published a C extension API shim library to Github.

The JRuby C extension shim library repository is here: http://github.com/wmeissner/jruby-cext

Last night I spent some time getting this building and working on OS X, and to my surprise I was able to get a (very) trivial C extension to work!

Here's the part in C. Note that this is identical to what you'd write if you were implementing the extension for Ruby:
#include <stdio.h>
#include <ruby.h>

VALUE HelloModule = Qnil;
VALUE HelloClass = Qnil;

VALUE say_hello(VALUE self, VALUE hello);
VALUE get_hello(VALUE self);

void
Init_defineclass()
{
HelloClass = rb_define_class("Hello", rb_cObject);
rb_define_method(HelloClass, "get_hello", get_hello, 0);
rb_define_method(HelloClass, "say_hello", say_hello, 1);
}

VALUE
say_hello(VALUE self, VALUE hello)
{
return Qnil;
}
VALUE
get_hello(VALUE self)
{
return rb_str_new2("Hello, World");
}

Here's the little snippit of Ruby code that loads and calls it. Note that the ModuleLoader logic would be hidden behind require/load in a final version of the C extension support.
require 'java'

m = org.jruby.cext.ModuleLoader.new
m.load(self, "defineclass")

h = Hello.new
puts "Hello.new returned #{h.inspect}"
puts "h.get_hello returns #{h.get_hello}"

Among the C API pieces we probably won't ever support are things like RSTRING, RARRAY, RHASH that give direct access to string, array, and hash internals, anything dealing with Ruby's threads or runtime, and so on. Basically the pieces that don't fit well into JNI (the Java VM C API) would not be supported.

It's also worth mentioning that this is really a user-driven venture. If you are interested in a C extension API for JRuby, then you'll need to help us get there. Not only are we plenty busy with Java and Ruby support, we are also not extension authors ourselves. Have a look at the repository, hop on the JRuby dev list, and we'll start collaborating.

JRuby in 2010

There's a lot more coming for JRuby in 2010. We're going to finally let you create "real" Java classes from Ruby code that you can compile against, serialize, annotate, and specify in configuration files. We're going to offer JRuby development support to help prioritize and accelerate bug fixes for commercial users that really need them. We're going to have a beautiful, simple deployment option in Engine Yard Cloud, with fire-and-forget for your JRuby-based applications (including any non-Ruby libraries and code you might use like Java, Scala or Clojure). And we're going to finally publish a JRuby book, with all the walkthroughs, reference material, and troubleshooting tips you'll need to adopt JRuby today.

It's going to be a great year...and a lot of fun.

27 comments:

Phil said...

As I'm looking into Android dev, this is pretty interesting. I'd heard reports that while precompiled Clojure works (as long as you don't use eval), the boot times were lousy, making it an iffy choice for Android. This actually had me curious about the use of Duby instead, since presumably JRuby would have the same boot time issues, though if that's not true I'd be thrilled. Have you tried Duby on dalvik yet?

Anonymous said...

Wow, impressive work Charles! Great to see JRuby improving its integration with the Java world.

Charles Oliver Nutter said...

Phil: Yes, Duby would probably work fine if it had all the features in place necessary to do Android dev. As it stands, it's missing a few key items like accessing other classes' public fields. But in theory with the missing features added it would work fine.

I'm hoping we're able to reduce the JRuby startup time on Android by stripping out a lot of initialization logic we don't need. It's pretty poor right now, though we've done basically no work to improve it.

I'm actually surprised Clojure had poor boot times, since the amount of code they generate is signficantly smaller than us. Perhaps it's because they opted to generate a class per function, and so many classes really confounds Dalvik?

grimen said...

I haven't had time to explore jruby more than in IRB...to take an explicit example: Will it be posible to run - say - HornetsEye ruby gem on jruby + android in the near future?

Charles Oliver Nutter said...

grimen: Absolutely! My desire is that you'd be able to write an app entirely in ruby, including the use of gems, and then package that all up as an Android application, possibly precompiling it all to improve performance. It would probably be easiest if instead of using RubyGems proper you used Bundler, since then the RubyGems logic would not need to be shipped.

grimen said...

@Charles: OK, fascinating. The reason I ask is that I might do my master thesis this spring on augmented reality, and I elaborated with the idea of using jruby on android. =) Follow-up question: Is there any roadmap for getting jruby + android rock 100%? With that I mean "stable enough for development at least".

Charles Oliver Nutter said...

grimen: In my head, there's a roadmap...but finding time and resources to work on it is rather tricky. There's a lot of stuff happening with JRuby right now, and Android is one of many projects. I really need more help to flesh it out and get it ready for prime time. Maybe you can help? :)

grimen said...

@Charles: More specifically...what would you want help with? I'm experienced in Ruby (and Java programming...well haven't touched in 2 years now), but if we talking c coding my passion is sort of non-existing. =)

P.S. I'm busy now with Rails, but when it's time for master thesis (in Mars at earliest) I need to do some research on android/jruby so maybe I ould help out somehow by then.

Charles Oliver Nutter said...

grimen: I think it would be useful to start playing with JRuby's current compiler and try to precompile Ruby code and get it to load on Android. There are some challenges, like the extra code it generates for method bindings, but I don't think there's anything we can't overcome. If you started playing with it and hit specific roadblocks, it would help me prioritize some of the heavy lifting on the JRuby side.

Jan Wedekind said...

@grimen: HornetsEye makes full use of the Ruby C-extension API (including memory allocations with 'xmalloc'). Furthermore it makes use of libJIT and Video4Linux. I'm quite sure that the current version of HornetsEye will not run on Android without modifications.

Charles Oliver Nutter said...

Jan: That is unfortunate. When will they learn!

grimen said...

@Jan: OK, thanks for the heads up. Will have to use some java library I guess.

Jan Wedekind said...

@charles: Since development of OpenMoko was suspended indefinitely, Maemo is probably the most promising project. But all the mobiles I am aware of are at best "mostly open". The other thing is that you need to utilise the GPU/GPGPU of the mobile in order to reach acceptable performance. But ordinary people like me usually have to wait to get access to new technology like this ;)

@grimen: No problem. The Marvin Project looks quite interesting. But I don't think it will run on the Android SDK. Another projet is ZXing which is a barcode reader running on Android.

Jan Wedekind said...

@charles @grimen Looks like I was wrong. I looked into it some more. According to Wikipedia it is possible to build and install C-libraries on Android (so far I thought you are required to use the Android SDK). Also I found some post related to V4L on Android. However according to Wikipedia, Android does not have X11 and the C-library is different. Furthermore I am not sure how well libjit supports ARM. So it might be possible to build HornetsEye for Android. So it might be possible to build HornetsEye (without X11 support) for android. However so far I haven't even tried to run HornetsEye with JRuby.

phoet said...

did you ever have a look at buildr (http://buildr.apache.org/) ? this already combines rake and ant pretty well.

grimen said...

@Jan: Interesting research! Regarding ARM, I found some info that hints about ARM-support for libjit is very much in progress:

http://code.google.com/p/libjit-linear-scan-register-allocator/wiki/WhyToLibJIT
http://old.nabble.com/libJIT-Linear-Scan-web-site-updated-td24004253.html

I plan to test out HornetsEye with jruby very soon (had issues on OS X, so will use ubuntu this time). Will post in HornetsEye forum about the outcome.

grimen said...

..and yea, thanks for the Marvin link, it might become very useful.

Jan Wedekind said...

@grimen Thanks for your response. I implemented malloc and currently I am working on multiarray. The plan is to rewrite the core to support lazy computation and split up HornetsEye into multiple "Gems". Furthermore I want to write a backend which makes use of GCC (something like RubyInline). I don't know how long that will take. But if there is an easy way to get part of HornetsEye to work with JRuby and on Android, that would be very interesting. There are various interesting things one can do on a mobile with a camera (panorama stitching, barcode reader, augmented reality, ...).

Charles Oliver Nutter said...

Jan: Perhaps you would be interested in helping us with the C extension shim for JRuby?

Phoet: Yes, we know of buildr, and it's a great tool if you want to (and are able to) follow the maven conventions. But it does not provide full two-way integration between ant and rake, which we would like to have out of the box for JRuby. I could see buildr being reworked to run atop JRuby's rake/ant integration in the future, or JRuby integrating maven logic more directly, but our current intention is to provide the lowest levels of integration without imposing project structure on users.

Jan Wedekind said...

@charles I was afraid someone would say that ;)
I would be interested in getting HornetsEye running with JRuby. I'll put it on my TODO list. At the moment however I want to implement lazy computation in order to get sufficient performance for doing real-time feature matching and hopefully 3D object recognition in the end.

Anonymous said...

Could you expand on this: "JRuby is just about the only mainstream JVM languge that can create *new* code while running on the device"

Is jruby doing something different to jython and groovy here?

Pascal Chatterjee said...

Nice work, I managed to get Ruboto IRB working perfectly on the emulator.

Sorry if I'm missing something obvious, but is it possible to execute ruby scripts with the application or are we limited to running in interactive mode?

Charles Oliver Nutter said...

comment_bot: Neither Jython nor Groovy nor Clojure nor Scala have an interpreted mode, which means all code they execute has to be compiled to JVM bytecode. Since the Dalvik VM on the device does not run JVM bytecode (it precompiles to Dalvik bytecode when installing) none of those languages can create new code while on the device.

Rhino (JavaScript) does have an interpreter, but I'm not sure anyone's using it for Android stuff.

Anonymous said...

Charles, I don't have your post-sun address. I hope to see you a Jfokus. Please drop me a line to my Sun address if you have a chance.

Thanks,

Ed

Charles Oliver Nutter said...

Pascal: Yes, basically any Ruby code you want to run on the device should run just fine. Currently, only interpreted mode is supported; but I'm adding improvements to JRuby's compiler and JIT to allow producing all your application's bytecode before building an app for the device. At any rate, if you ship JRuby's parser and interpreter, you can load arbitrary .rb source at any time.

Ed: Good to hear from you :) I'll be in town for most of the week, so perhaps we can grab a beer somewhere.

Michael said...

Rake/Ant integration and Maven gem can simplify several hybrid Ruby/Scala/Java tools I work on. Definitely worth having around. Many thanks!

Pascal Chatterjee said...

Charles: I gave that a try and made a little IRB + source code editor + script manager app that should make it more convenient to experiment with stuff on the phone.

Check out http://code.google.com/p/jruby-for-android/ if you're interested :)