Ruboto Is Your Friend

Ok, so I intentionally made my last post a bit of a "tease". You can't fault me for trying to drum up a little buzz, yeah? And hey, I spent almost as long fiddling with that logo as I did hacking JRuby to run on Android. Here it is again, just for good measure:


On Monday night, the local Ruby group (Ruby Users of Minnesota, or RUM...great buncha guys) hosted three talks: one on Android development, one on iPhone development, and one on migrating from Struts to JRuby a bit at a time. The Android talk kinda hooked me, even though I was working on last-minute JRuby 1.2RC1 issues and not really paying much attention (sorry, Justin).

I'd considered getting JRuby working on Android before, since it's compatible with most Java 1.5 language features, has a much more extensive library than any of the Java ME profiles (which hopefully will be remedied in future ME profiles), and represented the best chance for "mobile JRuby" to date. I had tweeted about it, scammed for a free G1 phone, and briefly browsed the online docs. I had even downloaded it back in early January...but I'd never bothered to try.

So late Monday night, I tried. And about an hour later it was running.

What I Did

There's really two sides to the Android SDK. There's the drag-and-drop fluffy-stuffy GUI in the form of a plugin for Eclipse. That was my first stop; I got it installed, created a sample project, and ran it in the emulator. It worked as expected, and I'll admit it made me want an Android phone a bit more. I'll be the first to admit I've been skeptical of Android, but at this point it's hard to argue with a totally open platform, especially since it has a shipping device now. So yeah, SDK plus sample app was easy and appetite-whetting.

Then I tried to pull in JRuby's main jar file. Nothing seemed to work right. I got errors about not having defined an "application" in some XML file, even though it was there. There was no obvious information on how to add third-party libraries to my app, and I certainly may have done it the wrong way. And of course my lack of knowledge about the structure of an Android app probably didn't help. But ultimately, since I didn't really need a full-on application, I started to dig around in the SDK for "another way".

Not one for reading documentation, I immediately started running the executables under "tools" with --help and guessing at combinations of arguments. Immediately I saw "emulator" and started that. Yay, an emulator! Then I saw dx, which looked intriguing. A-ha! It's the tool for converting an existing class or jar into Dalvik bytecode. A bit more fidding with flags, and I finally found the right incantation:

dx -JXmx1024M --dex --output=ruboto.jar jruby.jar

For the newbs: that's -JXmx1024M to allow dx to use up to a gig of memory, --dex to convert to Dalvik bytecode, and --output to specify an output file.

So, suddenly I had what I assumed was a Dalvik-ready ruboto.jar file. A quick jar -t confirmed that everything appeared to be there, along with a "classes.dex" file.

...
builtin/yaml.rb
builtin/yaml/store.rb
builtin/yaml/syck.rb
classes.dex
com/sun/jna/darwin/libjnidispatch.jnilib
com/sun/jna/freebsd-amd64/libjnidispatch.so
com/sun/jna/freebsd-i386/libjnidispatch.so
...

There were also a bunch of warnings about "Ignoring InnerClasses attribute for an anonymous inner class that doesn't come with an associated EnclosingMethod attribute." but warnings don't stop a true adventurer. I pressed on!

So, the next step was getting it into the emulator, eh? Hmm. Well there's no "upload" option in the emulator's OS X menu, and nothing obvious in the Android UI. There must be a tool. Like maybe a debugging tool of some kind... like a "jdb" but for Android. Hmm.....this "adb" executable looks promising...

$ ~/android-sdk-mac_x86-1.0_r2/tools/adb --help
Android Debug Bridge version 1.0.20
...

Ahh, bingo. And one of the adb subcommands was "push" for pushing files to the device. A few minutes and experiments later, I figured out incantation #2:

$ ~/android-sdk-mac_x86-1.0_r2/tools/adb push ruboto.jar ruboto.jar
failed to copy 'ruboto.jar' to 'ruboto.jar': Read-only file system

Or at least, I almost had it. Obviously the device was being closed-minded about the whole thing. So back to adb to run another subcommand and have a look around:

$ ~/android-sdk-mac_x86-1.0_r2/tools/adb shell
# ls
sqlite_stmt_journals
cache
sdcard
etc
system
sys
sbin
proc
init.rc
init.goldfish.rc
init
default.prop
data
root
dev

Hmm. "data". That looks promising. I mean, a "data" directory couldn't possibly be read-only, right? So let's give that a try.

$ ~/android-sdk-mac_x86-1.0_r2/tools/adb push ruboto.jar data/ruboto.jar
1702 KB/s (3249363 bytes in 1.863s)

BING! We have liftoff!

Ok, so we've "dexed" the jar, uploaded it to the emulator, and now we want to run it. Back into the shell we go!

There's obviously an sbin above, but it's pretty slim:

# ls sbin
adbd

Another debugging thingy I suppose. Maybe I'll have a look at that later. What about under "system"? I've gotten used to the bulk of my system living under something called "system" from running OS X. And as in that case, "system" was much more populous, with a bin directory containing all sorts of goodies. However one of them jumped out at me immediately:

# ls system/bin
am
app_process
cat
chmod
cmp
dalvikvm
date
dbus-daemon
dd
...

Oh, goodie, "dalvikvm". Could it possibly be the equivalent of the "java" command on a desktop? Could it really be that easy?

# dalvikvm -help

dalvikvm: [options] class [argument ...]
dalvikvm: [options] -jar file.jar [argument ...]

The following standard options are recognized:
-classpath classpath
...

It could! My hands began to tremble. My heart began to pound. Could I simply do

dalvikvm -jar ruboto.jar -e "puts 'hello'"

And expect it to work?

# dalvikvm -jar data/ruboto.jar -e "puts 'hello'"
-jar not yet handled
Dalvik VM unable to locate class 'data/ruboto/jar'
java.lang.NoClassDefFoundError: data.ruboto.jar
...

Curses! Ignoring for the moment how strange it seemed to have a -jar flag that simply doesn't work, I tried specifying -classpath and org.jruby.Main.

Aaaaaaaand...

It blew up with my first official JRuby-on-Android exception!

# dalvikvm -classpath ruboto.jar org.jruby.Main -e "puts 'hello'"
HugeEnumSet.java:102:in `next': java.lang.ArrayIndexOutOfBoundsException
from HugeEnumSet.java:52:in `next'
from Ruby.java:1237:in `initErrno'
...

Hmm. The code in question simply iterated over an EnumSet. After thinking through a few scenarios, I concluded this was not JRuby's fault. It seemed that I had discovered my first Android bug, the first time I tried to run anything on it. And that made me sad.

But only for a moment! The code in question turned out to be unimportant for a normal application; it was simply iterating over a set of Errno enums we use to report errors. Commented it out, and I was on to my next issue:

(I've lost the original error, but it was a VerifyError loading org.jruby.Ruby since it referenced BeanManager which referenced JMX classes. There is no JMX on Android.

Ok, VerifyError because of missing JMX stuff...that's no problem, I can just disable it for now. So, one more attempt, and if it fails I'm going to start doing iPhone development I SWEAR.

# dalvikvm -classpath data/ruboto.jar org.jruby.Main -e "puts 'hello'"
Error, could not compile; pass -d or -J-Djruby.jit.logging.verbose=true for more details
hello

SUCCESS!

More Details

Ok, so we all agree Android dodged a bullet there. But what's the real status of JRuby on Android?

It turns out there were very few changes necessary. I fixed the EnumSet stuff by just iterating over an Errno[] (EnumSet was not actually needed). I fixed the JMX stuff by creating a BeanManagerFactory (yay GOF) that loaded the JMX version via reflection, falling back on a dummy if that failed. And I fixed some warnings Dalvik was spouting about default BufferedReader and BufferedInputStream constructors by hardcoding specific buffer sizes (I think Dalvik is wrong here, and I'm arguing my case on the android-platform ML). And that's really all there was to it. JRuby pretty much "just worked".

Of course you see the "could not compile" error up there. What's up with that?

JRuby normally runs mixed-mode, interpreting Ruby code for a while and eventually compiling it down to Java bytecode if it's used enough. But we do try to immediately compile the target script, since it doesn't cost much and gives you better cold-start performance for simple scripts. The error above was simply JRuby reporting that it could not compile my little -e script. Why couldn't it? Because the JRuby compiler is generating JVM bytecode, not Dalvik bytecode. Dalvik does not run JVM bytecode. Here's the actual error you get:

# dalvikvm -classpath data/ruboto.jar org.jruby.Main -d -e "puts 'hello'"
could not compile: -e because of: "can't load this type of class file"
java.lang.UnsupportedOperationException: can't load this type of class file
at java.lang.VMClassLoader.defineClass(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:261)
at org.jruby.util.JRubyClassLoader.defineClass(JRubyClassLoader.java:22)
 ...

So that's caveat #1: this is currently only running in interpreted mode. To avoid the compiler warning, you can pass -X-C to JRuby to disable compilation entirely.

Unfortunately, interpretation means JRuby is none too fast at the moment. That may not matter if you're scripting a "real" app, but we'll definitely find ways to improve performance soon. That may mean providing an all-at-once compilation tool for Ruby code (we have an ahead-of-time (AOT) compiler right now, but it's per-file, and still expects to generate some code at runtime), or it may mean a second compiler that generates Dalvik bytecode. Either way...it's coming.

Caveat #2 is that a large number of libraries aren't working, especially any that depend on native code:

# dalvikvm -classpath data/ruboto.jar org.jruby.Main -X-C -e "require 'readline'"
-e:1:in `require': library `readline' could not be loaded: java.lang.VerifyError: org.jruby.ext.Readline (LoadError)
from -e:1
# dalvikvm -classpath data/ruboto.jar org.jruby.Main -X-C -e "require 'ffi'"
-e:1:in `require': library `ffi' could not be loaded: java.lang.ExceptionInInitializerError (LoadError)
from -e:1

And so on. There's nothing to say these libraries can't be made to work, but they're not working yet. And thankfully, our most important library seems to work fine:

# dalvikvm -classpath data/ruboto.jar org.jruby.Main -X-C -e "require 'java'; puts java.lang.System.get_property('java.vendor')"
The Android Project

And that leads me to caveat #3, better demonstrated than explained:

# dalvikvm -classpath data/ruboto.jar org.jruby.Main -X-C -e "require 'java'; import 'android.content.Context'"
Class.java:-2:in `getDeclaredMethods': java.lang.NoSuchMethodException
from ClassCache.java:137:in `getDeclaredMethods'
from Class.java:666:in `getDeclaredMethods'
from JavaClass.java:1738:in `getMethods'
 ...

Bummer, dude. There seems to be some feature (i.e. a bug) preventing some Android core classes from reflecting properly, which means that for the moment you may not be able to access them in JRuby.

Next Steps

Overall, I think it was a great success. We obviously weren't doing anything in critical JRuby code that Android could not handle. Kudos to the Android team for that, and kudos to us for still supporting Java 1.5. But success in software only leads to more opportunities:

• All the changes necessary to run JRuby on Android have already been shipped in JRuby 1.2RC1. So you can grab those files and dex them yourself, or wait for me to add Android-related build targets.
• Android's default stack size is incredibly small, 8kb. So for all but the most trivial Ruby code you're going to want to bump it up with -Xss. See the final snippit at the bottom of this post for an example. And of course you all know about using -Xmx to increase the max heap; it applies to Android as well.
  
• I need to report the bugs I've found in Android's bug tracker and provide some steps to reproduce them. I'll probably get to this in the next couple days. Hopefully they can be fixed quickly, and hopefully patched Android doesn't take too long to filter out to users.
• Meanwhile, I'll probably start poking at an all-at-once compilation mode, since I think that's simpler initially than emitting Dalvik bytecode. It's already done in my head. You'll run a command to "fully compile" a target script or scripts, and it will create the .class file it does now along with all the method binding .class files it normally generates at runtime. I've been planning this feature for a while anyway. With the "completely compiled" Ruby code you should be able to just "dex" it and upload to the device.
• Given that most people will probably want to ship precompiled code, and given the fact that many libraries will never work, we need to modularize JRuby a bit more so we can rip out unsupported libraries, parser guts, interpreter guts, and compiler guts. That should shrink the total size of the binary substantially. And I have other ideas for shrinking it too.
• We in the JRuby community also need to start brainstorming how to use this newfound power. Assuming the above items are all completed soon, what will we want to do with JRuby on Android? Build apps entirely in Ruby? Script existing ones? What Ruby features would we be willing to drop in order to boost Android-based performance a bit more? Hopefully this discussion can start in the comments and continue on the JRuby mailing lists.

The Bottom Line

JRuby works on Android, that much is certain. The remaining issues will get worked out. And I dare say this is probably the best way to get Ruby on any embedded device yet; after dexing, it's literally just "upload and run". So there's a great opportunity here. I'm excited.

And just one more example to show that not just JRuby itself, but also Ruby libraries that ship with it work (using the "complete" JRuby jar in this case):

# dalvikvm -Xss128k -classpath data/ruboto.jar org.jruby.Main -X-C -e "require 'irb'; IRB.start"
trap not supported or not allowed by this VM
irb(main):001:0> puts "Hello, JRuby on Android!"
Hello, JRuby on Android!
=> nil
irb(main):002:0>