Progress on WAR, RMagick, OpenSSL, Mutex Fronts

The JRuby community continues to pick up steam!

A number of JRubyists have recently started actively looking at the problem of deploying Rails apps as a WAR file. Some have working prototypes as well. However the most interesting development is that a number of them have joined the jruby-extras project on RubyForge to combine their efforts. I'll be helping to oversee their progress, but this is a perfect example of a community-driven project. I don't doubt they'll make great progress.

RMagick will soon have a full-featured equivalent for JRuby. Tom Palmer has been working on a Java-based RMagick for some time, and now has a version of his RMagickJr that can render some basic Gruff Graphs. He's been in communication with the RMagick creator, and it's likely that we'll start to see gems available soon. Tom's work will help ensure that Rails apps using RMagick for image processing can work seamlessly under JRuby as well. RMagickJr is also hosted in the jruby-extras project.

Ola Bini has been continuing his quest to bring the openssl library to JRuby. He says he's getting very close to having a working library, and it's been a long, hard road. Full support for openssl will mean all Ruby libraries that depend on it will work without modification on JRuby. It's quite an effort, and Ola deserves a lot of credit for making it happen.

At RubyConf 2006, Matz and Koichi made the announcement that Thread.critical was very likely to go away in Ruby 1.9.1/2.0. The reasons for this are simple, and well-known to the JRuby project: Thread.critical is incompatible with native threading. My own implementations of Thread.critical have ranged from a very strict version which frequently deadlocked to the current version which only enforces critical sections in a very loose sense. I am very pleased to hear that critical will go away, but that doesn't help us now. However, there's hope. MenTaLguY has recently taken on the challenge of implementing a fast Mutex for both the C and Java versions of Ruby (MRI and JRuby, respectively). As I understand it, the current C implementation he's built exceeds even low-level Thread.critical performance, and we both agree that a Java version should be extremely easy to construct using Java's built-in synchronization capabilities. A fast Mutex is the first step toward moving people off Thread.critical...and saving me doing yet another doomed reimplementation of it in JRuby.

Other news:

Tom and I spoke with some folks from the HotSpot VM team last night, and it was an extremely helpful discussion. We talked about Ruby's language design and quirkier features, the future of dynlangs on the VM, compilation and optimization strategies for dynlangs, and the current roadmap for JRuby development. Bottom line: everything we're doing is right, and if we keep on this course we'll rapidly approach their notion of an optimal Ruby implementation. We also agreed there's very little about Ruby that couldn't be compiled straight down to Java code. It was great vindication to hear that our "best guess" strategies for slowly redesigning, refactoring, and compiling JRuby are all on the right track. It was also great to hear that our confidence in the JVM has not been misplaced: it IS an excellent VM for dynlangs, and JRuby should eventually perform extremely well. The future of Ruby on the JVM is looking great.

I will be presenting JRuby again today at the Twin Cities Code Camp, as one of the few Java-based presentations (the rest being primarily .NET-related). I guess that's all there is to say about that...it's going to be a condensed version of the Gateway JUG presentation with fewer walkthroughs and a much shorter overall time.

Tom and I are also scheduling our trip to Europe in December. We'll be in Prague from the 5th to the 9th to meet with the NetBeans development team; in Antwerp the following week for JavaPolis; and in Rotterdam on the 19th for my JRuby presentation at Finalist. It remains to be seen if I'll spend some of the holidays in Europe or if my wife will join me, but if you'd like to propose any speaking engagements that could keep me in the Old Country, certainly let me know :)

Poll on Favorite Java Scripting Lang

There's an interesting little poll here on favorite Java-based scripting lang:

What is your favorite Java based scripting language?

It doesn't look like the poll has been hit too hard, but JRuby seems to score much better than I would have expected, given our relatively recent entry into the public eye. Groovy has garnered about twice as many votes, but of course it's been very public and available for several years.

Also interesting are the poor scores for Jython and Rhino. Jython is a great implementation, but it's pretty far behind at this point. I wish there were more resources to pour into it...perhaps soon. Rhino is also interesting...only four votes so far, even though it's really fast and being included in Java 6.

BTW, please don't go stuffing the ballots for JRuby or anything. I just figured I'd get this poll some exposure to see how things play out. It already looks really good for JRuby, and if you want to vote for Groovy or any of the others, be my guest.

Java SE 6 Passes Final JCP Approval

Huzzah! Java 6 has been approved by the JCP. The final results on JSR 270 show that the voting was 13 for, none opposed, with three not voting. As I understand it, this is the final step toward officially releasing Java 6 to the masses.

Java 6 is really an incredible piece of software. Once again the Sun engineers have boosted performance--in some JRuby benchmarks by as much as 20%. Beyond that, they've include the first round of native platform scripting support in the form of JSR 223, a scripting API intended to replace and improve upon the Bean Scripting Framework. Java 6 also includes native support for Javascript in the form of Rhino, just about the fastest mainstream JavaScript engine available.

Honestly, give Java 6 a shot. I've been using it constantly because of the performance gains, and JRuby support for 223 is already available. This is the first Java release during my time at Sun, and I'm extremely proud to be a part of this team.

Gateway JUG: Such Fun!

Last night I gave my 2+ hour talk on Ruby, JRuby, and JRuby on Rails to the Gateway JUG in St Louis, and it seems to have been a resounding success. We had extremely good attendance, pushing 50 folks from what I could tell, and everyone seemed to be very excited for the possibilities of Ruby on the Java platform. Many folks told me they were going to be looking into Ruby and Rails for future development, and others promised to contribute to the project however possible.

It was a great experiment for me to do this presentation, primarily because it was one long demo. There were a few slides to bridge things together, but ultimately I spent two hours typing into IRB, vi, and bash to demonstrate Ruby, JRuby, and building and running a simple JRuby on Rails app. There were a few glitches (I forgot a few metaprogramming methods, and my new migration initially failed because I ran it on an already-migrated database) but I managed to recover from everything and get all my demos across. I think the live walkthroughs coupled with a very enthusiastic and interactive crowd made a fun presentation for folks in attendance.

Afterwards I had a few beers with the locals and we shared our war stories about Java application development and excitement for a Ruby-filled future. Judging by their reactions to the talk and the stories they related, I think this JRuby thing is poised to really take off.

I've uploaded my Gateway JUG JRuby slides so you can see and share them, and Alex Miller from BEA took a transcript of the JRuby live walkthroughs.

Gateway JUG JRuby Presentation (Opendoc)
Gateway JUG JRuby Presentation (PDF)

Java EE Architect? Need a Job?

My former employer, Ventera Corporation, is still looking to fill my old position. I was their lead/senior Java EE architect, in charge of development of EE applications for their USDA Food and Nutrition Service contracts. I worked at FNS's downtown Minneapolis office, though a large part of the development team resides in Virginia at Ventera HQ.

Update: BTW, I'll be interviewing you, if that's any motivation for or against you applying...

If you're interested, contact Christina Pritchett at Ventera:

Email: cpritchett@ventera.com
Phone: 703.442.5223

And here's the job description Christina gave me...exciting fun for a Friday afternoon:

Senior Java Developer/Team Lead/Architect
Travel: 10% (back to our HDQA in Mclean, VA)
On-boarding: first few weeks will need to be spent in our McLean office

This individual will work on a development team for a large government client to develop, deliver and support a web-based application from design through implementation.

Responsibilities:

• Perform as a team lead – mentor junior staff and serve as the primary point of contact for the customer
• Perform as system architect role in the development of large-scale applications
• Develop prototypes from business and technical requirements/use case documents
• Conduct unit and integration testing
• Provide source code documentation and write reusable code; Maintain existing code base and component library
• Develop clear and concise documentation for work assignments
• Meet critical project timeframes
• Follow coding/design standards and J2EE Design Patterns
• Maintain application by implementing client enhancements as requested

Qualifications:

• 7+ years of relevant work experience - must have a minimum of 3 years as a tech lead and system architect (i.e. system design, performance tuning, and system prototyping & maintenance)
• Strong knowledge of Eclipse 3.1, WebLogic Platform 8.1, Hibernate, Spring, CVS (including branch based release management), Bugzilla (admin/server side), and Apache web server
• Ability to configure WebLogic server/cluster and troubleshoot issues related to WebLogic clusters
• Extensive knowledge about best practices across multiple disciplines including requirements analysis, architectural design, QA, project management, and configuration management
• Experience with Enterprise Java Beans development and related technologies (XML, JNDI, RMI, JMS, MDB) required
• Hands-on coding experience with: Java, JSP, STRUTS, HTML, EJB 2.0 (Session and Entity), JDBC, JAXP, JMS, and JTA required
• Experience with container managed persistence, container managed relationships and EJB query language
• Prior experience with Object-oriented analysis and design, including use case development, class modeling, J2EE design patterns (Factory, Façade, Singleton, Business Delegate, Service Locator, Dependency Injection, Data Access Object) and UML required
• Hands-on knowledge of application security support for e-commerce applications and relational database programming using SQL and JDBC required
• Ability to write and modify large ANT build scripts supporting builds and deploys in multiple environments
• Applied knowledge of JBoss and Configuration Management Processes is highly desirable
• Must have the ability to work in a team environment and work independently on client engagements
• Experience in professional consulting or like industry required
• Strong written & verbal communication skills required
• Keen interest in emerging Java technologies and industry best practices required
• Ability to set technical direction for application architecture and development discipline, and assess the impact of architecture changes on the business and application is required
• SCJP, SCJD and/or BEA Developer Certification is a plus
• Experience with SQL, JBoss, Hibernate and Spring strongly preferred

Education:

• BS in Computer Science, Engineering, or other related technical discipline required

Next Gig: Gateway JUG, St. Louis, Nov 7

Well the arrangements have all been made for my trip down to St. Louis for the November Gateway JUG meeting. I'll be presenting the usual fare, JRuby and JRuby on Rails stuff, except this time there's a few twists.

Twist One: I'll be presenting for 2+ hours

Yes, this one has me just a bit nervous...not because I think I'll have a problem presenting for that long or because I don't think I'll have enough material, but because I want to keep people awake the whole time. I've been told I'm a pretty good presenter, but even the best presentations can drag on after a while. Which leads me to the second twist:

Twist Two: I'll do 80-90% of the presentation manually

Yes, that's right...I'm going to be using IRB, command-line tools, IDEs, browsers, etc for most of the presentation. I think I've got my typing speed up to a comfortable enough level to keep people interested, and it's quite a bit more fun to watch things happen live than to see pictures or read slides. Plus it will be a unique challenge to do what basically amounts to a two-hour demo. You know me and challenges.

So a few links to information:

Gateway JUG - The Gateway JUG home page
JRuby: Bringing Ruby to the Java Platform - Description of the talk, my bio-of-the-month, directions and other info

And a rough overview of the talk (yes, this is the short, short list. I've got two pages for the long list):

• Intro to Ruby and JRuby
• Interactive demo of Ruby's major features
• What JRuby adds
• Interactive demo of JRuby
• Break
• Intro to Ruby on Rails
• What JRuby adds
• Building a simple Rails app
• JRuby status and future
• Conclusion

If you're going to be in the area on November 7, come watch me battle demo gremlins for two solid hours. It should be rife with action and adventure, love and laughs, guts and glory. Or at least you'll get to see both a JRuby and a JRuby on Rails presentation in one fun-filled night for free. That's worth it, no?

Update: There's an event flyer now as well.

Compiler Progress: MultiStub and Full-Script Compilation

I spent today hacking on the Ruby to Java compiler and made some good progress. Here's the highlights:

• It now parses a full script rather than just single method bodies.
• The toplevel of the script is given its own method, and defined methods get theirs.
• It uses MultiStub to implement the methods, so it will be faster than reflection.
• It's more aware of incoming arguments, rather than assuming a single argument as in the previous revision.
• It's generating a bit faster code, maybe 5-10% improvement.

MultiStub is our way of implementing many relocatable methods without generating a class-per-method or using reflection. We're using it today in our Enumerable implementation, and it works very well. Some numbers comparing our two reflection-based method binding techniques with MultiStub:

Invocation of a noop "test" method:

t = Time.now; 10000000.times { test }; puts Time.now - t
# test is alternately implemented using each of the three techniques

Control run (with no call to test in the block):

4.33s

ReflectionCallback-based (like Kernel methods today):

20.7s

ReflectedMethod-based (like most methods in normal classes):

19.3s

MultiStub-based (like Enumerable today):

14.9s

So simply switching to the MultiStub trims off around 20% for this benchmark. Removing the time to actually do 10M invocations of the block it comes closer to the 30% range. We're looking to start using MultiStub more in core classes. Anyway, back on topic...

What still needs to be done on the compiler:

• I didn't implement any additional nodes, so it only handles perhaps 20% of them.
  
• The toplevel method should define the contained methods as they're encountered. I'm wiring them all up manually in the test script right now.
• It doesn't have any smarts for binding Ruby method names to the generated MultiStub methods yet.

It's a big leap in the right direction though, since you can pass it a script and it will try to compile the whole thing to Java code. Here's the results of the recursive fib benchmark with the new compiler (calculating fib(30)):

Time for bi-recursive, interpreted: 14.859
Time for bi-recursive, compiled: 9.825

Ruby 1.8.5:

Time for bi-recursive, interpreted: 1.677

This was in the mid 10-second range previously, so this is the first time we've dropped below 10 seconds. This puts the compiled code around 6x as slow as Ruby for this benchmark, which is very method-call intensive. Still, it's a solid 33% improvement over the interpreted version...probably an even larger percentage improvement if we don't count method-call overhead. Now on to iterative results, which are very light on interpretation (calculating fib(500000)):

Time for iterative, interpreted: 58.681
Time for iterative, compiled: 58.345

JRuby sans ObjectSpace support:

Time for iterative, interpreted: 47.638
Time for iterative, compiled: 47.563

Ruby 1.8.5:

Time for iterative, interpreted: 50.770461

For the iterative benchmark we're still about on par with (or around 20% slower than) Ruby because there's no interpretation involved and Java's BigInteger is faster than Ruby's Bignum. When ObjectSpace is turned off (it's pure overhead for us), the iterative version runs faster in JRuby. Once we eliminate some method overhead, things should improve more.

Moving right along.

Transcript of Second Life Presentation

There's now a complete transcript of my JRuby presentation for the Rubyists of Second Life complete with my slides and all the inter-avatar discussion. It's raw and it's real, and it was a lot of fun.

Larry the Liquid has also posted a JRuby of Second Life Recap that gives readers a good idea what it was like to "be there" and has a few photos to show it off. I'm the one standing down by the screen, and though you can't really tell, I'm wearing a JRuby t-shirt.

If only it were as cheap and easy to print t-shirts in Real Life as it is in Second Life...

Rite, Rubinius, and Everything: Seeking the Answer to Ruby's Future

As most of you know, I was recently at RubyConf to see the latest developments in the Ruby world and to meet up with others interested in Ruby's future. A lot's happened in the past year, and Ruby has become the big ticket language in many circles. Developers from all parts of the world are sitting up and taking notice of the little gem that could, and those of us who love the language couldn't be happier about it.

This was my third RubyConf. I have attended since 2004, when I first came to Ruby as a rank newb. Immediately after arriving at RubyConf 2004 I started poking around for a Ruby implementation on the JVM. That brought me to JRuby, where my old friend and coworker had taken the reins. The rest you know...and it's not hard to see how Ruby has completely changed my life.

RubyConf this year was an outstanding collection of presentations and personalities. The applicability of the presentations seemed considerably higher than in previous years. Where last year many presentations were demonstrations of "this cool app I wrote" or explorations of problem domains I'll never approach, this year's topics seemed to cover whole subdomains of the programming ecosystem. They ranged from alternative Ruby implementations (two total plus two bridges), to natural language processing and generation (at least two I can think of), to Unicode and i18n (Tim Bray's excellent presentation and many discussions that followed), to networking, testing, graphics, and more. A whirlwind tour of damn near everything you might want to do with a programming language.

Beyond the standard conference fare, there was RejectConf, an ad-hoc sub-conference for all those who had their presentations rejected (or who missed deadlines, like me). It was a multi-hour flurry of 5-15 minute presentations on all those OTHER topics not covered in the main track. I gave a five-minute demo of JRuby's growing capabilities and NetBeans' nascent Ruby refactoring features--to many oohs and ahhs and even one f-word.

And then there was the hallway track, by far the most interesting part of the conference for me this year. I joined with several other Ruby implementers and interested parties for a full-on "implementers summit" Friday night. I met up with Ruby dignitaries, JRuby enthusiasts, and late-night hackers. I shared my pains implementing Ruby with others and had a good cry over some of Ruby's trickier-to-implement features. It was eating, sleeping, and breathing Ruby for three solid days. And now I'm home again.

I will here call out the more interesting presentations, discussions, and developments from my RubyConf 2006 adventure. I hope you will find these events as interesting and exciting as I did.

The Presentations

I won't pretend I loved every single presentation. I was bored by several and fell asleep during one. But there were some real gems this year, and they show the face of Ruby to come.

TAKAHASHI Masayoshi -- The History of Ruby

Takahashi-san kicked off the conference perfectly with his presentation on the history of all things Ruby. From Ruby's Pre-Historic Age in the mid-90's to the pre-Rails Modern Age, all the important facts and dates were delivered in Takahashi's trademark style. Everyone in the room learned something about Ruby's past and the long road it has traveled, including the rational for why Ruby came into existence in the first place and the name-that-almost-was: Coral.

Evan Phoenix -- Sydney and Rubinius: Hardcore Ruby

Evan is the creator of the semi-controversial "Sydney Ruby", an ambitious attempt to bring full native threading to Ruby 1.8. Sydney's drastic code changes prevented it from ever being merged into MRI, but the experience led Evan to attempt a more ambitious project: Rubinius.

Rubinius is, simply put, a Smalltalk-like self-bootstrapping VM for Ruby. A small microkernel provides memory management, IO, and other native interfaces, and then actual VM features are all implemented in a customized subset of Ruby. The resulting VM can then load in full-featured Ruby code and execute it like the existing interpreters do today.

Evan and I had talked at length online about his plans for Rubinius, and we both discovered we weren't alone in understanding some of the darker corners of Ruby's C implementation. Rubinius is a project to watch.

Zed Shaw - Iron Mongrel: Fuzzing, Auditing, Thrashing, Risk and The Ways Of Mongrel Destruction

This was my first time seeing Zed present, and I definitely agree he's very entertaining. Zed was at RubyConf to talk about RFuzz, a library for fuzzing HTTP services to test for security or stability issues. Fuzzing, for those of you who don't know, is generating a lot of random or pseudo-random noise (in this case, HTTP requests) and brutalizing some target service. Zed described how under fuzzed loads, many servers and libraries previously thought to be rock-solid crumble quickly and painfully. It seems to be in our nature to assume that all clients of our code will follow the rules we expect them to, even though we consciously know they won't. Fuzzing helps remind us.

It was also my first experience having Zed read my mind. Toward the end of his presentation, I raised my hand to ask about the possibility of using fuzzing against language implementations...to test parsers for robustness. Before I had a chance to ask, Zed talked about working on exactly that. He's planning to pull out the HTTP bits of RFuzz so it can be used as a general-purpose fuzzing tool, and being able to fuzz Ruby is already on his mind.

Saturday night Nick Sieger and I stayed up late hacking on stuff with Evan, Eric Hodel, Zed Shaw, and others. I had a chat with Zed about Mongrel-Java and we made some plans for how the future might play out. We agreed it would be best to just focus on getting 0.4 to work well and to explore the new Java support in Ragel. Zed also agreed it would be a good idea to provide a Mongrel gem that would work in JRuby. When Zed's ready to release 0.4, it's very likely there will be a new platform option: Java.

John Long - Radiant -- Content Managment Simplified

I mention Radiant not because it was a particularly stunning presentation or a particularly innovative application, but because it looked so polished...so businesslike. Other folks weren't terribly impressed, perhaps because it wasn't a lot of sound and fury. I was impressed for exactly the same reason: it's apps like this that are going to legitimize Ruby and Rails for prime-time business applications. We'll have to make an effort to get Radiant running in JRuby.

Tim Bray - I18n, M17n, Unicode, and all that

This was my second time watching Tim present. He does a great job of conveying a large amount of information in a very digestable way. His presentation helped everyone in the room understand a bit better how Unicode works, why it's important, and what Ruby and other languages can or should do to support it. I learned quite a bit myself.

Of particular interest were related events peripheral to the talk. Matz apparently was worried about what Tim might say, since so many folks are worried about Unicode support in Ruby and questions have been raised about Matz's m17n plan for Ruby 2.0. I think Tim was very equitable, however, laying out the facts and allowing Matz and everyone else to draw their own conclusions. Hopefully the community and the Ruby implementers will all take what they learned and run with it.

SASADA Koichi - YARV: on Rails?

I was pleasantly surprised by Koichi's progress on YARV. Getting Rails to run on an alternative implementation is no small feat, but that's exactly what Koichi demoed. Sure, it was just a simple scaffolded app, and much of Ruby's internals are still the same code under YARV, but it's excellent to see YARV starting to run real apps.

I was a bit disappointed, however, with the plan for native thread support in YARV. According to Koichi, the difficulty of native-threading Ruby will prevent full parallel thread support any time soon. Even when Ruby 1.9.1 with YARV merged is released in December 2007 (yes, that's right, over a year from now) it will only support one native thread running at a time. I understand Koichi has been given an almost impossible task, but I'd hoped that a year from now native threading would be real and robust.

Koichi did announce, however, that he would now officially have a fulltime job where one of his tasks is to work on YARV. He's got an office in Akihabara, and he sounded pretty excited about the future of his work. I talked with him a few times during the conference, and I think we're going to try to implement a YARV machine in JRuby. It seems to make sense, since Koichi has done much of the hard work determining an appropriate set of bytecodes and has a compiler already mostly working. Koichi is also excited about that possibility.

John Lam - You got your Ruby in my CLR!

John Lam is the creator of the RubyCLR project, a bridge from the Ruby runtime to the .NET CLR. He demonstrated how you can call CLR-hosted code from Ruby and implement interfaces that call back our of the CLR into Ruby code, much like JRuby supports today. He also demonstrated a very pretty Avalon-based Ruby console that works like IRB but with on-screen documentation and Intelli-Sense popup method completion.

John announced that he's been hired by Microsoft to work on their dynamic language initiatives--especially Ruby--so it seems he will be my counterpart there come January. john promised not to do evil, and he and I agreed we should not allow our differing employers to come in the way of providing world-class support for Ruby. I hope politics won't get in the way of innovation and creativity, as it so often does...and I believe that John agrees.

--

I'll check back in my next post with a summary of the implementer's summit and other excitement from the conference. This post is a bit long already :)

JRuby at Rubyists of Second Life TONIGHT

I should have tossed this in a blog entry earlier, but perhaps it's not too late.

I will be presenting JRuby to the Rubyists of Second Life this evening at 6PDT. Here's the announcement sent to Ruby-Talk, which has all the relevant info:

JRuby at Rubyists of Second Life

I hope you'll stop by and join in the conversation. I'll have a set of slides, but more importantly I'll be available for discussions on JRuby, Ruby, Java, and their future together. I'm "Headius Exodus" in SL. See you there!

JRuby on Rails: WEBrick vs AsyncWeb

Some fellow name TAKAI Naoto has posted an interesting comparison of JRuby on Rails running under WEBrick versus running under AsyncWeb:

TAKAI Naoto compares WEBrick and AsyncWeb and a comical translation.

Although the numbers he shows for WEBrick seem *awfully* slow (4-6 seconds per request...we haven't been that slow since JavaOne running Rails in "development" mode) what's most interesting is the speed gains he gets from AsyncWeb: something like a five times improvement. If we assume there would be an improvement running a more recent JRuby (0.9.1 should be considerably faster than all previous versions) and running Rails in production mode...well, this thing starts to look real-world ready.

He has a link to a snapshot...I'm investigating that now and will update this post when I know more.

See also Takai's post about about JRuby on Rails running under AsyncWeb:

TAKAI Naoto running JRuby on Rails with AsyncWeb and translation.

These sorts of things show real promise...AsyncWeb scales extremely well and is built upon Java's NIO library. A new contender enters the Rails front-ending competition!

Update: Ok, I've spent five minutes looking at the code, and it's cooler than I thought. He's got AsyncWeb and JRuby on Rails wired together using Spring, and it's a trivial amount of code to do it...like less code than WEBrick. I hope he's able to get a release out for this soon.

Here's a direct link to his rails-asyncweb snapshot. Super cool.

Another Year, Another Interpreter

It's been approximately a year since I launched into my first redesign of the JRuby interpreter engine, and a lot has happened since then. We've gotten Rails working as well as improving compatibility to the point where most other pure Ruby apps just work. We've filled out the base set of extensions and have active projects to complete what's left. We've built up a very active and intelligent community, all of whom are helping by contributing bug reports and patches. And perhaps most importantly for the project, Tom and I have been given the privilege of working on JRuby full time.

But none of that was because of the interpreter.

JRuby Interpreter History 101

The original JRuby interpreter, written by the original authors (Jan Arne Petersen and others), followed a pretty standard Visitor pattern. The AST nodes were visited in turn, and on each callback appropriate actions were performed. It was a perfect demonstration of Visitor in action, but the additional calls required to visit each node in turn badly impacted the Java call stack. At its lowest, before I began my work, JRuby on Windows under Java 1.4.2 could recurse no deeper than 300 levels in a simple single-recursive fib algorithm. And this while Ruby itself could go into the thousands.

However stack depth alone wasn't a good enough reason to work on a new interpreter. There were various ways we could clean up the stack size without a complete rework. Unfortunately, there were two Ruby features that at the time meant a new interpreter would be necessary: green threads and continuations.

Green Rubies

Ruby currently supports only green threading. The threading model is very simple; one native thread, the main process, pumps the entire interpreter. Once threads come into play, a signal or timer is set up using OS-specific calls. A new thread means a new set of global variables; variables that point into the thread's call stack, variable scopes, and current execution point. The signal/timer, once initiated, triggers a potential thread switch every 10ms by setting a global switch. On certain boundaries during execution, that switch is checked, and if it is set a call is made into the thread scheduler to set up the next thread for execution.

In short, a 10ms timeslicing green thread scheduler. And I believe that thread context switching is *voluntary*...it would be possible in a C extension or through specific Ruby scripts to prevent context switching at all.

But I digress. Green threading does carry with it some benefits. Because you are not dependent on the operating system for initialization and cleanup of thread data, you can manipulate threads much more easily. Ruby supports operations that native threading libraries forbid, like killing existing threads or stopping them indefinitely. Now it would not be such a problem if these operations were used sparingly, but during our testing we found that many libraries depend on them. So then we had a dilemma: how to support unsafe threading operations with safe native threads?

The initial solution, and the one we currently have in place, works reasonably well but could certainly be improved. On the same boundaries that the C implementation checks for a context switch, we check a set of runtime-global variables and locks. Various "unsafe" thread events trigger those locks and variables to change state. Kill sets a kill flag; upon encountering it the native JRuby thread will throw a ThreadKill exception. Entering a critical section sets a critical flag on all threads; when they reach it, they'll freeze in their tracks until the critical thread has completed. For the most part, the Ruby code behaves as it does in MRI, though backed by native threads. It's not foolproof, but it's sneaky and good enough for now. Emulation and simulation. Subterfuge and legerdemain.

A New Hope

When I launched into my redesign a year ago, I hoped to enable a different solution. I wanted to build a purely stackless Ruby interpreter that could support continuations and green threads. Because it would still run on top of a native-threaded VM, I planned to also support m:n threading, so JRuby could scale through a full range of threading options. And even though it was a very large task, the basic building blocks were already necessary: continuations would not be possible while deepening the Java stack (without really nasty tricks), and so green threading came along for the ride.

The first step was to make the interpreter stackless. This I mostly achieved by hacking the existing visitor into individual pieces. Each method that had previously been called during visitation was encapsulated in an Instruction implementation. The state associated with the interpreter was put into an EvaluationState object. Each instruction, when called, would receive an instance of EvaluationState and an InstructionContext that was usually just the AST node being traversed. The visitor now became a factory for Instructions; as it visited the AST nodes, instructions were pushed down into an instruction stack in the EvaluationState object; the previous deepening of the Java stack being emulated via a soft stack. As traversal went deeper, both the visitor and the instructions themselves would push more instructions down onto the stack. Execution meant popping the top instruction, running it, and returning back to the interpreter loop. An instruction could affect the state of the runtime or change the flow of execution by conditionally pushing further instructions on the stack. It was a fairly clever way to directly translate the old code into a stackless design.

As of this past spring, perhaps 80% of the AST nodes were fully represented with what I called "collapsed" instructions. This basically meant that for much of the AST, execution could proceed without the Java stack deepening. The tradeoff was in the additional overhead of maintaining a soft stack in the EvaluationState and a considerably more complicated interpreter design. But it worked, and after several months of tweaking it was faster than the original. Even better, it greatly improved our maximum stack depth.

The next step, if had we taken it, was to begin the painful task of making the rest of the interpreter stackless. The design I had in mind would have called for all Java-implemented methods to trampoline; in other words, calling a method would mean that whether it made an additional call or returned, control came back to the caller. The caller, in that design, would have been the overarching ThreadContext. Execution of a thread would begin by pointing a new ThreadContext at a starting node in the AST. As the thread executed, the stackless interpreter would maintain the Java stack depth at the ThreadContext level. Calls into Java-based methods would dispatch back to ThreadContext for deeper calls, so that any deepening of the Ruby stack did not actually deepen the Java stack. At any given time, ThreadContext could be frozen; the native thread pumping it could then be passed off to another ThreadContext, or used for other purposes. And of course, continuations would have been trivial to implement with this model, since the current running state of a ThreadContext could be saved at any time.

Whew! It was an ambitious goal, but with the mostly-stackless interpreter in place we were well on the way. I even had designs for how to compile Ruby code into stackless Java methods...so stackless that you could even pull off a continuation in the middle of a call. Like magic!

A Fork in the Road

So what happened? Sometime around JavaOne we heard about the Ruby KaiGi in Japan, a Ruby conference or get-together of some sort. If RubyConf is the big conference for us Westerners, this at least provided a mid-year update for English-speaking Rubyists. Matz was there, Koichi was there, and I believe other Ruby dignitaries made the trip as well.

And then Matz and Koichi dropped the bomb: Ruby 2.0 would support neither continuations nor green threads.

As many of you know, Koichi's next-generation Ruby interpreter engine, YARV, has been coming along in fits and starts. Because of the desire to keep existing extensions working and because of a lack of resources to work on Ruby internals, YARV has had many difficult problems to overcome. How do you write a next-generation interpreter without actually changing how the runtime works? Is it possible to do it without a new threading implementation, a new memory manager, a new garbage collector? Can you keep all your internal APIs exposed to C extensions and successfully migrate to a new interpreter design? I think the answer is that yes, you can, but it's really, really hard. And along with the announcement about continuations and green threads came the YARV/Rite (Ruby 2.0 + YARV) beta timeline: no earlier than Christmas 2007.

So we were faced with a decision. Do we continue along the much more complicated path toward green threads and continuations, knowing that they will eventually be unsupported by the language? Is it worth the effort to support them now if they'll go away?

We stewed over this decision for a week or two. We had been making all the right moves toward a stackless design, and were on the cusp of launching into the next set of battles. But it was a painful process.

Pragmatism

Eventually we decided that under the circumstances, the Ruby and JRuby communities would be better served by having a solid, native-threaded, continuation-free implementation on the JVM. We were unable to find any use of continuations in the most popular applications, and it seemed that very few people had a good reason to use them. In addition, there was growing concern over Ruby's lack of support for native threads; so there too was a opportunity for JRuby to shine.

Very little of the interpreter changed over the summer. We worked furiously in our spare hours getting Rails running, and were able to present it in a primitive form at JavaOne. I started work on a number of traditional compiler designs for JRuby that showed potential gains of 50-75% over the existing code. And most importantly, our community, compatibility, and test library all began to grow rapidly.

Toward the end of the summer, it became likely that we would join Sun Microsystems as full-time JRuby developers. Sun had wisely taken an interest in Ruby along with other scripting languages, and because we had shown some of the potential for Ruby on the JVM, they asked us to join their team. And just four weeks ago, I became a Sun employee.

JRuby Now and Into the Future

So where has all this led? JRuby has been getting more and more attention from folks within Sun, Rubyists around the world, and especially from Java developers anxious to escape from their Java-only prisons. Our compatibility is increasing faster than before; we've had over a hundred new bugs reported in the past few weeks...almost all of them with community-contributed patches. We have added our first non-Sun team member Ola Bini, a star of the JRuby community who has proven his dedication to making Ruby on the JVM succeed. And we have started to solidify our short-term goals for the project.

The primary goal remains the same: JRuby should be as close to 100% compatible with Ruby 1.8 as possible. Today, we are doing extremely well in this department. Rails generally works without issues, and most pure Ruby applications run without modification. There's still plenty of edge cases to iron out, but we're moving very rapidly now. Getting Rails to work as well as it does is already a major achievement.

We have also started to iron out what a JRuby 1.0 release should look like. A few major points come up again and again:

• Compatibility should be such that we can safely claim "Rails is supported"
• Java integration should look like we want it to look for the future, and should be performant, lightweight, and seamless
• All major codebase refactorings should be complete; this includes a solid design for wiring up Java-based method implementations, external extensions, and IO channels
• Unicode should be supported out-of-the-box, giving Ruby code access to everything Java is capable of
• Threading should work perfectly, both for JRuby-launched threads and for "adopted" threads from outside the runtime
• Performance should be markedly improved

This last bullet may sound a little less ambitious than the others. Performance has been a major concern of mine, but it's also the most difficult goal to quantify. If JRuby runs Rails almost as fast as Ruby, is that enough? Well, perhaps, but Rails is a very IO-intensive application, so it's not a particularly good measure of core JRuby performance. If JRuby is many times slower than Ruby for certain command-line tools, should a release be delayed? Perhaps, but it's not so cut-and-dried; a JRuby that's slow on the command-line may perform exceedingly well once hosted on a long-running server.

There is one piece of the puzzle, however, that performance neatly excludes: vastly complicated stackless green-threaded continuable interpreter engines.

Late-Night Hacking

We had basically made the decision not to continue down the green threading path this summer, but we had made no effort to reverse the work I had already done. We always suspected that a more traditional engine would perform better, but attempting another redesign was a very large task nobody wanted to tackle.

Last night I got bored of hunting around for juicy performance tidbits. I wanted to tackle something bigger.

About 9:00PM on Thursday, I started rewriting the interpreter engine to be a more straightforward switch-based affair. Instead of Instruction stacks and EvaluationState, or even Visitors, it would be a fast, concise switch statement, recursing as the AST deepened and affecting state changes along the way. I felt a certain sadness ripping up the old interpreter; I had spent weeks on that code last year, and was proud of what I'd accomplished. It's no small feat to turn a recursive, visitor-based interpreter into a stackless, instruction-based machine. Unfortunately, I knew that design would never serve JRuby the way we needed it to. It was designed for another purpose, and its complexity and cost gained us very little in the long term. And as of 9:00PM Friday, after working furiously for 24 hours, it was gone.

The new interpreter is, as I stated, a large switch statement. Each AST node is assigned an int, so the main "eval" call can quickly determine the correct code to execute for each. The new design does result in faster deepening of the Java stack (about a 15-20% hit to fib max-depth), but it still performs far better than the original visitor-based implementation. And although we'd only theorized up to this point, it does perform a good bit better than the Instruction-based engine.

Rake Installation Performance

Pretty much the first thing anyone does with Ruby is install some application or tool they need. And nine times out of ten, they install it with RubyGems.

One of our favorite benchmarks is a local RubyGems install of Rake, Ruby's answer to make. For reasons which are not yet clear, the documentation-generator RDoc--which is called as part of a RubyGem installation--performs extremely poorly under JRuby. RDoc is basically implemented as a series of source-code parsers that generate documentation based on special comment tags embedded in the code. And it's written in pure Ruby, so it's a good benchmark for JRuby.

We've been steadily improving performance. The following sets of numbers show our progress, all the way up through teh new interpreter:

JRuby 0.9.0, stackless interpreter, Java 6:
real 1m 58.465s
user 2m 1.671s
sys 0m 2.625s

JRuby 0.9.1 current trunk code, stackless interpreter, Java 6:
real 1m 10.488s
user 1m 13.075s
sys 0m 2.013s

JRuby headius branch, new interpreter, Java 6:
real 1m 0.489s
user 1m 5.220s
sys 0m 1.849s


What's Next for Performance

The numbers above spell it out pretty clearly; we've managed to double performance since the 0.9.0 release at the beginning of the summer. We've also managed to do that without writing a compiler and with a vast number of optimizations still on the table. Things are looking very good. Among future performance-enhancing changes:

• Reducing the cost of method calls. Currently a given method call generates an absurd amount of transient objects, like arrays copied into ArrayLists unwrapped into arrays, again and again. We must clean up the call path to minimize object churn.
• Building more pre-optimization into static structures in the system. Many areas of the system are cloned or regenerated again and again without ever changing. Blocks, for example, have only 5 mutable fields...but we clone the entire block for every invocation. By saving off the static bits of the system once, we lower the cost of all operations.
• ThreadContext is still alive and well; it has always represented the Ruby thread within the JRuby runtime, holding stacks for framing calls, scoping variables, and managing blocks. Unfortunately, the only way to access it is through a moderately expensive threadlocal call, and believe me we hit that call hard. Part of the new interpreter design helps limit those calls; further work in the rest of the runtime will help eliminate them.
• The bytecode compiler *will* happen. It's been on hold primarily because the runtime is still evolving. Because even compiled Ruby code will likely still have ties to the JRuby runtime, we need to first iron out how the runtime should look and act long-term.

I've had a great time this past year working on JRuby, and things are really starting to pick up speed. I'm sure the next year's going to be even crazier...especially once JRuby 1.0 comes around the corner.

Thanks all for your support. I'm having a blast.

New JRuby Syntax for Importing Java Classes

We've added a cleaner, simpler syntax to JRuby for including Java classes into a given namespace. It stemmed from a response of mine to a user on the JRuby mailing lists, and has been committed to trunk already. Almost everyone on the list agreed this new syntax is better.

Tom sums up the history of JRuby's Java include syntax in his blog post, but here's the executive summary.

Where before you would use this:

include_class "java.util.ArrayList"
include_class("java.lang.String") { |p,n| "J" + n }

You now can simply do this:

ArrayList = java.util.ArrayList
JString = java.lang.String

And this syntax also works directly; no assignment to a constant is required:

class MyClass < java.awt.event.ActionListener
...
end

Pretty slick, if you ask me.

Ola Bini Joins JRuby Development Team

We're proud and excited to announce that Ola Bini has officially joined the JRuby development team. Ola has proven to be one of the most valuable community contributors to JRuby, and he's shown that he's dedicated to making Ruby on the JVM a reality. He's the sole creator of the RbYAML and JvYAML projects that have enabled high-performance YAML support in JRuby, he single-handedly implemented Enumerable in Java, he has been the primary developer behind our Zlib support, and he's contributed numerous patches and fixes over the past 8 months.

Thank you Ola for all your help, and welcome to the team!

Performance Progress Since 0.9.0

I love comparing current trunk against 0.9.0. It always makes me happy.

Here are the numbers for a local gem install rake. The second set of numbers is trunk plus a ThreadContext-passing experimental patch I'm working on. Without my experimental patch, trunk benchmarks about four seconds slower.

0.9.0:
real 2m1.867s
user 2m2.158s
sys 0m2.768s

trunk + my patch:
real 1m22.221s
user 1m22.322s
sys 0m2.813s

That's an increase of 32%. Are we having fun yet?

The best part of this is that we haven't even started on a number of very promising performance changes, and we're still not compiling anything. Nibbling, nibbling away at overhead is paying serious dividends.

More People Who Get It

There's been some good blogging lately related to dynamic languages and their future in a Java-based world. I'm glad there's smart folks out there who really get it; the JVM is a great piece of engineering, and the Java platform is as rich and feature-complete as anything out there. Combined with dyntyped languages, anything's possible.

Steve Yegge: Ruby and Java and Stuff

Steve calls out some of Ruby's gems, Java's failings, and seems to agree that Sun's making the right move pushing for broader language support on the JVM. Oddly enough, he never calls out JRuby specifically, even when mentioning Ruby, Java, and JVM scripting in the same paragraph. www.jruby.org, Stevey.

Rick DeNatale: Performance Anxiety

Rick sums up a post on performance by John Duimovich nicely: dynamic languages absolutely do not have to be slow, and even a language as dynamic as Ruby should perform far better than it does today. Perhaps YARV will be the answer in the C world. I'll work my fingers to the bone making sure JRuby is the answer in the Java world.

Alex Russell: The Platform Strategy

Alex sees what the world is starting to see: that Java the platform was the reason it won, not Java the language. The indivisible trinity of platform, language and JVM is a thing of the past, and developers will be better off as a result. Alex is also right about the JRuby hiring by Sun; it's a good first step that must be followed up by continuing and increasing action. You have my promise to do what I must to ensure that happens.

Why Add Syntax When You Have Ruby?

A question came up this morning on the JRuby Users mailing list about a feature Jython and Groovy support: constructors that initialize bean attributes. The example given used a JLabel:

JLabel(text, bounds = bounds, foreground = color)

Which ends up being roughly equivalent to:

x = JLabel(text)
x.bound = bounds
x.foreground = color

Groovy has a similar syntax I won't illustrate here. So why doesn't Ruby support this, or perhaps why doesn't JRuby automatically support this syntax for Java types?

To answer, let's take a look at what it would take to add this to current Ruby with only existing features.

The equivalent syntax in Ruby or JRuby might look like:

SomeClass.new(text, :bounds => bounds, :foreground => foreground)

...or possibly using a block as in:

SomeClass.new(text) { @bounds = bounds, @foreground = foreground }

However there's no existing accomodation in the semantics of Ruby for these syntax to work out of the box. It would not be hard to write a short bit of code to allow constructing objects in this fashion, of course (using the block approach as an example):

class Class
 def construct(*baseargs, &initializer)
   x = self.new(*baseargs)
   x.instance_eval(&initializer) if initializer
   x
 end
end

...which would allow the block example above to work fine (with "construct" in place of "new"). For proxied Java objects, which don't actually have ruby instance vars (@whatever above) it would have to be a slightly different call:

JLabel.construct(text) { self.bounds = bounds, self.foreground =
foreground }

This works fine, but it's perhaps a little less beautiful. What about this impl instead?

class Class
 def construct(*baseargs)
   x = self.new(*baseargs)
   yield.each_pair { |k,v| x.send("#{k}=", *v) } if block_given?
   x
 end
end

Which would allow a variation:

JLabel.construct(text) {{ :bounds => bounds, :foreground =>
foreground }}

(the double {{}} is intentional; the block returns a hash of initializers)

The bottom line is that this kind of syntactic sugar in other languages can easily be added to Ruby through various techniques, and so features like field-initializing constructors don't need to be part of the core language or any of the implementations.

Does anyone still wonder why I love this language?

Interface Implementation Syntax and Open Classes

Tom and I got together and brainstormed the interface implementation issue today. We think we've come up with a reasonably solid solution.

First some background.

As most of you will know, the current way to implement a Java interface with Ruby code is to extend it:

include_class "java.awt.event.ActionListener"

class MyListener < ActionListener
...
end

This works fine for many cases, and it's great for a simple single-interface implementation. However it breaks down if you want to extend either a Java or Ruby class at the same time or if you want to implement multiple interfaces.

A few weeks back, we on the JRuby dev list kicked around the idea of using mixin
inheritance to do interface implementation:

class MyListener
  implement ActionListener
end

This has the advantage of allowing you to also extend a class and implement multiple interfaces, but there's a problem here. By the time we encounter MyListener, the class is already created and there's no opportunity to make such drastic changes as modifying the list of implemented interfaces.

In the case above, MyListener is already created as a pure Ruby class by the time we encounter the implement line...we can't then change it into a Ruby/Java proxy class. Even if we had a way to mark it ahead of time as a Java proxy, that proxy would have to be created already by the time we're in the class body. Ruby's unusual way of instantiating classes is to blame: all classes start out "blank" and the class body is basically eval'ed within that blank instance. With Java types, we do not have such flexibility.

So a new option comes into the debate today. It's not as clean, and it's not as Rubyish, but it should support Java typing and Java interface implementation very well:

include_class "java.util.AbstractList"
include_class "java.util.Map"

MapList = AbstractList.implement(Map)

class MyMapList < MapList
...
end

Or the shortcut version:

class MyMapList < AbstractList.implement(Map)

We will probably also continue to allow the single-inheritance shortcuts as well, since they're nice and clean:

class MyListener < ActionListener
end

...which is roughly synonymous with:

include_class("java.lang.Object") { |p, n| "J" + n }

class MyListener < JObject.implement(ActionListener)
end

The logic behind this approach (very similar to that being taken by RubyCLR and IronPython) is that a concrete class plus multiple interfaces as a whole represents a very rigid, specific type in the Java world. We do not have the flexibility to juggle the internals of those types after they're created, so having a very clear-cut way of specifying that combination of concrete + interfaces allows us to satisfy Java's typing requirements. We can then extend that with Ruby code, implement whatever we want, alter behavior, reopen classes, and so on. We're essentially creating a rigid top-level Java type with a "back door" for implementing its behavior with Ruby code under the covers.

We'll certainly want to try to coordinate with other projects addressing this same issue (Ruby.NET, RubyCLR, IronRuby, IronPython, Jython?), since we don't want multiple incompatible syntaxes for this stuff. You out there guys?

RailsConf Europe 2006: JRuby Slides and Demo Posted

I have posted my slides and demo on headius.com at the following URLs. The demo has some very basic instructions for getting it to work, but I don't plan to support this code in any way. It was fairly hacked together in the days before the conference, so you're on your own getting it working. Whether you can run it or not, however, it's fairly illustrative in how it accesses EJBs, javax.persistence entities, and JMX resources. A taste of things to come.

JRuby On Rails - RailsConf Europe 2006 (Keynote, original) Update: Fixed the URL
JRuby On Rails - RailsConf Europe 2006 (PDF)
JRuby On Rails - RailsConf Europe 2006 (OpenDoc Presentation)
JRuby On Rails - RailsConf Europe 2006 (PowerPoint)

JRuby on Rails Demo (.tar.bz2)

Conference Updates

Ok, so this post is as much for me as for you (so I have a post to refer back to) but some of you may be interested to know what conferences I'll be at. Things have gotten a bit busier since the Sun announcement.

BarCamp Milwaukee - September 30 - October 1, Milwaukee WI

I may make the trip down to Milwaukee for their upcoming BarCamp. Hopefully others from the cities will be willing to make the trip with me (and drive :), or I probably won't go.

RubyConf 2006 - October 20-22, Denver CO

Not presenting, but I'll be there and will probably try to organize a lightning session or something.

Gateway JUG - November 7, St Louis MO

The Gateway JUG has offered to fly me in for a JRuby-fest at their November 7 meeting. It ought to be a good sharing of ideas and meeting of the minds. Try visiting the Gateway JUG website if you might be interested in showing up for the meeting.

TC Code Camp - November 11, Bloomington MN

I'll be presenting JRuby and JRuby-related topics to coders at Microsoft's Bloomington campus.

JavaPolis 2006 - December 11-15, Antwerp, Belgium

Tom and I are officially on the schedule to present JRuby (on Rails, perhaps) at JavaPolis 2006. We'll follow that up with our Great Belgian Beer Tour, sampling brews in the Belgian backcountry. What could be better than that?

A Taste of Celebrity...and Pizza

I'm still decompressing a bit after a great conference and 14 hours of transit from London to San Jose, but blogs are starting to trickle in about RailsConf. Here's a particularly notable entry.

I'm still not entirely sure why he wanted a picture of me with his (girl)friend. Maybe it's a German thing?

I promise I'll get around to blogging more tomorrow. Time to de-lag and get some sleep before I'm pummeled with questions from Sun language and VM experts on Monday.

JRuby Get-Together Tonight in London

Yes, today is the day! We will be getting JRubyists together at the Fitzroy near the RailsConf conference center.

Here's a gmap to the Fitzroy. Show up around 6:30-7:00PM and I'll be there!

Pat Eyler: Gathering Reactions to the Sun Move

Pat Eyler has a good wrap up of some reactions around the Net to the recent Sun move.

More FAQs about JRuby and Sun

There's been a lot of buzz about Tom and my new jobs at Sun, and understandably so. The response so far has been overwhelmingly positive, and we're grateful for that. It's all very exciting.

There have been a few questions to come up again and again on blog postings and mailing lists, however, that I figured I could answer. Some are fear, some are uncertainty, and some are doubt, but almost all have good answers. I understand some folks have concerns and want to address those as best I can.

Note, my responses may not be Sun policy, but they're my policy. And I don't bend so easily.

1. Does the Sun move mean Groovy, Jython, BeanShell, and friends are being cut out of the picture? Has Sun chosen a winner in the dynamic languages realm?

Absolutely not, on both counts. I got involved in JRuby for one reason: Ruby was underrepresented in the Java world, and happened to be a very attractive language to me. Jython was fairly well-established and performed quite nicely. Groovy was gaining some traction and seeing an upswing in developer interest. JavaScript was scheduled to be included in Java 6. What about Ruby? JRuby didn't run most Ruby apps, had known major incompatibilities with C Ruby, and performance was very poor. Something had to be done.

I like to solve difficult problems, and JRuby certainly has presented some difficult problems. How do you take an existing interpreter that mostly works and--without breaking it--completely refactor and rewrite its internals to maximize performance and compatibility? Answer: very carefully. JRuby has come a long way this past year, and now we have leave to round things out on an accelerated schedule.

So what then? Naturally there's the tools angle (which I'll address in a moment), but will a JRuby 1.0 release mean the end for JRuby and continuing development of dynlangs on the JVM? Of course not! JRuby will continue to be a learning experience for all involved...inside and outside of Sun. Once we've solved JRuby's issues, why not find a way to raise all ships? Support for invokedynamic, open classes, and closures at the VM level? Hooks for code generation, alternative typing systems, and deeper threading, IO, and memory integration? Seamless cross-app connectivity for dynlangs + Java EE with all the tasty agility we've come to love?

This is only the beginning, my friends!

2. Sun has stated that Charles and Thomas have a mandate to think about/work on developer tools; does this mean Eclipse-based solutions are off the support list?

I want to make this perfectly clear...we will not discriminate between any users of JRuby. JRuby is a community project, first and foremost, and part of cultivating and growing that community is a rigid adherence to non-discriminatory support policies. We would be cutting off our right hand (or in my case, left hand) if we started picking and choosing who "deserves" support and help from the JRuby team. Nobody would benefit and everyone would suffer from such a move.

Of course, as Sun employees, we've got a major interest in ensuring NetBeans provides a world-class IDE for Ruby projects on the Java platform. But we were JRuby developers first, and we know that although we want people to migrate to NetBeans, we'll engender no community love by giving existing, mature Ruby IDE solutions the boot. Remember folks, competition is good. It would just be entirely unfair to deny NetBeans users the distinct joy of Ruby and Rails development. We're going to spread the love.

3. Will Charles and Thomas have to move to California?

Not at all. Tom and I will continue working from the Minneapolis area under Sun's remote-employee programs. You'll certainly see me and sometimes Tom at your favorite Ruby or Java conferences, but we'll continue to make home in flyover country. Moo.

4. Was this move in response to the IronPython 1.0 release this week?

Come on folks, you know how fast HR moves. You honestly think they could officially hire us in less than a month? This move has been in the making for several weeks..and man has it been hard to keep secret. I'm glad the truth is finally out and we can take a deep breath...before really getting down to business. (And really, for a move of this magnitude to only take "several weeks" is impressive by any measure...)

I wish the IronPython guys success...and Jim Hugunin is obviously in a similar position, doing what he loves. Hopefully our differing sponsorship won't get in the way of cross-pollenation and idea sharing. It would be a real shame if being hired to innovate by innovative companies actually hurt the innovation that got us here in the first place.

--

I'll plan to keep up with major questions as they come up. I really want this to be full disclosure, folks, as much as humanly possible. JRuby isn't my project or Tom's project--it's yours. You deserve to be in the loop (zero-day media blitz secrets notwithstanding, of course ;)

JRuby Steps Into the Sun

It's going to be a big news day for JRuby :)

The two core JRuby developers, myself and Thomas Enebo, will become employees at Sun Microsystems this month. Our charge? You guessed it...we're being hired to work on JRuby full-time.

Perhaps some of you may have seen this one coming. For others, it may be a pleasant surprise.

The primary goal is to give JRuby the attention it really needs. The potential for Ruby on the JVM has not escaped notice at Sun, and so we'll be focusing on making JRuby as complete, performant, and solid as possible. We'll then proceed on to help build out broader tool support for Ruby, answering calls by many in the industry for a "better" or "smarter" Ruby development experience. I'm also making it a personal priority to continue growing the JRuby community, foster greater cooperation between the Java and Ruby worlds, and work toward a "whole-platform" Ruby-on-JVM strategy for Sun.

And yes, JRuby will remain as open source as it is today. It just might start moving a bit faster (as if it weren't moving fast already!)

You can imagine how excited I am about this opportunity, and how pleased I am to know that Sun takes Ruby so seriously. Not only will I get to work on the project I've poured my heart into this past year, but I'll be able to do it while helping one of my favorite companies turn a technological corner. Naturally I've been talking with a myriad of folks at Sun over the past several weeks, so believe me when I say these guys really get it. The tide has turned and dynamic languages are on everyone's agenda. It's going to be quite a ride.

Thanks very much to Tim Bray, who has proven a fearless advocate for Ruby and dynamic languages at Sun. Thanks also to Gilad Bracha, Neal Gafter, James Gosling, Peter von der Ahé, and others fighting to improve Java and make the Java platform a dynlang-friendly place. Thanks also to my Sun interviewers for their excellent, thoughtful questions and ideas about the future of JRuby and dynlangs on the JVM. I'm looking forward to working with you all.

And finally, thanks to all you who use, contribute to, and blog about JRuby. You're a huge part of this, and I want you all to join us on this great Ruby adventure. Community involvement has gotten JRuby where it is today, and community involvement is vital to making Ruby on the JVM a reality.

Updates: I'll list substantive news items and blog postings as they come in (and I'll let your RSS aggregators deliver the one-liner posts ;)

Tim Bray: JRuby Love - Tim lays out some informative FAQs on the Sun move. He was our biggest Sun advocate, so he should know.

InfoQ: Sun Officially Back Ruby, Brings JRuby In-House - Includes a link to the newly-posted presentation we taped for InfoQ this past spring.

Thomas Enebo: JRuby Shines in Sun - My co-conspirator's almost-as-cleverly-named post about the event.

Bloglines Aggregation of JRuby articles - The blogosphere is very busy today!

InternetNews.com: JRuby Developers Join the Sun Fold - Andy Patrizio talks to Tim Bray and I about what this means for JRuby and for Sun.

InfoWorld: Sun picks up a gem in JRuby - Mostly a blog wrap-up, but a few good links and quotes.

ActiveRecord-JDBC 0.2.1 Released!

I told you it would only be a couple days!

ActiveRecord 0.2 has been released. (It's actually 0.2.1 because of a minor Gem glitch)

Ola wraps it up in the link above, but for the record it supports normal AR operations and basic migrations for the following databases:

• MySQL
• PostgreSQL
• Oracle
• HSQLDB
  
• Microsoft SQL Server (except for change_column_default)
• DB2 (except change_column, change_column_default, rename_column, remove_column,
  add_index, remove_index and rename_table)
• Derby (except change_column, change_column_default, remove_column, rename_column)
• FireBird (except change_column_default and rename_column)

Whew! And we really only started getting migrations and additional databases into the mix a week or two ago. Things go very fast when you have JDBC to handle all the heavy lifting!

I'm going to demo the AWDwR2 Depot app running on top of MySQL at Rails, and probably demonstrate one of the other databases Rails doesn't normally support as well (Derby perhaps? or SQL Server if I can get Parallels up before presenting?)

Usage

Ola has a Derby walkthrough with some unusual typos [Update: my mistake, he symlinked stuff like "gem" to "jem" to avoid collisions with MRI], but here's my walkthrough for MySQL in ultra-condensed form:

1. Install JRuby from trunk or my branch
2. Build it (not necessary once we get another release out)

   ant clean jar

3. Set up env vars for JRuby

   export JRUBY_HOME=[jruby dir]
   export PATH=[jruby dir]/bin:$PATH

4. Install Rails

   gem install rails -y --no-rdoc --no-ri

5. Install ActiveRecord-JDBC

   gem install activerecord-jdbc --no-rdoc --no-ri

6. Generate a Rails App

   rails ~/testapp

7. Go to the app

   cd ~/testapp

8. Modify database.yml

   development:
    adapter: jdbc
    driver: com.mysql.jdbc.Driver
    url: jdbc:mysql://localhost/testapp_development
    username: ...
    password: ...

9. Modify environment.rb:

   require File.join(File.dirname(__FILE__), 'boot')
   require 'active_record/connection_adapters/jdbc_adapter'

10. Add driver to CLASSPATH

    export CLASSPATH=$CLASSPATH:mysql.jar

11. Create a testapp_development database, grants for testapp user, and widgets table in MySQL (use migrations if you like)
12. Scaffold Widgets CRUD

    jruby script/generate scaffold widget

13. Start up the server

    jruby script/server

And that's about it. You've got a scaffolded widget page running in JRuby over JDBC to MySQL.

Pretty neat, eh?

Troubleshooting

• ActiveRecord-JDBC does not install: I had some trouble myself; I think something's up with either RubyForge or with our released gem. You can always download ActiveRecord-JDBC and install it locally, of course.
  
• scaffold fails with an error about "nonexistent jdbc adapter": Ensure you've added the require line to environment.rb (hopefully this will be unnecessary in the future)
• scaffold and script/server terminate without running: Make sure you've successfully installed the ActiveRecord-JDBC gem. The additional require in environment.rb causes Rails scripts to die silently if there are any errors.
• scaffold fails with the error "cannot convert NilClass into String": Make sure you've correctly specified the driver and url lines in database.yml
  

What's Next

We're going to continue adding database support, especially if you folks can contribute migrations impls for your database of choice. It's fun and easy!

We've also got a patch under discussion to allow adding JDBC database jars to [RAILS_ROOT]/lib and have the driver pick them up automatically. This could obviously be extended to Rails at large for service client jars, persistence framework jars, and so on. No more CLASSPATH manipulation!

We're hoping to get the Rails guys to include "jdbc" in the list of supported adapters out-of-the-box, so that environment.rb require is not necessary.

We're also going to see if there's a way to abstract out all the DDL code in core Rails, so we can all benefit from the same code.

More ActiveRecord Information

Supporting More Databases

A number of folks have asked about support for database X. We want to support all possible databases, of course, and JDBC gives us a huge head start on that. Many databases will only need a bit of tweaking to support all the basic CRUD operations. The additional work...small though it may be...is implementing the schema-manipulating methods of ActiveRecord. Here's a complete list...if you can provide examples for each operation, it would speed our supporting them:

• add_column
• add_index
• change_column
• change_column_default
• columns
• create_table
• drop_table
• initialize_schema_information
• native_database_types
• remove_column
• remove_index
• rename_column
• rename_table
• structure_dump
• table_alias_for
• table_alias_length

Check available docs for ActiveRecord or look at the existing adapters in ActiveRecord in SVN for examples on each.

Avoiding Modifying active_record.rb

There's now a thread going on the Rails core mailing list and a bug to address the issue of adding adapters to Rails. Hopefully one of two things will come out of this:

• "jdbc" will be added to the list of adapters available in core Rails
• a mechanism will be added (or fixed) to allow adding to the list of adapters from outside the core source

The former would be ideal, since no additional, special config changes would be needed for jdbc versus MySQL, for example. The latter would be an "ok" compromise, but would require modifying all apps to add "jdbc" to the adapter list. I'll report back when something comes of it.

Java Regex Broken for Alternation over Large Strings

This is a serious bummer.

I've been working through some of Rails's test cases this evening. Since I've got Depot running reasonably well now and I know I can easily wire together a simple Rails app that calls Hibernate or EJBs, I figured I'd try to get a better handle on the true state of Rails support.

As I'm running test cases--the majority of which appear to be passing, btw--I came across a couple goofy regex issues. The first is one we'd seen before, where dot refuses to match \r. In this case, adding DOT_ALL to the Pattern's flags seems to resolve it. I think we just hadn't noticed that flag before, so many other similar bugs might be resolved the same way. Not an easy issue to narrow down, but I found it and moved on.

So then I started seeing SystemStackErrors popping up within the CGI library. When I see stack overflows in JRuby, I usually figure it's one of two things:

1. Something's wrong with JRuby that's breaking a termination condition in Ruby code, so a recursion is never ending
2. JRuby itself is missing a termination and recursing endlessly

Pretty simple issues to track down...find where the recursion is happening, and figure out why it's not terminating. Except in this case, the recursion wasn't in either Ruby or JRuby code. It was in Java's regex library.

Blowing The Stack

O Java, how I love Thee. Every library I could want, all in one place. You care for me so well. It pains me to find your faults.

So it appears that the Java regex library's compiled patterns execute recursively. This alone wouldn't be particularly unusual, especially if the recursion was limited by the complexity of the regex itself. However in this case it seems the recursion is a factor of the string to match, rather than of the expression complexity. With a large enough input string, I can get Java's regex to blow the stack every time.

regex = /\A((?:.|\n)*?)?([\r\n]{1,2}|--)/n
long_string = ""
76.times { long_string += "xxxxxxxxxxxxxxxxxx" }

long_string.sub(r) {""}

This bit of Ruby code will cause java.util.regex.Pattern to blow the stack during matching. I'm sure this regex could be reduced to a simpler case, but I don't really need to go through the exercise of doing so; others have reported the same issue:

• Pattern.matches throws StackOverFlow Error - Closed, will not be fixed (reported in 2004)
  
• java.util.regex routines should use less stack space - In progress, request for enhancement (reported in 2002)
  

So the issue stands, unfortunately. I'm not sure I'd characterize it as an RFE...I'm no regex expert, but I would hope my regex engine wouldn't overflow based on large input, especially if that input was completely bogus as in my example above. I don't think that's unreasonable, especially considering that Ruby's regex engine appears to handle this case (and considerably larger cases) just fine. Correct me if I'm expecting too much.

So what's the damage? Well, since we currently use java.util.regex exclusively, and since Ruby libraries and apps very frequently use regex to match against very large strings, we'll have to migrate to a different regex library. Until that happens, however, Rails under JRuby will not support large multipart posts, at a minimum. There may be other places this has an effect, but multipart posting was the area I investigated.

So for now...I guess it's just busted. We'll begin evaluating alternative regex libraries posthaste. Any help from you folks would be appreciated...just take that regex or one from the bugs and try to match against extremely large input.

Rewrite the Regex?

Some may say that the regex should be rewritten to avoid the use of alternation. Perhaps so, perhaps there's a better way. However Ruby handles it fine, so requesting that the Rails folks or any other Ruby devs start tailoring their regex so they will work under JRuby is quite out of the question. If we can't run the same regex against the same input, we need a different regex library...that's all there is to it.

Besides, the bugs posted have additional regex that are even simpler. It just seems to be a limitation of the Java regex library that certain regex will blow the stack on large input.

For the record, here's the relevant bit of the stack. I get pages and pages of this, under both Java 5 and Java 6 beta 2...both 64-bit under Linux:

...
        at java.util.regex.Pattern$Branch.match(Pattern.java:3998)
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4052)
        at java.util.regex.Pattern$LazyLoop.match(Pattern.java:4241)
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4111)
        at java.util.regex.Pattern$BranchConn.match(Pattern.java:3962)
        at java.util.regex.Pattern$CharProperty.match(Pattern.java:3314)
        at java.util.regex.Pattern$Branch.match(Pattern.java:3998)
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4052)
        at java.util.regex.Pattern$LazyLoop.match(Pattern.java:4241)
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4111)
        at java.util.regex.Pattern$BranchConn.match(Pattern.java:3962)
        at java.util.regex.Pattern$CharProperty.match(Pattern.java:3314)
        at java.util.regex.Pattern$Branch.match(Pattern.java:3998)
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4052)
...

What a downer.

Using the ActiveRecord-JDBC Adapter

Ola Bini, Nick Sieger, and others have been making great progress on the ActiveRecord-JDBC adapter lately. I figure it's a good time for a brief update on progress, and a short walkthrough on how to use it.

Where We Are, Where We're Going

There has been one official release of the ActiveRecord-JDBC adapter, a 0.1 that has basic SQL support for MySQL. It's available as a gem called "ActiveRecord-JDBC" (case-insensitive). It provides normal MySQL support for most major ActiveRecord operations today, and it's mostly the same code we used for our JavaOne demo.

Although the JDBC libraries give us plenty of information on database typing and precision, they provide practically nothing for generating DDL. Because of this, a large part of current ActiveRecord capabilities have to be written outside of the JDBC-aware code. DDL differs between databases, so like the core Rails project, we're forced to implement the various schema-related methods of ActiveRecord differently for different database vendors.

Currently, we're just reproducing the same code available in core Rails, with a little manipulation to work with ActiveRecord-JDBC, but we're hoping to work with the Rails team to abstract out the non-driver-specific portions of the core ActiveRecord adapters so the JDBC version can leverage the same code. I'm also hoping we'll be able to get the "jdbc" adapter name added to core Rails in the active_record.rb file where such adapters are listed. This will allow JRuby users to install Rails and the JDBC adapter and immediately start using them, without additional steps.

We are planning a second release of the JDBC adapter very shortly. It should have reasonably solid migrations/DDL support for MySQL, Oracle, and Firebird, as well as some level of support for HSQLDB, Derby, and others. It's moving along quickly now that it's been spun off into its own jruby-extras project. You can go there or to the jruby-extras SVN repository to grab the bleeding-edge source from SVN.

Getting up and running with ActiveRecord-JDBC

Despite the design complexities and our DDL woes, getting an ActiveRecord-based app up and running on JDBC is now very simple. This walkthrough proceeds from a stock JRuby install, and is based on current JRuby trunk (though it may work ok with 0.9.0 as well...we'll have a new release out soon).

1. Ensure JRuby is set up and working correctly via the normal setup procedures
2. gem install rails -y --no-rdoc --no-ri (ri and rdoc generation are still too slow under JRuby)
3. gem install ActiveRecord-JDBC --no-rdoc --no-ri (this is the 0.1 version for now, but hopefully we'll have 0.2 out this week)
4. Edit lib/ruby/gems/1.8/gems/activerecord.../lib/active_record.rb, adding jdbc to the list of RAILS_CONNECTION_ADAPTERS toward the bottom of the file:

   RAILS_CONNECTION_ADAPTERS = %w( jdbc mysql postgresql sqlite firebird sqlserver db2 oracle sybase openbase )

5. Modify database.yml for your settings. For example, using MySQL:

   development:
     adapter: jdbc
     url: jdbc:mysql://localhost/testapp
     driver: com.mysql.jdbc.Driver
     username: testapp
     password: testapp
   

6. As always, ensure the appropriate JDBC jar files are in your CLASSPATH

That's pretty much all there is to it. We'll get that second release out in a day or two so that migrations work and additional databases are supported, but these instructions won't change. I'll post here when that release is out.

How Well Does It Work?

It seems to work very well so far. As part of my JRuby on Rails presentation at RailsConf, I'll be demonstrating the new Depot application from Agile Web Development with Rails 2nd Ed. The entire app seems to work very well under JRuby, and I only had to make one additional manual change in the database because JRuby doesn't yet work well with edge rails (on which the book's examples are currently based). It's quick and responsive, and will only improve as we continue on.

Also keep in mind that the DDL issue only limits the JDBC adapter's usefulness for migrations or schema-manipulation purposes. Without DDL generation, it will still work with many, many databases without major modifications, since JDBC really shines here. Hopefully a future version of JDBC will provide DDL metadata or tools for manipulating database schemas. Until then, we'll continue to work with the Rails folks to find a good solution for DDL under the ActiveRecord-JDBC adapter.

From Eclipse to NetBeans: Part 3

I got a lot of good tips from NetBeans developers and users about how to resolve some of the issues I ran into. A number of them, as I mostly expected, were related to my running a beta that's still under development. Others are resolved via modules and add-ins (that in many cases should perhaps be included as standard). And still other cases are resolved or scheduled to be resolved in NetBeans 6. So to be fair, I decided to continue my NetBeans adventure by diving to the bleeding edge of NetBeans: building directly from source and running JDK 1.6 Beta 2.

I won't go into the details, but getting NetBeans to build was an extremely simple matter (thanks to Tor Norbye from Sun). Set CVSROOT, checkout -P standard, cd nbbuild, ant. It's really that simple. I did need to up the maximum memory needed for ant, but that was a minor issue. After the build, I found "ant tryme" that starts up NetBeans, and I was off and running.

This article will be a short wrap-up of issues from Part 1, revisited based on tips, tricks, and NetBeans 6 current and planned features. Hopefully it will demonstrate to potential switchers that NetBeans has really turned a corner. I'll revisit Part 2 in the next article.

I used to be among the NetBeans detractors...and for good reason; NetBeans had a rocky start. But I'm rapidly growing fond of it, especially using NetBeans 6.

Salvation

I'll take the items from Part 1 point-by-point. There's some useful links inline as well, so feel free to skim this list for issues of particular interest to you.

1. Antialiasing - RESOLVED! When running under JDK 1.6, all text elements are immediately antialiased according to the host platform's settings. No additional configuration changes are needed.
2. SVN by default - RESOLVED, I think. My basic pull of NetBeans 6 from CVS included SVN support immediately. I believe it's intended to be in the full release by default as well.
3. Workspaces - OPEN. My main use for workspaces is to keep different sets of projects in different places, and to a lesser extent to have different configurations for those sets. No obvious support for this in NetBeans 6 yet.
4. Separate repository view - OPEN. Eclipse's repository view is a godsend when you're hunting around for projects, tags, and branches, or if you want to do any remote repository management. I still miss it, and there's no out-of-the-box support in NB 6 for a repository view.
5. Standard SVN layout support - OPEN. I would like NetBeans to understand what "trunk", "tags", and "branches" dirs in SVN are used for.
6. Delete project not deleting files - OPEN. I tried to delete a working copy locally and the files remained on disk.
7. Wizarditis - OPEN. The best example of this is perhaps the SCM wizard, where at least seven clicks are required for me to check out my JRuby branch. If I had a repository view, it might take three or four.
8. Show non-project files in project view - OPEN, but not a big deal. I've gotten accustomed to just using the Files view when I need access to non-Java source. I still think it's necessary to show other files in the project view, but it hasn't gotten in my way as much as I thought it might.
9. Settings dialog search - OPEN, but improving. I remember once being told that if people need to use a site map or search for your site, you're doing something wrong. NetBeans should actually be applauded for presenting common settings in a novice dialog and a full complement of expert settings a button-press away. In NB6, they appear to be moving some of the important advanced settings to the main dialog as well. It's a balancing act, but I think a lot of people will appreciate the easy dialog.
10. Browser selection - Doubly RESOLVED. With JDK 1.6, NB6 uses your platform's browser by default, and the setting has been moved to the basic settings dialog.
11. NetBeans is a goofy name - UNRESOLVABLE. There's too much branding and history behind the NetBeans name to change it, or so I'm told. Oh well, at least it's not limiting in any way, and it does have a nice mouthfeel to it unlike "Visual Studio 2005".
12. Eclipse keybindings - OPEN, but workable. The default Eclipse keybindings are missing some important ones, but you can go map them yourself easily enough. Unfortunately there's no equivalent to Ctrl-Shift-R, which is like a "super open" for any file in the workspace. I also haven't found an outline view shortcut like Ctrl-O, which hurts a bit.
13. Sorted members view - OPEN. However I can deal with the default sorting for now.
14. Show packages as a hierarchy - OPEN, but I've been using the Files view almost exclusively which already uses a hierarchy.
15. Anti-aliased UI - RESOLVED! Under JDK 1.6, UI elements are also antialiased! It looks great!
16. Collapsing comments - RESOLVED, but one bug. It turns out that only the large license block at the top of our JRuby source files refuses to collapse. Everything else, including JavaDocs and multi-line comments, collapses ok. Probably a bug in how NB determines what comments to collapse.
17. Improved coloration - RESOLVED SOON. Better coloration and other editor improvements are planned for NB6.
18. Memory usage - OPEN, but improved. NetBeans 6 is a little better, but still typically takes over 200MB and sometimes over 300MB of memory. Eclipse usually stays under 200MB for me. However I don't have any modules disabled, and NB6 comes with UI design, Java EE, DB support, XML support, Web design support and a ton of other stuff out-of-the-box. That balances things out a lot.
19. FIXME support - RESOLVED via external module. The NetBeans Tasklist looks quite a bit more sophisticated than Eclipse's support.
20. Clear search results - OPEN, but workable. Search for nothing and the results clear. However I conversed with Roman Strobl, NetBeans Evangelist, and he agreed that was a little cumbersome.
21. Right-click editor menu with SCM operations - RESOLVED. You can't get those options right-clicking in the editor, but you can right-click the editor tab for the same effect. Hooray!
22. Odd behavior of SVN menus in file view - RESOLVED. This and several other SVN issues I ran into appear to have been fixed in recent 5.5 and 6.0 development builds. Subversion support seems to be tighter in NB6, or so it seems.
23. Slow start-up - OPEN, but fading fast. It's not a large perceptible difference between Eclipse and NetBeans now, and NetBeans is loading a lot more stuff. It's very nearly RESOLVED.

Hopefully it's apparent that NetBeans is arriving fast. If I were to make predictions, I'd say that NetBeans 5.5 will be a really powerful and useful release, especially considering suport for Java SE 5 and Java EE 5 out-of-the-box. I believe the bugs I've encountered will be wrapped up, and 5.5 will be a compelling alternative to "Eclipse-plus-ten-thousand-plugins", which functionality it easily matches.

I would also predict that NB6 may finally be the big release that surpasses Eclipse for day-one usability and functionality. There's just so much packed into the base product, I don't think Eclipse itself is even a fair comparison. WSAD may be the only real competitor at that point, and I'd prefer not to run that monster (and I also prefer not to use WebSphere.)

One area I think NetBeans needs to improve is in making development releases easily available. I have to dig down through multiple pages, dropdowns and license-acceptance forms to get at the 5.5 and 6.0 dev releases, which have so far been very stable and very nice. To save some time, here's a link to the NetBeans Download form. "Daily" builds are obvious enough, but I believe "Q-Build" builds are probably "QA" builds, perhaps equivalent to Eclipse's "integration builds" or "milestones". Q-Builds might be a good place to get started for you Pro users out there.