Friday, October 31, 2008

FFI for Ruby Now Available

One of the largest problems plaguing Ruby implementations (and plaguing some other language implementations, so I hear from my Pythonista friends) is the ever-painful story of "extensions". In general, these take the form of a dynamic library, usually written in C, that plugs into and calls Ruby's native API as exposed through ruby.h and libruby. Ignoring for the moment the fact that this API exposes way more of Ruby's internals than it should, extensions present a very difficult problem for other implementations:

Do we support them or not?

In many cases, this question is answered for us; most extensions require access to object internals we can't expose, or can't expose without extremely expensive copying back and forth. But there's also a silver lining: the vast majority of C-based extensions exist solely to wrap another library.

Isn't it obvious what's needed here?

This problem has been tackled by a number of libraries on a number of platforms. On the JVM, there's Java Native Access (JNA). On Python, there's ctypes. And even on Ruby, there's the "dl" stdlib, wrapping libdl for programmatic access to dynamic libraries. But dl is not widely used, because of real or perceived bugs and a rather arcane API. Something better is needed.

Enter FFI.

FFI stands for Foreign Function Interface. FFI has been implemented in various libraries; one of them, libffi, actually serves as the core of JNA, allowing Java code to load and call arbitrary C libraries. libffi allows code to load a library by name, retrieve a pointer to a function within that library, and invoke it, all without static bindings, header files, or any compile phase.

In order to address a need early in Rubinius's dev cycle, Evan Phoenix came up with an FFI library for Rubinius, wrapping the functionality of libffi in a friendly Ruby DSL-like API.

A simple FFI script calling the C "getpid" function:

require 'ffi'

module GetPid
extend FFI::Library

attach_function :getpid, [], :uint
end

puts GetPid.getpid
Because JRuby already ships with JNA, and because FFI could fulfill the C-extension needs of almost all Ruby users, we endeavored to create a compatible implementation. And by we I mean Wayne Meissner.

Wayne is one of the primary maintainers of JNA, and has recently spent time on a new higher-performance version of it called JFFI. Wayne also became a JRuby committer this spring, and perhaps his most impressive contribution to date is a full FFI library for JRuby, based on JNA (eventually JFFI, once we migrate fully) and implementing the full set of what we and Evan agreed would be "FFI API 1.0". We shipped the completed FFI support in JRuby 1.1.4.

The "Passwd" and "Group" structures for functions like 'getpwuid':
module Etc
class Passwd < FFI::Struct
layout :pw_name, :string, 0,
:pw_passwd, :string, 4,
:pw_uid, :uint, 8,
:pw_gid, :uint, 12,
:pw_dir, :string, 20,
:pw_shell, :string, 24
end
class Group < FFI::Struct
layout :gr_name, :string, 0,
:gr_gid, :uint, 8
end
end
In JRuby 1.1.5, we've taken another step forward with the API, adding support for callbacks. How would you represent a callback you pass into a C function from Ruby? How else! As a block!

Binding and calling "qsort" with an array of integers:
require 'ffi'

module LibC
extend FFI::Library
callback :qsort_cmp, [ :pointer, :pointer ], :int
attach_function :qsort, [ :pointer, :int, :int, :qsort_cmp ], :int
end

p = MemoryPointer.new(:int, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
i1 = p1.get_int32(0)
i2 = p2.get_int32(0)
i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"
But what good is having such a library if it doesn't run everywhere? Up until recently, only Rubinius and JRuby supported FFI, which made our case for cross-implementation use pretty weak. Even though we were getting good use out of FFI, there was no motivation for anyone to use it in general, since the standard Ruby implementation had no support.

That is, until Wayne pulled another rabbit out of his hat and implemented FFI for C Ruby as well. The JRuby team is proud to announce a wholly non-JRuby library: FFI is now available on Ruby 1.9 and Ruby 1.8.6/7, in addition to JRuby 1.1.4+ and Rubinius (though Rubinius does not yet support callbacks).

Session showing installation and use of FFI in C Ruby:
$ sudo gem install ffi
Password:
Building native extensions. This could take a while...
Successfully installed ffi-0.1.1
1 gem installed
Installing ri documentation for ffi-0.1.1...
Installing RDoc documentation for ffi-0.1.1...
[headius @ cnutter:~]
$ irb
>> require 'ffi'
=> true
>> module RubyFFI
>> extend FFI::Library
>> attach_function :getuid, [], :uint
>> end
=> #<FFI::Invoker:0x1fe8c>
>> puts RubyFFI.getuid
501
=> nil
>>
Our hope with JRuby's support of FFI and our release of FFI for C Ruby is that we may finally escape the hell of C extensions. Next time you need to call out to a C library, don't write a wrapper shim in C! Write it using FFI, and it will work across implementations without recompile.

Here's some links to docs on FFI. As with most open-source projects, documentation is a little light right now, but hopefully that will change.

Calling C from JRuby
Rubinius's Foreign Function Interface
On the Rubinius FFI

A key feature that's not well documented is the use of FFI's templating system to generate bindings based on the current platform's header files. Here's a sample from the "Etc" module above.

Etc module template, showing how to pull in header files and inspect a struct definition:
module Etc
class Passwd < FFI::Struct
@@@
struct do |s|
s.include "sys/types.h"
s.include "pwd.h"

s.name "struct passwd"
s.field :pw_name, :string
s.field :pw_passwd, :string
s.field :pw_uid, :uint
s.field :pw_gid, :uint
s.field :pw_dir, :string
s.field :pw_shell, :string
end
@@@
end
class Group < FFI::Struct
@@@
struct do |s|
s.include "sys/types.h"
s.include "grp.h"

s.name "struct group"
s.field :gr_name, :string
s.field :gr_gid, :uint
end
@@@
end
end
As more docs come to my attention, I'll update this post and add links to the JRuby wiki page. For those of you interested in the Ruby FFI project itself, check out the Ruby FFI project on Kenai. And feel free to hunt down any of the JRuby team, including Wayne Meissner, on the JRuby mailing lists or in #jruby on FreeNode.

Update: Wayne has posted a follow-up with more details here: More on Ruby FFI

32 comments:

jwmittag said...

Just out of interest: have you talked to any of the other implementations yet? I don't know about GemStone/S, but both Parrot and .NET have some sort of FFI and so it shouldn't probably too hard to add FFI to Cardinal, Ruby.NET or IronRuby.

Also, have you talked to Matz about maybe getting FFI into the Ruby Spec, maybe even deprecating the Extension interface for, say, the YARV 2.0 or Rite timeframe and maybe the upcoming ISO Spec?

Raphaël Valyi said...

Really sounds like a great move! I hope the MRI guys will follow even if unfortunately a few guys might do their best to avoid confrontation with JRuby on purpose and prefer a C extensions balkanization to consolidate their small influence circles. Hope the general interest will win here.

A question I have is: do you know if the Jython guys are benefiting from this work and in which ways? I would really see great Python softwares such as OpenERP to run on the JVM (Java would get a killer ERP; best oss potential SAP killer would just run on Java), Jython seems good enough at this stage but the C libraries support seems even a greater issue with Jython than it is with JRuby given that the 'maturity' of Python brought it way to much to the native extensions trend and almost every real application would require C bindings at some point. Any idea?

And aside from Jython, might all the other JVM languages, including the Java language, benefit from that kind of lib to set up bridges with the non Java world? Could you imagine a Swing application backed with GTK using such a lib (just curious, I consider Swing better even if its momentum isn't as good)?


Finally, I assume no C library call might fit in the Java security sandbox, correct? Java might have lost the RIA battle. But, given the recent plugin progresses for instance or the Flash VM limitations that would just be a shame. But RIA means a fine grained security sandbox model. Would that be an all or nothing model: like JNA calls are non secured or could we imagine somethings like JNA calls to signed+certified C libs might be allowed: like a C call to GTK or anything else stable and secured would be allowed. Sorry if that's a dumb question, but could such things be done?


Congrats as usual anyway.

Raphaël Valyi.

Luis said...

Awesome work indeed!

FFI for C Ruby yay!

I only found problematic to wotk with broken kenai site and workaround hardcoded stuff in the building process.

Of course, I'm talking to make FFI work on Windows, which is pretty much working for libffi.

Anyhow, great work from Wayne!

Anonymous said...

I agree with jwmittag. A *standard* FFI interface would be a big thing for Ruby.

Look at how every Common Lisp implementation has its own FFI (which is now thankfully handled by UFFI, but it took a long while), and see what a mess you can have when you don't have a standard FFI.

Charles Oliver Nutter said...

jsmittag: I think it's come up in discussions with John Lam of IronRuby, and I don't suspect they'll have any trouble building an FFI of their own. Also, I brought up FFI to Matz this past spring and he didn't seem opposed to it. I think he needs to be shown how it's better than DL, which I didn't have a chance to do at the time. I think the API Evan and we have assembled will easily become a de-facto standard, at any rate.

raphaël: I think the release of the gem pretty much guarantees that FFI will become a commonly-used API now. Even if there are anti-Java or anti-JRuby zealots out there, the API is nice enough to draw them in anyway. And the more libraries we see backed by FFI, the better for all implementations.

We are also working closely with the Jython guys. Wayne has spoken with them about JNA and JFFI use, and it's almost certain they'll work with him to get Python's ctypes library functioning as well.

I don't expect that C calls will ever be kosher inside a secure container, but for many other domains you mention--like GUI development--they'll probably prove very useful. And honestly, I think it's time that Java included a standard library for calling C libs. Hopefully this will help push things in that direction.

luis: I've never had any trouble with Kenai, but do report issues you see. They're standing by and fixing issues as fast as they can while simultaneously building out features. I think it's going to be a great site.

As for building on Windows, please, please submit patches back. We definitely want to make the gem as solid as possible on all platforms, and none of us are well-versed in Windows dev. (I did a bunch of it years ago, but it's all been pushed out to make room for new information)

anonymous: With this gem, the same FFI code will now work on C Ruby, JRuby, and Rubinius, and I suspect that IronRuby and others will follow suit. We're now well on the path toward a standard FFI API, even if it's not currently "blessed" as such.

rledge21 said...

Can't wait to give it a try, great work!

Wayne Meissner said...

I added a bit about FFI over at http://wmeissner.blogspot.com/2008/11/more-on-ruby-ffi.html

A few other things:
Jython already implements ctypes using code from PyPy and an adapter to implement PyPy's rawffi via JNA. This should allow them to run anything that uses the ctypes API. Not exactly sure how much does that, versus implementing functionality via the Python native API.

@Raphael
Have a look at jgir (http://live.gnome.org/JGIR) - its a project from Colin Walters (he hangs out in #jruby as walters) that automagically generates java APIs for GObject (e.g. GTK+) libraries. It uses ASM+JNA to do the java->C bridging.

@jwmittag
As for making a standard FFI api, I'm all for it if other implementations desire it. I haven't talked to other implementations, because the FFI API is still very new, and I don't think it is fully developed yet - i.e. it will be evolving as people use it.

Luis said...

Thanks Charles!

I had some random errors while using kenai, but seems these errors got fixed (was 3 weeks ago, which are ages in this field, hehehe).

Indeed I'll supply some patches, as soon get a healthy way to build the C extensions more independently of the platform.

Bluescreen303 said...

This sounds nice, I've tried a few times to extend ruby programs with C++ code, which is even harder, since you have to write a c layer that sits between ruby and c++ which handles c++'s OO features.

It takes quite some time and it feels like 'dumb' work.

So is FFI able to handle c++ libraries as well? How does it wrap the C++ classes in that case?

And on a different subject... Haskell has an FFI too (even called FFI). Is this just a coincidence or is this project inspired by haskell's? any thoughts on calling haskell code from ruby?

Bob Aman said...

Far as I'm concerned, this is huge. Quite possibly one of the most significant contributions JRuby has made thus far. Awesome, awesome work, and many thanks.

Charles Oliver Nutter said...

Bluescreen303: I talked with Wayne a bit about C++, and largely the problems that face C++ are that you can't programmatically bind to a C++ library unless you know how the method names have been mangled. So FFI in its current form doesn't change that fact any.

But there's a light at the end of the tunnel. Wayne believes we could implement something RubyInline-like that works within the FFI module bodies to generate a small amount of C code. So in essence, it would be RubyInline without the automatic type conversion and with the sole purpose of creating a simple shim library inline with FFI method bindings. So in that scenario you'd still use FFI, but it would be binding to snippits of C code you include; it wouldn't hurt implementation independence, since the eventual call to your C code is still via FFI, but it would allow doing a bit more binding logic than we can easily do from within Ruby, including calling out to C++.

If you're interested in this, and maybe interested in helping, I'd suggest you join the Ruby FFI mailing lists on the kenai.com project page and let us know.

http://kenai.com/projects/ruby-ffi/lists

Christopher Parker said...

This is beautiful! Beautiful!

The examples work as advertised.

I am going to check out the code to see what is happening so that I understand how to use it. It looks like there is a winner here. I have played around with writing ruby extensions, but the current system (before ffi) were pretty nasty.

There are a lot of libraries out there that need to be wrapped.

Beoran said...

One thing that's also not clear is how to register a custom free function (from the C side) for memory that also has been allocated from by the C library, so we can let the garbage collector handle freeing that memory.

For example in SDL you need to do

image = SDL_LoadBMP(file_name);
...
SDL_FreeSurface(image);

However, it's undesirable to have to do this manually in a garbage collected language.

C Ruby 'dl' can do this, but can FFI do this too? If so how? And if not, I think you will need this.

Anyway, looks like an interesting beginning. Keep up the good work.

Charles Oliver Nutter said...

Beoran: What's wrong with putting it in a finalizer? The finalizer proc holds a reference to the the memory to be freed, and frees it once that happens.

If not, feel free to submit a patch that adds the functionality to Ruby-FFI in a way that will work across implementations. :)

Beoran said...

What do you mean by setting a finalizer? Maybe we're talking about the same thing? In DL on ruby 1.9.x , you'd do something like:

require 'dl/import'
module SDL
extend DL::Importer
extern "void * SDL_LoadBMP(char *)"
FREE_SURFACE = extern "void SDL_FreeSurface(void *)"
end

surfp = SDL.SDL_LoadBMP('cute.bmp')
surfp.free = SDL::FREE_SURFACE
# Now, I don't have to worry about calling SFL_FreeSurface manually anymore
surfp = nil
# Garbage collector knows what to do.

Perhaps this is already possible in ffi, but I couldn't find it.

And yes, I'm willing to help you implement this if it's not already there.

Matthew King said...

N.B.: The 0.1.1 gem doesn't appear to have the "templating system to generate bindings based on the current platform's header files".

FFI::Generator does exist in the Hg repo, though.

Wayne Meissner said...

@beoran
Garbage collection of arbitrary pointers is not currently supported. It wouldn't be that hard, I would probably do something like:

module SDL
class SDL_Surface < FFI::AutoPtr
def free(ptr)
SDL.SDL_FreeSurface(ptr)
end
end
attach_function :SDL_LoadBMP, [ :string ], SDL_Surface
attach_function :SDL_SDL_FreeSurface, [ :pointer ], :void
end

Then you just use it as:
surface = SDL.SDL_LoadBMP("foo.bmp")
# Do whatever with surface
surface = nil # releases surface


That pushes the pointer garbage collection setup back to the declaration, and removes it from the point of use.

Beoran said...

@Wayne:

Well, yes, that would be nice, I guess, but wouldn't it be even better if you could simply call .free on an instance of MemoryPointer, passing in either a block callback or a symbol of the function to call?

I'm interesting in wrapping SDL though FFI so I can also use it on JRuby for my big game project. Of course, I want an easy API. To be honest, I kinda like the DL API because it's easy to use once you get to know it. It can already parse C quite well, so perhaps you can find something to borrow in it's Ruby code?


Anyway, if you'd like me to help you out a bit, I already subscribed as an observer to the FFI project with username BeorAegul on the Kenai site, so perhaps we could talk further that way if you like.

Anonymous said...

How can I compile the following code under Linux that is avaible under jruby ffi?
/* File : example.c */

/* A global variable */
double Foo = 3.0;

int gcd(int x, int y) {
int g;
g = y;
while (x > 0) {
g = x;
x = y % x;
y = g;
}
return g;
}

Charles Oliver Nutter said...

anonymous: I'm not sure I understand the question, so I'd recommend joining the Ruby FFI project mailing lists at the Kenai project page:

http://kenai.com/projects/ruby-ffi

Anonymous said...

The gcd C Code is only a example. I try make a library for JRuby. I did

gcc -c -fPIC gcd.c -o gcd.o
gcc -shared -o libgcd.so gcd.o -lc

and my Ruby code looks like this
----------------------------------
require 'ffi'

module Test
extend FFI::Library
attach_function :gcd, [:int, :int], :int
end

Test.gcd(3,3)
---------------------------------

But if start jruby I get following error:
/home/flashdog/jruby-1.1.4/lib/ruby/site_ruby/1.8/ffi/ffi.rb:328:in `create_invoker': Function 'gcd' not found! (Looking in 'c' or this process) (FFI::NotFoundError)
from /home/Anonymous/jruby-1.1.4/lib/ruby/site_ruby/1.8/ffi/ffi.rb:380:in `attach_function'
from gcd.rb:5

What do I wrong?

Charles Oliver Nutter said...

anonymous: I believe you just need to get the library loaded, which you could do via

ffi_lib "gcd"

in the module body. If you have further issues, I recommend asking on the Ruby FFI mailing lists :)

Anonymous said...

Thank you

ffi_lib "libgcd" # works

My first problem:
#doesn't work
ffi_lib "/swig-ruby/simple/libgcd"
---------------------
$ jruby gcd.rb
/home/Anonymous/jruby-1.1.5/lib/ruby/site_ruby/1.8/ffi/ffi.rb:224:in `create_invoker': Function 'gcd' not found! (Looking in '/swig-ruby/simple/libgcd' or this process) (FFI::NotFoundError)
from /home/Anonymous/jruby-1.1.5/lib/ruby/site_ruby/1.8/ffi/ffi.rb:328:in `create_invoker'
from /home/Anonymous/jruby-1.1.5/lib/ruby/site_ruby/1.8/ffi/ffi.rb:415:in `attach_function'
from gcd.rb:6
---------------------


#doesn't work
ffi_lib "swig-ruby/simple/libgcd"
---------------------
$ jruby gcd.rb
/home/Anonymous/jruby-1.1.5/lib/ruby/site_ruby/1.8/ffi/ffi.rb:224:in `create_invoker': Function 'gcd' not found! (Looking in 'swig-ruby/simple/libgcd' or this process) (FFI::NotFoundError)
from /home/Anonymous/jruby-1.1.5/lib/ruby/site_ruby/1.8/ffi/ffi.rb:328:in `create_invoker'
from /home/Anonymous/jruby-1.1.5/lib/ruby/site_ruby/1.8/ffi/ffi.rb:415:in `attach_function'
from gcd.rb:6
---------------------


My second problem:
I compiled gcd.rb with "jrubyc gcd.rb", but if I tried to run it with java I get following error:
$ jrubyc gcd.rb
Compiling gcd.rb to class gcd
$ java -cp /home/Anonymous/jruby-1.1.5/lib/jruby.jar:. gcd
Exception in thread "main" null:1:in `const_missing': uninitialized constant Test::FFI (NameError)
from gcd.rb:4
...internal jruby stack elided...
from Module.const_missing(gcd.rb:4)
from (unknown).(unknown)(:1)
from (unknown).(unknown)(:1)


How can I fix the problems with the path and Jrubyc/Java?

Brian said...

I have a small patch, but am I just missing it or is there really no way to submit bugs/patches without signing up for this Kenai thing? Do not want.

Charles Oliver Nutter said...

Brian: Kenai is just the project hosting site, like RubyForge, SourceForge, etc. As with those services, if you want to submit a bug or patch you generally need to sign up.

Anonymous said...

Hello,
I have a problem with Arrays ( http://kenai.com/projects/ruby-ffi/lists/users/archive/2008-11/message/20 ).
What do I wrong?

Anonymous said...

http://kenai.com/projects/ruby-ffi/lists/users/
archive/2008-11/message/20

Anonymous said...

I'd like to know whom to contact about Ruby-FFI. I have started to test this and it looks good, but the fact that I have not found full documentation and there seems to be no project "owner" does not inspire confidence.

Also, there are gaps, one of which I have filled ad hoc with the following code:

module FFI
class Pointer

def read_double
get_float64(0)
end
def write_double(obj)
put_float64(0, obj)
end

def read_array_of_double(length)
get_array_of_double(0, length)
end

def write_array_of_double(ary)
put_array_of_double(0, ary)
end

end
end

Feel free to contact me on ak90@zoom1000.com

-Alex

Dave Ekhaus said...

Hi

I'm trying to get a simple FFI example to work. I created a .so containing a function called 'mylib_factorial' - obviously a simple factorial function.

I created a ruby Module and ruby test.rb script. When I try and run my test.rb I get the following error ...

/usr/lib/ruby/gems/1.8/gems/ffi-0.5.0/lib/ffi/library.rb:77:in `attach_function': Function 'mylib_factorial' not found in [/home/dekhaus/mylib/mylib.so] (FFI::NotFoundError)

My ffi_lib 'call' looks like this (from within my ruby Module) ...

ffi_lib "/home/dekhaus/mylib/mylib.so"

Can I get some help please ? I really don't want to write a C Extension

Thanks
Dave

Charles Oliver Nutter said...

Derek: You should check out the ruby-ffi project on kenai.com. Your problem may be a 32/64-bit issue or you may not have the signatures matched up quite right. Folks on the ruby-ffi mailing list will be able to assist you.

Dave Ekhaus said...

Hi Charles

Not sure if your last post was directed to me as my name is not Derek.

For what its worth - I did submit a question to ruby-ffi project at kenai - or at least I think I did. The feedback wasn't very clear.

Thanks
Dave

Charles Oliver Nutter said...

Oops, sorry about that Dave. You might also try to catch us and other JRubyists using FFI on #jruby on irc.freenode.net, and maybe we can provide better help there.