More on Ruby FFI

Charlie announced FFI for Ruby. I did the implementation, but I hate writing documentation ... or announcements, so I left that part up to him. This is mostly a response to some of the comments.

Why implement a new API instead of the existing DL one? Mostly because we (JRuby) wanted to run the syslog and zlib modules from Rubinius, and they were written using their FFI syntax. There also seemed to be a dearth of software that actually used DL, and it had somewhat ... arcane syntax.

The Rubinius API on the other hand was DSL-like, and relatively easy to implement, so it was pretty much a no-brainer. This doesn't mean we won't ever implement DL, the infrastructure developed for FFI is sufficient that a DL api could be layered over the top, most likely all in ruby. We already have the beginnings of Win32::API implemented this way.

What about sharing with other implementations? Its unlikely there will be much code sharing - there is already a good chunk of higher-level implementation shared between JRuby-ffi, ruby-ffi, and rubinius, but this is more by necessity than design.

The code is constantly evolving, and trying to define some sort of implementation layer would be like nailing jelly to the wall. The ruby-ffi implementation is all BSD licensed, so that shouldn't be an obstacle.

A better way is what we're going to be doing - defining the FFI API and developing enough specs that it should be easy for an alternative implementation to correctly implement the FFI API.

I haven't really been collaborating with other implementations besides Rubinius yet, because I didn't want to get bogged down in design-by-committee. The rough goal with ruby-ffi is to make it the definitive implementation where the API, specs and documentation are kept, and where new features are designed & implemented.

Whats performance like? For MRI, its a bit unknown. It shouldn't be too bad, but I wanted to get a release out and have people using it and giving feedback, so I didn't bother to tune things at all. There are a couple of places where there is ruby code in the call path, since I needed to munge things around, and couldn't work out how to do what I needed using the C API. Those things are pretty small, and can be fixed up as we go along.

The JRuby implementation on the other hand is very fast. Calling some posix functions via FFI is as fast as calling via jna-posix - and in some cases faster. The path from calling a ruby method, to marshalling the arguments, and calling the C function is very streamlined - the call path is all java, so it gets JIT'd easily, and it avoids boxing parameters where possible.

Basically all the performance tricks JRuby has to implement ruby methods as fast as it can, apply to FFI methods. The JRuby implementation also supports pluggable backends, which means that a faster implementation might be possible on other JVMs (e.g. Maxine, or IKVM.net) and loaded at runtime.

What is the future? For ruby-ffi, in the short term, I'm going to be expanding the specs to cover as much as possible, and waiting to see how people are actually using it in the real world.

I have also started working on jaffl (Java Abstracted Foreign Function Layer), which takes the pluggable backend idea from JRuby's FFI implementation and repurposes it for java code. The idea is that in future, jna-posix and friends will use jaffl as their java FFI layer, with a JNA, JFFI or some future backend. Eventually, JRuby's FFI will also switch over to use jaffl.