OSGi: OBR in the cluster, a return of experience

I've been working for some time on an OBR based deployment framework for clusters of identical JVMs. The framework is implemented and in use within Alcatel-Lucent's application server, and I'd like to share some ways, thoughts and returns of experience, so that maybe we can all come to agree on some best practice on the subject. The discussion is open :)

Before we dive in the subject, I'd like to say the good news is, there's almost nothing to add: OSGi has it all already (provided we resurrect some deprecated headers!)
Bad news is, there's no such thing as a free lunch!

Let's try and state the problem briefly: what we're trying to do is

• to deploy "grids of JVMs", that is, bunches of identical nodes, from scratch (no host framework has been previously deployed for us),
• take full advantage of the OBR to automatically resolve transitive dependencies expressed between bundles,
• to support multiple implementations of one service, either exclusive or complementary
• to do so in an "industrial" manner: interactivity is not an option,
• to do so in a safe manner, avoiding "virus" bundles as far as we can.

What do you mean "from scratch"?

Our problem here is not to deploy applications inside an existing framework but to assemble a new framework, along with the application.
So the first of these requirements puts us in an awkward position already.
Traditionally, the OBR resolver assumes it runs in the host framework, and it only pulls from the repository the bundles it doesn't have already.

But then who took care of building that host framework? What bundles does it contain? What version? How do I ensure that all my nodes are identical if I have 2 different processes, one for deploying the host and another one for pulling the application from the OBR? And why should my application even need an OBR resolver if it's not its purpose?
Even worse, if the resolver is called independently from each host framework, and if I update the OBR before deploying a new host (say I need to scale up a few days later), I may get a different resolution for the new host. This breaks my requirement to ensure that all nodes run the exact same software.

So we initially had to modify the resolver a bit, to resolve independently of locally installed bundles, so that one JVM can use the resolver to build another JVM from scratch: that is bring everything, starting with the OSGi framework bundles themselves (the feature is now available in the Felix resolver). The resolution can then be done only once and saved as list of versionned bundles, that will be used as such for assembling each JVM.

The OBR addresses packages; what about services? 

The second and most tricky part is to pull the services implementations from the OBR.
By nature, the OBR will only resolve the transitive dependencies, essentially expressed via the Import-Package and Export-Package manifest headers. Package wiring is a static operation and packages can be versionned and imported with a range to pick amongst different options.

But as we all design clean service architectures built upon the service registry, our package imports all point to an empty shell: the service API. All implementations are properly hidden in private packages. Resolve this with an OBR and you'll end up with a bunch of API bundles that do nothing!

One quick answer is to just package the service implementation along with its API, but
1 - it doesn't resolve the main issue: how do we pick one (or several) implementations of a service amongst our OBR options?
2 - it's actually "dangerous", as we'll see further (in the "virus" section!)

The two main issues with services are, well, the same as their strength! Dynamicity and Cardinality.

Cardinality: when expressing dependency on a service, we may need
- optionally one
- any random implementation (not recommended!)
- one specific implementation (either "hard coded" with a service filter, or to be chosen at deployment time)
- all available implementations (typically plugins)
- a chosen subset of the available implementations

Dynamicity: services can be dynamically created at runtime, and for instance, another component can be (by design) locked in a service dependency until the service is dynamically created. This is very practical pattern for IOs for example. But how can we express this statically in an OBR?

So let's review what standard OSGi tools we have at hand (if we dig a bit for deprecated ones!)

Import/Export-Package, we've already mentioned, is great but not enough.

Import/Export-Service: great! this looks just like what we need!

This header gives us the transitive dependencies that we need on the service layer.. but,

• first bummer, Bnd doesn't automatically generate these headers like it does for packages, so all these service dependencies we've been coding in java, annotations or xml, must now be expressed a second time for the OBR
• second bummer, even if Bnd could generate these automatically, it wouldn't work! Take the dynamic service example, generate an Import-Service automatically, and you get a required dependency that will never resolve in the OBR. (We can still use Import-Service, but then we need to manually add an Export-Service in the bundle that may eventually register the service at runtime)
• third bummer, being (half way?) deprecated, the Import/Export-Service headers don't support the extended header syntax, that would allow to specify things like cardinality or service filters. (There's an open OSGi issue on the subject https://www.osgi.org/bugzilla/show_bug.cgi?id=70)
• and version? You'd think it would be nice for the resolver to pick the highest service version available, or a range, like it does with packages. But as several providers of a service may have different development cycles, the version for a service isn't really relevant; version 2 of provider 2 may be much newer than version 16 of provider 1.

So Import/Export-Service is very limited.
If we keep these limitations in mind though, it is a very convenient expression for the many simple in-house cases where we know there will just be one provider (but even then, the virus bundle is lurking around!).


Bundle-Category: what does this have to do with anything? it's just for display!

Well it's not just display and the good news is that the OBR resolver can take a Bundle-Category constraint as input for its resolution. This is a prefect expression for the "plugins" use case, when you have many possible implementors and you can't know them all in advance.

Ideally, it would be nice to be able to express dependency to a category directly from a bundle, but as it is, it's already quite useful.

Require-Bundle: the return of the terrible child!

We all know why Require-Bundle has been demonized: it implies the creation of one giant classloader containing all the required bundles, and as such it breaks the fine grained modularity of OSGi. OK.
But if we forget about this runtime aspect for a minute and look at it from an OBR perspective, it's just the simple expression of an explicit assembly. Exactly what we need to artificially create a "transitive dependency" to the chosen implementation bundles.
So with a simple adaptation of it's use, this header becomes a very convenient tool in the OBR. Here's what we can do:

• create empty bundles, only providing a manifest with a Require-Bundle header that points to all specific service implementations that need to be specified
• tag the bundle with Bundle-Category: assembly
• use this "assembly bundle" as the root of the OBR resolution, and when deploying the bundles into an actual JVM, simply strip off all the assembly bundles, so that you don't end up with that one big classloader.

Of course, these assemblies don't have to be monolitic; it's perfectly fine to split them into smaller reuseable "assembly" components. The point is that they simply add the missing glue to the OBR.


The "Virus bundle"! Now what bundles should actually be started?!

Wow. We're almost there!
But now consider the following use case (which is exactly how I originally bumped into the issue).
I'm trying to assemble a Jetty web server with a web application, and my OBR happens to contain this:

• org.mortbay.jetty.server.jar
• org.mortbay.jetty.servlet-api.jar
• my.web.application.war
• my.assembly.jar (Require-Bundle: org.mortbay.jetty.server, my.web.application)
• and, org.apache.felix.http.jetty.jar

It turns out that org.apache.felix.http.jetty.jar also embeds, and exports, the servlet APIs.
So when I ask the resolver to resolve my.assembly:

• it pulls org.mortbay.jetty.server and my.web.application as requested,
• and following the package imports, it needs to pull the servlet APIs

With my luck of course, it always picked org.apache.felix.http.jetty as exporter of the servlet API, not the plain library bundle.

So I deploy this, start all the bundles, and I end up with a port conflict, with both instances of Jetty (mortbay and felix) trying to start concurrently!
This is what I call the virus bundle: I never meant to bring Felix's Jetty (least start it!), but it was dragged along by transitivity, for the packages it exports.

What? All this, and it just doesn't work!
We need a finer control on what bundles are started and what bundles should only be installed. And a we certainly can't go with an "Are you sure?" dialog box for each bundle starting!




So we'll consider Require-Bundle and Import-Service as requirements that a the corresponding bundle should be started. We can simply scan the resolved bundles to automatically generate a whitelist of bundles to be started, in the form of an OSGi/LDAP filter:

• each Require-Bundle translates into a Bundle-SymbolicName appended to the whitelist,
• each Import-Service translates into a Export-Service appended to the whitelist.

This way, if an "activable" bundle is dragged only to be used as a library; it just won't be started.

Note that, even though convenient, the Import-Service rule should only apply to the well known in-house services: in an uncontrolled OBR, it is quite a breach and should probably be removed.

Conclusion?

My point here is that there is not much to invent and, providing some simple conventions and a little intelligence in the deployer, the OBR has everything we need to assemble clusters of JVMs (actually not only JVMs but other processes as well).

On the other hand, things could be easier, through standard conventions and better support of the extended header syntax for all headers.

OSGi Service Diagnostics

In my series "OSGi and Scala are the two best things that ever happened to the JVM", I'd like to introduce a small OSGi project written in Scala that I started on GitHub. The project aims at easing diagnostics of OSGi services and finding about missing dependencies.

Typically in a large system with many cascading dependencies managed by different trackers such as DeclarativeService, DependencyManager, iPojo or others, tracking the root cause of a top level service not being started can become very cumbersome. When building service oriented architectures, it is often the case that a single missing requirement will lock a full stack of services, but to find that one requirement is like finding a needle in a haystack!

Typically, if A depends on B which depends on C which depends on D, and D is nowhere to be found, I need only show the "C -> D" missing requirement; if D is resolved, then the whole stack is unlocked.

Similarly, if D is known by another dependency management system, but unregistered because it depends on E which is missing, then only the "D -> E" requirement is relevant.

So the basic idea here is to ask each dependency manager instance about its unresolved dependencies, merge all answers and filter the result to keep only the root causes. For now, only plugins for Felix SCR and Felix DependencyManager are implemented, but additional plugins for iPojo, Blueprint or others could easily be developed either in Scala or Java by anyone interested.

From this, and with the help of the Raphaël Graffle javascript library, I wrote a WebConsole plugin to graph both registered services and missing dependencies.

Here's what it looks like on basic sample of unsatisfied dependencies:

and for registered services, the arrows point from bundles to the services they use:

The colors are random and don't mean anything. The "bubbles" can be dragged around to help readability. If the graph of all services grows too large and difficult to read, a filter is provided to focus only on a subset of packages for instance.

Enough blabbering, try it for yourself:

> git clone https://github.com/joune/servicediagnostics.git
> cd servicediagnostics
> mvn install
> run.sh

Using Scala to test your OSGi Java

There we go for my first attempt at sharing some geek thoughts!

You'll read about how to

• make use of ScalaTest's neat "Specs" syntax (aka "Behaviour Driven Development"), or any other test framework for that matter
• while leveraging the use of OSGi dependency injection
• and keep the boiler plate code to a minimum!

As many Java old timers, I'm attracted to Scala because of the many good reasons you'll find all over the internet these days, but as many, my company is reluctant to let go of the good old (annoying) Java.

So one simple compromise we found was to use Scala to write our automated tests; It won't break any production code, and yet allow us to get our hands dirty with some actual Scala code and know better where we want to go from there.

That only, I felt, was an idea worth sharing because It seems like many of us are in a similar situation!

If it turns out that our tests are more concise, easier to write and maintain and fully compatible with our code base, it should open the doors for introducing Scala into production code.

Now comes the meat.

The code I'd like to test is packaged a "service oriented OSGi bundle". By that, I mean that the code must run in an OSGi environment, so that the service i'm testing is properly injected with its dependencies (and their transitive dependencies).

While this is has many benefits in terms of architecture, it can quickly become a bummer when it comes to tests. Testing outside of OSGi, I would need to mock and/or wire everything by hand, which ends up in a lot of boilerplate code for my test (and I'm lazy).

So there we go!

Step one: create an activator to delay the test Runner

As I mentioned we'll use ScalaTest (but we could use any other), but since we are running in OSGi we can't simply launch the Runner directly. The point here, is to wait for our service dependencies to be available before we run the tests.
I could just use a plain ServiceTracker here, but I'm still lazy, so I'll use the great DependencyManager library by Marcel Offermans to do the work for me.

I haven't found a way to directly inject the dependencies into the test class itself because the Runner will instantiate this class so instead I'll just keep my dependencies accessible via a separate singleton object. (The cake pattern won't help me here because dependencies are wired dynamically after the creation of the object)

I want to start running my tests, only once all my dependencies are available, so I'll call the test Runner from the start method of my Dependencies object.

import org.osgi.framework.Bundle
import org.osgi.framework.BundleContext

import org.apache.felix.dm.DependencyActivatorBase
import org.apache.felix.dm.DependencyManager

import org.scalatest.tools.Runner
import scala.actors.Actor._

class Activator extends DependencyActivatorBase
{
  override def init(bc:BundleContext, dm:DependencyManager) =
  {
    dm.add(createComponent
      .setImplementation(Dependencies)
      .add(createServiceDependency
        .setService(classOf[SomeService])
        .setAutoConfig("service")
        .setRequired(true)))
  }

  override def destroy(bc:BundleContext, dm:DependencyManager) = Unit
}

object Dependencies { 
  var service:SomeService = _ 

  def start = actor { 
    Runner.run(Array("-o", "-s", classOf[SomeServiceSpec].getName))
    System.exit(0)
  }
}

// note: if you're using the previous (/current) version of DependencyManager:
//  - the package is org.apache.felix.dependencymanager
//  - use createService instead of createComponent

Notice here that the test Runner is launched in a separate thread (using the actor block) so that we're not holding a lock in the framework's main thread while starting our service.(The "-o" option will output the tests results to the console instead of launching the graphical interface)

And that's it for the boiler plate! Appreciate that the service I'm about to test (SomeService) could have a dozen of dependencies, which could also have dependencies... whatever the complexity behind, I'll start running my test only once everything is properly wired!

And I don't need to mock anything; I'm testing the real thing. Though I could still use mocks if I wanted to, to fill in for some missing service, and I would do that from the same Activator, by simply registering the mock services into the OSGi registry and letting the wiring happen the same way.

Step two: write the actual test

Now comes the treat; a nice clean BDD test focusing only on what it needs to test and not on how to mock the missing parts, in which you can use/learn the full functional and DSL power of Scala!

import org.scalatest.FlatSpec
import org.scalatest.matchers.ShouldMatchers

class SomeServiceSpec extends FlatSpec with ShouldMatchers {

  "SomeService" should "say Hi in return to hello" in {
     Dependencies.service.hello should equal ("Hi")
  }

  it should "say 'See you' in return to bye" in {
     Dependencies.service.bye should equal ("See you")
  }
}

Please forgive the terrible lack of imagination regarding the purpose of the test, but that's not the point. Simply notice how the tested service is accessed via the Dependencies holder.

Step three: wrapping it up

Before wrapping up the post, we'll need to wrap some library jars to turn them into OSGi bundles!

The Bnd tool by Peter Kriens is your friend here. On top of building your own OSGi bundles, it can turn any plain old jar into an OSGi bundles with one simple command.

The scala library used to be provided as an OSGi bundle (I'm pretty sure!) but I can't find it anymore. Instead, the OSGi packaging seems to be targeting specifically the Eclipse environment, and what they did was to package the scala-compiler.jar as an OSGi bundle and add a Require-Bundle to include scala.library! Pretty strange if you ask me. And to make things worse there's also an Require-Bundle on org.apache.ant! That's close to evil!

Anyway, we won't bother and simply run:

> java -jar bnd.jar wrap scala-library.jar
> mv scala-library.bar scala-library-osgi.jar

Similarly for scalatest:

> java -jar bnd.jar wrap scalatest-1.3.jar
> mv scalatest-1.3.bar scalatest-1.3.osgi.jar

Now there's an additional trick for the ScalaTest bundle: It will need to dynamically load the test code from our bundle which it doesn't know anything about. So we'll add the -evil but sometimes you have to- "DynamicImport-Package: *" header to its Manifest, otherwise we'll run into a ClassNotFoundException when loading the test.

Finally package your own test bundle. And don't forget to export your test's package so that ScalaTest can load it!
The bnd directives could look like:

Bundle-Name: My First OSGi Scala Test
Bundle-SymbolicName: me.myself.service.test
Export-Package: me.myself.service.test
Bundle-Activator: me.myself.service.test.Activator 

Step four: create a launcher for the OSGi framework

This optional actually. I'm using Felix, but I don't want to use a configuration file. For testing I'd rather simply give the list of required bundles as an argument to the launcher. So let's write a quick custom Launcher:

import java.util.HashMap
import java.util.ServiceLoader
import org.osgi.framework.launch.FrameworkFactory
import org.apache.felix.main.AutoProcessor

object Launcher
{
  def main(bundles:Array[String]) = 
  {
    val factory:FrameworkFactory = ServiceLoader.load(classOf[FrameworkFactory],
      getClass.getClassLoader).iterator.next
    val felixProps = new HashMap[String, String]
    felixProps.put("felix.auto.start", bundles.map("file:"+_).mkString(" "))
    felixProps.put("org.osgi.framework.bootdelegation", "sun.*")
    val felix = factory.newFramework(felixProps)
    felix.init
    AutoProcessor.process(felixProps, felix.getBundleContext)
    felix.start
  }
}

This is quite close to the Java version. Simply notice the nice one-liner to set the list of auto.start bundles.

Also notice the explicit bootdelegation on sun.* because the scala library makes use of some sun classes which are not automatically re-exported by the framework.

Step five: finally run your test!

Assuming we've packaged our test into me.myself.service.test.jar, and the service we're testing is in me.myself.service.jar :

scala \
  -classpath org.apache.felix.main.jar:classes/ \
  me.myself.service.test.Launcher \
    me.myself.service.test.jar \
    me.myself.service.jar \
    org.apache.felix.main.jar\
    org.apache.felix.dependencymanager.jar\
    org.osgi.compendium.jar\
    scala-library.osgi.jar\
    scalatest-1.3.osgi.jar

This should output something like:

Run starting. Expected test count is: 2
SomeServiceSpec:
SomeService 
- should say Hi in return to hello
- should say 'See you' in return to bye
Run completed in 158 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 2, failed 0, ignored 0, pending 0

That's it for my thought of the day!
It won't feed the poor or stop the wars, but don't hesitate to drop a note if you found it useful :)
Any suggestion on how to make this sexier-stronger-faster-lighter-better is naturally welcome!