Chapter 18  Remote Method Invocation

• What is RMI?
• Babel RMI Concepts
• Babel RMI Usage
• Babel Object Servers
• Non-Blocking Babel RMI

18.1  What is RMI?

In classic Babel, all calls are in-process. That is, everything Babel generates is loaded into the same process. This means inter-language Babel calls use the same mechanisms as normal function calls. This makes calls between languages extremely fast. However, many systems also have a need for Remote Procedure Calls (RPC), that is, calls made between different processes, or even machines over a network. Remote Method Invocation (RMI) is Babel’s answer to this need.

RMI is Object Oriented RPC. However, unlike RPC where calls are made to procedures on a specified machine, in RMI calls are made on objects. The call is run on whichever machine the object resides on.

There are several reasons why an application may choose to use RMI. The main reasons are wrapping code tied to particular hardware, wrapping code tied to a particular operating system release, coarse-grained parallel execution, or greater encapsulation. With RMI, you can make code that’s tied to a particular machine available to programs running on other platforms. You can utilize multi-CPU systems to concurrently solve problems using RMI. RMI can solve problems that sometimes occur when you put two codes in the same address space. For example, two Fortran codes may use the same logical unit numbers (similar to C file numbers), or two codes may both need a customized form of a third party library. Bringing both codes into the same process may cause a symbol collision for the third party library, and one code gets the wrong version of the library.

Despite the radical low-level differences between RMI and classic Babel, the user interfaces are nearly identical. In fact, if a library writer does not care if an object is remote or not, they simply do not need to know. RMI support requires a few simple calls to set up the infrastructure, but almost everything else is handled automatically by the Babel runtime library. Babel also has a few RMI support functions and a special remote constructor.

18.2  Babel RMI Concepts

For normal function calls, arguments are passed by initializing registers and pushing things onto the system stack. In RMI, function arguments and return values are passed by a network protocol. From a programmers point of view, the only difference between normal and network function calls is that network function calls have more failure modes. Anything that can disturb a network connection, such as a router going offline, can cause a RMI call to fail.

Conceptually, the RMI view of the world can be thought of as 1 or more Babel Object Servers (BOSs) that a client can connect to in order to create or use objects on those servers. Of course, any server can also connect as a client to any other server, and any client can become a server simply by starting up a BOS of it’s own.

This makes Babel RMI very flexible, and accepting of whatever client-server relationships the application writers choose to use. Web Services users of Babel tend to use traditional client-server models, while scientific distributed systems users tend toward peer-to-peer usage.

18.2.1  RMI Protocols

The first thing any user of Babel RMI has to do is choose a Babel RMI protocol. Babel RMI can use any protocol that implements the Babel RMI API, but a client and a server using different protocols probably cannot communicate.

A Babel RMI protocol may be built on any low level protocol (such as TCP/IP) that the protocol implementer wishes to use. This should not affect the user at all. The protocol controls the details of how arguments and return values are converted to a stream of bytes that can be shipped across the network and read by the other process.

Currently there is only one protocol that fully implements the Babel RMI API, and it is included with Babel in the runtime/sidlx directory. It is called “Simple Protocol.” However, there are many other protocols currently under development (at least four.) Soon there will be a number of choices, including protocols specifically tuned for high-performance scientific computing, web services, and CORBA[] compatibility.

18.2.2  Babel Object Server

The next thing that a user needs is a BOS to connect to. The BOS is implemented by the protocol writers as a library. As mentioned before, the BOS may be run by itself, by a small driver program, or run as part of a program that also acts as a client. There is an example of a small driver program in contrib/babel-rmi/orb.

The BOS is accessed by a protocol specific URL. A URL is a string the uniquely identifies a network resource. Most people are aware of Internet URLs like: http://www.llnl.gov/CASC/components/babel.html (which is the URL for the Babel web page). Babel RMI also uses URLs, but they are mostly protocol specific. Babel RMI only uses the portion of the URL up to the first non-alphanumeric character to identify the protocol that is being used. The rest of the URL is passed on to the protocol. This means that while “Simple Protocol” URLs look like this:

simhandle://pc3.nowhere.com:9999/

one can also imagine URLs of the form:

weird://05:16:5B:BD:E1:73/

for the weird protocol, or:

weird+SSL://05:16:5B:BD:E1:73/

for running the weird protocol over SSL. Babel RMI itself does not attempt to parse anything past the first non-alphanumeric character, so most of the URL is entirely protocol dependent.

18.2.3  Object Creation and Connection

There are two main ways of accessing objects on a remote server, creation and connection.

A client may create a remote object with the Babel built-in static _createRemote(in string URL) method. This asks the remote server given in the URL to create an object. For example, the C function

foo_Bar b = foo_Bar__createRemote(‘‘simhandle://pc3:9999/’’);

will create an object of type foo.Bar on the server running on pc3 port 9999 using the “Simple Protocol.”

foo_Bar b may now be passed around exactly like a normal Babel object, except that all calls on b will actually run on pc3.

However, in some cases, an object already exists on a remote server that the user just wants to access. In this case, the object can be connected to via the built-in static _connect(in string URL) method. The only difference is, in this case the URL must include an object ID to uniquely identify the object desired on the BOS. For example:

foo_Bar b = foo_Bar__connect(‘‘simhandle://pc3:9999/Bar1025’’);

here again, foo_Bar b may now be used exactly like any other Babel object. To relate back to normal Babel, connection is kind of like an active addRef. The user actively goes and gets his own reference to a given object.

_connect is actually used by Babel internally whenever objects are passed remotely as arguments. In fact, users will probably rarely use connect directly, most often it will be done automatically by Babel when objects are passed remotely. _connect is exposed to the user mostly for Web Services, where the objectID may always be the same, and for special boot strapping uses.

18.2.4  RMI Arguments

All basic types are passed by value in Babel RMI. They are actually copied across the network. This is reasonable since they are small. Arrays are also copy-only, so anytime an array is passed remotely through Babel RMI, it is actually copied to the remote machine.

Because arrays are copy-only for RMI, there is a noticeable difference in the behavior of an in array argument between a non-RMI call and an RMI call. For the non-RMI call, the code implementing the method can change elements of the incoming array. Because the caller and callee share the same array, the caller’s copy of the array will also be changed. For an RMI call, even if the code implementing the method changes elements of the incoming array, the caller’s copy can never be modified because the client and server each have a distinct copy of the array data.

There are two ways one can pass objects in Babel RMI, by reference, and by copy. The default method is pass-by-reference. For example, server A calls a function foo on server B, and passes a local object Bar as an argument. In this case A will actually pass the URL of Bar to B, B will then call _connect on the URL, which connects back to the object Bar on A.

Pass-by-copy (also called serialization) is different. Pass-by-copy means that a new object is actually created locally on B, and filled in with the values from the object Bar on A. The result is two distinct local objects, one on A and one on B. In order to pass by copy, copy must be used as an argument modifier in the SIDL file. For example:

copy Bar retBar(copy in Foo f)

This sidl function takes a copy of a Foo and returns a copy of a Bar.

Passing by copy also requires the object being passed implements sidl.io.Serializable:

Serializable declares two methods, packObj and unPackObj. packObj serializes the internal object data to a string. unPackObj reinstates the data into the new object by unserializing it from a string. The library developer must implement these functions because Babel does not know what data is in the object, or how it should be serialized. Examples of packObj and unpackObj implementations can be found in sidl.rmi.SIDLException and sidl.rmi.NetworkException.

18.2.5  Casting Remote Objects

Babel RMI casting works the same as normal Babel casting, the user calls casts an object to a new type, and gets a new reference back of the object of the new type. In normal Babel, the new reference points to the same IOR object as the old reference. This is because all Babel objects are internally represented as the type they were created as, so casting is simply a matter of checking if the internal Babel type extends the target type or not.

Babel RMI objects are more complex, a cast may result in a new stub. If _connect() is called on a remote object, the object can be connected as a super type of its actual type, such as an interface. If this object is later cast to a more derived type, a new local object stub must be created. These two stubs must be deleteRef’d individually.

Here is an example where foo_Quux extends foo_Bar. The first is what the user should do, the second is an error.

Do not do this:

18.3  Babel RMI Usage

The previous section generally covered the capabilities of RMI. This section covers the actual usage of those features.

18.3.1  Adding Protocols

In a normal Babel RMI client program, the first thing that needs to be done is adding any protocols that the user plans to use to the ProtocolFactory. The ProtocolFactory is a mapping from protocol prefix that normally proceeds a URL, and the protocol’s actual implementation. The only method the user ever needs to access is addProtocol.

static bool addProtocol( in string prefix, in string typeName );

addProtocol takes the protocol prefix and the fully qualified SIDL protocol typename. It returns TRUE on success. For example, normally the shortname for the “Simple Protocol” protocol is “simhandle.” So we would call the ProtocolFactory like this:

sidl_rmi__addProtocol(‘‘simhandle’’,’’sidl.rmi.SimHandle’’);

Now Babel RMI knows what to call when it encounters a URL prefixed by “simhandle://”.

18.3.2  Built-in Functions

We already covered two important built-in Babel RMI functions in Section 18.2.3. They were the _create[Remote] and _connect static built-in functions, which remotely create an object or connect to an already existing remote object respectively.

There are three other important RMI related built-in functions that a user may find useful. The first two are related:

bool _isRemote();

bool _isLocal();

_isLocal() and _isRemote() are opposites. _isRemote() returns TRUE if the object it is called on is a remote object. _isLocal(), on the other hand, returns TRUE if the object is implemented locally.

Many Babel RMI users will never need these functions. If you don’t care where an object exists, or you already know statically, these methods are totally superfluous. However, since calls on remote objects have serious performance implications, we included these functions for convenience.

There is one other important RMI related built-in:

string _getURL();

This function returns the URL of the object it is called on. This is straight forward for Remote objects, but for local objects it may have some interesting side effects. First, if there is no BOS running locally, no local objects can be exported remotely. Therefore no local object will have a URL. In this case _getURL() returns NULL. However, if there is a BOS running, then local object may have a URL. In this case, if the object has already been exported, _getURL() will return the URL in the sidl.rmi.InstanceRegistry, if the object is not in the InstanceRegistry, Babel will add it, thereby automatically generating a URL for the object. To reiterate, a possible side-effect of calling _getURL() is that the object it is called on my be added to the InstanceRegistry.

18.3.3  Passing Objects from a client

If there is no BOS running locally, (that is, your process is strictly a client) you cannot expose your local objects to remote machines by reference. However, that doesn’t mean you can’t pass objects around. The client can still pass references to remote objects on other remote servers, and can still pass both local and remote objects by copy. For this section, we will take our examples from this SIDL:

From the above SIDL, you can see that the following C code is perfectly legal for a client:

It’s legal because the remote call returns a reference to another remote object, the client never actually exports any of it’s local objects.

The following chunk is also legal, because it passes a remote object to a different remote server. (Passing it to the same remote server would be OK too.)

And the following is ALSO legal, because clients can pass local objects remotely by copy, they just can’t pass local objects by reference. (This allows users to drive a remote simulation on a cluster from a regular workstation with nothing but a simple client.)

However this final bit of code will throw an exception if run by a client that has no BOS:

18.4  Babel Object Servers

Now that we’ve seen how to use a client, we will take a look at running a Babel Object Server.

18.4.1  Starting up a Babel Object Server

Babel Object Servers are generally easy to start up, although each BOS may have a different construction interface. Here is an example of starting up the “Simple Protocol”

Notice that before the server is run, requestPort must be called. There are actually two versions of requestPort: requestPort, and requestPort[InRange]. requestPort takes one argument, a TCP port number (integer). The port number is the TCP port that the BOS should listen to for connections. requestPort[InRange] takes two integers, which denote a range of ports the BOS may try. Because only one server can listen on any TCP port, if the port is already in use by another program, requestPort may fail. requestPort[InRange] gets around this by giving the BOS a range of ports to try. The BOS will try ports in this range until the whole range is exhausted or it could successfully bind to a port.

run returns a long. This return argument is meant to hold the thread id of the thread waiting for connections. The user may wish to join on the thread in order to keep the “Simple Protocol” server from exiting prematurely. (We are now past the “Simple Protocol” specific portion of this section)

After calling run the server is running, but you won’t be able to export any local objects until you register the server with the sidl.rmi.ServerRegistry. Every BOS must be registered with the ServerRegistry, and therefore every BOS must implement the sidl.rmi.ServerInfo interface. This interface is what allows the server to interact with the ServerRegistry.

The ServerRegistry is a singleton class that Babel RMI uses internally to interface with the BOS. It interfaces through the sidl.rmi.ServerInfo interface:

Simply cast the BOS to a ServerInfo and register it with the ServerRegistry.

The user is never really meant to use the ServerInfo interface. In some cases a user may wish to call getExceptions() through the ServerRegistry. getExceptions() is an advanced function. Usually, if an exception is raised in the BOS by a remote call, the exception is returned back to the caller. However, in some cases this is not possible. In those cases the BOS logs the exceptions. Later, a user may use getExceptions to get the logged exceptions.

NOTE: Currently the ServerRegistry can only handle one ServerInfo. This means that Babel can effectively only support one BOS at a time for exporting local objects. (There are hairy ways around this) This is because there are a lot of issues that appear when a user can export objects with a number of different protocols that we have not dealt with. This may be researched further in the future.

18.4.2  Publishing Babel Objects

Once you have a BOS up and running, you are free to export your local object to remote servers. (And, depending on your BOS, remote clients may be able to create and access objects on your BOS.) Exporting an object basically means that remote Babel processes can access the object. In implementation, this means that the object is accessible through the sidl.rmi.InstanceRegistry. The InstanceRegistry maps objectIDs to objects, and vice-versa. This is what allows a remote client to get a handle to your object with nothing more than a URL.

There are 3 ways for an object to end up in the InstanceRegistry. The first, and easiest, is simply to pass a local object by-reference as an argument in a remote call. The last example in 18.3.3 works if a BOS is running.

Another possibility is simply to call _getURL() on the local object when a BOS is running. This will add the object to the InstanceRegistry, so theoretically a remote client could access it. Although realistically the remote client would have to get the URL somehow.

The third possibility is to add it to the InstanceRegistry yourself. The InstanceRegistry class:

calling registerInstance by itself results in the same thing as calling _getURL on the object, it puts the object in the registry, and returns a unique objectID. However, by calling registerInstance[ByString], the user can supply their own objectID. This is useful for WebServices and bootstrapping. It is possible to explicitly publish an object with a special name. In fact, the InstanceRegistry allows aliasing, the same object can be in the registry multiple times with different names.

However, there is one issue with using registerInstance[ByString]. What if there is already an object in the registry with that name? There are two possible cases, if the object under that name is the same object you are trying to register, the call is idempotent, it does nothing. However, if a different object in the registry already has that name, the InstanceRegistry registers the new object under a similar, but unique name. Usually a combination of the instanceID passed in by the user and a unique integer. This is usually the correct thing to do, but if the user really wants the object under the original name, they must call removeInstance[ByString] on the object that currently has that name, and re-register the new object.

NOTE: The InstanceRegistry does not addRef objects when they are inserted into it. You must not destroy an object you wish to be accessible remotely. This means that if you create an object, insert it into the instanceRegistry, and then deleteRef it, it will be destroyed. You must keep a reference to it until you wish to remove it from the InstanceRegistry. (The InstanceRegistry does, however, addRef an objects that are gotten from it. If you call getInstance[ByString], you will get a reference to that object and are free to deleteRef it.)

18.4.3  De-publishing Babel Objects

There are two ways to remove an object from the InstanceRegistry. The first, and most automatic, is for it’s reference count to reach 0. When an object is destroyed it is automatically removed from the InstanceRegistry.

The other way to remove an object is to call removeInstance[ByString] or removeInstance[ByClass]. These will remove the objects from the registry without destroying them. They do not addRef however. So, if you create an object, insert it into the InstanceRegistry, remove it from the InstanceRegistry and then deleteRef it, it will be destroyed. (Assuming no one else has addRef’d it in the meantime.)

18.5  Non-Blocking Babel RMI

Non-Blocking RMI is an even more advanced topic, but it is essential to high-speed distributed computing. Non-Blocking RMI allows the user to mix work and communication. Many scientific computing related methods may take a very long time to complete, and the client might like to do some work while waiting. Non-blocking calls return immediately after sending the information to the server. When the response comes back, the user can make a special call to access the data. During the time between the send and the receive, the client is free to do other work.

There are two types of Non-blocking RMI in Babel, Nonblocking and oneway. Both are declared as attributes on the method in SIDL. The difference is that with oneway communication, the client does not expect any return values. A oneway method will not even return an exception, unless it occurs during communication with the server. On the other hand, a non-blocking call can have return arguments. The user will send a request, and get a sidl.rmi.Ticket. Later, the user may use the Ticket to receive the out arguments.

18.5.1  Protocols

18.5.2  Nonblocking SIDL

The SIDL declaring calls to be nonblocking and/or oneway:

Notice that the nonblocking call may take any arguments, but only in arguments are allowed for the oneway call.

18.5.3  Tickets

As mentioned previously, non-blocking RMI uses the class sidl.rmi.Ticket to handle the return values of non-blocking methods. There are actually two interfaces implemented by the Protocol that are used. sidl.rmi.Ticket and sidl.rmi.TicketBook

sidl.rmi.TicketBook is, obviously, a collection of Tickets. A Ticket represents the out arguments of a single non-blocking call. The user may test() if the call has returned yet, or block() until it does. The user can also get an empty TicketBook.

The TicketBook is a little more complex. It extends Ticket as well as creating some of it’s own functions. It is mostly just to allow a user to make a large amount of nonblocking calls and work while they return. This is a common design paradigm in highly parallel scientific computing. In the case of TicketBook, it is assumed the user will input a bunch of Tickets with IDs. Then he can either block() on all of them (waitall), test() to see if any have returned, or block on removeReady (waitany). removeReady will return the id that the Ticket was inserted with so that the user may identify it. Perhaps with a case statement.

One odd thing about TicketBook is that you can insert multiple tickets with the same name. TicketBook will not warn you or throw an exception if you double up on the same name. If two different Tickets are put in the TicketBook with the same name, there is guarantee about what order they will come out in, even if you remove by name.

18.5.4  Non-blocking Usage

The examples in this section will be written in C using the SIDL file given in Section 18.5.2.

Calling a oneway Babel RMI function is syntactically exactly like calling a normal Babel function. The difference is just the danger of not being able to receive any exceptions beyond the initial communication. Example:

Non-blocking calls are a bit more complex, requiring Tickets in order to get the return values. Here’s an example program, now using a non-blocking call. Notice that the inout argument y is passed as an in argument in the send (as a value), and an out argument in the recv (as a pointer).

Now, next is an example of a more complex program, that utilizes the power of TicketBooks to make multiple remote calls, work, and deal with the responses when they return.