The power of “magical” APIs.

When in my last post I reviewed what (in my opinion) we have done right while developing the Devil Framework I rediscovered its API management system. I'm so used to it that I take it for something granted. Its concepts are really simple, but as everybody knows, behind simple concepts powerful features can be found.

The API management system is the way Devil Framework's clients and servers expose functionalities that can be used either internally either externally: plugins use this system to implement and share new APIs. I want to describe the more interesting server side implementation as it provides an extended set of features in comparison of the client side. I'll show also some examples using the remote shell connected to a framework's component.

Easy Of Use

The Api class provides an easy system to register, overwrite and use new APIs. When you are writing code in a shell or for plugins that provides dynamic execution of code, the base shared Api object can be accessed through the global env object as env.api.

>>> env.api
'<Class Api instance@-485f8134>'

To get a list of registered methods:

>>> env.api.methods ()
[ 'distro.remove_manager',
  'distro.list_managers',
  'system_monitor.get_display_pages',
  'wiki.set_is_template',
  ...
]

Calling a registered API is as easy as:

>>> env.api.manager.get_id ()
'cid_1'

Calling the same API from a local GUI console is just as easy, except you make the call through the RPC system:

>>> env.rpc.manager.get_id ()
'cid_1'

Other ways for calling the same API are:

>>> env.api['manager.get_id'] ()
'cid_1'
>>> env.api['manager'].get_id ()
'cid_1'
>>> get_id = env.rpc.manager.get_id
>>> get_id ()
'cid_1'

or any variation on the theme.

Registering a new API is quite easy:

>>> def test_1 (*args, **kargs):
...  print 'ARGS:', args
...  print 'KARGS:', kargs
...  
>>> api_id = env.api.add ('my_tests.test_1', test_1)
>>> api_id
6422
>>> env.api.my_tests.test_1 (1, 2, a='A', b='B')
ARGS: (1, 2)
KARGS: {'a': 'A', 'b': 'B'}

You can register methods, functions and objects. Must be noted that returned values must be standard Python data types (string, long, float, complex, list, tuple, dictionary). This limitation can be circumvented with the use of RPC codecs (I'll explain them later in the network section). When an object is registered as an API, it exposes all of its methods but none of it's properties.

>>> class A (object):
...  p1 = 42
...  def m1 (self):
...   return 7
...  def m2 (self, v=6):
...   return v
...   
>>> class B (object):
...  def m3 (self):
...   return 'Hello'
...   
>>> a = A ()
>>> a.b = B ()
>>> env.api.add ('my_tests.test_2', a)
7959
>>> env.api.my_tests.test_2.p1
'my_tests.test_2.p1'
>>> env.api.my_tests.test_2.p1 == 42
False
>>> env.api.my_tests.test_2.b.m3 ()
'Hello'

Flexibility

To understand why I think the API management system is quite flexible, just take a look of what the api.add and api.get methods can do.

The api.add method accepts the following parameters:

• name: name of the API. Must be a valid Python method name, or a set of valid method names joined with a dot.

• api: function, method or object to be associated to the given name.

• replace: when an API with the given name is already registered, if True replace it, if False (default) raise an exception.

• replaceable: if True (default) the API can be replaced. A replaceable API is not removed from the system when overwritten, but pushed down the “stack”: it will be put back in place whenever the API that overwrote it is removed. The “stack” is of unlimited depth and the “pops” need not to be in reverse “push” order. If access to the old API is needed, remember to store a pointer to it using the api.get method.

• cacheable: True (default) to enable the caching system for this particular API.

• security: specifies what kind of security should be associated to the API. If None (default) the default security policy is applied. There are four security modes available:

  • “Public”: anybody can access this API.

  • “Private”: this API can be accessed from the “owning” component only.

  • “Protected”: checks the resources/privileges system before granting access to the API.

  • “Dynamic”: the given method/function returns (security_type, new_method, args, kargs, info). This values are than used as the “real” API to be called.

The api.get method accepts the following parameters:

• name: name of the API you want.

• default: the value returned if the API is not found (default None).

• wrapper: if True (default) the returned value is a wrapper to the top of the API stack. If the stack is modified the wrapper automatically points to the top. If False the real API is returned.

Network Transparency

As seen above, API calls can be seamlessly made either locally either remotely. Remote calls were made using the rpc prefix that has to different meanings depending where it's used.

On the client, the remote call is performed on the Application Server component the client is directly connected to:

>>> env.rpc.manager.get_id ()
'cid_1'

On the server, the remote call is performed on the parent component (if any).

>>> env.rpc.manager.get_id ()
'cid_1'

To perform a call on any other child node the node ID must be used:

>>> cid = env.api.components.resolve ('C1')
>>> cid
['cid_fbb2cc44_6b2f_11db_b3ef_00138f4756ee']
>>> env.api[env.api.components.resolve ('C1')[0]].manager.get_id ()
'cid_fbb2cc44_6b2f_11db_b3ef_00138f4756ee'

Or if the call has to be made from a client:

>>> env.rpc.cid_fbb2cc44_6b2f_11db_b3ef_00138f4756ee.manager.get_id ()
'cid_fbb2cc44_6b2f_11db_b3ef_00138f4756ee'

Has you can see a component ID can be found using the components.resolve API, which has its counterpart in the components.lookup API:

>>> env.api.components.lookup ('cid_fbb2cc44_6b2f_11db_b3ef_00138f4756ee')
u'C1'

We use globally unique fixed IDs and not names so we can change names at will without the need of updating internal structures and checking for name conflicts.

Encoders and Decoders

As said before, the RPC system can transport basic Python data types only. To resolve this problem we have implemented a sub-system for configuring data encoders and decoders that transparently convert non-basic data types to basic ones and vice versa.

An example will better explain how codecs work. On the server side:

ecm = EncoderManager ()
ecm.add_codec ('database_manager.managers', server_encoder=db_encoder)

def db_encoder (method, result):
    if method.endswith ('.get_result'):
        result = convert_rows (result)
    return result

On the client side:

ecm = EncoderManager ()
ecm.remove_codec ('database_manager.managers', client_decoder=db_decoder)

def db_decoder (method, result):
    if method.endswith ('.get_result'):
        result = convert_result (result)
    return result

On the server side an encoder is installed and it will be called on any RPC call made by a client on an API whose name starts with 'database_manager.managers'. The encoder converts the result if the API call is one that needs encoding. As you can see, you can really do whatever you want on the values to be returned (checks, logging, etc.). The same thing is done on the client side except that e client decoder is installed.

The kind of codec (server_encoder, server_decoder, client_encoder, client_decoders) tells the Encoder Manager and the RPC system where to use the it (client or server) and when to use it (an encoder is used when an API receives a remote call, a decoder is used when a call to a remote API is made).

Codecs can be disabled and re-enabled thread-safely: we use this feature when a plugin acts as a proxy and no data encoding/decoding is needed.

Exceptions Management

Exceptions are simply propagated back to the caller:

>>> env.rpc.manager.get_id (123)
Exception.Base <Internal Error>
Internal Error
Traceback (most recent call last):
  File "/root/DLD/lib/python2.4/threading.py", line 442, in __bootstrap
    
  File "/home/alex/Projects/DLevel/Utilities/Threads/ThreadPool.py", line 59, in run
    job ()
  File "/home/alex/Projects/DLevel/Devil/Common/Plugin/Manager2.py", line 110, in __call__
    return Job.__call__ (self)
--- <exception caught here> ---
  File "/home/alex/Projects/DLevel/Utilities/Threads/ThreadPool.py", line 101, in __call__
    self.callback (
  File "../Plugins/RPCServer/RPCServer.py", line 661, in __call
    
  File "/home/alex/Projects/DLevel/Devil/Common/Api.py", line 201, in apply
    return apply (f, args, kargs)
exceptions.TypeError: get_id() takes exactly 1 argument (2 given)

Security

Access to Protected APIs is filtered by the authorization system. It tests for access rights and performs execution audits. To achieve this the authorization system uses associations between users/groups, resources and APIs. Users are given permissions to do actions on specific resources, and APIs are assigned each to a specific resource's permission.

I know this is a fogy description, but I'll explain how the authentication and authorization system works in a later article.

Implementation Details

The API name resolution, the foundation block on which the API system is based, is handled by the following class:

class MagicMethod (object):
    """Magic class to extract remote method name, args and kargs"""
    def __init__ (self, name, applyfunc, item_to_attr=0):
       self.__apply = applyfunc
       self.__name = name
       self.__item_to_attr = item_to_attr
                                
    def __getattr__ (self, name):
        return MagicMethod (self.__name + '.' + str (name), self.__apply, self.__item_to_attr)
                
    def __getitem__ (self, name):
        if  self.__item_to_attr:
            return MagicMethod (self.__name + '.' + str (name), self.__apply, 1)
       else:
            return MagicMethod (self.__name + '[' + repr (name) + ']', self.__apply , 0)
                
    def __call__ (self, *args, **kargs):
        return self.__apply (self.__name, args, kargs)
                
    def __str__ (self):
        return self.__name
                
    __repr__ = __str__

Some sample code will explain how the MagicMethod class work:

>>> class M (object):
...     def __getattr__ (self, name):
...             return MagicMethod (name, self.__test)
...     def __test (self, name, args, kargs):
...             return (name, args, kargs)
...             
>>> mi = M ()
>>> mi.just.a.test ('hello', who='world')
('just.a.test', ('hello',), {'who': 'world'})

Conclusion

Re-reading this post, I hope no one will put it into the top 3 of the worst poetry in the known universe .

To distract the innocent reader an inspirational masterpiece:

Oh freddled gruntbuggly, 
Thy micturations are to me 
As plurdled gabbleblotchits 
On a lurgid bee. 
Groop, I implore thee, my foonting turlingdromes 
And hooptiously drangle me 
With crinkly bindlewurdles, 
Or I will rend thee in the gobberwarts with my blurglecruncheon, 
See if I don't!

Prostetnic Vogon Jeltz