Event Horizon: Devil Framework's event management system

A story about events, filters, situations and some other details where the Devil (Framework) lives.

Bolz and me recently had a deep-though brainstorming day analyzing and reevaluating the event processing engine embedded into the Devil Framework's Application Server. We had just taken a deep look in some Complex Event Processing systems (especially Colar8) so we wanted to verify that our system was in good shape among its buddies.

Architecture

The Application Server is designed around "events". Its conceptual design is really very simple:

• Events are simple data structures that contain structured data collected from some "data generator". Practically you can consider events has Python dictionaries with some utility methods added and a way to access attributes either as dictionaries keys (es. "event['attr']") either like class attributes (es. "event.attr"). Each event has a "type", a unique "uid" and can be marked as "read_only". There's no practical limit on what can be contained into an event attribute as long as the data type can be "marshaled" by Python (events are transmitted over the network and we use marshal to serialize them for intra-node communication).

• Data is collected by input plugins. This plugins can be instantiated and configured (automatically or by a "policy") on any node of the distributed system, at any time and without system downtime. When an input plugin is installed, it registers what kind of events it generates into the Events Description Repository so that other plugins can easily do some introspection stuff on the generated events (es. "The field 'temperature' from the 'refrigerator_status' event is expressed in 'Celsius Degrees'."). Once activated the plugin starts collecting data from whatever source it is designed to interact with: collected data is converted and normalized into one of the registered events.

• Events generated by input plugins are than piped into the local mangle queue. Mangle plugins are used to register interest into events posted into the mangle queue. This plugins can modify or drop any event they receive: as the mangle process is a serial one (and not a parallel one as for input and output) subsequent plugins will receive the modified events (or if dropped no event at all). Two of the most important mangle plugins are the Filters Manager and the Situations Manager and I'll describe them later.

• Finally, "survived" events are processed by output plugins. As their name says, this plugins are the counterpart of the input ones. Events they receive are translated into a suitable form and than sent to the data recipient. Output recipients can be anything from other system nodes (and by default all events are processed by the Queue Out plugin that sends them to the parent node) or to connected IceBridge consoles or to databases or to whatever they are designed to communicate to.

As you can see this is just a simple input-process-output chain. The real power lies in the modular architecture that allows adding and removing plugins as needed, without downtime, and by the policy-based dynamic configuration system that can change system's modus operandi at specified times or when some kind of event occurs.

Plugins

Now I'm going to briefly describe three event-related plugins provided with the Devil Framework: the Filters Manager, the Situations Manager and the Events Manager.

Filters Manager

The Filters Manager is really two mangle plugins in one as it installs itself as the first and the last plugin to receive events form the mangle queue. This plugin implements and manages a distributed, hierarchical event filtering system.

A filter is a processing entity that takes events as input and evaluates a Python expression (that must return True or False) against each one of them. Different Python code is than executed based on the filter's return value and the events are (not) passed to subsequent filters.

Filters can be defined on each node or in a global repository and referenced by a node filter (filters can be nested into other filters). By default no filter is active and so no event is passed to any filter. To activate a filter you must add to the Pre or/and Post Autostart Filters' list.

A filter definition includes:

• Name. A unique name for the filter. You can organize filters naming them with the classical Python dot form: "test.filter_1" will create a folder "test"with a filter "filter_1" inside. At this time names are sticky; you can not rename a filter (but you can copy and paste a filter into any node).

• Flow On True/False. You can change the filters' invocation flow depending if the filter evaluates to True or False. You can instruct the system to evaluate the "sibling" filter (if any), to go back to the parent filter or to just drop out the filter tree.

• Send To Children On True/False. You can select if an event should be passed to the filter's children nodes (if any) when the filter evaluates to True or False.

• "on_init ()", "on_destroy ()" let you write filter initialization and destruction code. Every global variable defined into the initialization code is made available to the other filter's code properties.

• "eval ()" is the evaluation code executed for every event that is passed to the filter. Must eval to True or False (es. "event.temperature > 50").

• "on_true ()" and "on_false ()" are code properties that are executed respectively when the filter evals to True and False.

• Threaded "on_true ()/on_false ()" properties are used to tell the system that the above code must be executed in a separate thread.

• In the "Filters" you can add sub-filters. They will be evaluated only if the configuration flags above are properly set.

• In the "Macros" tab you can define local macros. Macros are like global functions that are made available in the environment of all the code properties of the filter and once executed use the global and local environment of the code that invokes them. Moreover child filters inherit parent macros, and all filters inherit global macros.

Situations Manager

Situations are a kind of filters, but with memory and continuous evaluation (filters are only evaluated when an event is received). Management is very similar to the filters' one: there are global and local repositories of situations, facts, macros and autostart list (to specify which situations must be made active at startup).

Facts? What's a fact?

Facts are a situation's building blocks. A fact is something like "the light number 5 is turned on" or "there's somebody in the room 1" and when you question it it return its status (True or False). You can configure a fact to refresh itself (reevaluating the eval code without any incoming event) every X seconds, to limit the number of status changes, to be activated by only some kind of events or by events satisfying some condition (es. "event.timestamp < last_event_timestamp + 10" to accept an event only if it occurs less than 10 seconds after the last accepted event).

Situations can be defined by any number of facts and can have any number of child situations (that can be activated on status changes). Situation can be in one of two states ("entered" or "exited") depending on the return value of the "eval (timestamp)" callback. An example may be if "light_1_on_fact and not somebody_in_room_1_fact" then "close_light (1)" and emit event "light_closed" with attribute "number" set to 1. You can also limit the number of times a situation can be reused before being destroyed.

To explain the concepts better here it's how the situation's engine works:

• Facts registered by situations are continuously re-evaluated at the defined intervals. Facts can change status and on status change the "allowed changes" counter is incremented and proper code is executed.

• Each new event (if any) is passed to all the facts interested in its kind and archived into the interested situations (if needed). If a fact changes status, the relative situation is reevaluated.

• When a situation is (re)evaluated it can send "enter" and "exit" events, create new child situations, reset itself and execute proper code.

Once you grab how this sub-system works you can implement really complex event processing solutions with relative easy. And remember that from all the various callbacks you can access the full API of the system.

Events Manager

Well, the Events Manager has a misleading name, as the events it refers to are not the events I've talked about till now. This plugin augments the framework with a GUI like event subscription system, something like Trolltech Qt's slots and signals. Slots are just names to which signal handler are binded to. This signal handlers are invoked when a signal with the right name is emitted.

I want to talk about this plugin because it's an example of how you can use events: "GUI events" are propagated to other nodes and connected consoles encapsulating them into regular events.

The API is very simple:

• With "events_manager.bind (slot, uid, handler, priority=0, **kargs)" a callback function (a signal handler) is binded to a slot. You can assign a priority to the handler to ensure an ordered invocation. You can pass a set of named arguments to be passed to the handler when called. The "uid" argument is an user provided unique ID used as reference to the binding.

• With "events_manager.unbind (slot, uid)" you remove a callback (or a set of callbacks with the same uid) from the slot bindings.

• With "events_manager.enabled (slot, b=None, data=None)" you can query if a slot is enabled (with the "b" argument set to "None") or enable/disable a slot (and if you pass a "time.time ()" value as "data" the slot will be disabled till the given time is reached).

• With "events_manager.emit (slot, routing=None, blocking=False, multicast=False, **kargs)" a new event is created and posted for processing. The "routing" argument is used to define event propagation to consoles: "local" (default) sends the event only to the consoles directly connected to the node, "parents" sends the event to all the consoles connected to a parent nodes, while "children" sends the event to all consoles connected to child nodes.

When an handler is invoked it receives as its first argument the event. An event is a Python dictionary defined as follows:

event = {
        'slot': slot, # slot name
        'routing': routing, # routing policy (None, 'local', ['local', ...])
        'kargs': kargs, # optional arguments
        'user_uid': user_uid, # user id of the event emitter
        'user_info': user_info, # user_info structure of the event emitter
        'drop': False, # drop event flag, set it to True to drop event and prevent calling other handler
        'multicast': False, # multicasting flag
        'blocking': False, # True to block till all handlers are executed
}

A cool feature is that an IceBridge console can bind local handlers to slots on the Application Server it is connected to:

api.events_manager.bind ('rpc:remote_slot', uid, handler)

However you can not "transparently" emit node's events from the console (but you can always use the Events Manager node's API).

The Visualization Side of Life

OK, but when all those events reach an IceBridge console, what can we do with them?

First of all there's a way to intercept and filter incoming events before any console sub-system can take a look at them, using the "api.events_server.add_filter (uid, name, filter)" API. We use this API to process incoming "GUI events" and system wide users' messages. If a filter function returns True, the event is removed from the processing queue.

Another way to get events is using the "api.events_server.add_observer (uid, name, callback, pre_filters=False)" API. The difference between observers and filters is that observers can not dump events from the processing queue.

Views

Then the events will encounter some (opened) view. Views are GUI interfaces developed using the Views Editor and that can be displayed using the Views Browser or embedded in any custom window or dialog (to be more precise, the Views Editor is just an operational mode of the Views Browser; you can start editing a view also while you are using it).

In a view you can insert and edit a whole set of standard (and not so standard) widgets like you do in your preferred GUI editor (an the Views Editor is heavily inspired by Qt Designer), with the difference that you can also add/edit Python code to them and test the view "live", just switching to "view" mode (you can see that the view is always live, even in edit mode, looking at the bottom/right corner of the picture above, where an embedded panel displays the CPU status of an Application Server).

All views and their widgets can receive incoming events but by default they do not react to them. You must set the desired target's (view or widget) "update_mode" property to "wait": this tells the target that it will wait for events (all kinds by default, but you can also select which kind of events you are interested into) and that every X seconds, if any event was received, it must emit the "events" signal. Then you just need to write some code into the "on_events (events)" target's signal handler. That's it!

Final Words Of Self Overindulgence

Mmmmmm..... well...... at the end I think our little Complex Event Processing system can stand quite proud between its pals. Sure, it has its weaknesses (compared to the C based engines it's relatively slow) but it has also its strengths (very, very flexible and customizable, and once you know Python, you are ready to go).

Pat-pat..... ;-)