Open Stack

Hello all,

I wrote a small proof of concept for the realtime communication in 
Horizon[1][2]. The code is on Gerrit awaiting your feedback. It's just 
two smallish commits:

https://review.openstack.org/30462
https://review.openstack.org/30463

I would very much like anyone interested to try it out and send their 
thoughts. Below are notes on the implementation and a few things that we 
should hash out.


## The Realtime library

The PoC is written in Python and integrated with Horizon. We're all 
familiar with Python and we can leverage the existing infrastructure 
(views and templates, the Request objects for API authentication, deploy 
both the realtime portion and the traditional code on a single server 
and the common RPC API).

As for the specific transport implementation, I picked 
gevent-socketio[3]. I wanted to use a Socket.IO-compatible solution 
because it supports several fallbacks for WebSockets and allows us to 
switch to Node.js later if necessary.

There are a few Python socket.io implementations out there:

https://github.com/abourget/gevent-socketio
https://github.com/MrJoes/tornadio2
https://github.com/stephenmcd/django-socketio

django-socketio currently supports an out-of-date version of the 
Socket.IO protocol. We may want to revisit it once they get up to date.

The development of tornadio2 seems to be less active, and 
gevent-socketio seems to be more readily integrated into various web 
frameworks. But it seems either would work.


## Deployment

No matter what backend we end up using, the deployment will get more 
complicated. The WSGI specification doesn't support websockets-like 
transfer so this (or any other solution I could find) won't work with 
vanilla WSGI servers (most notably mod_wsgi).

The socket.io part of Horizon can be served via Gevent or Gunicorn and 
since both support WSGI, we can serve the rest of the app alongside. An 
alternative is to keep using mod_wsgi for the non-realtime part and a 
separate process for realtime.

While this complicates things, it isn't worse than using Node.js: there 
we must have two servers: one for Django and one for Node.


## Functionality

The code doesn't do much at the moment. It's more about laying the 
ground work and having something tangible we can discuss. So the only 
functionality right now is that in the /project/instances/ page, the 
change in the instances' statuses are continuously updated using 
socket.io instead of the existing ajax polling.

If you have a few instances running, when you visit the page and then 
change their status out of band (e.g. via the command line), this should 
be reflected in the UI.

I purposefully avoided two thornier issues that need to be resolved 
before this is of any use. These should be discussed and decided before 
we jump in and start to write more code:

1. The current way of updating the UI is to send the new chunk of HTML 
to the browser. For the purposes of the realtime communication, it would 
(I think) make more sense to send just the updated data and update the 
UI client-side.

We can of course keep sending the HTML chunks, if that's the way you all 
want to go, but it limits what we can do.

2. Horizon is mostly a request/response UI. To be able to push changes 
to the browsers, the server needs to figure out when things actually 
changed. We could receive the notifications via RPC or we poll for the 
changes internally and then broadcast them to the browsers.

I wonder if we could leverage Ceilometer here since it already listens 
in on the message bus. Could it be configured to send interesting 
information to Horizon (as opposed to being polled for it)?


I'm not sure whether it makes sense to push the code or not, but I tried 
to make sure I didn't break the standard set up.

Please let me know what you think.

Thanks,
Tomas Sedovic


[1]: https://blueprints.launchpad.net/horizon/+spec/realtime-communication
[2]: https://etherpad.openstack.org/havana-horizon-realtime
[3]: https://github.com/abourget/gevent-socketio