Open Stack

On 06/12/2017 03:17 PM, James E. Blair wrote:
> Clint Byrum <clint at fewbar.com> writes:
> 
>> Excerpts from corvus's message of 2017-06-09 13:11:00 -0700:
>>> Clark Boylan <cboylan at sapwetik.org> writes:
>>>
>>>> I'm wary of this simply because it looks a lot like repeating
>>>> OpenStack's (now failed) decision to stick web servers in a bunch of
>>>> python processes then do cooperative multithreading with them along with
>>>> all your application logic. It just gets complicated. I also think this
>>>> underestimates the value of using tools people are familiar with (wsgi
>>>> and flask) particularly if making it easy to jump in and building
>>>> community is a goal.
>>>
>>> I agree that mixing an asyncio based httpserver with application logic
>>> using cooperative multithreading is not a good idea.  Happily that is
>>> not the proposal.  The proposal is that the webserver be a separate
>>> process from the rest of Zuul, it would be an independently scaleable
>>> component, and *only* the webserver would use asyncio.
>>>
>>
>> I'm not totally convinced that having an HTTP service in the scheduler
>> that gets proxied to when appropriate is the worst idea in the short term,
>> since we already have one and it already works reasonably well with paste,
>> we just want to get rid of paste faster than we can refactor it out by
>> making a ZK backend.
>>
>> Even if we remove paste and create a web tier aiohttp thing, we end up
>> writing most of what would be complex about doing it in-process in the
>> scheduler. So, to tack gearman on top of that, versus just letting the
>> reverse proxy do its job, seems like extra work.
> 
> What I'd like to get out of this conversation is a shared understanding
> of what the web tier for Zuul should look like in the future, so that we
> can know where we want to end up eventually, but *not* a set of
> additional requirements for Zuul v3.0.  In other words, I think this is
> a long-term, rather than short-term conversation.
> 
> The way I see it is that we're adding a bunch of new functionality to an
> area of Zuul that we've traditionally kept very simple.  We're growing
> from a simple JSON endpoint to support websockets, event injection via
> hooks, and a full-blown API for historic data.
> 
> That last item in particular calls out for a real web framework.  Since
> it is new work and has substantial interaction with the web framework,
> it would be good to know what our end state is, so that folks working on
> it can go ahead and head in that direction.
> 
> The other aspects, which are largely already implemented, can be ported
> over in the fullness of time.
> 
> We do not need to change how we are doing webhooks or log streaming for
> Zuul v3.0.
> 
> In fact, I imagine that at least initially, we would implement something
> in openstack-infra like what you describe, Clint.  We will have an
> Apache server which proxies status.json requests and webhooks to
> zuul-scheduler, and proxies websocket requests to the streaming server.
> 
> As time permits, we can incorporate those into a comprehensive web
> server with the framework we choose.
> 
> Does that sound like a good plan?
> 
> Does aiohttp alone fit the bill as Monty suggests, or do we need to
> consider something else?

As a followup. We have discovered that autobahn is problematic with the 
manner in which we want to run it, and that aiohttp works great - so the 
websocket streaming will be switching to autobahn. It has also been 
verified that this works great running inside of a thread, so we're fine 
with the use cases we're hoping for.

With that in mind, I believe the path should be for the new server being 
written for the console-log streaming to be called "zuul-web" and that 
it should serve the console-log websocket at /console-stream or 
/console-log or something.

We can then use our Apache frontend to serve /status from the 
zuul-scheduler and /console-stream from zuul-web - and in the fullness 
of time we can potentially move the webhooks and the status page from 
the scheduler to zuul-web should we choose.

Additionally, as we look at things like dashboards, they can go directly 
into zuul-web and be written with aiohttp. As zuul-web is stateless, 
it's a great candidate for a pure scaleout model.

Additionally, if a deployer would prefer to run it in a pre-fork WSGI 
model, it comes with a uvloop enabled Gunicorn Worker:

aiohttp.GunicornUVLoopWebWorker

http://aiohttp.readthedocs.io/en/stable/deployment.html#nginx-gunicorn

There's a good model for:
- simple in-process - good for unit tests and also for AIO deploys
- in-process scale-out - deployer uses something like kubernetes and 
gets scale by just running more than one container
- worker pool - running a pool of workers on a machine with Apache or 
nginx with gunicorn managing

Which I believe means all of our bases are covered and there's no need 
to introduce additional technologies.

Monty