Open Stack

----- Original Message -----
> >>> I was blindly assuming we want to pull in eventlet support, with the
> >>> implicit understanding that we will be doing some form of timeslicing and
> >>> async io bound waiting in the API... but would like to hear others weigh
> >>> in before I add the monkey_patch and stub code around script startup.
> >>
> >> I'm not so sure that bringing in eventlet should be done by default. It
> >> adds complexity and if most/all of the API calls will be doing some call
> >> to a native C library like libmysql that blocks, I'm not sure there is
> >> going to be much benefit to using eventlet versus multiplexing the
> >> servers using full OS processes -- either manually like some of the
> >> projects do with the workers=N configuration and forking, or using more
> >> traditional multiplexing solutions like running many mod_wsgi or uwsgi
> >> workers inside Apache or nginx.
> >>
> >
> > What about callouts to heat/keystone APIs?
> 
> Sure, it's possible to do that with eventlet. It's also possible to do
> that with a queueing system. For example, the API server could send an
> RPC message to a queue and a separate process could work on executing
> the individual tasks associated with a particular application build-pack.

I guess I was asking because event patterns are generally more efficient for memory than multiprocess, assuming that the underlying codebase isn't fighting the event system at every step.  Are your concerns with eventlet based on that mismatch (bugs, problems with eventlet across the various projects and libraries that OpenStack uses) or more that you believe we should start, at the very beginning, with the pattern of building everything as a distributed ordered task flow since we know at least some of our interactions are asynchronous?  There are at least a few network IO operations that will be synchronous to our flows - while they are not likely to be a large percentage of the API time, they may definitely block threads for a period of time.

> 
> The design of Zuul [1], especially with the relatively recent addition
> of Gearman and the nodepool orchestrator [2] that the openstack-infra
> folks wrote would, IMO, be a worthy place to look for inspiration, since
> Solum essentially is handling a similar problem domain -- distributed
> processing of ordered tasks.
> 
> Best,
> -jay

Familiar with Gearman, will look through nodepool/Zuul.