[openstack-dev] [nova] blueprint about multiple workers supported in nova-scheduler

Sylvain Bauza sbauza at redhat.com
Thu Mar 5 09:12:39 UTC 2015


Le 05/03/2015 08:54, Rui Chen a écrit :
> We will face the same issue in multiple nova-scheduler process case, 
> like Sylvain say, right?
>
> Two processes/workers can actually consume two distinct resources on 
> the same HostState.
>

No. The problem I mentioned was related to having multiple threads 
accessing the same object in memory.
By running multiple schedulers on different hosts and listening to the 
same RPC topic, it would work - with some caveats about race conditions 
too, but that's unrelated to your proposal -

If you want to run multiple nova-scheduler services, then just fire them 
up on separate machines (that's HA, eh) and that will work.

-Sylvain

>
>
>
> 2015-03-05 13:26 GMT+08:00 Alex Xu <soulxu at gmail.com 
> <mailto:soulxu at gmail.com>>:
>
>     Rui, you still can run multiple nova-scheduler process now.
>
>
>     2015-03-05 10:55 GMT+08:00 Rui Chen <chenrui.momo at gmail.com
>     <mailto:chenrui.momo at gmail.com>>:
>
>         Looks like it's a complicated problem, and nova-scheduler
>         can't scale-out horizontally in active/active mode.
>
>         Maybe we should illustrate the problem in the HA docs.
>
>         http://docs.openstack.org/high-availability-guide/content/_schedulers.html
>
>         Thanks for everybody's attention.
>
>         2015-03-05 5:38 GMT+08:00 Mike Bayer <mbayer at redhat.com
>         <mailto:mbayer at redhat.com>>:
>
>
>
>             Attila Fazekas <afazekas at redhat.com
>             <mailto:afazekas at redhat.com>> wrote:
>
>             > Hi,
>             >
>             > I wonder what is the planned future of the scheduling.
>             >
>             > The scheduler does a lot of high field number query,
>             > which is CPU expensive when you are using sqlalchemy-orm.
>             > Does anyone tried to switch those operations to
>             sqlalchemy-core ?
>
>             An upcoming feature in SQLAlchemy 1.0 will remove the vast
>             majority of CPU
>             overhead from the query side of SQLAlchemy ORM by caching
>             all the work done
>             up until the SQL is emitted, including all the function
>             overhead of building
>             up the Query object, producing a core select() object
>             internally from the
>             Query, working out a large part of the object fetch
>             strategies, and finally
>             the string compilation of the select() into a string as
>             well as organizing
>             the typing information for result columns. With a query
>             that is constructed
>             using the “Baked” feature, all of these steps are cached
>             in memory and held
>             persistently; the same query can then be re-used at which
>             point all of these
>             steps are skipped. The system produces the cache key based
>             on the in-place
>             construction of the Query using lambdas so no major
>             changes to code
>             structure are needed; just the way the Query modifications
>             are performed
>             needs to be preceded with “lambda q:”, essentially.
>
>             With this approach, the traditional session.query(Model)
>             approach can go
>             from start to SQL being emitted with an order of magnitude
>             less function
>             calls. On the fetch side, fetching individual columns
>             instead of full
>             entities has always been an option with ORM and is about
>             the same speed as a
>             Core fetch of rows. So using ORM with minimal changes to
>             existing ORM code
>             you can get performance even better than you’d get using
>             Core directly,
>             since caching of the string compilation is also added.
>
>             On the persist side, the new bulk insert / update features
>             provide a bridge
>             from ORM-mapped objects to bulk inserts/updates without
>             any unit of work
>             sorting going on. ORM mapped objects are still more
>             expensive to use in that
>             instantiation and state change is still more expensive,
>             but bulk
>             insert/update accepts dictionaries as well, which again is
>             competitive with
>             a straight Core insert.
>
>             Both of these features are completed in the master branch,
>             the “baked query”
>             feature just needs documentation, and I’m basically two or
>             three tickets
>             away from beta releases of 1.0. The “Baked” feature itself
>             lives as an
>             extension and if we really wanted, I could backport it
>             into oslo.db as well
>             so that it works against 0.9.
>
>             So I’d ask that folks please hold off on any kind of
>             migration from ORM to
>             Core for performance reasons. I’ve spent the past several
>             months adding
>             features directly to SQLAlchemy that allow an ORM-based
>             app to have routes
>             to operations that perform just as fast as that of Core
>             without a rewrite of
>             code.
>
>             > The scheduler does lot of thing in the application, like
>             filtering
>             > what can be done on the DB level more efficiently. Why
>             it is not done
>             > on the DB side ?
>             >
>             > There are use cases when the scheduler would need to
>             know even more data,
>             > Is there a plan for keeping `everything` in all
>             schedulers process memory up-to-date ?
>             > (Maybe zookeeper)
>             >
>             > The opposite way would be to move most operation into
>             the DB side,
>             > since the DB already knows everything.
>             > (stored procedures ?)
>             >
>             > Best Regards,
>             > Attila
>             >
>             >
>             > ----- Original Message -----
>             >> From: "Rui Chen" <chenrui.momo at gmail.com
>             <mailto:chenrui.momo at gmail.com>>
>             >> To: "OpenStack Development Mailing List (not for usage
>             questions)" <openstack-dev at lists.openstack.org
>             <mailto:openstack-dev at lists.openstack.org>>
>             >> Sent: Wednesday, March 4, 2015 4:51:07 AM
>             >> Subject: [openstack-dev] [nova] blueprint about
>             multiple workers supported   in nova-scheduler
>             >>
>             >> Hi all,
>             >>
>             >> I want to make it easy to launch a bunch of scheduler
>             processes on a host,
>             >> multiple scheduler workers will make use of multiple
>             processors of host and
>             >> enhance the performance of nova-scheduler.
>             >>
>             >> I had registered a blueprint and commit a patch to
>             implement it.
>             >>
>             https://blueprints.launchpad.net/nova/+spec/scheduler-multiple-workers-support
>             >>
>             >> This patch had applied in our performance environment
>             and pass some test
>             >> cases, like: concurrent booting multiple instances,
>             currently we didn't find
>             >> inconsistent issue.
>             >>
>             >> IMO, nova-scheduler should been scaled horizontally on
>             easily way, the
>             >> multiple workers should been supported as an out of box
>             feature.
>             >>
>             >> Please feel free to discuss this feature, thanks.
>             >>
>             >> Best Regards
>             >>
>             >>
>             >>
>             __________________________________________________________________________
>             >> OpenStack Development Mailing List (not for usage
>             questions)
>             >> Unsubscribe:
>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>             <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>             >>
>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>             >
>             >
>             __________________________________________________________________________
>             > OpenStack Development Mailing List (not for usage questions)
>             > Unsubscribe:
>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>             <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>             >
>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>             __________________________________________________________________________
>             OpenStack Development Mailing List (not for usage questions)
>             Unsubscribe:
>             OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>             <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150305/2ac8b220/attachment.html>


More information about the OpenStack-dev mailing list