Open Stack

I agree that we should embrace eventual consistency (under certain cases), but it begs the question of what are u eventually consistent on (maybe u shouldn't be eventually consistent on resource knowledge). U don't have to be eventually consistent on all the things.

So lets assume we are always consistent about static-like resources and then you could offer a 'consistent' scheduler that has no races; u could offer a less-consistent scheduler if there was a pluggable way to do this. But then the question becomes how does the 'consistent' scheduler reserve resources on a compute-node, before actually asking that compute-node to do the work required to fulfill the resource request, this is where I think the reservation process would be useful (of course it then also brings along the question of what do u do about reservation timeouts and cleaning up inactive/unfulfilled reservations). Think of this as planning how to carve up a cake before u carve it up. Nova has enough knowledge (or should) to know what the cake currently looks like (with-in reason, aka minus the dynamic eventually consistent resources) and therefore it should be able to know how to plan the cake carving, before actually doing the cake carving.

This is similar/the same issue (?) that cinder is dealing with with its work on having a defined state-machine (see: https://etherpad.openstack.org/p/CinderTaskFlowFSM) and integrating with taskflow to gain reliable workflows. Personally I prefer a slower (optimize it later) and reliable consistent scheduler & workflow that keeps my operations people sane over a eventually consistent one that has a higher chance of making them insane ;)

Anyway, that’s my current brain dump (and cake analogy, ha).

-Josh

From: Joe Gordon <joe.gordon0 at gmail.com<mailto:joe.gordon0 at gmail.com>>
Date: Monday, November 18, 2013 5:33 PM
To: Joshua Harlow <harlowja at yahoo-inc.com<mailto:harlowja at yahoo-inc.com>>
Cc: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Nova] Does Nova really need an SQL database?




On Mon, Nov 18, 2013 at 4:47 PM, Joshua Harlow <harlowja at yahoo-inc.com<mailto:harlowja at yahoo-inc.com>> wrote:
An idea related to this, what would need to be done to make the DB have the exact state that a compute node is going through (and therefore the scheduler would not make unreliable/racey decisions, even when there are multiple schedulers). It's not like we are dealing with a system which can not know the exact state (as long as the compute nodes are connected to the network, and a network partition does not occur).


Good question, I don't have a clear idea of the amount of work required to do this.

So maybe if we think about ways to correctly reserve resources, and keep up to date information about reserved resources we could then eliminate the race and eliminate the retries entirely?

What is the trade off here? What benefits do we get at what cost? I have a vague idea but just want to be explicit here.  Also for 'cloudy' things we embrace the eventually consistent model, and I don't think we should drop that.


From: Joe Gordon <joe.gordon0 at gmail.com<mailto:joe.gordon0 at gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Monday, November 18, 2013 3:32 PM

To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Nova] Does Nova really need an SQL database?




On Mon, Nov 18, 2013 at 4:08 PM, yunhong jiang <yunhong.jiang at linux.intel.com<mailto:yunhong.jiang at linux.intel.com>> wrote:
On Mon, 2013-11-18 at 14:09 -0800, Joe Gordon wrote:
>
> Phil Day discussed this at the summit and I have finally gotten around
> to posting a POC of this.
>
> https://review.openstack.org/#/c/57053/

Hi, Joe, why you think the DB is not exact state in your followed commit
message? I think the DB is updated to date by resource tracker, am I
right (the resource tracker get the underlying resource information
periodically but I think that information is mostly static). And I think
the scheduler retry mainly comes from the race condition of multiple
scheduler instance.


You answered the question yourself, the compute nodes (indirectly) update the DB periodically, so the further you are from the last periodic update the less up to date the DB is.

Its there for both reasons.  But yes it was originally put there because of the multi scheduler race condition.


"We already have the concept that the DB isn't the exact state of the
world, right now it's updated every 10 seconds. And we use the scheduler
retry mechanism to handle cases where the scheduler was wrong. "


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131119/ae981735/attachment-0001.html>