Open Stack

On 30/10/13 15:27 -0400, Jay Pipes wrote:
>Has anyone looked into using concoord for distributed locking?
>
>https://pypi.python.org/pypi/concoord

Looks interesting!

-Angus

>
>Best,
>-jay
>
>On 10/30/2013 02:39 PM, Joshua Harlow wrote:
>>So my idea here was that to break the abstraction for heat into 3 parts.
>>
>>Pardon my lack of heat terminology/knowledge if I miss something.
>>
>>1. The thing that receives the API request (I would assume an api server
>>here).
>>
>>I would expect #1 to parse something into a known internal format. Whether
>>this is tasks or jobs or something is up to heat, so this might of been my
>>lack of understanding heat concepts here, but usually an API request
>>translates into some internal format. Maybe this is the parser or
>>something else (not sure really).
>>
>>Lets assume for now that it parses the API request into some tasks + flow
>>(what taskflow provides).
>>
>>So then it becomes a question of how what do u do with those tasks & flows
>>(what I call stage #2).
>>
>>- https://wiki.openstack.org/wiki/TaskFlow#Two_thousand_foot_view
>>
>>To me this is where taskflow 'shines' in that it has an engine concept
>>which can run in various manners (the tasks and flow are not strongly
>>associated with a engine). One of these engines is planned to be a
>>distributed one (but its not the only one) and with that engine type it
>>would have to interact with some type of job management system (or it
>>would have to provide that job management system - or a simple version
>>itself), but the difference is that the about tasks and flows (and the
>>links/structure between them) is still disconnected from the actual engine
>>that runs those tasks & flows. So this to mean means that there is
>>plugabbility with regard to execution, which I think is pretty great.
>>
>>If that requires rework of the heat model, way of running, maybe its for
>>the better? Idk.
>>
>>As taskflow is still newish, and most projects in openstack have there own
>>distributed model (conductors, rpc process separation), we wanted to focus
>>on having the basic principles down, and the review
>>https://review.openstack.org/#/c/47609/ I am very grateful for jessica
>>working her hardest to get that in a nearly there state. So yes, taskflow
>>will continue on the path/spirit of 47609, and contributions are welcome
>>of course :-)
>>
>>Feel free to also jump on #openstack-state-management since it might be
>>easier to just chat there in the end with other interested parties.
>>
>>-Josh
>>
>>On 10/30/13 11:10 AM, "Steven Dake" <sdake at redhat.com> wrote:
>>
>>>On 10/30/2013 10:42 AM, Clint Byrum wrote:
>>>>So, recently we've had quite a long thread in gerrit regarding locking
>>>>in Heat:
>>>>
>>>>https://review.openstack.org/#/c/49440/
>>>>
>>>>In the patch, there are two distributed lock drivers. One uses SQL,
>>>>and suffers from all the problems you might imagine a SQL based locking
>>>>system would. It is extremely hard to detect dead lock holders, so we
>>>>end up with really long timeouts. The other is ZooKeeper.
>>>>
>>>>I'm on record as saying we're not using ZooKeeper. It is a little
>>>>embarrassing to have taken such a position without really thinking
>>>>things
>>>>through. The main reason I feel this way though, is not because
>>>>ZooKeeper
>>>>wouldn't work for locking, but because I think locking is a mistake.
>>>>
>>>>The current multi-engine paradigm has a race condition. If you have a
>>>>stack action going on, the state is held in the engine itself, and not
>>>>in the database, so if another engine starts working on another action,
>>>>they will conflict.
>>>>
>>>>The locking paradigm is meant to prevent this. But I think this is a
>>>>huge mistake.
>>>>
>>>>The engine should store _all_ of its state in a distributed data store
>>>>of some kind. Any engine should be aware of what is already happening
>>>>with the stack from this state and act accordingly. That includes the
>>>>engine currently working on actions. When viewed through this lense,
>>>>to me, locking is a poor excuse for serializing the state of the engine
>>>>scheduler.
>>>>
>>>>It feels like TaskFlow is the answer, with an eye for making sure
>>>>TaskFlow can be made to work with distributed state. I am not well
>>>>versed on TaskFlow's details though, so I may be wrong. It worries me
>>>>that TaskFlow has existed a while and doesn't seem to be solving real
>>>>problems, but maybe I'm wrong and it is actually in use already.
>>>>
>>>>Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper
>>>>has some real advantages over using the database. But there is hesitance
>>>>because it is not widely supported in OpenStack. What say you, OpenStack
>>>>community? Should we keep ZooKeeper out of our.. zoo?
>>>
>>>I will -2 any patch that adds zookeeper as a dependency to Heat.
>>>
>>>The rest of the idea sounds good though.  I spoke with Joshua about
>>>TaskFlow Friday as a possibility for solving this problem, but TaskFlow
>>>presently does not implement a distributed task flow. Joshua indicated
>>>there was a celerity review at https://review.openstack.org/#/c/47609/,
>>>but this would introduce a different server dependency which suffers
>>>from the same issues as Zookeeper, not to mention incomplete AMQP server
>>>support for various AMQP implementations.  Joshua indicated using a pure
>>>AMQP implementation would be possible for this job but is not implemented.
>>>
>>>I did get into a discussion with him about the subject of breaking the
>>>tasks in the flow into "jobs", which led to the suggestion that the
>>>parser should be part of the API server process (then the engine could
>>>be responsible for handling the various jobs Heat needs). Sounds like
>>>poor abstraction, not to mention serious rework required.
>>>
>>>My take from our IRC discussion was that TaskFlow is not a job
>>>distribution system (like Gearman) but an in-process workflow manager.
>>>These two things are different.  I was unclear if Taskflow could be made
>>>to do both, while also operating under already supported AMQP server
>>>infrastructure that all of OpenStack relies on currently.  If it could,
>>>that would be fantastic, as we would only have to introduce a library
>>>dependency vs a full on server dependency with documentation, HA and
>>>scalability concerns.
>>>
>>>Regards
>>>-steve
>>>
>>>>_______________________________________________
>>>>OpenStack-dev mailing list
>>>>OpenStack-dev at lists.openstack.org
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>_______________________________________________
>>>OpenStack-dev mailing list
>>>OpenStack-dev at lists.openstack.org
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>_______________________________________________
>>OpenStack-dev mailing list
>>OpenStack-dev at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev