[openstack-dev] Nova workflow management update

John Garbutt john at johngarbutt.com
Thu May 2 11:16:03 UTC 2013


Right now that is 9pm in the UK and in the middle of a Concert Band
rehearsal (not johnthetubaguy for nothing...), which sucks, but there
is never a perfect time for these things.

John

On 1 May 2013 19:47, Joshua Harlow <harlowja at yahoo-inc.com> wrote:
> There is a
> https://wiki.openstack.org/wiki/Meetings#Orchestration_team_meeting which
> I have gotten approval from sandy to take over.
>
> Might as well use that time (since the time-slot is unused and the meeting
> name fits), does that sound good?
>
> -JOsh
>
> On 5/1/13 11:02 AM, "John Garbutt" <john at johngarbutt.com> wrote:
>
>>Scheduling guys started a weekly meeting, maybe this warrants the same
>>approach?
>>Or should we just go for a slot inside the nova meeting?
>>
>>John
>>
>>On 1 May 2013 18:35, Joshua Harlow <harlowja at yahoo-inc.com> wrote:
>>> Wow, very interesting that the work documented in your paper is so
>>>similar!
>>>
>>> Very impressive :-)
>>>
>>> Lets definitely chat sometime. Ping me on IRC or email and lets arrange
>>> something.
>>>
>>> There seems to be a lot of interested parties, so it might just make
>>>sense
>>> to arrange some kind of organized time for all. Since the more we align
>>>the
>>> more we can accomplish together.
>>>
>>> -Josh
>>>
>>> From: Changbin Liu <changbin.liu at gmail.com>
>>>
>>> Reply-To: OpenStack Development Mailing List
>>> <openstack-dev at lists.openstack.org>
>>> Date: Wednesday, May 1, 2013 8:51 AM
>>> To: OpenStack Development Mailing List
>>><openstack-dev at lists.openstack.org>
>>>
>>> Subject: Re: [openstack-dev] Nova workflow management update
>>>
>>> Hi Joshua,
>>>
>>> First, +1 on your documents and code!
>>>
>>> My name is Changbin Liu, from AT&T Labs Research. Together with Yun Mao
>>> (also from AT&T), we once worked on a research prototype cloud
>>>controller
>>> which supports "transactional" cloud resource orchestration (e.g., state
>>> rollback, concurrency, consistency issues, etc), and we used ZooKeeper
>>>to
>>> provide HA and serve as state store. I believe your work shares many
>>> interesting similarities with ours, and we would be very happy to join
>>>your
>>> efforts here to improve Nova state management.
>>>
>>> FYI: our work is documented in a paper:
>>> https://www.usenix.org/system/files/conference/atc12/atc12-final41_0.pdf
>>>
>>> Please let us know when we can set up meetings to discuss more.
>>>
>>>
>>>
>>> Thanks
>>>
>>> Changbin
>>>
>>>
>>> On Fri, Apr 26, 2013 at 11:10 PM, Joshua Harlow <harlowja at yahoo-inc.com>
>>> wrote:
>>>>
>>>> Great to hear all the encouraging feedback!
>>>>
>>>> Since likely people like to see code I thought I'd just throw out some
>>>> pointers to what the prototype is doing.
>>>>
>>>> - https://github.com/yahoo/NovaOrc/blob/master/nova/orc/manager.py#L187
>>>> (new manager, could be part of conductor, TBD)
>>>> - Workflow to fulfill the create request @
>>>> https://github.com/yahoo/NovaOrc/blob/master/nova/orc/manager.py#L216
>>>> - Refactored run_instance states/plugins @
>>>> https://github.com/yahoo/NovaOrc/tree/master/nova/orc/states
>>>> - Potential pieces of new workflow library @
>>>>
>>>>https://github.com/yahoo/NovaOrc/blob/master/nova/orc/states/__init__.py
>>>>
>>>> Work is in progress to add the zookeeper part in (which will vastly
>>>> increase HA, concurrency and add things like distributed resumption).
>>>> --- Related to discussion @
>>>>
>>>>http://lists.openstack.org/pipermail/openstack-dev/2013-April/007881.htm
>>>>l
>>>>
>>>> For those that are interested we are hoping to figure out the
>>>>coordination
>>>> and inter/intra team+project direction shortly (the harder problem
>>>>IMHO).
>>>>
>>>> More details & code to be landed soon!
>>>>
>>>> On 4/26/13 3:14 PM, "Patil, Tushar" <Tushar.Patil at nttdata.com> wrote:
>>>>
>>>> >+1 on all the work done by Joshua and Rohit so far.
>>>> >I think in a very highly scalable solutions like Nova, we need a
>>>> >framework to resume/retry/rollback various stages of vm in a
>>>>structured
>>>> >way.
>>>> >I can see this is driving in that direction.
>>>> >
>>>> >- Tushar
>>>> >
>>>> >>-----Original Message-----
>>>> >>From: Mike Wilson [mailto:geekinutah at gmail.com]
>>>> >>Sent: Friday, April 26, 2013 8:30 AM
>>>> >>To: OpenStack Development Mailing List
>>>> >>Subject: Re: [openstack-dev] Nova workflow management update
>>>> >>
>>>> >>Very exciting to see this happening. Structured state management is
>>>> >>sorely needed. I also like the plan of attack, tackling creation of
>>>>an
>>>> >>instance first is a good way to feel this one out.
>>>> >>
>>>> >>-Mike
>>>> >>
>>>> >>
>>>> >>On Fri, Apr 26, 2013 at 9:21 AM, Senhua Huang (senhuang)
>>>> >><senhuang at cisco.com> wrote:
>>>> >>
>>>> >>
>>>> >>      +1 on the great work on initiating the design and
>>>>implementation
>>>> >>of a more structured and "manageable" provision manager.
>>>> >>      +1 on the documentation.
>>>> >>
>>>> >>      Having a state management separated from resource selection is
>>>> >>very helpful for group scheduling, cross compute/storage/network
>>>> >>scheduling, as well as improved migration solution. It makes Nova
>>>>more
>>>> >>modular, easier to track and reasoning about errors, and easier to
>>>> >>scale.
>>>> >>
>>>> >>      Thanks,
>>>> >>      Senhua
>>>> >>
>>>> >>
>>>> >>      From: <Karajgi>, Rohit <Rohit.Karajgi at nttdata.com>
>>>> >>      Reply-To: OpenStack Development Mailing List <openstack-
>>>> >>dev at lists.openstack.org>
>>>> >>      Date: Thursday, April 25, 2013 11:07 PM
>>>> >>      To: OpenStack Development Mailing List <openstack-
>>>> >>dev at lists.openstack.org>
>>>> >>
>>>> >>      Subject: Re: [openstack-dev] Nova workflow management update
>>>> >>
>>>> >>
>>>> >>
>>>> >>      +1x on the really well written plan and wikis.
>>>> >>
>>>> >>
>>>> >>
>>>> >>      It just goes to show how important having a Structured State
>>>> >>management is to Nova
>>>> >>
>>>> >>      for making it highly reliable and resilient across all APIs.
>>>> >>
>>>> >>
>>>> >>
>>>> >>      We would eventually want to see Nova become SuperNova!! J
>>>> >>
>>>> >>
>>>> >>
>>>> >>      Regards,
>>>> >>
>>>> >>      Rohit
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>      From: Adrian Otto [mailto:adrian.otto at rackspace.com]
>>>> >>      Sent: Friday, April 26, 2013 9:28 AM
>>>> >>      To: OpenStack Development Mailing List
>>>> >>      Cc: OpenStack Development Mailing List
>>>> >>      Subject: Re: [openstack-dev] Nova workflow management update
>>>> >>
>>>> >>
>>>> >>
>>>> >>      Joshua,
>>>> >>
>>>> >>
>>>> >>
>>>> >>      I'm one of the Rackers helping to add development resources to
>>>> >>Convection. I also work on the OASIS CAMP TC as an editor. I want to
>>>> >>help create a reusable task system that helps Nova, Heat, and many
>>>>other
>>>> >>OpenStack projects. I am happy to see this progressing. The recent
>>>> >>collaboration among the Nova and Heat/Convection teams is very
>>>> >>encouraging. Thanks for all your efforts to form a sensible written
>>>> >>plan.
>>>> >>
>>>> >>
>>>> >>
>>>> >>      I'd like to take an editorial pass through the
>>>> >>StructuredStateManagement wiki, and help tighten up the definitions a
>>>> >>bit. Some wording changes may avoid some of the more overloaded
>>>> >>technical terms (but there are practically no pure ones left). I'm
>>>> >>planning to make a few edits to the wiki page (so there will be
>>>>diffs),
>>>> >>but if another approach is preferred, I'm open to that too. Please
>>>> >>advise.
>>>> >>
>>>> >>
>>>> >>
>>>> >>      Thanks,
>>>> >>
>>>> >>
>>>> >>
>>>> >>      Adrian
>>>> >>
>>>> >>
>>>> >>      On Apr 25, 2013, at 5:16 PM, "Joshua Harlow" <harlowja at yahoo-
>>>> >>inc.com> wrote:
>>>> >>
>>>> >>              Since I wanted to make sure everyone was aware of this,
>>>> >>since some of you might have missed the summit session and I'd like
>>>> >>discussions so we can land code in havana.
>>>> >>
>>>> >>
>>>> >>
>>>> >>              For those that missed the session & associated
>>>>material.
>>>> >>
>>>> >>
>>>> >>
>>>> >>              - https://etherpad.openstack.org/the-future-of-orch
>>>> >>(session details + discussion ...)
>>>> >>
>>>> >>
>>>> >>
>>>> >>              The summary of what I am trying to do is to move nova
>>>>away
>>>> >>from having ad-hoc tasks and move it toward having a central entity
>>>>(not
>>>> >>a single entity, but a central one, one that can be horizontally
>>>> >>scalable) which can execute these tasks on-behalf of nova-compute.
>>>>This
>>>> >>central entity (a new orchestrator or conductor...) would centrally
>>>> >>manage
>>>> >>the workflow that nova goes through when completing an API request
>>>>and
>>>> >>would do so in a organized, controlled and resumable manner (it would
>>>> >>also support rollbacks and more...). The reasons why what exists
>>>> >>currently
>>>> >>may not be optimal/good are listed in that etherpad, so I won't
>>>>repeat
>>>> >>them here.
>>>> >>
>>>> >>
>>>> >>
>>>> >>              For example this is a possible diagram for the
>>>> >> run_instance
>>>> >>'workflow' under this new scheme: http://imgur.com/sYOVz5X
>>>> >>
>>>> >>
>>>> >>
>>>> >>              Nttdata and y! have been pursuing how to refactor this
>>>>in
>>>> >> a
>>>> >>well thought out design, and even have prototype code @
>>>> >>https://github.com/Yahoo/NovaOrc which has some of these changes (see
>>>> >>the last 4-10 commits). The prototype was shown in the session but
>>>>feel
>>>> >>free to check out the code, if you setup with that code - its based
>>>>on
>>>> >>stable/grizzly, it should run (note that no external api changes
>>>> >>occurred).
>>>> >>
>>>> >>
>>>> >>
>>>> >>              Some of the outcomes of that meeting I received that
>>>>are
>>>> >>relevant here:
>>>> >>
>>>> >>
>>>> >>
>>>> >>              - HEAT may have a convection library (WIP -
>>>> >>https://wiki.openstack.org/wiki/Convection
>>>> >><https://wiki.openstack.org/wiki/Convection> ) that this workflow
>>>> >>restructuring can use.
>>>> >>
>>>> >>              --- Note: If this code is created quickly (creating a
>>>> >> solid
>>>> >>core) then it seems like we can use this code in nova itself and
>>>>start
>>>> >>restructuring nova into using this code. This of course then allows
>>>>HEAT
>>>> >>to use said library also, and nova as well (and likely creates future
>>>> >>capabilities for something like http://aws.amazon.com/swf). The talk
>>>> >>about this I think is just being started, but it seems like a solid
>>>>core
>>>> >>can be created in a week or two.
>>>> >>
>>>> >>              --- The documentation for my attempt at what I would
>>>>like
>>>> >>this central library to do where put @
>>>> >>https://etherpad.openstack.org/task-system (thx for the heat team for
>>>> >>starting that pad)
>>>> >>
>>>> >>              - There was an ask to document more the overall design
>>>>and
>>>> >>how to accomplish it. I have started this @
>>>> >>https://wiki.openstack.org/wiki/StructuredStateManagement (input is
>>>> >>welcome)
>>>> >>
>>>> >>              --- More details are at
>>>> >>https://wiki.openstack.org/wiki/StructuredStateManagementDetails
>>>>(WIP)
>>>> >>since I didn't want to clutter the main page up...
>>>> >>
>>>> >>              --- Other thoughts of mine at
>>>> >>http://lists.openstack.org/pipermail/openstack-dev/2013-
>>>> >>April/007881.html (with other code associated with it)
>>>> >>
>>>> >>              - There was an ask on how conductor fits into this
>>>> >> picture,
>>>> >>this is still being worked out and discussed (thoughts welcome!)
>>>> >>
>>>> >>              - There was talk about how live migration/resizing can
>>>> >> take
>>>> >>advantage of such a workflow like system to become more secure
>>>>(details
>>>> >>on another email)
>>>> >>
>>>> >>              --- This one involves planning, where imho i would like
>>>> >>nova/heat groups to focus on this core library, and when adjusting
>>>>the
>>>> >>live migration/resize path they should use said core library. If not
>>>>a
>>>> >>core library then the prototype code I have created above (along with
>>>> >>nttdata) can be altered to focus on those paths instead of the
>>>>initial
>>>> >>prototype path of 'run_instance'.
>>>> >>
>>>> >>              - More blueprints - I have started a few @
>>>> >>https://wiki.openstack.org/wiki/StructuredStateManagement#Blueprints
>>>> >>
>>>> >>              - Make a plan on how to get this into mainline, started
>>>> >>this @
>>>>
>>>>>>https://wiki.openstack.org/wiki/StructuredStateManagement#Plan_of_reco
>>>>>>rd
>>>> >>
>>>> >>
>>>> >>
>>>> >>              Discussion is always welcome! I believe we can make
>>>>this
>>>> >>happen (and in all honesty must make it happen).
>>>> >>
>>>> >>
>>>> >>
>>>> >>              I know there are others interested in this
>>>>idea/solution,
>>>> >>so if they want to chime in that would be wonderful :-)
>>>> >>
>>>> >>
>>>> >>
>>>> >>              -Josh
>>>> >>
>>>> >>
>>>> >>
>>>> >>              _______________________________________________
>>>> >>              OpenStack-dev mailing list
>>>> >>              OpenStack-dev at lists.openstack.org
>>>> >>              http://lists.openstack.org/cgi-
>>>> >>bin/mailman/listinfo/openstack-dev
>>>> >>
>>>> >>
>>>> >>
>>>>__________________________________________________________________
>>>> >>____
>>>> >>      Disclaimer:This email and any attachments are sent in strictest
>>>> >>confidence for the sole use of the addressee and may contain legally
>>>> >>privileged, confidential, and proprietary data. If you are not the
>>>> >>intended recipient, please advise the sender by replying promptly to
>>>> >>this email and then delete and destroy this email and any attachments
>>>> >>without any further use, copying or forwarding
>>>> >>
>>>> >>
>>>> >>      _______________________________________________
>>>> >>      OpenStack-dev mailing list
>>>> >>      OpenStack-dev at lists.openstack.org
>>>> >>
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>
>>>> >>
>>>> >>
>>>> >
>>>> >______________________________________________________________________
>>>> >Disclaimer:This email and any attachments are sent in strictest
>>>> >confidence for the sole use of the addressee and may contain legally
>>>> >privileged, confidential, and proprietary data.  If you are not the
>>>> >intended recipient, please advise the sender by replying promptly to
>>>>this
>>>> >email and then delete and destroy this email and any attachments
>>>>without
>>>> >any further use, copying or forwarding
>>>> >
>>>> >_______________________________________________
>>>> >OpenStack-dev mailing list
>>>> >OpenStack-dev at lists.openstack.org
>>>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>_______________________________________________
>>OpenStack-dev mailing list
>>OpenStack-dev at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



More information about the OpenStack-dev mailing list