Open Stack

On Sat, Apr 16, 2016, at 12:37 AM, Steven Dake (stdake) wrote:
> 
> 
> On 4/15/16, 6:38 PM, "Clark Boylan" <cboylan at sapwetik.org> wrote:
> 
> >On Fri, Apr 15, 2016, at 02:39 PM, Steven Dake (stdake) wrote:
> >> Hey fellow OpenStackers,
> >> 
> >> Kolla 1.0.0 has a fatal flaw in its design related to data containers.
> >> The details are outlined here:
> >> 
> >>http://docs.openstack.org/developer/kolla/liberty-deployment-warning.html
> >> 
> >> Our plan to rectify this involves taking what is in stable/mitaka and
> >> making it stable/liberty and placing 1-3 patches on top of
> >>stable/liberty
> >> to make it deploy the liberty version of openstack.  These 1-3 patches
> >> include metadata like repo locations, upstream tarball locations, etc.
> >> No actual code.  This is a one time thing; in the future we will be
> >> following a strict bug-backport only policy.
> >> 
> >> Our options include:
> >> 1. make a megapatch diff of liberty vs mitaka and merge that, with a
> >> patch on top to fix the repos
> >> Example here:
> >> https://review.openstack.org/#/c/306625/
> >> 
> >> 2. Tag the tip of stable/liberty with liberty-early-demise, delete the
> >> liberty branch, then create a new stable/liberty branch from the tip of
> >> stable/mitaka
> >> 
> >> 3. Tag the tip of stable/liberty with liberty-early-demise, and run git
> >> reset -hard origin/stable/mitaka outside of gerrit.  Tonyb says our
> >>setup
> >> prevents non-fast-forwarded pushes so this might not be viable.
> >> 
> >> This was tonyb's work here:
> >> http://paste.openstack.org/raw/494295/
> >> 
> >> What we want to avoid is jamming 1k+ patches through our CI system and
> >> having the core reviewers have to deal with acking all those patches, or
> >> overloading gerrit and breaking things there.
> >> 
> >> I am far from a git wizard, and don't really know the best way to
> >> proceed, other  than we must have a liberty 1.1.0 deliverable with our
> >> mitaka bits + liberty repo pointers.  I'd really like to preserve
> >> history.  What we need is for stable/liberty to be an exact duplicate of
> >> stable/mitaka codwise, while also preserving history (which option 1
> >> doesn't do).
> >> 
> >> Can the Kolla community get an ack on the options from the
> >>Infrastructure
> >> team one way or another, so I can get the ball rolling on our end and
> >> sync up with Doug on the plan?
> >
> >Is there some reason that the normal backport process used on all of the
> >other projects wouldn't work here? Backport the bug fixes you need from
> >the newer branches then release a 1.1.0.
> >
> >Clark
> 
> Clark,
> 
> Great question.  Thanks for the response :)
> 
> We have over 1000 patches delta between stable/mitaka and stable/liberty
> and some of them will conflict.  But I guess we could do that and jam up
> the gerrit/jenkins with that and the core reviewers with approving all
> those backports.
> 
> Unfortunately because of the critical flaw, we really can't just
> cherrypick ~30% of the commits - we need pretty much the whole repo
> intact.  Its like a big onion unpeel.  In essence we need a docker
> feature
> called named volumes to correct the critical flaw.  Before we were using
> data containers which results in data loss.  To gain access to named
> volumes with Ansible 1.9.4, we had to write our own docker module called
> kolla_docker.py.  kolla_docker.py is different then the Ansible bulit-in
> module to integrate with Docker.  The Ansible built-in module only works
> with docker 1.8.2(hard pin) and named volumes were introduced in Docker
> 1.9.0.  Ansible Inc has stopped maintenance on Ansible 1.9 series and is
> focusing all of their efforts on Ansible 2.0.  Porting our code base to
> Ansible 2.0 is planned for Newton but realistically its a 2-3 month job.
> IIUC Ansible 2.0 still has a hard pin on 1.8.2 when used with their
> Docker
> module - so we still prefer to use our own module to decouple Ansible's
> release cycle from the fast moving Docker upstream (6 y stream releases a
> year).
> 
> Therefore, kolla-docker is a requirement to remove the hard pin and gain
> access to named volumes.  We have approximately 300-500 (super rough
> guess) patches to port our code base to use this kolla_docker.py module.
> We really don't have any consistent way to identify the relevant patches.
> I have concerns with this type of backport approach, as we would
> invalidate all of the testing we have done to date with Mitaka and
> Liberty
> - it would basically be forking our own code base for one version of
> Kolla.  
> 
> It is actually much more complex then I've outlined above - there are
> other things in play here, involving how our diagnostics system works
> (and
> won't work with docker 1.9 and the current Liberty code) meaning we have
> to backport our diagnostics system intact as well.  There are other parts
> of Kolla that fit into the same category - lots of interdependencies
> mostly centered around kolla_docker.py.
> 
> Both Doug and Tony mentioned a backport of this type is not ideal, and a
> better solution should be found.  I agree completely - at this point the
> viable options are listed above.
> 
> I recognize the above approaches aren't how repos should be managed in
> OpenStack (or anywhere else for that matter).  We have corrected our
> deficiencies with backport management this cycle.  We are ready to go
> with
> bug-fix only backports from now into the far future because we fixed the
> pain with how Ansible handles Docker integration via kolla-docker.py.
> 
> The mission of Kolla is to provide Operators a smooth experience
> deploying
> and operating OpenStack.  We would be in violation of our mission in my
> opinion if we did only a partial backport of our repo.
> 
> The only other alternative is to abandon the Liberty repository entirely,
> which has been discussed.  We believe this would prematurely and
> permanently damage the Kolla project's viability in the eyes of
> Operators,
> and result in overall damage to OpenStack in general.
> 
> We do plan to tag a 1.1.0 release as soon as one of the above approaches
> is executed - which we think is about 1-2 days of dev work if the repos
> were equivalent.  I could see backporting individually using the standard
> process taking months and resulting in completely broken software.  Our
> best option at this point is to take mitaka as is and point the tarball
> and repo files we depend on at the proper liberty locations.
> 
> FWIW the liberty patch has already been produced (against master) - and
> while not complete, results in a working from-source deploy.  A second
> patch needs to be developed to make from-binary deploys work.  Reference:
> 
> https://review.openstack.org/#/c/306630/

In this case option 2 is probably your best bet. You should be able to
coordinate that with the release team as they have permissions to create
and delete branches on the Kolla project. The basic process would be tag
old branch, abandon all existing changes on stable/liberty, delete the
branch, create the branch with a new head, push and merge the fixes to
switch to liberty things.

The thing to keep in mind is all of your devs and users will need to
forcefully update their local stable/liberty branches as this will be a
non fastforward change if liberty and mitaka/master have diverged. If
this is going to be a big problem then you should consider the backport
process (either commit by commit or squashed).

Clark