Open Stack

On Sat, Apr 16, 2016 at 07:16:27PM +0000, Steven Dake (stdake) wrote:
> 
> 
> On 4/16/16, 11:59 AM, "Clark Boylan" <cboylan at sapwetik.org> wrote:
> 
> >On Sat, Apr 16, 2016, at 12:37 AM, Steven Dake (stdake) wrote:
> >> 
> >> 
> >> On 4/15/16, 6:38 PM, "Clark Boylan" <cboylan at sapwetik.org> wrote:
> >> 
> >> >On Fri, Apr 15, 2016, at 02:39 PM, Steven Dake (stdake) wrote:
> >> >> Hey fellow OpenStackers,
> >> >> 
> >> >> Kolla 1.0.0 has a fatal flaw in its design related to data
> >>containers.
> >> >> The details are outlined here:
> >> >> 
> >> 
> >>>>http://docs.openstack.org/developer/kolla/liberty-deployment-warning.ht
> >>>>ml
> >> >> 
> >> >> Our plan to rectify this involves taking what is in stable/mitaka and
> >> >> making it stable/liberty and placing 1-3 patches on top of
> >> >>stable/liberty
> >> >> to make it deploy the liberty version of openstack.  These 1-3
> >>patches
> >> >> include metadata like repo locations, upstream tarball locations,
> >>etc.
> >> >> No actual code.  This is a one time thing; in the future we will be
> >> >> following a strict bug-backport only policy.
> >> >> 
> >> >> Our options include:
> >> >> 1. make a megapatch diff of liberty vs mitaka and merge that, with a
> >> >> patch on top to fix the repos
> >> >> Example here:
> >> >> https://review.openstack.org/#/c/306625/
> >> >> 
> >> >> 2. Tag the tip of stable/liberty with liberty-early-demise, delete
> >>the
> >> >> liberty branch, then create a new stable/liberty branch from the tip
> >>of
> >> >> stable/mitaka
> >> >> 
> >> >> 3. Tag the tip of stable/liberty with liberty-early-demise, and run
> >>git
> >> >> reset -hard origin/stable/mitaka outside of gerrit.  Tonyb says our
> >> >>setup
> >> >> prevents non-fast-forwarded pushes so this might not be viable.
> >> >> 
> >> >> This was tonyb's work here:
> >> >> http://paste.openstack.org/raw/494295/
> >> >> 
> >> >> What we want to avoid is jamming 1k+ patches through our CI system
> >>and
> >> >> having the core reviewers have to deal with acking all those
> >>patches, or
> >> >> overloading gerrit and breaking things there.
> >> >> 
> >> >> I am far from a git wizard, and don't really know the best way to
> >> >> proceed, other  than we must have a liberty 1.1.0 deliverable with
> >>our
> >> >> mitaka bits + liberty repo pointers.  I'd really like to preserve
> >> >> history.  What we need is for stable/liberty to be an exact
> >>duplicate of
> >> >> stable/mitaka codwise, while also preserving history (which option 1
> >> >> doesn't do).
> >> >> 
> >> >> Can the Kolla community get an ack on the options from the
> >> >>Infrastructure
> >> >> team one way or another, so I can get the ball rolling on our end and
> >> >> sync up with Doug on the plan?
> >> >
> >> >Is there some reason that the normal backport process used on all of
> >>the
> >> >other projects wouldn't work here? Backport the bug fixes you need from
> >> >the newer branches then release a 1.1.0.
> >> >
> >> >Clark
> >> 
> >> Clark,
> >> 
> >> Great question.  Thanks for the response :)
> >> 
> >> We have over 1000 patches delta between stable/mitaka and stable/liberty
> >> and some of them will conflict.  But I guess we could do that and jam up
> >> the gerrit/jenkins with that and the core reviewers with approving all
> >> those backports.
> >> 
> >> Unfortunately because of the critical flaw, we really can't just
> >> cherrypick ~30% of the commits - we need pretty much the whole repo
> >> intact.  Its like a big onion unpeel.  In essence we need a docker
> >> feature
> >> called named volumes to correct the critical flaw.  Before we were using
> >> data containers which results in data loss.  To gain access to named
> >> volumes with Ansible 1.9.4, we had to write our own docker module called
> >> kolla_docker.py.  kolla_docker.py is different then the Ansible bulit-in
> >> module to integrate with Docker.  The Ansible built-in module only works
> >> with docker 1.8.2(hard pin) and named volumes were introduced in Docker
> >> 1.9.0.  Ansible Inc has stopped maintenance on Ansible 1.9 series and is
> >> focusing all of their efforts on Ansible 2.0.  Porting our code base to
> >> Ansible 2.0 is planned for Newton but realistically its a 2-3 month job.
> >> IIUC Ansible 2.0 still has a hard pin on 1.8.2 when used with their
> >> Docker
> >> module - so we still prefer to use our own module to decouple Ansible's
> >> release cycle from the fast moving Docker upstream (6 y stream releases
> >>a
> >> year).
> >> 
> >> Therefore, kolla-docker is a requirement to remove the hard pin and gain
> >> access to named volumes.  We have approximately 300-500 (super rough
> >> guess) patches to port our code base to use this kolla_docker.py module.
> >> We really don't have any consistent way to identify the relevant
> >>patches.
> >> I have concerns with this type of backport approach, as we would
> >> invalidate all of the testing we have done to date with Mitaka and
> >> Liberty
> >> - it would basically be forking our own code base for one version of
> >> Kolla.  
> >> 
> >> It is actually much more complex then I've outlined above - there are
> >> other things in play here, involving how our diagnostics system works
> >> (and
> >> won't work with docker 1.9 and the current Liberty code) meaning we have
> >> to backport our diagnostics system intact as well.  There are other
> >>parts
> >> of Kolla that fit into the same category - lots of interdependencies
> >> mostly centered around kolla_docker.py.
> >> 
> >> Both Doug and Tony mentioned a backport of this type is not ideal, and a
> >> better solution should be found.  I agree completely - at this point the
> >> viable options are listed above.
> >> 
> >> I recognize the above approaches aren't how repos should be managed in
> >> OpenStack (or anywhere else for that matter).  We have corrected our
> >> deficiencies with backport management this cycle.  We are ready to go
> >> with
> >> bug-fix only backports from now into the far future because we fixed the
> >> pain with how Ansible handles Docker integration via kolla-docker.py.
> >> 
> >> The mission of Kolla is to provide Operators a smooth experience
> >> deploying
> >> and operating OpenStack.  We would be in violation of our mission in my
> >> opinion if we did only a partial backport of our repo.
> >> 
> >> The only other alternative is to abandon the Liberty repository
> >>entirely,
> >> which has been discussed.  We believe this would prematurely and
> >> permanently damage the Kolla project's viability in the eyes of
> >> Operators,
> >> and result in overall damage to OpenStack in general.
> >> 
> >> We do plan to tag a 1.1.0 release as soon as one of the above approaches
> >> is executed - which we think is about 1-2 days of dev work if the repos
> >> were equivalent.  I could see backporting individually using the
> >>standard
> >> process taking months and resulting in completely broken software.  Our
> >> best option at this point is to take mitaka as is and point the tarball
> >> and repo files we depend on at the proper liberty locations.
> >> 
> >> FWIW the liberty patch has already been produced (against master) - and
> >> while not complete, results in a working from-source deploy.  A second
> >> patch needs to be developed to make from-binary deploys work.
> >>Reference:
> >> 
> >> https://review.openstack.org/#/c/306630/
> >
> >In this case option 2 is probably your best bet. You should be able to
> >coordinate that with the release team as they have permissions to create
> >and delete branches on the Kolla project. The basic process would be tag
> >old branch, abandon all existing changes on stable/liberty, delete the
> >branch, create the branch with a new head, push and merge the fixes to
> >switch to liberty things.
> >
> >The thing to keep in mind is all of your devs and users will need to
> >forcefully update their local stable/liberty branches as this will be a
> >non fastforward change if liberty and mitaka/master have diverged. If
> >this is going to be a big problem then you should consider the backport
> >process (either commit by commit or squashed).
> >
> >Clark
> 
> Clark,
> 
> I don't think this git issue (checking out a new git repo) will be a big
> problem for our user base but I'll point them at this thread just to make
> certain.
> 
Why not highly recommend to your user base to move to mitaka ASAP?  Since this
will be your latest stable branch.

Seems like a lot of work, and disruption, to liberty for a mitaka release plus 3
patches.

> Regards
> -steve
> >
> 
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra