Open Stack

On 4/16/16, 11:59 AM, "Clark Boylan" <cboylan at sapwetik.org> wrote:

>On Sat, Apr 16, 2016, at 12:37 AM, Steven Dake (stdake) wrote:
>> 
>> 
>> On 4/15/16, 6:38 PM, "Clark Boylan" <cboylan at sapwetik.org> wrote:
>> 
>> >On Fri, Apr 15, 2016, at 02:39 PM, Steven Dake (stdake) wrote:
>> >> Hey fellow OpenStackers,
>> >> 
>> >> Kolla 1.0.0 has a fatal flaw in its design related to data
>>containers.
>> >> The details are outlined here:
>> >> 
>> 
>>>>http://docs.openstack.org/developer/kolla/liberty-deployment-warning.ht
>>>>ml
>> >> 
>> >> Our plan to rectify this involves taking what is in stable/mitaka and
>> >> making it stable/liberty and placing 1-3 patches on top of
>> >>stable/liberty
>> >> to make it deploy the liberty version of openstack.  These 1-3
>>patches
>> >> include metadata like repo locations, upstream tarball locations,
>>etc.
>> >> No actual code.  This is a one time thing; in the future we will be
>> >> following a strict bug-backport only policy.
>> >> 
>> >> Our options include:
>> >> 1. make a megapatch diff of liberty vs mitaka and merge that, with a
>> >> patch on top to fix the repos
>> >> Example here:
>> >> https://review.openstack.org/#/c/306625/
>> >> 
>> >> 2. Tag the tip of stable/liberty with liberty-early-demise, delete
>>the
>> >> liberty branch, then create a new stable/liberty branch from the tip
>>of
>> >> stable/mitaka
>> >> 
>> >> 3. Tag the tip of stable/liberty with liberty-early-demise, and run
>>git
>> >> reset -hard origin/stable/mitaka outside of gerrit.  Tonyb says our
>> >>setup
>> >> prevents non-fast-forwarded pushes so this might not be viable.
>> >> 
>> >> This was tonyb's work here:
>> >> http://paste.openstack.org/raw/494295/
>> >> 
>> >> What we want to avoid is jamming 1k+ patches through our CI system
>>and
>> >> having the core reviewers have to deal with acking all those
>>patches, or
>> >> overloading gerrit and breaking things there.
>> >> 
>> >> I am far from a git wizard, and don't really know the best way to
>> >> proceed, other  than we must have a liberty 1.1.0 deliverable with
>>our
>> >> mitaka bits + liberty repo pointers.  I'd really like to preserve
>> >> history.  What we need is for stable/liberty to be an exact
>>duplicate of
>> >> stable/mitaka codwise, while also preserving history (which option 1
>> >> doesn't do).
>> >> 
>> >> Can the Kolla community get an ack on the options from the
>> >>Infrastructure
>> >> team one way or another, so I can get the ball rolling on our end and
>> >> sync up with Doug on the plan?
>> >
>> >Is there some reason that the normal backport process used on all of
>>the
>> >other projects wouldn't work here? Backport the bug fixes you need from
>> >the newer branches then release a 1.1.0.
>> >
>> >Clark
>> 
>> Clark,
>> 
>> Great question.  Thanks for the response :)
>> 
>> We have over 1000 patches delta between stable/mitaka and stable/liberty
>> and some of them will conflict.  But I guess we could do that and jam up
>> the gerrit/jenkins with that and the core reviewers with approving all
>> those backports.
>> 
>> Unfortunately because of the critical flaw, we really can't just
>> cherrypick ~30% of the commits - we need pretty much the whole repo
>> intact.  Its like a big onion unpeel.  In essence we need a docker
>> feature
>> called named volumes to correct the critical flaw.  Before we were using
>> data containers which results in data loss.  To gain access to named
>> volumes with Ansible 1.9.4, we had to write our own docker module called
>> kolla_docker.py.  kolla_docker.py is different then the Ansible bulit-in
>> module to integrate with Docker.  The Ansible built-in module only works
>> with docker 1.8.2(hard pin) and named volumes were introduced in Docker
>> 1.9.0.  Ansible Inc has stopped maintenance on Ansible 1.9 series and is
>> focusing all of their efforts on Ansible 2.0.  Porting our code base to
>> Ansible 2.0 is planned for Newton but realistically its a 2-3 month job.
>> IIUC Ansible 2.0 still has a hard pin on 1.8.2 when used with their
>> Docker
>> module - so we still prefer to use our own module to decouple Ansible's
>> release cycle from the fast moving Docker upstream (6 y stream releases
>>a
>> year).
>> 
>> Therefore, kolla-docker is a requirement to remove the hard pin and gain
>> access to named volumes.  We have approximately 300-500 (super rough
>> guess) patches to port our code base to use this kolla_docker.py module.
>> We really don't have any consistent way to identify the relevant
>>patches.
>> I have concerns with this type of backport approach, as we would
>> invalidate all of the testing we have done to date with Mitaka and
>> Liberty
>> - it would basically be forking our own code base for one version of
>> Kolla.  
>> 
>> It is actually much more complex then I've outlined above - there are
>> other things in play here, involving how our diagnostics system works
>> (and
>> won't work with docker 1.9 and the current Liberty code) meaning we have
>> to backport our diagnostics system intact as well.  There are other
>>parts
>> of Kolla that fit into the same category - lots of interdependencies
>> mostly centered around kolla_docker.py.
>> 
>> Both Doug and Tony mentioned a backport of this type is not ideal, and a
>> better solution should be found.  I agree completely - at this point the
>> viable options are listed above.
>> 
>> I recognize the above approaches aren't how repos should be managed in
>> OpenStack (or anywhere else for that matter).  We have corrected our
>> deficiencies with backport management this cycle.  We are ready to go
>> with
>> bug-fix only backports from now into the far future because we fixed the
>> pain with how Ansible handles Docker integration via kolla-docker.py.
>> 
>> The mission of Kolla is to provide Operators a smooth experience
>> deploying
>> and operating OpenStack.  We would be in violation of our mission in my
>> opinion if we did only a partial backport of our repo.
>> 
>> The only other alternative is to abandon the Liberty repository
>>entirely,
>> which has been discussed.  We believe this would prematurely and
>> permanently damage the Kolla project's viability in the eyes of
>> Operators,
>> and result in overall damage to OpenStack in general.
>> 
>> We do plan to tag a 1.1.0 release as soon as one of the above approaches
>> is executed - which we think is about 1-2 days of dev work if the repos
>> were equivalent.  I could see backporting individually using the
>>standard
>> process taking months and resulting in completely broken software.  Our
>> best option at this point is to take mitaka as is and point the tarball
>> and repo files we depend on at the proper liberty locations.
>> 
>> FWIW the liberty patch has already been produced (against master) - and
>> while not complete, results in a working from-source deploy.  A second
>> patch needs to be developed to make from-binary deploys work.
>>Reference:
>> 
>> https://review.openstack.org/#/c/306630/
>
>In this case option 2 is probably your best bet. You should be able to
>coordinate that with the release team as they have permissions to create
>and delete branches on the Kolla project. The basic process would be tag
>old branch, abandon all existing changes on stable/liberty, delete the
>branch, create the branch with a new head, push and merge the fixes to
>switch to liberty things.
>
>The thing to keep in mind is all of your devs and users will need to
>forcefully update their local stable/liberty branches as this will be a
>non fastforward change if liberty and mitaka/master have diverged. If
>this is going to be a big problem then you should consider the backport
>process (either commit by commit or squashed).
>
>Clark

Clark,

I don't think this git issue (checking out a new git repo) will be a big
problem for our user base but I'll point them at this thread just to make
certain.

Regards
-steve
>