Open Stack

On 4/16/16, 12:48 PM, "Paul Belanger" <pabelanger at redhat.com> wrote:

>On Sat, Apr 16, 2016 at 07:16:27PM +0000, Steven Dake (stdake) wrote:
>> 
>> 
>> On 4/16/16, 11:59 AM, "Clark Boylan" <cboylan at sapwetik.org> wrote:
>> 
>> >On Sat, Apr 16, 2016, at 12:37 AM, Steven Dake (stdake) wrote:
>> >> 
>> >> 
>> >> On 4/15/16, 6:38 PM, "Clark Boylan" <cboylan at sapwetik.org> wrote:
>> >> 
>> >> >On Fri, Apr 15, 2016, at 02:39 PM, Steven Dake (stdake) wrote:
>> >> >> Hey fellow OpenStackers,
>> >> >> 
>> >> >> Kolla 1.0.0 has a fatal flaw in its design related to data
>> >>containers.
>> >> >> The details are outlined here:
>> >> >> 
>> >> 
>> 
>>>>>>http://docs.openstack.org/developer/kolla/liberty-deployment-warning.
>>>>>>ht
>> >>>>ml
>> >> >> 
>> >> >> Our plan to rectify this involves taking what is in stable/mitaka
>>and
>> >> >> making it stable/liberty and placing 1-3 patches on top of
>> >> >>stable/liberty
>> >> >> to make it deploy the liberty version of openstack.  These 1-3
>> >>patches
>> >> >> include metadata like repo locations, upstream tarball locations,
>> >>etc.
>> >> >> No actual code.  This is a one time thing; in the future we will
>>be
>> >> >> following a strict bug-backport only policy.
>> >> >> 
>> >> >> Our options include:
>> >> >> 1. make a megapatch diff of liberty vs mitaka and merge that,
>>with a
>> >> >> patch on top to fix the repos
>> >> >> Example here:
>> >> >> https://review.openstack.org/#/c/306625/
>> >> >> 
>> >> >> 2. Tag the tip of stable/liberty with liberty-early-demise, delete
>> >>the
>> >> >> liberty branch, then create a new stable/liberty branch from the
>>tip
>> >>of
>> >> >> stable/mitaka
>> >> >> 
>> >> >> 3. Tag the tip of stable/liberty with liberty-early-demise, and
>>run
>> >>git
>> >> >> reset -hard origin/stable/mitaka outside of gerrit.  Tonyb says
>>our
>> >> >>setup
>> >> >> prevents non-fast-forwarded pushes so this might not be viable.
>> >> >> 
>> >> >> This was tonyb's work here:
>> >> >> http://paste.openstack.org/raw/494295/
>> >> >> 
>> >> >> What we want to avoid is jamming 1k+ patches through our CI system
>> >>and
>> >> >> having the core reviewers have to deal with acking all those
>> >>patches, or
>> >> >> overloading gerrit and breaking things there.
>> >> >> 
>> >> >> I am far from a git wizard, and don't really know the best way to
>> >> >> proceed, other  than we must have a liberty 1.1.0 deliverable with
>> >>our
>> >> >> mitaka bits + liberty repo pointers.  I'd really like to preserve
>> >> >> history.  What we need is for stable/liberty to be an exact
>> >>duplicate of
>> >> >> stable/mitaka codwise, while also preserving history (which
>>option 1
>> >> >> doesn't do).
>> >> >> 
>> >> >> Can the Kolla community get an ack on the options from the
>> >> >>Infrastructure
>> >> >> team one way or another, so I can get the ball rolling on our end
>>and
>> >> >> sync up with Doug on the plan?
>> >> >
>> >> >Is there some reason that the normal backport process used on all of
>> >>the
>> >> >other projects wouldn't work here? Backport the bug fixes you need
>>from
>> >> >the newer branches then release a 1.1.0.
>> >> >
>> >> >Clark
>> >> 
>> >> Clark,
>> >> 
>> >> Great question.  Thanks for the response :)
>> >> 
>> >> We have over 1000 patches delta between stable/mitaka and
>>stable/liberty
>> >> and some of them will conflict.  But I guess we could do that and
>>jam up
>> >> the gerrit/jenkins with that and the core reviewers with approving
>>all
>> >> those backports.
>> >> 
>> >> Unfortunately because of the critical flaw, we really can't just
>> >> cherrypick ~30% of the commits - we need pretty much the whole repo
>> >> intact.  Its like a big onion unpeel.  In essence we need a docker
>> >> feature
>> >> called named volumes to correct the critical flaw.  Before we were
>>using
>> >> data containers which results in data loss.  To gain access to named
>> >> volumes with Ansible 1.9.4, we had to write our own docker module
>>called
>> >> kolla_docker.py.  kolla_docker.py is different then the Ansible
>>bulit-in
>> >> module to integrate with Docker.  The Ansible built-in module only
>>works
>> >> with docker 1.8.2(hard pin) and named volumes were introduced in
>>Docker
>> >> 1.9.0.  Ansible Inc has stopped maintenance on Ansible 1.9 series
>>and is
>> >> focusing all of their efforts on Ansible 2.0.  Porting our code base
>>to
>> >> Ansible 2.0 is planned for Newton but realistically its a 2-3 month
>>job.
>> >> IIUC Ansible 2.0 still has a hard pin on 1.8.2 when used with their
>> >> Docker
>> >> module - so we still prefer to use our own module to decouple
>>Ansible's
>> >> release cycle from the fast moving Docker upstream (6 y stream
>>releases
>> >>a
>> >> year).
>> >> 
>> >> Therefore, kolla-docker is a requirement to remove the hard pin and
>>gain
>> >> access to named volumes.  We have approximately 300-500 (super rough
>> >> guess) patches to port our code base to use this kolla_docker.py
>>module.
>> >> We really don't have any consistent way to identify the relevant
>> >>patches.
>> >> I have concerns with this type of backport approach, as we would
>> >> invalidate all of the testing we have done to date with Mitaka and
>> >> Liberty
>> >> - it would basically be forking our own code base for one version of
>> >> Kolla.  
>> >> 
>> >> It is actually much more complex then I've outlined above - there are
>> >> other things in play here, involving how our diagnostics system works
>> >> (and
>> >> won't work with docker 1.9 and the current Liberty code) meaning we
>>have
>> >> to backport our diagnostics system intact as well.  There are other
>> >>parts
>> >> of Kolla that fit into the same category - lots of interdependencies
>> >> mostly centered around kolla_docker.py.
>> >> 
>> >> Both Doug and Tony mentioned a backport of this type is not ideal,
>>and a
>> >> better solution should be found.  I agree completely - at this point
>>the
>> >> viable options are listed above.
>> >> 
>> >> I recognize the above approaches aren't how repos should be managed
>>in
>> >> OpenStack (or anywhere else for that matter).  We have corrected our
>> >> deficiencies with backport management this cycle.  We are ready to go
>> >> with
>> >> bug-fix only backports from now into the far future because we fixed
>>the
>> >> pain with how Ansible handles Docker integration via kolla-docker.py.
>> >> 
>> >> The mission of Kolla is to provide Operators a smooth experience
>> >> deploying
>> >> and operating OpenStack.  We would be in violation of our mission in
>>my
>> >> opinion if we did only a partial backport of our repo.
>> >> 
>> >> The only other alternative is to abandon the Liberty repository
>> >>entirely,
>> >> which has been discussed.  We believe this would prematurely and
>> >> permanently damage the Kolla project's viability in the eyes of
>> >> Operators,
>> >> and result in overall damage to OpenStack in general.
>> >> 
>> >> We do plan to tag a 1.1.0 release as soon as one of the above
>>approaches
>> >> is executed - which we think is about 1-2 days of dev work if the
>>repos
>> >> were equivalent.  I could see backporting individually using the
>> >>standard
>> >> process taking months and resulting in completely broken software.
>>Our
>> >> best option at this point is to take mitaka as is and point the
>>tarball
>> >> and repo files we depend on at the proper liberty locations.
>> >> 
>> >> FWIW the liberty patch has already been produced (against master) -
>>and
>> >> while not complete, results in a working from-source deploy.  A
>>second
>> >> patch needs to be developed to make from-binary deploys work.
>> >>Reference:
>> >> 
>> >> https://review.openstack.org/#/c/306630/
>> >
>> >In this case option 2 is probably your best bet. You should be able to
>> >coordinate that with the release team as they have permissions to
>>create
>> >and delete branches on the Kolla project. The basic process would be
>>tag
>> >old branch, abandon all existing changes on stable/liberty, delete the
>> >branch, create the branch with a new head, push and merge the fixes to
>> >switch to liberty things.
>> >
>> >The thing to keep in mind is all of your devs and users will need to
>> >forcefully update their local stable/liberty branches as this will be a
>> >non fastforward change if liberty and mitaka/master have diverged. If
>> >this is going to be a big problem then you should consider the backport
>> >process (either commit by commit or squashed).
>> >
>> >Clark
>> 
>> Clark,
>> 
>> I don't think this git issue (checking out a new git repo) will be a big
>> problem for our user base but I'll point them at this thread just to
>>make
>> certain.
>> 
>Why not highly recommend to your user base to move to mitaka ASAP?  Since
>this
>will be your latest stable branch.
>
>Seems like a lot of work, and disruption, to liberty for a mitaka release
>plus 3
>patches.

Paul,

Good question.

Operators have historically deployed n-1 or n-2 of OpenStack, so the first
opportunity they would actually deploy Mitaka with Kolla is when Occata
development begins.  That essentially puts a 6 month delay in deployments
using Kolla.

The Kolla community expects instead Operators will deploy stable/liberty
which will be highly stable, because Mitaka is highly stable and builds on
the iterative development we have done for the last two cycles.  Then when
they feel comfortable with field reports of Mitaka success, they might
feel like upgrading to Mitaka :)

Regards
-steve

>
>> Regards
>> -steve
>> >
>> 
>> 
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra