Open Stack

On 4/15/16, 6:38 PM, "Clark Boylan" <cboylan at sapwetik.org> wrote:

>On Fri, Apr 15, 2016, at 02:39 PM, Steven Dake (stdake) wrote:
>> Hey fellow OpenStackers,
>> 
>> Kolla 1.0.0 has a fatal flaw in its design related to data containers.
>> The details are outlined here:
>> 
>>http://docs.openstack.org/developer/kolla/liberty-deployment-warning.html
>> 
>> Our plan to rectify this involves taking what is in stable/mitaka and
>> making it stable/liberty and placing 1-3 patches on top of
>>stable/liberty
>> to make it deploy the liberty version of openstack.  These 1-3 patches
>> include metadata like repo locations, upstream tarball locations, etc.
>> No actual code.  This is a one time thing; in the future we will be
>> following a strict bug-backport only policy.
>> 
>> Our options include:
>> 1. make a megapatch diff of liberty vs mitaka and merge that, with a
>> patch on top to fix the repos
>> Example here:
>> https://review.openstack.org/#/c/306625/
>> 
>> 2. Tag the tip of stable/liberty with liberty-early-demise, delete the
>> liberty branch, then create a new stable/liberty branch from the tip of
>> stable/mitaka
>> 
>> 3. Tag the tip of stable/liberty with liberty-early-demise, and run git
>> reset -hard origin/stable/mitaka outside of gerrit.  Tonyb says our
>>setup
>> prevents non-fast-forwarded pushes so this might not be viable.
>> 
>> This was tonyb's work here:
>> http://paste.openstack.org/raw/494295/
>> 
>> What we want to avoid is jamming 1k+ patches through our CI system and
>> having the core reviewers have to deal with acking all those patches, or
>> overloading gerrit and breaking things there.
>> 
>> I am far from a git wizard, and don't really know the best way to
>> proceed, other  than we must have a liberty 1.1.0 deliverable with our
>> mitaka bits + liberty repo pointers.  I'd really like to preserve
>> history.  What we need is for stable/liberty to be an exact duplicate of
>> stable/mitaka codwise, while also preserving history (which option 1
>> doesn't do).
>> 
>> Can the Kolla community get an ack on the options from the
>>Infrastructure
>> team one way or another, so I can get the ball rolling on our end and
>> sync up with Doug on the plan?
>
>Is there some reason that the normal backport process used on all of the
>other projects wouldn't work here? Backport the bug fixes you need from
>the newer branches then release a 1.1.0.
>
>Clark

Clark,

Great question.  Thanks for the response :)

We have over 1000 patches delta between stable/mitaka and stable/liberty
and some of them will conflict.  But I guess we could do that and jam up
the gerrit/jenkins with that and the core reviewers with approving all
those backports.

Unfortunately because of the critical flaw, we really can't just
cherrypick ~30% of the commits - we need pretty much the whole repo
intact.  Its like a big onion unpeel.  In essence we need a docker feature
called named volumes to correct the critical flaw.  Before we were using
data containers which results in data loss.  To gain access to named
volumes with Ansible 1.9.4, we had to write our own docker module called
kolla_docker.py.  kolla_docker.py is different then the Ansible bulit-in
module to integrate with Docker.  The Ansible built-in module only works
with docker 1.8.2(hard pin) and named volumes were introduced in Docker
1.9.0.  Ansible Inc has stopped maintenance on Ansible 1.9 series and is
focusing all of their efforts on Ansible 2.0.  Porting our code base to
Ansible 2.0 is planned for Newton but realistically its a 2-3 month job.
IIUC Ansible 2.0 still has a hard pin on 1.8.2 when used with their Docker
module - so we still prefer to use our own module to decouple Ansible's
release cycle from the fast moving Docker upstream (6 y stream releases a
year).

Therefore, kolla-docker is a requirement to remove the hard pin and gain
access to named volumes.  We have approximately 300-500 (super rough
guess) patches to port our code base to use this kolla_docker.py module.
We really don't have any consistent way to identify the relevant patches.
I have concerns with this type of backport approach, as we would
invalidate all of the testing we have done to date with Mitaka and Liberty
- it would basically be forking our own code base for one version of
Kolla.  

It is actually much more complex then I've outlined above - there are
other things in play here, involving how our diagnostics system works (and
won't work with docker 1.9 and the current Liberty code) meaning we have
to backport our diagnostics system intact as well.  There are other parts
of Kolla that fit into the same category - lots of interdependencies
mostly centered around kolla_docker.py.

Both Doug and Tony mentioned a backport of this type is not ideal, and a
better solution should be found.  I agree completely - at this point the
viable options are listed above.

I recognize the above approaches aren't how repos should be managed in
OpenStack (or anywhere else for that matter).  We have corrected our
deficiencies with backport management this cycle.  We are ready to go with
bug-fix only backports from now into the far future because we fixed the
pain with how Ansible handles Docker integration via kolla-docker.py.

The mission of Kolla is to provide Operators a smooth experience deploying
and operating OpenStack.  We would be in violation of our mission in my
opinion if we did only a partial backport of our repo.

The only other alternative is to abandon the Liberty repository entirely,
which has been discussed.  We believe this would prematurely and
permanently damage the Kolla project's viability in the eyes of Operators,
and result in overall damage to OpenStack in general.

We do plan to tag a 1.1.0 release as soon as one of the above approaches
is executed - which we think is about 1-2 days of dev work if the repos
were equivalent.  I could see backporting individually using the standard
process taking months and resulting in completely broken software.  Our
best option at this point is to take mitaka as is and point the tarball
and repo files we depend on at the proper liberty locations.

FWIW the liberty patch has already been produced (against master) - and
while not complete, results in a working from-source deploy.  A second
patch needs to be developed to make from-binary deploys work.  Reference:

https://review.openstack.org/#/c/306630/


Hope the context helps.

Regards
-steve

>