[openstack-dev] [Fuel] Getting rid of Docker containers on the Fuel master node
Igor Kalnitsky
ikalnitsky at mirantis.com
Fri Nov 20 20:55:00 UTC 2015
Hey Timur,
> I think we can change all docker-based code from our tests / scripts
> in 2-3 days
That sounds good.
> Do we really want to remove docker containers from master node?
Yes, we do. Currently we're suffering from using container-based
architecture on master node, and since we've decided to change our
*upgrade* approach (where we stop gain benefits from containers) it would
be nice to get rid of them and fix a bunch of docker-related bugs.
> How long it will take to provide the experimental MOS 8.0 build
> without docker containers?
I think we need to ask Vladimir Kozhukalov here.
> Are we ready to change the date of MOS 8.0 release and make this
> change?
No, we don't ready to change release date. If we don't have time for it,
let's postpone it till 9.0.
Regards,
Igor
On Fri, Nov 20, 2015 at 12:41 PM Timur Nurlygayanov <
tnurlygayanov at mirantis.com> wrote:
> Hi Igor and Alexander,
>
> >But I'd like to hear from QA how do we rely on container-based
>> infrastructure? Would it be hard to change our sys-tests in short
>> time?
>
> QA team hadn't significant dependencies from docker images in our tests
> [0], I think we can change all docker-based code from our tests / scripts
> in 2-3 days, but it is hard to predict when ISO without docker images will
> pass all SWARM / Tempest tests.
>
> And one more time:
>
>> Of course, we can fix BVT / SWARM tests and don't use docker images in
>> our test suite (it shouldn't be really hard) but we didn't plan these
>> changes and in fact these changes can affect our estimates for many tasks.
>
>
> Do we really want to remove docker containers from master node? How long
> it will take to provide the experimental MOS 8.0 build without docker
> containers?
> Are we ready to change the date of MOS 8.0 release and make this change?
>
> [0]
> https://github.com/openstack/fuel-qa/search?p=2&q=docker&utf8=%E2%9C%93
>
>
> On Fri, Nov 20, 2015 at 7:57 PM, Bogdan Dobrelya <bdobrelia at mirantis.com>
> wrote:
>
>> On 20.11.2015 17:31, Vladimir Kozhukalov wrote:
>> > Bogdan,
>> >
>> >>> So, we could only deprecate the docker feature for the 8.0.
>> >
>> > What do you mean exactly when saying 'deprecate docker feature'? I can
>> > not even imagine how we can live with and without docker containers at
>> > the same time. Deprecation is usually related to features which directly
>> > impact UX (maybe I am wrong).
>>
>> I may be understood this [0] wrong, and the docker containers are not
>> user-visible, but that depends on the which type of users do we mean :-)
>> Sorry, for being not clear.
>>
>> [0]
>>
>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/073576.html
>>
>> >
>> > Guys,
>> >
>> > When you estimate risks of the docker removal, please take into account
>> > not only release deadlines but also the overall product quality. The
>> > thing is that continuing using containers makes it much more complicated
>> > (and thus less stable) to implement new upgrade flow (upgrade tarball
>> > can not be used any more, we need to re-install the host system).
>> > Switching from Centos 6 to Centos 7 is also much more complicated with
>> > docker. Every single piece of Fuel system is going to become simpler and
>> > easier to support.
>> >
>> > Of course, I am not suggesting to jump overboard into cold water without
>> > a life jacket. Transition plan, checklist, green tests, even spec etc.
>> > are assumed without saying (after all, I was not born yesterday). Of
>> > course, we won't merge changes until everything is green. What is the
>> > problem to try to do this and postpone if not ready in time? And please
>> > do not confuse these two cases: switching from plain deployment to
>> > containers is complicated, but switching from docker to plain is much
>> > simpler.
>> >
>> >
>> >
>> >
>> > Vladimir Kozhukalov
>> >
>> > On Fri, Nov 20, 2015 at 6:47 PM, Bogdan Dobrelya <
>> bdobrelia at mirantis.com
>> > <mailto:bdobrelia at mirantis.com>> wrote:
>> >
>> > On 20.11.2015 15:10, Timur Nurlygayanov wrote:
>> > > Hi team,
>> > >
>> > > I think it too late to make such significant changes for MOS 8.0
>> now,
>> > > but I'm ok with the idea to remove docker containers in the future
>> > > releases if our dev team want to do this.
>> > > Any way, before we will do this, we need to plan how we will
>> perform
>> > > updates between different releases with and without docker
>> containers,
>> > > how we will manage requirements and etc. In fact we have a lot of
>> > > questions and haven't answers, let's prepare the spec for this
>> change,
>> > > review it, discuss it with developers, users and project
>> management team
>> > > and if we haven't requirements to keep docker containers on
>> master node
>> > > let's remove them for the future releases (not in MOS 8.0).
>> > >
>> > > Of course, we can fix BVT / SWARM tests and don't use docker
>> images in
>> > > our test suite (it shouldn't be really hard) but we didn't plan
>> these
>> > > changes and in fact these changes can affect our estimates for
>> many tasks.
>> >
>> > I can only add that features just cannot be removed without a
>> > deprecation period of 1-2 releases.
>> > So, we could only deprecate the docker feature for the 8.0.
>> >
>> > >
>> > > Thank you!
>> > >
>> > >
>> > > On Fri, Nov 20, 2015 at 4:44 PM, Alexander Kostrikov
>> > > <akostrikov at mirantis.com <mailto:akostrikov at mirantis.com>
>> > <mailto:akostrikov at mirantis.com <mailto:akostrikov at mirantis.com>>>
>> > wrote:
>> > >
>> > > Hello, Igor.
>> > >
>> > > >But I'd like to hear from QA how do we rely on
>> container-based
>> > > infrastructure? Would it be hard to change our sys-tests in
>> short
>> > > time?
>> > >
>> > > At first glance, system tests are using docker only to fetch
>> logs
>> > > and run shell commands.
>> > > Also, docker is used to run Rally.
>> > >
>> > > If there is an action to remove docker containers with
>> carefull
>> > > attention to bvt testing, it would take couple days to fix
>> system tests.
>> > > But time may be highly affected by code freezes and active
>> features
>> > > merging.
>> > >
>> > > QA team is going to have Monday (Nov 23) sync-up - and it is
>> > > possible to get more exact information from all QA-team.
>> > >
>> > > P.S.
>> > > +1 to remove docker.
>> > > -1 to remove docker without taking into account
>> deadlines/other
>> > > features.
>> > >
>> > > On Thu, Nov 19, 2015 at 10:27 PM, Igor Kalnitsky
>> > > <ikalnitsky at mirantis.com <mailto:ikalnitsky at mirantis.com>
>> > <mailto:ikalnitsky at mirantis.com <mailto:ikalnitsky at mirantis.com>>>
>> > wrote:
>> > >
>> > > Hey guys,
>> > >
>> > > Despite the fact I like containers (as deployment unit),
>> we
>> > > don't use
>> > > them so. That means I +1 idea to drop containers, just
>> because I
>> > > believe that would
>> > >
>> > > * simplify a lot of things
>> > > * helps get rid of huge amount of hacks
>> > > * increase master node deployment
>> > > * release us from annoying support of upgrades /
>> rollbacks that
>> > > proved
>> > > to be non-working well
>> > >
>> > > But I'd like to hear from QA how do we rely on
>> container-based
>> > > infrastructure? Would it be hard to change our sys-tests
>> in short
>> > > time?
>> > >
>> > > Thanks,
>> > > Igor
>> > >
>> > >
>> > > On Thu, Nov 19, 2015 at 10:31 AM, Vladimir Kuklin
>> > > <vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>
>> > <mailto:vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>>> wrote:
>> > > > Folks
>> > > >
>> > > > I guess it should be pretty simple to roll back -
>> install
>> > > older version and
>> > > > restore the backup with preservation of /var/log
>> directory.
>> > > >
>> > > > On Thu, Nov 19, 2015 at 7:38 PM, Sergii Golovatiuk
>> > > > <sgolovatiuk at mirantis.com
>> > <mailto:sgolovatiuk at mirantis.com> <mailto:sgolovatiuk at mirantis.com
>> > <mailto:sgolovatiuk at mirantis.com>>>
>> > > wrote:
>> > > >>
>> > > >> Hi,
>> > > >>
>> > > >> On Thu, Nov 19, 2015 at 5:50 PM, Matthew Mosesohn
>> > > <mmosesohn at mirantis.com <mailto:mmosesohn at mirantis.com>
>> > <mailto:mmosesohn at mirantis.com <mailto:mmosesohn at mirantis.com>>>
>> > > >> wrote:
>> > > >>>
>> > > >>> Vladimir,
>> > > >>>
>> > > >>> The old site.pp is long out of date and should just be
>> > > recreated from the
>> > > >>> content of all the other $service-only.pp files.
>> > > >>>
>> > > >>> My main question is how do we propose to do a
>> rollback from
>> > > an update (in
>> > > >>> theory, from 8.0 to 9.0, then back to 8.0)? Should we
>> > > hardcode persistent
>> > > >>> data directories (or symlink them?) to
>> > > >>> /var/lib/fuel/$fuel_version/$service_name, as we are
>> doing
>> > > behind the scenes
>> > > >>> currently with Docker? If we keep that mechanism in
>> place,
>> > > all the existing
>> > > >>> puppet modules can be used without any modifications.
>> On the
>> > > same note,
>> > > >>> upgrade/rollback is the same as backup and restore,
>> that
>> > > means our restore
>> > > >>> should follow a similar approach.
>> > > >>> -Matthew
>> > > >>
>> > > >>
>> > > >> There only one idea I have is to do dual partitioning
>> system.
>> > > The similar
>> > > >> approach is implemented in CoreOS.
>> > > >>
>> > > >>>
>> > > >>>
>> > > >>> On Thu, Nov 19, 2015 at 6:36 PM, Bogdan Dobrelya
>> > > <bdobrelia at mirantis.com <mailto:bdobrelia at mirantis.com>
>> > <mailto:bdobrelia at mirantis.com <mailto:bdobrelia at mirantis.com>>>
>> > > >>> wrote:
>> > > >>>>
>> > > >>>> On 19.11.2015 15:59, Vladimir Kozhukalov wrote:
>> > > >>>> > Dear colleagues,
>> > > >>>> >
>> > > >>>> > As might remember, we introduced Docker containers
>> > on the
>> > > master node
>> > > >>>> > a
>> > > >>>> > while ago when we implemented first version of Fuel
>> > > upgrade feature.
>> > > >>>> > The
>> > > >>>> > motivation behind was to make it possible to
>> rollback
>> > > upgrade process
>> > > >>>> > if
>> > > >>>> > something goes wrong.
>> > > >>>> >
>> > > >>>> > Now we are at the point where we can not use our
>> > tarball
>> > > based upgrade
>> > > >>>> > approach any more and those patches that deprecate
>> > > upgrade tarball has
>> > > >>>> > been already merged. Although it is a matter of a
>> > > separate discussion,
>> > > >>>> > it seems that upgrade process rather should be
>> based on
>> > > kind of backup
>> > > >>>> > and restore procedure. We can backup Fuel data on
>> an
>> > > external media,
>> > > >>>> > then we can install new version of Fuel from
>> > scratch and
>> > > then it is
>> > > >>>> > assumed backed up Fuel data can be applied over
>> > this new Fuel
>> > > >>>> > instance.
>> > > >>>>
>> > > >>>> A side-by-side upgrade, correct? That should work as
>> > well.
>> > > >>>>
>> > > >>>> > The procedure itself is under active development,
>> > but it
>> > > is clear that
>> > > >>>> > rollback in this case would be nothing more than
>> just
>> > > restoring from
>> > > >>>> > the
>> > > >>>> > previously backed up data.
>> > > >>>> >
>> > > >>>> > As for Docker containers, still there are potential
>> > > advantages of
>> > > >>>> > using
>> > > >>>> > them on the Fuel master node, but our current
>> > > implementation of the
>> > > >>>> > feature seems not mature enough to make us benefit
>> > from the
>> > > >>>> > containerization.
>> > > >>>> >
>> > > >>>> > At the same time there are some disadvantages like
>> > > >>>> >
>> > > >>>> > * it is tricky to get logs and other information
>> (for
>> > > example, rpm
>> > > >>>> > -qa) for a service like shotgun which is run
>> inside
>> > > one of
>> > > >>>> > containers.
>> > > >>>> > * it is specific UX when you first need to run
>> > > dockerctl shell
>> > > >>>> > {container_name} and then you are able to debug
>> > > something.
>> > > >>>> > * when building IBP image we mount directory
>> from the
>> > > host file
>> > > >>>> > system
>> > > >>>> > into mcollective container to make image build
>> > faster.
>> > > >>>> > * there are config files and some other files
>> which
>> > > should be shared
>> > > >>>> > among containers which introduces unnecessary
>> > > complexity to the
>> > > >>>> > whole system.
>> > > >>>> > * our current delivery approach assumes we wrap
>> into
>> > > rpm/deb
>> > > >>>> > packages
>> > > >>>> > every single piece of the Fuel system. Docker
>> > images
>> > > are not an
>> > > >>>> > exception. And as far as they depend on other
>> rpm
>> > > packages we
>> > > >>>> > forced
>> > > >>>> > to build docker-images rpm package using kind
>> of
>> > > specific build
>> > > >>>> > flow. Besides this package is quite big (300M).
>> > > >>>> > * I'd like it to be possible to install Fuel not
>> from
>> > > ISO but from
>> > > >>>> > RPM
>> > > >>>> > repo on any rpm based distribution. But it is
>> > double
>> > > work to
>> > > >>>> > support
>> > > >>>> > both Docker based and package based approach.
>> > > >>>>
>> > > >>>> There is another point, the containers long build
>> > time when
>> > > installing
>> > > >>>> the master node.
>> > > >>>>
>> > > >>>> >
>> > > >>>> > Probably some of you can give other examples.
>> > Anyway, the
>> > > idea is to
>> > > >>>> > get
>> > > >>>> > rid of Docker containers on the master node and
>> > switch to
>> > > plane
>> > > >>>> > package
>> > > >>>> > based approach that we used before.
>> > > >>>> >
>> > > >>>> > As far as there is nothing new here, we just need
>> > to use
>> > > our old
>> > > >>>> > site.pp
>> > > >>>> > (with minimal modifications), it looks like it is
>> > possible to
>> > > >>>> > implement
>> > > >>>> > this during 8.0 release cycle. If there are no
>> > principal
>> > > objections,
>> > > >>>> > please give me a chance to do this ASAP (during
>> 8.0), I
>> > > know it is a
>> > > >>>> > huge risk for the release, but still I think I can
>> > do this.
>> > > >>>> >
>> > > >>>> >
>> > > >>>> >
>> > > >>>> >
>> > > >>>> > Vladimir Kozhukalov
>> > > >>>> >
>> > > >>>> >
>> > > >>>> >
>> > > >>>> >
>> > >
>> >
>> __________________________________________________________________________
>> > > >>>> > OpenStack Development Mailing List (not for usage
>> > questions)
>> > > >>>> > Unsubscribe:
>> > > >>>> >
>> > >
>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > > >>>> >
>> > >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > >>>> >
>> > > >>>>
>> > > >>>>
>> > > >>>> --
>> > > >>>> Best regards,
>> > > >>>> Bogdan Dobrelya,
>> > > >>>> Irc #bogdando
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > >
>> __________________________________________________________________________
>> > > >>>> OpenStack Development Mailing List (not for usage
>> questions)
>> > > >>>> Unsubscribe:
>> > > >>>>
>> > >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > > >>>>
>> > >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > >
>> __________________________________________________________________________
>> > > >>> OpenStack Development Mailing List (not for usage
>> questions)
>> > > >>> Unsubscribe:
>> > > >>>
>> > >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > > >>>
>> > >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > >>>
>> > > >>
>> > > >>
>> > > >>
>> > >
>> __________________________________________________________________________
>> > > >> OpenStack Development Mailing List (not for usage
>> questions)
>> > > >> Unsubscribe:
>> > >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > > >>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Yours Faithfully,
>> > > > Vladimir Kuklin,
>> > > > Fuel Library Tech Lead,
>> > > > Mirantis, Inc.
>> > > > +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
>> > > > +7 (926) 702-39-68 <tel:%2B7%20%28926%29%20702-39-68>
>> > > > Skype kuklinvv
>> > > > 35bk3, Vorontsovskaya Str.
>> > > > Moscow, Russia,
>> > > > www.mirantis.com <http://www.mirantis.com>
>> > <http://www.mirantis.com>
>> > > > www.mirantis.ru <http://www.mirantis.ru>
>> > <http://www.mirantis.ru>
>> > > > vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>
>> > <mailto:vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>>
>> > > >
>> > > >
>> > >
>> __________________________________________________________________________
>> > > > OpenStack Development Mailing List (not for usage
>> questions)
>> > > > Unsubscribe:
>> > >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > > >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > >
>> > >
>> > >
>> __________________________________________________________________________
>> > > OpenStack Development Mailing List (not for usage
>> questions)
>> > > Unsubscribe:
>> > >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Kind Regards,
>> > >
>> > > Alexandr Kostrikov,
>> > >
>> > >
>> > > Mirantis, Inc.
>> > >
>> > > 35b/3, Vorontsovskaya St., 109147, Moscow, Russia
>> > >
>> > >
>> > > Tel.: +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
>> > > Tel.: +7 (925) 716-64-52 <tel:%2B7%20%28906%29%20740-64-79>
>> > >
>> > > Skype: akostrikov_mirantis
>> > >
>> > > E-mail:akostrikov at mirantis.com
>> > <mailto:E-mail%3Aakostrikov at mirantis.com>
>> > <mailto:elogutova at mirantis.com <mailto:elogutova at mirantis.com>>
>> > >
>> > > _www.mirantis.com <http://www.mirantis.com>
>> > <http://www.mirantis.ru/>__
>> > > __www.mirantis.ru <http://www.mirantis.ru>
>> > <http://www.mirantis.ru/>_
>> > >
>> > >
>> > >
>> __________________________________________________________________________
>> > > OpenStack Development Mailing List (not for usage questions)
>> > > Unsubscribe:
>> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Timur,
>> > > Senior QA Engineer
>> > > OpenStack Projects
>> > > Mirantis Inc
>> > >
>> > >
>> > >
>> >
>> __________________________________________________________________________
>> > > OpenStack Development Mailing List (not for usage questions)
>> > > Unsubscribe:
>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> >
>> >
>> > --
>> > Best regards,
>> > Bogdan Dobrelya,
>> > Irc #bogdando
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Irc #bogdando
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
>
> Timur,
> Senior QA Engineer
> OpenStack Projects
> Mirantis Inc
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151120/814dc22a/attachment.html>
More information about the OpenStack-dev
mailing list