[openstack-dev] [Fuel] Getting rid of Docker containers on the Fuel master node
Timur Nurlygayanov
tnurlygayanov at mirantis.com
Fri Nov 20 20:39:45 UTC 2015
Hi Igor and Alexander,
>But I'd like to hear from QA how do we rely on container-based
> infrastructure? Would it be hard to change our sys-tests in short
> time?
QA team hadn't significant dependencies from docker images in our tests
[0], I think we can change all docker-based code from our tests / scripts
in 2-3 days, but it is hard to predict when ISO without docker images will
pass all SWARM / Tempest tests.
And one more time:
> Of course, we can fix BVT / SWARM tests and don't use docker images in our
> test suite (it shouldn't be really hard) but we didn't plan these changes
> and in fact these changes can affect our estimates for many tasks.
Do we really want to remove docker containers from master node? How long it
will take to provide the experimental MOS 8.0 build without docker
containers?
Are we ready to change the date of MOS 8.0 release and make this change?
[0] https://github.com/openstack/fuel-qa/search?p=2&q=docker&utf8=%E2%9C%93
On Fri, Nov 20, 2015 at 7:57 PM, Bogdan Dobrelya <bdobrelia at mirantis.com>
wrote:
> On 20.11.2015 17:31, Vladimir Kozhukalov wrote:
> > Bogdan,
> >
> >>> So, we could only deprecate the docker feature for the 8.0.
> >
> > What do you mean exactly when saying 'deprecate docker feature'? I can
> > not even imagine how we can live with and without docker containers at
> > the same time. Deprecation is usually related to features which directly
> > impact UX (maybe I am wrong).
>
> I may be understood this [0] wrong, and the docker containers are not
> user-visible, but that depends on the which type of users do we mean :-)
> Sorry, for being not clear.
>
> [0]
>
> http://lists.openstack.org/pipermail/openstack-dev/2015-September/073576.html
>
> >
> > Guys,
> >
> > When you estimate risks of the docker removal, please take into account
> > not only release deadlines but also the overall product quality. The
> > thing is that continuing using containers makes it much more complicated
> > (and thus less stable) to implement new upgrade flow (upgrade tarball
> > can not be used any more, we need to re-install the host system).
> > Switching from Centos 6 to Centos 7 is also much more complicated with
> > docker. Every single piece of Fuel system is going to become simpler and
> > easier to support.
> >
> > Of course, I am not suggesting to jump overboard into cold water without
> > a life jacket. Transition plan, checklist, green tests, even spec etc.
> > are assumed without saying (after all, I was not born yesterday). Of
> > course, we won't merge changes until everything is green. What is the
> > problem to try to do this and postpone if not ready in time? And please
> > do not confuse these two cases: switching from plain deployment to
> > containers is complicated, but switching from docker to plain is much
> > simpler.
> >
> >
> >
> >
> > Vladimir Kozhukalov
> >
> > On Fri, Nov 20, 2015 at 6:47 PM, Bogdan Dobrelya <bdobrelia at mirantis.com
> > <mailto:bdobrelia at mirantis.com>> wrote:
> >
> > On 20.11.2015 15:10, Timur Nurlygayanov wrote:
> > > Hi team,
> > >
> > > I think it too late to make such significant changes for MOS 8.0
> now,
> > > but I'm ok with the idea to remove docker containers in the future
> > > releases if our dev team want to do this.
> > > Any way, before we will do this, we need to plan how we will
> perform
> > > updates between different releases with and without docker
> containers,
> > > how we will manage requirements and etc. In fact we have a lot of
> > > questions and haven't answers, let's prepare the spec for this
> change,
> > > review it, discuss it with developers, users and project
> management team
> > > and if we haven't requirements to keep docker containers on master
> node
> > > let's remove them for the future releases (not in MOS 8.0).
> > >
> > > Of course, we can fix BVT / SWARM tests and don't use docker
> images in
> > > our test suite (it shouldn't be really hard) but we didn't plan
> these
> > > changes and in fact these changes can affect our estimates for
> many tasks.
> >
> > I can only add that features just cannot be removed without a
> > deprecation period of 1-2 releases.
> > So, we could only deprecate the docker feature for the 8.0.
> >
> > >
> > > Thank you!
> > >
> > >
> > > On Fri, Nov 20, 2015 at 4:44 PM, Alexander Kostrikov
> > > <akostrikov at mirantis.com <mailto:akostrikov at mirantis.com>
> > <mailto:akostrikov at mirantis.com <mailto:akostrikov at mirantis.com>>>
> > wrote:
> > >
> > > Hello, Igor.
> > >
> > > >But I'd like to hear from QA how do we rely on container-based
> > > infrastructure? Would it be hard to change our sys-tests in
> short
> > > time?
> > >
> > > At first glance, system tests are using docker only to fetch
> logs
> > > and run shell commands.
> > > Also, docker is used to run Rally.
> > >
> > > If there is an action to remove docker containers with carefull
> > > attention to bvt testing, it would take couple days to fix
> system tests.
> > > But time may be highly affected by code freezes and active
> features
> > > merging.
> > >
> > > QA team is going to have Monday (Nov 23) sync-up - and it is
> > > possible to get more exact information from all QA-team.
> > >
> > > P.S.
> > > +1 to remove docker.
> > > -1 to remove docker without taking into account deadlines/other
> > > features.
> > >
> > > On Thu, Nov 19, 2015 at 10:27 PM, Igor Kalnitsky
> > > <ikalnitsky at mirantis.com <mailto:ikalnitsky at mirantis.com>
> > <mailto:ikalnitsky at mirantis.com <mailto:ikalnitsky at mirantis.com>>>
> > wrote:
> > >
> > > Hey guys,
> > >
> > > Despite the fact I like containers (as deployment unit), we
> > > don't use
> > > them so. That means I +1 idea to drop containers, just
> because I
> > > believe that would
> > >
> > > * simplify a lot of things
> > > * helps get rid of huge amount of hacks
> > > * increase master node deployment
> > > * release us from annoying support of upgrades / rollbacks
> that
> > > proved
> > > to be non-working well
> > >
> > > But I'd like to hear from QA how do we rely on
> container-based
> > > infrastructure? Would it be hard to change our sys-tests
> in short
> > > time?
> > >
> > > Thanks,
> > > Igor
> > >
> > >
> > > On Thu, Nov 19, 2015 at 10:31 AM, Vladimir Kuklin
> > > <vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>
> > <mailto:vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>>> wrote:
> > > > Folks
> > > >
> > > > I guess it should be pretty simple to roll back - install
> > > older version and
> > > > restore the backup with preservation of /var/log
> directory.
> > > >
> > > > On Thu, Nov 19, 2015 at 7:38 PM, Sergii Golovatiuk
> > > > <sgolovatiuk at mirantis.com
> > <mailto:sgolovatiuk at mirantis.com> <mailto:sgolovatiuk at mirantis.com
> > <mailto:sgolovatiuk at mirantis.com>>>
> > > wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> On Thu, Nov 19, 2015 at 5:50 PM, Matthew Mosesohn
> > > <mmosesohn at mirantis.com <mailto:mmosesohn at mirantis.com>
> > <mailto:mmosesohn at mirantis.com <mailto:mmosesohn at mirantis.com>>>
> > > >> wrote:
> > > >>>
> > > >>> Vladimir,
> > > >>>
> > > >>> The old site.pp is long out of date and should just be
> > > recreated from the
> > > >>> content of all the other $service-only.pp files.
> > > >>>
> > > >>> My main question is how do we propose to do a rollback
> from
> > > an update (in
> > > >>> theory, from 8.0 to 9.0, then back to 8.0)? Should we
> > > hardcode persistent
> > > >>> data directories (or symlink them?) to
> > > >>> /var/lib/fuel/$fuel_version/$service_name, as we are
> doing
> > > behind the scenes
> > > >>> currently with Docker? If we keep that mechanism in
> place,
> > > all the existing
> > > >>> puppet modules can be used without any modifications.
> On the
> > > same note,
> > > >>> upgrade/rollback is the same as backup and restore,
> that
> > > means our restore
> > > >>> should follow a similar approach.
> > > >>> -Matthew
> > > >>
> > > >>
> > > >> There only one idea I have is to do dual partitioning
> system.
> > > The similar
> > > >> approach is implemented in CoreOS.
> > > >>
> > > >>>
> > > >>>
> > > >>> On Thu, Nov 19, 2015 at 6:36 PM, Bogdan Dobrelya
> > > <bdobrelia at mirantis.com <mailto:bdobrelia at mirantis.com>
> > <mailto:bdobrelia at mirantis.com <mailto:bdobrelia at mirantis.com>>>
> > > >>> wrote:
> > > >>>>
> > > >>>> On 19.11.2015 15:59, Vladimir Kozhukalov wrote:
> > > >>>> > Dear colleagues,
> > > >>>> >
> > > >>>> > As might remember, we introduced Docker containers
> > on the
> > > master node
> > > >>>> > a
> > > >>>> > while ago when we implemented first version of Fuel
> > > upgrade feature.
> > > >>>> > The
> > > >>>> > motivation behind was to make it possible to
> rollback
> > > upgrade process
> > > >>>> > if
> > > >>>> > something goes wrong.
> > > >>>> >
> > > >>>> > Now we are at the point where we can not use our
> > tarball
> > > based upgrade
> > > >>>> > approach any more and those patches that deprecate
> > > upgrade tarball has
> > > >>>> > been already merged. Although it is a matter of a
> > > separate discussion,
> > > >>>> > it seems that upgrade process rather should be
> based on
> > > kind of backup
> > > >>>> > and restore procedure. We can backup Fuel data on an
> > > external media,
> > > >>>> > then we can install new version of Fuel from
> > scratch and
> > > then it is
> > > >>>> > assumed backed up Fuel data can be applied over
> > this new Fuel
> > > >>>> > instance.
> > > >>>>
> > > >>>> A side-by-side upgrade, correct? That should work as
> > well.
> > > >>>>
> > > >>>> > The procedure itself is under active development,
> > but it
> > > is clear that
> > > >>>> > rollback in this case would be nothing more than
> just
> > > restoring from
> > > >>>> > the
> > > >>>> > previously backed up data.
> > > >>>> >
> > > >>>> > As for Docker containers, still there are potential
> > > advantages of
> > > >>>> > using
> > > >>>> > them on the Fuel master node, but our current
> > > implementation of the
> > > >>>> > feature seems not mature enough to make us benefit
> > from the
> > > >>>> > containerization.
> > > >>>> >
> > > >>>> > At the same time there are some disadvantages like
> > > >>>> >
> > > >>>> > * it is tricky to get logs and other information
> (for
> > > example, rpm
> > > >>>> > -qa) for a service like shotgun which is run
> inside
> > > one of
> > > >>>> > containers.
> > > >>>> > * it is specific UX when you first need to run
> > > dockerctl shell
> > > >>>> > {container_name} and then you are able to debug
> > > something.
> > > >>>> > * when building IBP image we mount directory from
> the
> > > host file
> > > >>>> > system
> > > >>>> > into mcollective container to make image build
> > faster.
> > > >>>> > * there are config files and some other files
> which
> > > should be shared
> > > >>>> > among containers which introduces unnecessary
> > > complexity to the
> > > >>>> > whole system.
> > > >>>> > * our current delivery approach assumes we wrap
> into
> > > rpm/deb
> > > >>>> > packages
> > > >>>> > every single piece of the Fuel system. Docker
> > images
> > > are not an
> > > >>>> > exception. And as far as they depend on other
> rpm
> > > packages we
> > > >>>> > forced
> > > >>>> > to build docker-images rpm package using kind of
> > > specific build
> > > >>>> > flow. Besides this package is quite big (300M).
> > > >>>> > * I'd like it to be possible to install Fuel not
> from
> > > ISO but from
> > > >>>> > RPM
> > > >>>> > repo on any rpm based distribution. But it is
> > double
> > > work to
> > > >>>> > support
> > > >>>> > both Docker based and package based approach.
> > > >>>>
> > > >>>> There is another point, the containers long build
> > time when
> > > installing
> > > >>>> the master node.
> > > >>>>
> > > >>>> >
> > > >>>> > Probably some of you can give other examples.
> > Anyway, the
> > > idea is to
> > > >>>> > get
> > > >>>> > rid of Docker containers on the master node and
> > switch to
> > > plane
> > > >>>> > package
> > > >>>> > based approach that we used before.
> > > >>>> >
> > > >>>> > As far as there is nothing new here, we just need
> > to use
> > > our old
> > > >>>> > site.pp
> > > >>>> > (with minimal modifications), it looks like it is
> > possible to
> > > >>>> > implement
> > > >>>> > this during 8.0 release cycle. If there are no
> > principal
> > > objections,
> > > >>>> > please give me a chance to do this ASAP (during
> 8.0), I
> > > know it is a
> > > >>>> > huge risk for the release, but still I think I can
> > do this.
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > > >>>> > Vladimir Kozhukalov
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > >
> >
> __________________________________________________________________________
> > > >>>> > OpenStack Development Mailing List (not for usage
> > questions)
> > > >>>> > Unsubscribe:
> > > >>>> >
> > >
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > > >>>> >
> > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >>>> >
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Best regards,
> > > >>>> Bogdan Dobrelya,
> > > >>>> Irc #bogdando
> > > >>>>
> > > >>>>
> > > >>>>
> > >
> __________________________________________________________________________
> > > >>>> OpenStack Development Mailing List (not for usage
> questions)
> > > >>>> Unsubscribe:
> > > >>>>
> > >
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > > >>>>
> > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > >
> __________________________________________________________________________
> > > >>> OpenStack Development Mailing List (not for usage
> questions)
> > > >>> Unsubscribe:
> > > >>>
> > >
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > > >>>
> > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >>>
> > > >>
> > > >>
> > > >>
> > >
> __________________________________________________________________________
> > > >> OpenStack Development Mailing List (not for usage
> questions)
> > > >> Unsubscribe:
> > >
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > > >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Yours Faithfully,
> > > > Vladimir Kuklin,
> > > > Fuel Library Tech Lead,
> > > > Mirantis, Inc.
> > > > +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
> > > > +7 (926) 702-39-68 <tel:%2B7%20%28926%29%20702-39-68>
> > > > Skype kuklinvv
> > > > 35bk3, Vorontsovskaya Str.
> > > > Moscow, Russia,
> > > > www.mirantis.com <http://www.mirantis.com>
> > <http://www.mirantis.com>
> > > > www.mirantis.ru <http://www.mirantis.ru>
> > <http://www.mirantis.ru>
> > > > vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>
> > <mailto:vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>>
> > > >
> > > >
> > >
> __________________________________________________________________________
> > > > OpenStack Development Mailing List (not for usage
> questions)
> > > > Unsubscribe:
> > >
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >
> > >
> > >
> __________________________________________________________________________
> > > OpenStack Development Mailing List (not for usage
> questions)
> > > Unsubscribe:
> > >
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Kind Regards,
> > >
> > > Alexandr Kostrikov,
> > >
> > >
> > > Mirantis, Inc.
> > >
> > > 35b/3, Vorontsovskaya St., 109147, Moscow, Russia
> > >
> > >
> > > Tel.: +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
> > > Tel.: +7 (925) 716-64-52 <tel:%2B7%20%28906%29%20740-64-79>
> > >
> > > Skype: akostrikov_mirantis
> > >
> > > E-mail:akostrikov at mirantis.com
> > <mailto:E-mail%3Aakostrikov at mirantis.com>
> > <mailto:elogutova at mirantis.com <mailto:elogutova at mirantis.com>>
> > >
> > > _www.mirantis.com <http://www.mirantis.com>
> > <http://www.mirantis.ru/>__
> > > __www.mirantis.ru <http://www.mirantis.ru>
> > <http://www.mirantis.ru/>_
> > >
> > >
> > >
> __________________________________________________________________________
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Timur,
> > > Senior QA Engineer
> > > OpenStack Projects
> > > Mirantis Inc
> > >
> > >
> > >
> >
> __________________________________________________________________________
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> >
> > --
> > Best regards,
> > Bogdan Dobrelya,
> > Irc #bogdando
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Timur,
Senior QA Engineer
OpenStack Projects
Mirantis Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151120/aad02531/attachment.html>
More information about the OpenStack-dev
mailing list