[openstack-dev] [Fuel] Getting rid of Docker containers on the Fuel master node

Igor Kalnitsky ikalnitsky at mirantis.com
Fri Nov 20 20:55:00 UTC 2015


Hey Timur,

> I think we can change all docker-based code from our tests / scripts
> in 2-3 days

That sounds good.


> Do we really want to remove docker containers from master node?

Yes, we do. Currently we're suffering from using container-based
architecture on master node, and since we've decided to change our
*upgrade* approach (where we stop gain benefits from containers) it would
be nice to get rid of them and fix a bunch of docker-related bugs.


> How long it will take to provide the experimental MOS 8.0 build
> without docker containers?

I think we need to ask Vladimir Kozhukalov here.


> Are we ready to change the date of MOS 8.0 release and make this
> change?

No, we don't ready to change release date. If we don't have time for it,
let's postpone it till 9.0.

Regards,
Igor

On Fri, Nov 20, 2015 at 12:41 PM Timur Nurlygayanov <
tnurlygayanov at mirantis.com> wrote:

> Hi Igor and Alexander,
>
> >But I'd like to hear from QA how do we rely on container-based
>> infrastructure? Would it be hard to change our sys-tests in short
>> time?
>
> QA team hadn't significant dependencies from docker images in our tests
> [0], I think we can change all docker-based code from our tests / scripts
> in 2-3 days, but it is hard to predict when ISO without docker images will
> pass all SWARM / Tempest tests.
>
> And one more time:
>
>> Of course, we can fix BVT / SWARM tests and don't use docker images in
>> our test suite (it shouldn't be really hard) but we didn't plan these
>> changes and in fact these changes can affect our estimates for many tasks.
>
>
> Do we really want to remove docker containers from master node? How long
> it will take to provide the experimental MOS 8.0 build without docker
> containers?
> Are we ready to change the date of MOS 8.0 release and make this change?
>
> [0]
> https://github.com/openstack/fuel-qa/search?p=2&q=docker&utf8=%E2%9C%93
>
>
> On Fri, Nov 20, 2015 at 7:57 PM, Bogdan Dobrelya <bdobrelia at mirantis.com>
> wrote:
>
>> On 20.11.2015 17:31, Vladimir Kozhukalov wrote:
>> > Bogdan,
>> >
>> >>> So, we could only deprecate the docker feature for the 8.0.
>> >
>> > What do you mean exactly when saying 'deprecate docker feature'? I can
>> > not even imagine how we can live with and without docker containers at
>> > the same time. Deprecation is usually related to features which directly
>> > impact UX (maybe I am wrong).
>>
>> I may be understood this [0] wrong, and the docker containers are not
>> user-visible, but that depends on the which type of users do we mean :-)
>> Sorry, for being not clear.
>>
>> [0]
>>
>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/073576.html
>>
>> >
>> > Guys,
>> >
>> > When you estimate risks of the docker removal, please take into account
>> > not only release deadlines but also the overall product quality. The
>> > thing is that continuing using containers makes it much more complicated
>> > (and thus less stable) to implement new upgrade flow (upgrade tarball
>> > can not be used any more, we need to re-install the host system).
>> > Switching from Centos 6 to Centos 7 is also much more complicated with
>> > docker. Every single piece of Fuel system is going to become simpler and
>> > easier to support.
>> >
>> > Of course, I am not suggesting to jump overboard into cold water without
>> > a life jacket. Transition plan, checklist, green tests, even spec etc.
>> > are assumed without saying (after all, I was not born yesterday). Of
>> > course, we won't merge changes until everything is green. What is the
>> > problem to try to do this and postpone if not ready in time? And please
>> > do not confuse these two cases: switching from plain deployment to
>> > containers is complicated, but switching from docker to plain is much
>> > simpler.
>> >
>> >
>> >
>> >
>> > Vladimir Kozhukalov
>> >
>> > On Fri, Nov 20, 2015 at 6:47 PM, Bogdan Dobrelya <
>> bdobrelia at mirantis.com
>> > <mailto:bdobrelia at mirantis.com>> wrote:
>> >
>> >     On 20.11.2015 15:10, Timur Nurlygayanov wrote:
>> >     > Hi team,
>> >     >
>> >     > I think it too late to make such significant changes for MOS 8.0
>> now,
>> >     > but I'm ok with the idea to remove docker containers in the future
>> >     > releases if our dev team want to do this.
>> >     > Any way, before we will do this, we need to plan how we will
>> perform
>> >     > updates between different releases with and without docker
>> containers,
>> >     > how we will manage requirements and etc. In fact we have a lot of
>> >     > questions and haven't answers, let's prepare the spec for this
>> change,
>> >     > review it, discuss it with developers, users and project
>> management team
>> >     > and if we haven't requirements to keep docker containers on
>> master node
>> >     > let's remove them for the future releases (not in MOS 8.0).
>> >     >
>> >     > Of course, we can fix BVT / SWARM tests and don't use docker
>> images in
>> >     > our test suite (it shouldn't be really hard) but we didn't plan
>> these
>> >     > changes and in fact these changes can affect our estimates for
>> many tasks.
>> >
>> >     I can only add that features just cannot be removed without a
>> >     deprecation period of 1-2 releases.
>> >     So, we could only deprecate the docker feature for the 8.0.
>> >
>> >     >
>> >     > Thank you!
>> >     >
>> >     >
>> >     > On Fri, Nov 20, 2015 at 4:44 PM, Alexander Kostrikov
>> >     > <akostrikov at mirantis.com <mailto:akostrikov at mirantis.com>
>> >     <mailto:akostrikov at mirantis.com <mailto:akostrikov at mirantis.com>>>
>> >     wrote:
>> >     >
>> >     >     Hello, Igor.
>> >     >
>> >     >     >But I'd like to hear from QA how do we rely on
>> container-based
>> >     >     infrastructure? Would it be hard to change our sys-tests in
>> short
>> >     >     time?
>> >     >
>> >     >     At first glance, system tests are using docker only to fetch
>> logs
>> >     >     and run shell commands.
>> >     >     Also, docker is used to run Rally.
>> >     >
>> >     >     If there is an action to remove docker containers with
>> carefull
>> >     >     attention to bvt testing, it would take couple days to fix
>> system tests.
>> >     >     But time may be highly affected by code freezes and active
>> features
>> >     >     merging.
>> >     >
>> >     >     QA team is going to have Monday (Nov 23) sync-up - and it is
>> >     >     possible to get more exact information from all QA-team.
>> >     >
>> >     >     P.S.
>> >     >     +1 to remove docker.
>> >     >     -1 to remove docker without taking into account
>> deadlines/other
>> >     >     features.
>> >     >
>> >     >     On Thu, Nov 19, 2015 at 10:27 PM, Igor Kalnitsky
>> >     >     <ikalnitsky at mirantis.com <mailto:ikalnitsky at mirantis.com>
>> >     <mailto:ikalnitsky at mirantis.com <mailto:ikalnitsky at mirantis.com>>>
>> >     wrote:
>> >     >
>> >     >         Hey guys,
>> >     >
>> >     >         Despite the fact I like containers (as deployment unit),
>> we
>> >     >         don't use
>> >     >         them so. That means I +1 idea to drop containers, just
>> because I
>> >     >         believe that would
>> >     >
>> >     >         * simplify a lot of things
>> >     >         * helps get rid of huge amount of hacks
>> >     >         * increase master node deployment
>> >     >         * release us from annoying support of upgrades /
>> rollbacks that
>> >     >         proved
>> >     >         to be non-working well
>> >     >
>> >     >         But I'd like to hear from QA how do we rely on
>> container-based
>> >     >         infrastructure? Would it be hard to change our sys-tests
>> in short
>> >     >         time?
>> >     >
>> >     >         Thanks,
>> >     >         Igor
>> >     >
>> >     >
>> >     >         On Thu, Nov 19, 2015 at 10:31 AM, Vladimir Kuklin
>> >     >         <vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>
>> >     <mailto:vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>>> wrote:
>> >     >         > Folks
>> >     >         >
>> >     >         > I guess it should be pretty simple to roll back -
>> install
>> >     >         older version and
>> >     >         > restore the backup with preservation of /var/log
>> directory.
>> >     >         >
>> >     >         > On Thu, Nov 19, 2015 at 7:38 PM, Sergii Golovatiuk
>> >     >         > <sgolovatiuk at mirantis.com
>> >     <mailto:sgolovatiuk at mirantis.com> <mailto:sgolovatiuk at mirantis.com
>> >     <mailto:sgolovatiuk at mirantis.com>>>
>> >     >         wrote:
>> >     >         >>
>> >     >         >> Hi,
>> >     >         >>
>> >     >         >> On Thu, Nov 19, 2015 at 5:50 PM, Matthew Mosesohn
>> >     >         <mmosesohn at mirantis.com <mailto:mmosesohn at mirantis.com>
>> >     <mailto:mmosesohn at mirantis.com <mailto:mmosesohn at mirantis.com>>>
>> >     >         >> wrote:
>> >     >         >>>
>> >     >         >>> Vladimir,
>> >     >         >>>
>> >     >         >>> The old site.pp is long out of date and should just be
>> >     >         recreated from the
>> >     >         >>> content of all the other $service-only.pp files.
>> >     >         >>>
>> >     >         >>> My main question is how do we propose to do a
>> rollback from
>> >     >         an update (in
>> >     >         >>> theory, from 8.0 to 9.0, then back to 8.0)? Should we
>> >     >         hardcode persistent
>> >     >         >>> data directories (or symlink them?) to
>> >     >         >>> /var/lib/fuel/$fuel_version/$service_name, as we are
>> doing
>> >     >         behind the scenes
>> >     >         >>> currently with Docker? If we keep that mechanism in
>> place,
>> >     >         all the existing
>> >     >         >>> puppet modules can be used without any modifications.
>> On the
>> >     >         same note,
>> >     >         >>> upgrade/rollback is the same as backup and restore,
>> that
>> >     >         means our restore
>> >     >         >>> should follow a similar approach.
>> >     >         >>> -Matthew
>> >     >         >>
>> >     >         >>
>> >     >         >> There only one idea I have is to do dual partitioning
>> system.
>> >     >         The similar
>> >     >         >> approach is implemented in CoreOS.
>> >     >         >>
>> >     >         >>>
>> >     >         >>>
>> >     >         >>> On Thu, Nov 19, 2015 at 6:36 PM, Bogdan Dobrelya
>> >     >         <bdobrelia at mirantis.com <mailto:bdobrelia at mirantis.com>
>> >     <mailto:bdobrelia at mirantis.com <mailto:bdobrelia at mirantis.com>>>
>> >     >         >>> wrote:
>> >     >         >>>>
>> >     >         >>>> On 19.11.2015 15:59, Vladimir Kozhukalov wrote:
>> >     >         >>>> > Dear colleagues,
>> >     >         >>>> >
>> >     >         >>>> > As might remember, we introduced Docker containers
>> >     on the
>> >     >         master node
>> >     >         >>>> > a
>> >     >         >>>> > while ago when we implemented first version of Fuel
>> >     >         upgrade feature.
>> >     >         >>>> > The
>> >     >         >>>> > motivation behind was to make it possible to
>> rollback
>> >     >         upgrade process
>> >     >         >>>> > if
>> >     >         >>>> > something goes wrong.
>> >     >         >>>> >
>> >     >         >>>> > Now we are at the point where we can not use our
>> >     tarball
>> >     >         based upgrade
>> >     >         >>>> > approach any more and those patches that deprecate
>> >     >         upgrade tarball has
>> >     >         >>>> > been already merged. Although it is a matter of a
>> >     >         separate discussion,
>> >     >         >>>> > it seems that upgrade process rather should be
>> based on
>> >     >         kind of backup
>> >     >         >>>> > and restore procedure. We can backup Fuel data on
>> an
>> >     >         external media,
>> >     >         >>>> > then we can install new version of Fuel from
>> >     scratch and
>> >     >         then it is
>> >     >         >>>> > assumed backed up Fuel data can be applied over
>> >     this new Fuel
>> >     >         >>>> > instance.
>> >     >         >>>>
>> >     >         >>>> A side-by-side upgrade, correct? That should work as
>> >     well.
>> >     >         >>>>
>> >     >         >>>> > The procedure itself is under active development,
>> >     but it
>> >     >         is clear that
>> >     >         >>>> > rollback in this case would be nothing more than
>> just
>> >     >         restoring from
>> >     >         >>>> > the
>> >     >         >>>> > previously backed up data.
>> >     >         >>>> >
>> >     >         >>>> > As for Docker containers, still there are potential
>> >     >         advantages of
>> >     >         >>>> > using
>> >     >         >>>> > them on the Fuel master node, but our current
>> >     >         implementation of the
>> >     >         >>>> > feature seems not mature enough to make us benefit
>> >     from the
>> >     >         >>>> > containerization.
>> >     >         >>>> >
>> >     >         >>>> > At the same time there are some disadvantages like
>> >     >         >>>> >
>> >     >         >>>> >   * it is tricky to get logs and other information
>> (for
>> >     >         example, rpm
>> >     >         >>>> >     -qa) for a service like shotgun which is run
>> inside
>> >     >         one of
>> >     >         >>>> > containers.
>> >     >         >>>> >   * it is specific UX when you first need to run
>> >     >         dockerctl shell
>> >     >         >>>> >     {container_name} and then you are able to debug
>> >     >         something.
>> >     >         >>>> >   * when building IBP image we mount directory
>> from the
>> >     >         host file
>> >     >         >>>> > system
>> >     >         >>>> >     into mcollective container to make image build
>> >     faster.
>> >     >         >>>> >   * there are config files and some other files
>> which
>> >     >         should be shared
>> >     >         >>>> >     among containers which introduces unnecessary
>> >     >         complexity to the
>> >     >         >>>> >     whole system.
>> >     >         >>>> >   * our current delivery approach assumes we wrap
>> into
>> >     >         rpm/deb
>> >     >         >>>> > packages
>> >     >         >>>> >     every single piece of the Fuel system. Docker
>> >     images
>> >     >         are not an
>> >     >         >>>> >     exception. And as far as they depend on other
>> rpm
>> >     >         packages we
>> >     >         >>>> > forced
>> >     >         >>>> >     to build docker-images rpm package using kind
>> of
>> >     >         specific build
>> >     >         >>>> >     flow. Besides this package is quite big (300M).
>> >     >         >>>> >   * I'd like it to be possible to install Fuel not
>> from
>> >     >         ISO but from
>> >     >         >>>> > RPM
>> >     >         >>>> >     repo on any rpm based distribution. But it is
>> >     double
>> >     >         work to
>> >     >         >>>> > support
>> >     >         >>>> >     both Docker based and package based approach.
>> >     >         >>>>
>> >     >         >>>> There is another point, the containers long build
>> >     time when
>> >     >         installing
>> >     >         >>>> the master node.
>> >     >         >>>>
>> >     >         >>>> >
>> >     >         >>>> > Probably some of you can give other examples.
>> >     Anyway, the
>> >     >         idea is to
>> >     >         >>>> > get
>> >     >         >>>> > rid of Docker containers on the master node and
>> >     switch to
>> >     >         plane
>> >     >         >>>> > package
>> >     >         >>>> > based approach that we used before.
>> >     >         >>>> >
>> >     >         >>>> > As far as there is nothing new here, we just need
>> >     to use
>> >     >         our old
>> >     >         >>>> > site.pp
>> >     >         >>>> > (with minimal modifications), it looks like it is
>> >     possible to
>> >     >         >>>> > implement
>> >     >         >>>> > this during 8.0 release cycle. If there are no
>> >     principal
>> >     >         objections,
>> >     >         >>>> > please give me a chance to do this ASAP (during
>> 8.0), I
>> >     >         know it is a
>> >     >         >>>> > huge risk for the release, but still I think I can
>> >     do this.
>> >     >         >>>> >
>> >     >         >>>> >
>> >     >         >>>> >
>> >     >         >>>> >
>> >     >         >>>> > Vladimir Kozhukalov
>> >     >         >>>> >
>> >     >         >>>> >
>> >     >         >>>> >
>> >     >         >>>> >
>> >     >
>> >
>> __________________________________________________________________________
>> >     >         >>>> > OpenStack Development Mailing List (not for usage
>> >     questions)
>> >     >         >>>> > Unsubscribe:
>> >     >         >>>> >
>> >     >
>> >      OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> >      <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >         >>>> >
>> >     >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >         >>>> >
>> >     >         >>>>
>> >     >         >>>>
>> >     >         >>>> --
>> >     >         >>>> Best regards,
>> >     >         >>>> Bogdan Dobrelya,
>> >     >         >>>> Irc #bogdando
>> >     >         >>>>
>> >     >         >>>>
>> >     >         >>>>
>> >     >
>>  __________________________________________________________________________
>> >     >         >>>> OpenStack Development Mailing List (not for usage
>> questions)
>> >     >         >>>> Unsubscribe:
>> >     >         >>>>
>> >     >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> >      <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >         >>>>
>> >     >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >         >>>
>> >     >         >>>
>> >     >         >>>
>> >     >         >>>
>> >     >         >>>
>> >     >
>>  __________________________________________________________________________
>> >     >         >>> OpenStack Development Mailing List (not for usage
>> questions)
>> >     >         >>> Unsubscribe:
>> >     >         >>>
>> >     >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> >      <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >         >>>
>> >     >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >         >>>
>> >     >         >>
>> >     >         >>
>> >     >         >>
>> >     >
>>  __________________________________________________________________________
>> >     >         >> OpenStack Development Mailing List (not for usage
>> questions)
>> >     >         >> Unsubscribe:
>> >     >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> >      <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >         >>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >         >>
>> >     >         >
>> >     >         >
>> >     >         >
>> >     >         > --
>> >     >         > Yours Faithfully,
>> >     >         > Vladimir Kuklin,
>> >     >         > Fuel Library Tech Lead,
>> >     >         > Mirantis, Inc.
>> >     >         > +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
>> >     >         > +7 (926) 702-39-68 <tel:%2B7%20%28926%29%20702-39-68>
>> >     >         > Skype kuklinvv
>> >     >         > 35bk3, Vorontsovskaya Str.
>> >     >         > Moscow, Russia,
>> >     >         > www.mirantis.com <http://www.mirantis.com>
>> >     <http://www.mirantis.com>
>> >     >         > www.mirantis.ru <http://www.mirantis.ru>
>> >     <http://www.mirantis.ru>
>> >     >         > vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>
>> >     <mailto:vkuklin at mirantis.com <mailto:vkuklin at mirantis.com>>
>> >     >         >
>> >     >         >
>> >     >
>>  __________________________________________________________________________
>> >     >         > OpenStack Development Mailing List (not for usage
>> questions)
>> >     >         > Unsubscribe:
>> >     >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> >      <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >         >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >         >
>> >     >
>> >     >
>>  __________________________________________________________________________
>> >     >         OpenStack Development Mailing List (not for usage
>> questions)
>> >     >         Unsubscribe:
>> >     >
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> >      <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >
>> >     >
>> >     >
>> >     >
>> >     >     --
>> >     >
>> >     >     Kind Regards,
>> >     >
>> >     >     Alexandr Kostrikov,
>> >     >
>> >     >
>> >     >     Mirantis, Inc.
>> >     >
>> >     >     35b/3, Vorontsovskaya St., 109147, Moscow, Russia
>> >     >
>> >     >
>> >     >     Tel.: +7 (495) 640-49-04 <tel:%2B7%20%28495%29%20640-49-04>
>> >     >     Tel.: +7 (925) 716-64-52 <tel:%2B7%20%28906%29%20740-64-79>
>> >     >
>> >     >     Skype: akostrikov_mirantis
>> >     >
>> >     >     E-mail:akostrikov at mirantis.com
>> >     <mailto:E-mail%3Aakostrikov at mirantis.com>
>> >     <mailto:elogutova at mirantis.com <mailto:elogutova at mirantis.com>>
>> >     >
>> >     >     _www.mirantis.com <http://www.mirantis.com>
>> >     <http://www.mirantis.ru/>__
>> >     >     __www.mirantis.ru <http://www.mirantis.ru>
>> >     <http://www.mirantis.ru/>_
>> >     >
>> >     >
>> >     >
>>  __________________________________________________________________________
>> >     >     OpenStack Development Mailing List (not for usage questions)
>> >     >     Unsubscribe:
>> >     >     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> >      <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >
>> >     >
>> >     >
>> >     >
>> >     > --
>> >     >
>> >     > Timur,
>> >     > Senior QA Engineer
>> >     > OpenStack Projects
>> >     > Mirantis Inc
>> >     >
>> >     >
>> >     >
>> >
>>  __________________________________________________________________________
>> >     > OpenStack Development Mailing List (not for usage questions)
>> >     > Unsubscribe:
>> >     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >     >
>> >
>> >
>> >     --
>> >     Best regards,
>> >     Bogdan Dobrelya,
>> >     Irc #bogdando
>> >
>> >
>>  __________________________________________________________________________
>> >     OpenStack Development Mailing List (not for usage questions)
>> >     Unsubscribe:
>> >     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >     <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Irc #bogdando
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
>
> Timur,
> Senior QA Engineer
> OpenStack Projects
> Mirantis Inc
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151120/814dc22a/attachment.html>


More information about the OpenStack-dev mailing list