[openstack-dev] [Fuel] Master node upgrade

Oleg Gelbukh ogelbukh at mirantis.com
Wed Nov 18 22:58:38 UTC 2015


We are going to address the problem using the following approach:

1. Backup settings from the Fuel 7.0 Master node, including configuration
files for bootstrap script, data from state database, security keys and
certs, etc
2. Restore settings on top of freshly installed Fuel 8.0 Master.
3. Upload database dump into the DB of Fuel 8.0 Master.
3. Perform required actions to apply migrations to Fuel state DB.

In this case, rollback scenario is either revert to Fuel 7.0 Master (if it
wasn't reinstalled), or apply the same procedure to fresh Fuel 7.0 Master
installation.

This scenario introduces different upgrade workflow vs what upgrade tarball
used. We will update user documentation with the new workload. Operators
will have to consider changes to their processes in accordance with the new
workflow.

I will update this list once we have some progress on this task. You can
also track it in the following blueprint:

https://blueprints.launchpad.net/fuel/+spec/upgrade-master-node-centos7

--
Best regards,
Oleg Gelbukh

On Tue, Nov 10, 2015 at 8:52 AM, Vladimir Kuklin <vkuklin at mirantis.com>
wrote:

> Evgeniy
>
> I am not sure you addressed me, but, anyway, - yes, we will have a
> situation with old containers on new host node. This will be identical to
> old host node from database migration point of view.
>
> On Tue, Nov 10, 2015 at 7:38 PM, Evgeniy L <eli at mirantis.com> wrote:
>
>> Hi Vladimir,
>>
>> Just to make sure that we are on the same page. We'll have to use upgrade
>> script anyway, since you will need to run database migration and register
>> new releases.
>>
>> Thanks,
>>
>> On Monday, 9 November 2015, Vladimir Kozhukalov <vkozhukalov at mirantis.com>
>> wrote:
>>
>>> Looks like most people thing that building backup/re-install approach is
>>> more viable. So, we certainly need to invent completely new upgrade from
>>> and thus my suggestion is disable building/testing upgrade tarball right
>>> now, because anyway it makes no sense.
>>>
>>>
>>> Vladimir Kozhukalov
>>>
>>> On Fri, Nov 6, 2015 at 8:21 PM, Vladimir Kuklin <vkuklin at mirantis.com>
>>> wrote:
>>>
>>>> Just my 2 cents here - let's do docker backup and roll it up onto brand
>>>> new Fuel 8 node.
>>>>
>>>> On Fri, Nov 6, 2015 at 7:54 PM, Oleg Gelbukh <ogelbukh at mirantis.com>
>>>> wrote:
>>>>
>>>>> Matt,
>>>>>
>>>>> You are talking about this part of Operations guide [1], or you mean
>>>>> something else?
>>>>>
>>>>> If yes, then we still need to extract data from backup containers. I'd
>>>>> prefer backup of DB in simple plain text file, since our DBs are not that
>>>>> big.
>>>>>
>>>>> [1]
>>>>> https://docs.mirantis.com/openstack/fuel/fuel-7.0/operations.html#howto-backup-and-restore-fuel-master
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Oleg Gelbukh
>>>>>
>>>>> On Fri, Nov 6, 2015 at 6:03 PM, Matthew Mosesohn <
>>>>> mmosesohn at mirantis.com> wrote:
>>>>>
>>>>>> Oleg,
>>>>>>
>>>>>> All the volatile information, including a DB dump, are contained in
>>>>>> the small Fuel Master backup. There should be no information lost unless
>>>>>> there was manual customization done inside the containers (such as puppet
>>>>>> manifest changes). There shouldn't be a need to back up the entire
>>>>>> containers.
>>>>>>
>>>>>> The information we would lose would include the IP configuration
>>>>>> interfaces besides the one used for the Fuel PXE network and any custom
>>>>>> configuration done on the Fuel Master.
>>>>>>
>>>>>> I want #1 to work smoothly, but #2 should also be a safe route.
>>>>>>
>>>>>> On Fri, Nov 6, 2015 at 5:39 PM, Oleg Gelbukh <ogelbukh at mirantis.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Evgeniy,
>>>>>>>
>>>>>>> On Fri, Nov 6, 2015 at 3:27 PM, Evgeniy L <eli at mirantis.com> wrote:
>>>>>>>
>>>>>>>> Also we should decide when to run containers
>>>>>>>> upgrade + host upgrade? Before or after new CentOS is installed?
>>>>>>>> Probably
>>>>>>>> it should be done before we run backup, in order to get the latest
>>>>>>>> scripts for
>>>>>>>> backup/restore actions.
>>>>>>>>
>>>>>>>
>>>>>>> We're working to determine if we need to backup/upgrade containers
>>>>>>> at all. My expectation is that we should be OK with just backup of DB, IP
>>>>>>> addresses settings from astute.yaml for the master node, and credentials
>>>>>>> from configuration files for the services.
>>>>>>>
>>>>>>> --
>>>>>>> Best regards,
>>>>>>> Oleg Gelbukh
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> On Fri, Nov 6, 2015 at 1:29 PM, Vladimir Kozhukalov <
>>>>>>>> vkozhukalov at mirantis.com> wrote:
>>>>>>>>
>>>>>>>>> Dear colleagues,
>>>>>>>>>
>>>>>>>>> At the moment I'm working on deprecating Fuel upgrade tarball.
>>>>>>>>> Currently, it includes the following:
>>>>>>>>>
>>>>>>>>> * RPM repository (upstream + mos)
>>>>>>>>> * DEB repository (mos)
>>>>>>>>> * openstack.yaml
>>>>>>>>> * version.yaml
>>>>>>>>> * upgrade script itself (+ virtualenv)
>>>>>>>>>
>>>>>>>>> Apart from upgrading docker containers this upgrade script makes
>>>>>>>>> copies of the RPM/DEB repositories and puts them on the master node naming
>>>>>>>>> these repository directories depending on what is written in openstack.yaml
>>>>>>>>> and version.yaml. My plan was something like:
>>>>>>>>>
>>>>>>>>> 1) deprecate version.yaml (move all fields from there to various
>>>>>>>>> places)
>>>>>>>>> 2) deliver openstack.yaml with fuel-openstack-metadata package
>>>>>>>>> 3) do not put new repos on the master node (instead we should use
>>>>>>>>> online repos or use fuel-createmirror to make local mirrors)
>>>>>>>>> 4) deliver fuel-upgrade package (throw away upgrade virtualenv)
>>>>>>>>>
>>>>>>>>> Then UX was supposed to be roughly like:
>>>>>>>>>
>>>>>>>>> 1) configure /etc/yum.repos.d/nailgun.repo (add new RPM MOS repo)
>>>>>>>>> 2) yum install fuel-upgrade
>>>>>>>>> 3) /usr/bin/fuel-upgrade (script was going to become lighter,
>>>>>>>>> because there should have not be parts coping RPM/DEB repos)
>>>>>>>>>
>>>>>>>>> However, it turned out that Fuel 8.0 is going to be run on Centos
>>>>>>>>> 7 and it is not enough to just do things which we usually did during
>>>>>>>>> upgrades. Now there are two ways to upgrade:
>>>>>>>>> 1) to use the official Centos upgrade script for upgrading from 6
>>>>>>>>> to 7
>>>>>>>>> 2) to backup the master node, then reinstall it from scratch and
>>>>>>>>> then apply backup
>>>>>>>>>
>>>>>>>>> Upgrade team is trying to understand which way is more
>>>>>>>>> appropriate. Regarding to my tarball related activities, I'd say that this
>>>>>>>>> package based upgrade approach can be aligned with (1) (fuel-upgrade would
>>>>>>>>> use official Centos upgrade script as a first step for upgrade), but it
>>>>>>>>> definitely can not be aligned with (2), because it assumes reinstalling the
>>>>>>>>> master node from scratch.
>>>>>>>>>
>>>>>>>>> Right now, I'm finishing the work around deprecating version.yaml
>>>>>>>>> and my further steps would be to modify fuel-upgrade script so it does not
>>>>>>>>> copy RPM/DEB repos, but those steps make little sense taking into account
>>>>>>>>> Centos 7 feature.
>>>>>>>>>
>>>>>>>>> Colleagues, let's make a decision about how we are going to
>>>>>>>>> upgrade the master node ASAP. Probably my tarball related work should be
>>>>>>>>> reduced to just throwing tarball away.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Vladimir Kozhukalov
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> __________________________________________________________________________
>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>> Unsubscribe:
>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> __________________________________________________________________________
>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>> Unsubscribe:
>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> __________________________________________________________________________
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Yours Faithfully,
>>>> Vladimir Kuklin,
>>>> Fuel Library Tech Lead,
>>>> Mirantis, Inc.
>>>> +7 (495) 640-49-04
>>>> +7 (926) 702-39-68
>>>> Skype kuklinvv
>>>> 35bk3, Vorontsovskaya Str.
>>>> Moscow, Russia,
>>>> www.mirantis.com <http://www.mirantis.ru/>
>>>> www.mirantis.ru
>>>> vkuklin at mirantis.com
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151118/16bb8ac8/attachment.html>


More information about the OpenStack-dev mailing list