Open Stack

Today I encountered an issue which is caused by the lack of automatic version checks.

One of the users of Fuel Client asked me why it doesn’t work on their environment. I realized, that the issue was caused by wrong version specification in the Fuel Client requirements list.

I analyzed python-fuelclient’s requirements, RPM’s package requirements and Global Requirements and found that none of them was compatible to another one. And the issue was not that there is only one package, that is not compatible, but that there is only one package, that IS compatible.

Fuel Client is a set of very simple single threaded scripts so detecting problems in it is relatively simple. However, we also have complex components like Nailgun and different kinds of agents, where a version mismatch may produce errors which will take months to discover.

My proposal of sticking to requirements partially fixes this issue. The only thing I’d like to add to that scheme is this:

 - Add a CI jod that on daily basis checks whether python-requirements are in line with the requirements in RPM specs.


- romcheg


> 26 бер. 2015 о 12:07 Roman Prykhodchenko <me at romcheg.me> написав(ла):
> 
> So guys, I think it’s reasonable to find a consensus on this thread.
> 
>> I think this rule fits fine within the general frame of Roman's proposal:
>> 
>> - if the base distro already has a package that satisfies OpenStack
>> global requirements (or Fuel requirements), the distro package is
>> used;
>> - else, OSCI mirror should contain the maximum version of a
>> requirement that matches its version specification.
> 
> Yup, that also fits well. The only note I’d like to make here is that OS services are tested against the latest version of requirements. So perhaps we want to test them against the versions, supplied in distos, if those requirements are used.
> 
> 
> - romcheg
> 
>> 19 бер. 2015 о 19:14 Dmitry Borodaenko <dborodaenko at mirantis.com> написав(ла):
>> 
>> Maciej,
>> 
>> Maintaining multiple versions of the same package concurrently and
>> tracking their compatibility with the many different components of
>> OpenStack and Fuel creates additional work on many different levels,
>> from spec branch management to repo management to validation to
>> container building and so on. Unified global requirements help avoid
>> such work where it isn't necessary (and when you look close enough
>> into each specific case you're likely to find that it's never really
>> necessary).
>> 
>> -DmitryB
>> 
>> On Thu, Mar 19, 2015 at 4:04 AM, Maciej Kwiek <mkwiek at mirantis.com> wrote:
>>> I guess it would depend on how many docker containers are running on master
>>> node and if we are able to pull off such stunt :).
>>> 
>>> I am not familiar with the amount of work needed to do sth like that, so the
>>> proposition may be silly. Just let me know if it is.
>>> 
>>> On Thu, Mar 19, 2015 at 11:51 AM, Dmitry Burmistrov
>>> <dburmistrov at mirantis.com> wrote:
>>>> 
>>>> Folks,
>>>> 
>>>>>>> Correct me if I am wrong, but isn't it what we have containers on
>>>>>>> master node for?
>>>>>>> On the master node itself conflicts won’t happen because the
>>>>>>> components are run in their containers.
>>>> 
>>>> Do you propose to use _separate_ package repository for each docker
>>>> container? (It means separate gerrit project for each package of each
>>>> container, including openstack projects)
>>>> 
>>>> On Thu, Mar 19, 2015 at 1:16 PM, Roman Prykhodchenko <me at romcheg.me>
>>>> wrote:
>>>>> Folks,
>>>>> 
>>>>>> I assume you meant: "If a requirement that previously was only in Fuel
>>>>>> Requirements is merged to Global Requirements, it should be removed
>>>>>> from *Fuel* Requirements».
>>>>> 
>>>>> Exactly.
>>>>> 
>>>>>> I'm not sure it's good idea.
>>>>>> We should stay so close to upstream distro as we can. It should be
>>>>>> very important reason to update package against it's upstream distro
>>>>> 
>>>>> 
>>>>> The issue is the following: OpenStack’s components are tested against
>>>>> those versions of dependencies, that are specified in their requirements.
>>>>> IIRC those requirements are set up by pip so CI nodes contain latest
>>>>> versions of python packages that match their specs.
>>>>> 
>>>>> The question is whether it’s required to ship OpenStack services with
>>>>> packages from a distro or with packages, that are used for testing.
>>>>> 
>>>>>> Splitting of repositories doesn't help to solve python packages
>>>>>> conflicts because master node uses a number of openstack components.
>>>>> 
>>>>> On the master node itself conflicts won’t happen because the components
>>>>> are run in their containers.
>>>>> 
>>>>> 
>>>>> - romcheg
>>>>> 
>>>>> 
>>>>>> 19 бер. 2015 о 10:47 Dmitry Burmistrov <dburmistrov at mirantis.com>
>>>>>> написав(ла):
>>>>>> 
>>>>>> Roman, all
>>>>>> 
>>>>>>>>>      - OSCI mirror should contain the maximum version of a
>>>>>>>>> requirement that matches its version specification.
>>>>>> I'm not sure it's good idea.
>>>>>> We should stay so close to upstream distro as we can. It should be
>>>>>> very important reason to update package against it's upstream distro
>>>>>> version.
>>>>>> If we update some package, we should maintain it too. Tracking bugs,
>>>>>> CVEs and so on. The more packages, the more efforts to support it.
>>>>>> 
>>>>>>>>> - Set up CI jobs to notify OSCI team if either Global Requirements
>>>>>>>>> or
>>>>>>>>> Fuel Requirements are changed.
>>>>>>>>> - Set up requirements proposal jobs that will automatically propose
>>>>>>>>> changes to all fuel projects once either of requirements lists was
>>>>>>>>> changed
>>>>>> Need to be discussed and investigated.
>>>>>> 
>>>>>> Sebastian, Dmitry,
>>>>>> 
>>>>>> 
>>>>>>>>> There are some plans (unfortunately I do not know details, so maybe
>>>>>>>>> someone from OSCI could tell more) to split those repositories
>>>>>> Splitting of repositories doesn't help to solve python packages
>>>>>> conflicts because master node uses a number of openstack components.
>>>>>> Anyway it should be done for 7.0 milestone in order to separatre
>>>>>> master-node distro from environment one. (e.g. if we going to move
>>>>>> master-node to debian)
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, Mar 19, 2015 at 12:01 AM, Dmitry Borodaenko
>>>>>> <dborodaenko at mirantis.com> wrote:
>>>>>>> Roman,
>>>>>>> 
>>>>>>> I like this proposal very much, thanks for the idea and for putting
>>>>>>> together a straightforward process.
>>>>>>> 
>>>>>>> I assume you meant: "If a requirement that previously was only in Fuel
>>>>>>> Requirements is merged to Global Requirements, it should be removed
>>>>>>> from *Fuel* Requirements".
>>>>>>> 
>>>>>>> Sebastian,
>>>>>>> 
>>>>>>> We have found ways to resolve the conflicts between clvm and docker,
>>>>>>> and between ruby versions 1.8 and 2.1, without introducing a separate
>>>>>>> package repo for Fuel master. I've updated the blueprint to make note
>>>>>>> of that:
>>>>>>> 
>>>>>>> https://blueprints.launchpad.net/fuel/+spec/separate-repo-for-master-node
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Mar 18, 2015 at 9:25 AM, Sebastian Kalinowski
>>>>>>> <skalinowski at mirantis.com> wrote:
>>>>>>>> I assume that you considered a situation when we have a common
>>>>>>>> repository
>>>>>>>> with RPMs for Fuel master and for nodes.
>>>>>>>> There are some plans (unfortunately I do not know details, so maybe
>>>>>>>> someone
>>>>>>>> from OSCI could tell more) to split those repositories. How this
>>>>>>>> workflow
>>>>>>>> will work with those separated repos? Will we still need it?
>>>>>>>> 
>>>>>>>> Thanks!
>>>>>>>> Sebastian
>>>>>>>> 
>>>>>>>> 2015-03-18 11:04 GMT+01:00 Roman Prykhodchenko <me at romcheg.me>:
>>>>>>>>> 
>>>>>>>>> Hi folks,
>>>>>>>>> 
>>>>>>>>> before you say «romcheg, go away and never come back again!», please
>>>>>>>>> read
>>>>>>>>> the story that caused me to propose this and the proposed solution.
>>>>>>>>> Perhaps
>>>>>>>>> it makes you reconsider :)
>>>>>>>>> 
>>>>>>>>> As you know for different reasons, among which are being able to set
>>>>>>>>> up
>>>>>>>>> everything online and bringing up-to-date packages, we maintain an
>>>>>>>>> OSCI
>>>>>>>>> repository which is used for building ISOs and deploying OpenStack
>>>>>>>>> services.
>>>>>>>>> Managing that repo is a pretty hard job. Thus a dedicated group of
>>>>>>>>> people is
>>>>>>>>> devoted to perform that duty, they are always busy because of a lot
>>>>>>>>> of
>>>>>>>>> responsibilities they have.
>>>>>>>>> 
>>>>>>>>> At the same time Fuel’s developers are pretty energetic and always
>>>>>>>>> want to
>>>>>>>>> add new features to Fuel. For that they love to use different
>>>>>>>>> libraries,
>>>>>>>>> many of which aren’t in the OSCI mirror yet. So they ask OSCI guys
>>>>>>>>> to add
>>>>>>>>> more and more of those and I guess that’s pretty fine except one
>>>>>>>>> little
>>>>>>>>> thing — sometimes those libraries conflict with ones, required by
>>>>>>>>> OpenStack
>>>>>>>>> services.
>>>>>>>>> 
>>>>>>>>> To prevent that from happening someone has to check every patch
>>>>>>>>> against
>>>>>>>>> the OSCI repo and OpenStack’s global requirements, to detect whether
>>>>>>>>> a
>>>>>>>>> version bump or adding a new library is required an whether it can
>>>>>>>>> be
>>>>>>>>> performed. As you can guess, there’s too much of a human factor so
>>>>>>>>> statistically no one does that until problems appear. Moreover,
>>>>>>>>> theres’
>>>>>>>>> nothing but a «it’s not compatible with OpenStack» yelling from OSCI
>>>>>>>>> team
>>>>>>>>> that stops developers to change dependencies in Fuel.
>>>>>>>>> 
>>>>>>>>> All the stuff described above causes sometimes tremendous time
>>>>>>>>> losses and
>>>>>>>>> is very problem-prone.
>>>>>>>>> 
>>>>>>>>> I’d like to propose to make everyone’s life easier by following
>>>>>>>>> these
>>>>>>>>> steps:
>>>>>>>>> 
>>>>>>>>> - Create a new project called Fuel Requirements, all changes to it
>>>>>>>>> should
>>>>>>>>> go through a standard review procedure
>>>>>>>>> - We strict ourselves to use only packages from both Fuel
>>>>>>>>> Requirements
>>>>>>>>> and Global Requirements for the version of OpenStack, Fuel is
>>>>>>>>> installing in
>>>>>>>>> the following manner:
>>>>>>>>>  - If a requirement is in Global Requirements, the version spec in
>>>>>>>>> all
>>>>>>>>> Fuel’s components should be exactly like that.
>>>>>>>>>      - OSCI mirror should contain the maximum version of a
>>>>>>>>> requirement
>>>>>>>>> that matches its version specification.
>>>>>>>>>  - If a requirement is not in the global requirements list, then
>>>>>>>>> Fuel
>>>>>>>>> Requirements list should be used to check whether all Fuel’s
>>>>>>>>> components
>>>>>>>>> require the same version of a library/package.
>>>>>>>>>      - OSCI mirror should contain the maximum version of a
>>>>>>>>> requirement
>>>>>>>>> that matches its version specification.
>>>>>>>>>  - If a requirement that previously was only in Fuel Requirements
>>>>>>>>> is
>>>>>>>>> merged to Global Requirements, it should be removed from Global
>>>>>>>>> Requirements
>>>>>>>>> - Set up CI jobs in both OpenStack CI and FuelCI to check all
>>>>>>>>> patches
>>>>>>>>> against both Global Requirements and Fuel Requirements and block, if
>>>>>>>>> either
>>>>>>>>> of checks doesn’t pass
>>>>>>>>> - Set up CI jobs to notify OSCI team if either Global Requirements
>>>>>>>>> or
>>>>>>>>> Fuel Requirements are changed.
>>>>>>>>> - Set up requirements proposal jobs that will automatically propose
>>>>>>>>> changes to all fuel projects once either of requirements lists was
>>>>>>>>> changed,
>>>>>>>>> just like it’s done for OpenStack projects.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> These steps may look terribly hard, but most of the job is already
>>>>>>>>> done in
>>>>>>>>> OpenStack projects and therefore it can be reused for Fuel.
>>>>>>>>> Looking forward for your feedback folks!
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> - romcheg
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> __________________________________________________________________________
>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>> Unsubscribe:
>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> __________________________________________________________________________
>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>> Unsubscribe:
>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Dmitry Borodaenko
>>>>>>> 
>>>>>>> 
>>>>>>> __________________________________________________________________________
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>> 
>>>>>> 
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> 
>>>>> 
>>>>> 
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> 
>>>> 
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> 
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>> 
>> 
>> 
>> --
>> Dmitry Borodaenko
>> 
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150401/b2a1a418/attachment.pgp>