[openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

Emilien Macchi emilien at redhat.com
Fri Oct 14 11:55:33 UTC 2016


On Thu, Oct 13, 2016 at 10:30 PM, Davanum Srinivas <davanum at gmail.com> wrote:
> Please see below:
>
> On Thu, Oct 13, 2016 at 9:33 PM, Matt Riedemann
> <mriedem at linux.vnet.ibm.com> wrote:
>> On 10/13/2016 7:47 PM, Emilien Macchi wrote:
>>>
>>> Greetings OpenStack,
>>>
>>> == Background
>>>
>>> Since the beginning of OpenStack (or almost), devstack has been used
>>> as a common tool to deploy OpenStack in CI environment. Most of
>>> OpenStack projects (if not all) that are written in Python use it to
>>> deploy the different components.
>>> While devstack became popular and the reference in term of deployment
>>> tool for continuous integration, devstack doesn't deploy OpenStack in
>>> production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
>>> It means things might (and did) break when deploying OpenStack outside
>>> devstack, for different reasons. Some examples:
>>>
>>> * until recently, SSL was not tested, and I believe some projects
>>> still don't test with SSL enabled.
>>> * IPv6 is not tested everywhere.
>>> * Production scenarios, with HA (HAproxy or/and Pacemaker) are not tested.
>>>
>>> My point here, is that devstack has been doing very good job for its
>>> simplicity (written in bash) and its large adoption by projects to
>>> make CI, though we might consider adding more coverage to make sure it
>>> works outside devstack.
>>> As an example, Puppet OpenStack modules CI is using a devstack-like
>>> job (with 3 scenarios), called puppet-openstack-integration [1] but we
>>> also run TripleO and Fuel CI jobs, to increase coverage and give a
>>> better feedback on testing.
>>>
>>>
>>> == Proposal
>>>
>>> This is not about removing devstack! The idea is to add more coverage
>>> in an iterative way, with some other tools.
>>> We started some months ago by running TripleO CI jobs in Ironic and
>>> Heat gates (experimental pipeline) because TripleO is high consumer of
>>> Ironic and Heat.
>>> Also, we recently added our TripleO multinode job in Nova experimental
>>> pipeline (doc here [2]).
>>> Now, we are moving forward with python-openstackclient and osc-lib.
>>>
>>> I started to draft a document about how we could increase coverage in
>>> different projects:
>>>
>>> https://docs.google.com/spreadsheets/d/1bLg-uEGrQXyRZ-FuR6pf1WT4XN0-6MrlfqEShI7xMxg/edit#gid=0
>>>
>>> (feel free to add your project and give your opinion directly in the
>>> spreadsheet).
>>>
>>> The intention here is to discuss with teams interested by such CI
>>> coverage. We don't want to slow down or break your gate (example with
>>> TripleO, our jobs are non-voting outside TripleO and take ~45 min);
>>> but reduce the feedback loop between developers and deployment tools
>>> used in production.
>>> We don't expect developers to investigate why new CI jobs would fail
>>> (ex: a Nova developer to look at TripleO CI job), it would be unfair.
>>> Just some horizontal communication would be enough, IRC or email, to
>>> inform that a patch might break a CI job outside devstack.
>>> I also want to mention that the proposal is not only about TripleO. I
>>> used this tool in my examples because I'm working on it but obviously
>>> this proposal should be open to Big Tent projects that also deploy
>>> OpenStack.
>>>
>>> Please give any feedback, and let's make OpenStack testing stronger!
>>> Thanks for reading so far,
>>>
>>> [1] https://github.com/openstack/puppet-openstack-integration
>>> [2] https://review.openstack.org/#/c/381838/
>>>
>>
>> I don't really have a problem with wanting to run non-devstack deployment
>> tool jobs against project changes on-demand (experimental queue job). That's
>> why I approved the change to add that TripleO job to nova's experimental
>> queue.
>>
>> The experimental queue is only on-demand though, so reviewers have to be
>> conscious of running it and even then people don't think to check the
>> results, or a failure might not be obvious as to what caused it (my patch,
>> or is this job always broken and is thus in the experimental queue, like the
>> nova-lxc job?).
>>
>> For better or worse devstack is at least universally used and it's THE
>> default thing we point newcomers to when getting started if they want to
>> quickly and easily get a development environment with a running openstack on
>> a single-node up and running to kick the tires. I can't say the same for the
>> plethora of other deployment projects out there like kolla, ansible, salt,
>> puppet, chef, tripleo/packstack/rdo, fuel, etc, etc. I think that's what's
>> really caused the lack of universal adoption of anything besides devstack in
>> our CI environment. And love it or hate it, I think anyone that's been
>> around for awhile and tries to debug gate failures is at least used to
>> hacking on devstack and knows how it works to a certain extent.
>>
>> Anyway, as I said, I've got no problem with getting some additional optional
>> non-voting coverage in other projects besides devstack to at least try and
>> prevent breaking changes. I worry about trying to move various deployment
>> jobs into the check queue for multiple projects though, as I think that
>> would put a pretty serious strain on resources for non-voting jobs, which
>> we'd like to avoid I think.
>>
>> My two cents.
>
>
> Matt, Emilien,
>
> there's one more option, run jobs daily and add the output to the
> openstack health dashboard.
> example - http://status.openstack.org/openstack-health/#/g/build_name/periodic-ironic-py35-with-oslo-master

TripleO already has periodic jobs that test OpenStack from master, but
the periodic frequency is 24 hours which is too long to wait for
testing.

I'm also aware about OpenStack Infra desire to reduce the number of
periodic jobs (tell me if I'm wrong).
An alternative would be to have a periodic job pipeline that runs jobs
every 3 or 6 hours, this frequency would be good enough to tell us
when things started to break and the window of time where patches got
merged in master would be less important than for 24 hours, so
investigation would be easier. It sounds like to me an alternative to
run tripleo jobs outside tripleo CI.

Thanks,

>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi



More information about the OpenStack-dev mailing list