Open Stack

On Tue, Sep 19, 2017 at 6:51 PM Andrea Frittoli <andrea.frittoli at gmail.com>
wrote:

> Dear Bats,
>
> thanks everyone for participating to the Queens PTG - in person and
> remotely - and for making it a successful and enjoyable event.
>
> Here's a summary of what we discussed and achieved during the week.
> This report is far from complete - please complement it with QA related
> stories I may have missed.
>
> Unfortunately I ran out of bat stickers, I hope to have some more for the
> next time we meet :)
>
> Pike Retrospective
> -------------------------
> We experimented this time with running a retrospective [0].
> The input for the retrospective was almost exclusively from members of the
> QA team, which means that either the retrospective was poorly advertised by
> me, or that we did a nice job for the community during the QA cycle. I hope
> the latter :)
>
> The outcome of the retrospective was mostly 'yay' for tasks completed, and
> 'next cycle' for things we did not have time to work on.
> A few extra things that came up were:
> - bug triage and fix: we may need a bug czar and more automation to stay
> on top of the bug queue
> - elastic recheck categorisation: we may need an e-r czar to ensure we
> don't leave gate failures and races uncategorised and accumulating
> - meetings are mostly attended in the APAC time zone and very seldom
> attended by non-qa folks. We will consider shortening / reducing them and
> complementing them with QA office hours
>
> Monitoring the Gate
> --------------------------
> At the beginning of the Pike cycle we had a number of stability issues in
> the gate.
> To prevent issues from accumulating over time we discussed a few ideas
> about monitoring the status of the gate and providing more feedback to
> reviewers to help catch patches that may introduce issues [1].
> There are a few things that can be done relatively easily (or that we
> already have) in terms of data collection: job duration and failure rates,
> aggregated dstat data, resources created by tests.
> We miss OpenStack Health contributors to create new views for this data.
> Links from gerrit and zuul dashboard into OpenStack Health would help
> making the data discoverable.
>
> Tempest
> ------------
> A large chunk of the patches required to mark test.py as a stable
> interface for plugins was merged during the summit.
> It was good to have them merged during PTG so it was easier to fix
> resulting issues in Tempest plugins - at least Manila and Sahara needed a
> small patch.
> There are two patch series left [2][3] which should have very little
> impact on plugins - we'll try to merge them as soon as possible to avoid
> disruptions later in the Queens cycle.
>
> We worked with a few project teams on the goal to split Tempest plugins to
> a dedicated repo.
> The step by step process [4] linked to the goal includes an example
> section - we hope to get more and more examples in there from team who
> already went through the process.
>
> Devstack
> ------------
> We discussed about devstack runtime. The time to setup an OpenStack cloud
> using devstack seems to have increased over time [5][6] - it would be good
> to investigate why and see if there is time we can save in the gate and on
> each developer laptop :) Between August and September the average runtime
> in the gate increased by about 200s.
>
> Upgrade Testing
> ----------------------
> Rolling upgrade testing via grenade is important for project that seek
> obtaining the support rolling upgrade tag [7].
> While the scope of Grenade is rather fixed, it should be possible to
> support ordering (or relative ordering) in project updates.
>
> Policy Testing
> ------------------
> We discussed what's next for the Queens cycle - support for multi-policy
> testing is the largest chunk of work planned for now.
>
> stestr
> -------
> The migration in ostestr to use stestr internally happened shortly before
> the PTG [8] and we worked through the PTG to amend any deviation in
> behaviour that this may have caused.
> Next in the todo list is to run stestr natively in Tempest, bypassing
> ostestr completely.
> The plan is for this to lead the way for projects to gradually remove the
> dependency to ostestr completely.
>
> HA Testing
> ---------------
> We talked quite a bit about HA and non-functional testing in general.
> Non-functional testing is not a good fit for gate testing, since it's not
> as reliable as functional / integration testing and it often produces
> results which needs to be interpreted by a human being.
> It also has strong dependencies to the deployment tooling and
> architecture. Until now most of OpenStack non-functional testing has been
> done by vendors and operators using downstream tools.
>
> SamP and guatamdivgi presented to the QA team a proposal for an Ansible
> based framework for HA testing [9]. Plugins will allow to seamlessly port
> different test scenario against different cloud architecture, thus
> rendering the framework of interest as a general testing tool for
> OpenStack. The same concept can be extended to non-functional testing in
> general.
>
> It's not clear yet if any of this could run as part of OpenStack CI.  We
> hope to see a PoC after a couple of months in the Queens cycle.
> The NFV ecosystem would be happy to see publicly available non-functional
> tests for OpenStack as well - if we had a common framework for such tests
> we'd have an opportunity to attract contributions into tests from them as
> well.
>
> All things Zuul V3
> -----------------------
> Zuul v3 will provide a DB backend for test runs data, which we want to
> integrate as a new data source for OpenStack Health, so that we can then
> provide a full picture of all job runs in O-H.
>
> Job names will change with zuul v3. While we found a simple solution to
> keep OpenStack Health data functional with Zuul v3 provided metadata, we
> will still loose 6 months of historical data in the process of migration,
> for which we don't have a solution yet.
>
> In terms of Zuul v3 jobs, we started working on Zuul v3 native Tempest
> jobs [10]. this will be the basis for most jobs that run Tempest against
> devstack.
> The patch is still WIP, but it already runs the full Tempest run, with
> only one test failure which I need to investigate.
>
> QA Help Room
> --------------------
> The help room was helpful (heh, I couldn't resist) mostly on day 1.
> It was good to have dedicated time allocated to answer questions, since it
> gave us space to open the laptop and work on code directly to answer
> specific questions.
>
> There were no questions asked during day 2. We used the time to join other
> sessions and work on QA topics, but perhaps in Dublin we could consider
> having a single help room day.
>
> Cross-Community
> ------------------------
> NFV:
> It was very interesting to discuss with the OPNFV community about CI and
> test processes and architecture.
> The aim on both sides to re-use / share tools and best practices, share
> test results and avoid duplicate efforts where relevant.
>
> K8S:
> We discussed with dmellado the possibility of having a basic k8s client
> hosted in a Tempest plugin [11], that could be used by k8s related
> OpenStack project to write OpenStack / k8s scenario tests. The main
> advantages of a k8s client written in Tempest format and hosted by
> OpenStack would be a uniform interface with the OpenStack Tempest client
> and a stable testing API.
>
> QA Themes for Queens
> --------------------------------
> We summarised the topics the QA team will focus on through the Queens
> cycle in out queens priority etherpad [11].
> The main etherpad for QA at the PTG in Denver is available as well [12].
>
> QA Team Pictures
> ------------------------
> I'll forward the pictures as soon as I have a high-res copy available :)
>


Team Pictures:

https://photos.app.goo.gl/R6cuI7lAD6YKzwfF2



>
> Thanks for reading!
>
> Andrea Frittoli (andreaf)
>
> [0] https://etherpad.openstack.org/p/qa-pike-retrospective
> [1] https://etherpad.openstack.org/p/qa-pike-gate-performance
> [2]
> https://review.openstack.org/#/q/topic:test_module_stable+(status:open+OR+status:merged
> )
> [3]
> https://review.openstack.org/#/q/topic:bp/consistent-service-method-names+status:open+owner:%22Ghanshyam+Mann+%253Cghanshyammann%2540gmail.com%253E%22
>
> [4] https://etherpad.openstack.org/p/tempest-separate-plugin
> [5]
> http://lists.openstack.org/pipermail/openstack-dev/2017-September/122271.html
> [6]
> http://status.openstack.org/openstack-health/#/test/devstack?resolutionKey=day&duration=P6M
>
> [7]
> https://governance.openstack.org/tc/reference/tags/assert_supports-rolling-upgrade.html
> [8]
> http://lists.openstack.org/pipermail/openstack-dev/2017-September/122135.html
>
> [9] https://etherpad.openstack.org/p/qa-queens-ptg-destructive-testing
> [10] https://review.openstack.org/#/c/504246/
> [11] https://etherpad.openstack.org/p/qa-queens-priorities
> [12] https://etherpad.openstack.org/p/qa-queens-ptg
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170919/78127237/attachment-0001.html>