Open Stack

I'm Cc'ing this to the openstack-infra ML but setting MFT to direct
subsequent discussion to the openstack-dev ML so we can hopefully
avoid further cross-posting as much as possible. If you're replying
on a particular session topic, please update the Subject so that the
subthreads are easier to keep straight.

Apologies for the _extreme_ delay in getting this composed and sent
out!


Firehose
--------

https://etherpad.openstack.org/p/ocata-infra-firehose

This was primarily a brainstorming/roadmap session on possible
future plans for the firehose.openstack.org service. Discussed was
potential to have Zuul (post-v3) both consume and emit events over
MQTT, as well as having StoryBoard (should probably support an
analog of the events handled by lpmqtt at a minimum, probably easy
to add given it already has an RabbitMQ one) and Nodepool event
streams. The gerritbot consumer PoC was mentioned, but determined
that it would be better to shelve that until planning for an
Errbot-based gerritbot reimplementation is fleshed out.

We talked about how the current logstash MQTT stream implementation,
while interesting, has significant scaling (especially bandwidth)
issues with the volume of logging we do in tests while only offering
limited benefit. We could potentially make use of it in concern with
a separate logstash for production server and Ansible logs, but it's
efficacy for our job logs was called into question.

We also spent much of the timeslot talking about possible
integration with FedMesg (particularly that they're considering
pluggable backend support which could include an MQTT
implementation), which yields much opportunity for collaboration
between our projects.

One other topic which came up was how to do a future HA
implementation, either by having publishers send to multiple brokers
and configure consumers to have a primary/fallback behavior or my
trying to implement a more integrated clustering solution with load
balancing proxies. We concluded that current use cases don't demand
anywhere near 100% message delivery and 100% uptime, so we can dig
deeper when there's an actual use case.


Status update and plans for task tracking
-----------------------------------------

https://etherpad.openstack.org/p/ocata-infra-community-task-tracking

As is traditional, we held a fishbowl on our ongoing task tracking
woes. We started with a brief introduction of stakeholders who
attended and the groups whose needs they were there to represent.
After that, some presentation was made of recent StoryBoard
development progress since Austin (including Gerrit integration,
private story support for embargoed security issues, improved event
timelines, better discoverability for boards and worklists, flexible
task prioritization), as well as the existing backlog of blockers.

We revisited the Infra spec on task tracking
http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html
for the benefit of those present, and Kendall Nelson (diablo_rojo)
agreed to pick up and continue with the excellent stakeholder
blocking issues outreach/coordination work begun by Anita Kuno
(anteaya).


Next steps for infra-cloud
--------------------------

https://etherpad.openstack.org/p/ocata-infra-infra-cloud

This was sort of a catch-all opportunity to hash out current plans
and upcoming needs for the infra-cloud. We determined that the
current heterogeneous hardware in the in-progress "chocolate" region
should be split into two homogeneous regions named "chocolate" and
"strawberry" (our "vanilla" region was already homogeneous). We also
talked about ongoing work to get a quote from OSUOSL for hosting the
hardware so that we can move it out of HPE data centers, and
attempting to find funding once we have some figures firmed up.

There were also some sideline discussions on possible monitoring and
high-availability options for the underlying services.
Containerization was, as always, brought up but the usual "not a fit
for this use case" answers abounded. It was further suggested that
using infra-cloud resources for things like special TripleO tests or
Docker registry hosting were somehow in scope, but there are other
solutions to these needs which should not be conflated with the
purpose of the infra-cloud effort.


Interactive infra-cloud debugging
---------------------------------

https://etherpad.openstack.org/p/ocata-infra-infra-cloud-debugging

The original intent for this session was to try to gather
leaders/representatives from the various projects that we're relying
on in the infra-cloud deployment and step through an interactive
session debugging the sorts of failures we see arise on the servers.
The idea was that this would be potentially educational for some
since this is a live bare metal "production" deployment of Nova,
Neutron, Keystone, Glance, et cetera with all the warts and rough
edges that operators handle on a daily basis but our developers may
not have directly experienced.

As well-intentioned as it was, the session suffered from several
issues. First and foremost we didn't realize the Friday morning
workroom we got was going to lack a projector (only so many people
can gather around one laptop, and if it's mine then fewer still!).
Trying to get people from lots of different projects to show up for
the same slot on a day that isn't for cross-project sessions is
pretty intractable. And then there's the fact that we were all
approaching burnout as it was the last day of the week and coffee
was all the way at the opposite end of the design summit space. :/

Instead the time was spent partly continuing the "future of
infra-cloud" discussion, and partly just talking about random things
like troubleshooting CI jobs (some people misunderstood the session
description and thought that's what we had planned) or general Infra
team wishlist items. Not a complete waste, but some lessons learned
if we ever want to try this idea again at a future summit.


Test environment expectations
-----------------------------

https://etherpad.openstack.org/p/ocata-infra-test-env-expectations

After the morning break we managed to perk back up again and discuss
test platform expectations. This was a remarkably productive
brainstorming session where we assembled a rough list of
expectations developers can and, more importantly, can't make about
the systems on which our CI jobs run. The culmination of these
musings can since be found in a shiny new page of the Infra Manual:

http://docs.openstack.org/infra/manual/testing.html


Xenial jobs transition for stable/newton
----------------------------------------

https://etherpad.openstack.org/p/ocata-infra-xenial-stable-newton

Another constructive session right on the heels of the last...
planning the last mile of the cut-over from Ubuntu 14.04 to 16.04
testing. We confirmed that we would switch all jobs for
stable/newton as well as master (since the implementation started
early in the Newton cycle and we need to be consistent across
projects in a stable branch). We decided to set a date (which
incidentally is TODAY) to finalize the transition. The plan was
announced to the dev ML a month ago:

http://lists.openstack.org/pipermail/openstack-dev/2016-November/106906.html

The (numerous) changes in flight today to switch the lingering jobs
are covered under a common review topic:

https://review.openstack.org/#/q/topic:st-nicholas-xenial


Unconference afternoon
----------------------

https://etherpad.openstack.org/p/ocata-infra-contributors-meetup

At this stage things were starting to wind up and a lot of people
with early departures had already bowed out. Those of us who
remained were treated to our own room for the first time in many
summits (no offense to the Release and QA teams, but it was nice to
not need to share for a change). Since we were a little more at
liberty to set our own pace this time we treated it as a sort of
home base from which many of us set forth to pitch in on
Infra-related planning discussions in other teams' spaces, then
regroup and disseminate what we'd done (from translation platform
upgrades to release automation designs).

We also got in some good one-on-one time to work through topics
which weren't covered in scheduled sessions, such as Zuul v3 spec
additions or changes to the pep8 jobs to guard against missing sdist
build dependencies. As the afternoon progressed and the crowd
dwindled further we said our goodbyes and split up into smaller
groups to go out for one last meal, commiserate with those who found
themselves newly in search of employment and generally celebrate a
successful week in Barcelona.


That concludes my recollection of these sessions over the course of
the week--thanks for reading this far--feel free to follow up (on
the openstack-dev ML please) with any corrections/additions. Many
thanks to all who attended, and to those who could not: we missed
you. I hope to see lots of you again at the PTG in Atlanta, only a
couple months away now. Don't forget to register and book your
flights/lodging!
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: Digital signature
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20161206/f79ff058/attachment.pgp>