<p dir="ltr"><br>
On 19 May 2016 5:38 pm, "Paul Belanger" <<a href="mailto:pabelanger@redhat.com">pabelanger@redhat.com</a>> wrote:<br>
><br>
> On Thu, May 19, 2016 at 03:50:15PM +0100, Derek Higgins wrote:<br>
> > On 18 May 2016 at 13:34, Paul Belanger <<a href="mailto:pabelanger@redhat.com">pabelanger@redhat.com</a>> wrote:<br>
> > > On Wed, May 18, 2016 at 12:22:55PM +0100, Derek Higgins wrote:<br>
> > >> On 6 May 2016 at 14:18, Paul Belanger <<a href="mailto:pabelanger@redhat.com">pabelanger@redhat.com</a>> wrote:<br>
> > >> > On Tue, May 03, 2016 at 05:34:55PM +0100, Steven Hardy wrote:<br>
> > >> >> Hi all,<br>
> > >> >><br>
> > >> >> Some folks have requested a summary of our summit sessions, as has been<br>
> > >> >> provided for some other projects.<br>
> > >> >><br>
> > >> >> I'll probably go into more detail on some of these topics either via<br>
> > >> >> subsequent more focussed threads an/or some blog posts but what follows is<br>
> > >> >> an overview of our summit sessions[1] with notable actions or decisions<br>
> > >> >> highlighted. I'm including some of my own thoughts and conclusions, folks<br>
> > >> >> are welcome/encouraged to follow up with their own clarifications or<br>
> > >> >> different perspectives :)<br>
> > >> >><br>
> > >> >> TripleO had a total of 5 sessions in Austin I'll cover them one-by-one:<br>
> > >> >><br>
> > >> >> -------------------------------------<br>
> > >> >> Upgrades - current status and roadmap<br>
> > >> >> -------------------------------------<br>
> > >> >><br>
> > >> >> In this session we discussed the current state of upgrades - initial<br>
> > >> >> support for full major version upgrades has been implemented, but the<br>
> > >> >> implementation is monolithic, highly coupled to pacemaker, and inflexible<br>
> > >> >> with regard to third-party extraconfig changes.<br>
> > >> >><br>
> > >> >> The main outcomes were that we will add support for more granular<br>
> > >> >> definition of the upgrade lifecycle to the new composable services format,<br>
> > >> >> and that we will explore moving towards the proposed lightweight HA<br>
> > >> >> architecture to reduce the need for so much pacemaker specific logic.<br>
> > >> >><br>
> > >> >> We also agreed that investigating use of mistral to drive upgrade workflows<br>
> > >> >> was a good idea - currently we have a mixture of scripts combined with Heat<br>
> > >> >> to drive the upgrade process, and some refactoring into discrete mistral<br>
> > >> >> workflows may provide a more maintainable solution. Potential for using<br>
> > >> >> the existing SoftwareDeployment approach directly via mistral (outside of<br>
> > >> >> the heat templates) was also discussed as something to be further<br>
> > >> >> investigated and prototyped.<br>
> > >> >><br>
> > >> >> We also touched on the CI implications of upgrades - we've got an upgrades<br>
> > >> >> job now, but we need to ensure coverage of full release-to-release upgrades<br>
> > >> >> (not just commit to commit).<br>
> > >> >><br>
> > >> >> -------------------------------<br>
> > >> >> Containerization status/roadmap<br>
> > >> >> -------------------------------<br>
> > >> >><br>
> > >> >> In this session we discussed the current status of containers in TripleO<br>
> > >> >> (which is to say, the container based compute node which deploys containers<br>
> > >> >> via Heat onto an an Atomic host node that is also deployed via Heat), and<br>
> > >> >> what strategy is most appropriate to achieve a fully containerized TripleO<br>
> > >> >> deployment.<br>
> > >> >><br>
> > >> >> Several folks from Kolla participated in the session, and there was<br>
> > >> >> significant focus on where work may happen such that further collaboration<br>
> > >> >> between communities is possible. To some extent this discussion on where<br>
> > >> >> (as opposed to how) proved a distraction and prevented much discussion on<br>
> > >> >> supportable architectural implementation for TripleO, thus what follows is<br>
> > >> >> mostly my perspective on the issues that exist:<br>
> > >> >><br>
> > >> >> Significant uncertainty exists wrt integration between Kolla and TripleO -<br>
> > >> >> there's largely consensus that we want to consume the container images<br>
> > >> >> defined by the Kolla community, but much less agreement that we can<br>
> > >> >> feasably switch to the ansible-orchestrated deployment/config flow<br>
> > >> >> supported by Kolla without breaking many of our primary operator interfaces<br>
> > >> >> in a fundamentally unacceptable way, for example:<br>
> > >> >><br>
> > >> >> - The Mistral based API is being implemented on the expectation that the<br>
> > >> >> primary interface to TripleO deployments is a parameters schema exposed<br>
> > >> >> by a series of Heat templates - this is no longer true in a "split stack"<br>
> > >> >> model where we have to hand off to an alternate service orchestration tool.<br>
> > >> >><br>
> > >> >> - The tripleo-ui (based on the Mistral based API) consumes heat parameter<br>
> > >> >> schema to build it's UI, and Ansible doesn't support the necessary<br>
> > >> >> parameter schema definition (such as types and descriptions) to enable<br>
> > >> >> this pattern to be replicated. Ansible also doesn't provide a HTTP API,<br>
> > >> >> so we'd still have to maintain and API surface for the (non python) UI to<br>
> > >> >> consume.<br>
> > >> >><br>
> > >> >> We also discussed ideas around integration with kubernetes (a hot topic on<br>
> > >> >> the Kolla track this summit), but again this proved inconclusive beyond<br>
> > >> >> that yes someone should try developing a PoC to stimulate further<br>
> > >> >> discussion. Again, significant challenges exist:<br>
> > >> >><br>
> > >> >> - We still need to maintain the Heat parameter interfaces for the API/UI,<br>
> > >> >> and there is also a strong preference to maintain puppet as a tool for<br>
> > >> >> generating service configuration (so that existing operator integrations<br>
> > >> >> via puppet continue to function) - this is a barrier to directly<br>
> > >> >> consuming the kolla-kubernetes effort directly.<br>
> > >> >><br>
> > >> >> - A COE layer like kubernetes is a poor fit for deployments where operators<br>
> > >> >> require strict control of service placement (e.g exactly which nodes a service<br>
> > >> >> runs on, IP address assignments to specific nodes etc) - this is already<br>
> > >> >> a strong requirement for TripleO users and we need to figure out if/how<br>
> > >> >> it's possible to control container placement per node/namespace.<br>
> > >> >><br>
> > >> >> - There are several uncertainties regarding the HA architecture, such as<br>
> > >> >> how do we achieve fencing for nodes (which is currently provided via<br>
> > >> >> pacemaker), in particular the HA model for real production deployments<br>
> > >> >> via kubernetes for stateful services such as rabbit/galera is unclear.<br>
> > >> >><br>
> > >> >> Overall a session with much discussion, but further prototyping and<br>
> > >> >> discussion is required before we can define a definitive implementation<br>
> > >> >> strategy (several folks are offering to be involved in this).<br>
> > >> >><br>
> > >> >> ---------------------------------------------<br>
> > >> >> Work session (Composable Services and beyond)<br>
> > >> >> ---------------------------------------------<br>
> > >> >><br>
> > >> >> In this session we discussed the status of the currently in-progress work<br>
> > >> >> to decompose our monolithic manifests into per-service profiles[3] in<br>
> > >> >> puppet-tripleo, then consume these profiles via per-service templates in<br>
> > >> >> tripleo-heat-templates[4][5], and potential further work to enable fully<br>
> > >> >> composable (including user defined) roles.<br>
> > >> >><br>
> > >> >> Overall there was agreement that the composable services work and puppet<br>
> > >> >> refactoring are going well, but that we need to improve velocity and get<br>
> > >> >> more reviewers helping to land the changes. There was also agreement that<br>
> > >> >> a sub-team should form temporarily to drive the remaining work[6], that<br>
> > >> >> we should not land any new features in the "old" template architecture and<br>
> > >> >> relatedly that tripleo cores should help rebase and convert currently<br>
> > >> >> under-review changes to the new format where needed to ease the transition.<br>
> > >> >><br>
> > >> >> I described a possible approach to providing fully composable roles that<br>
> > >> >> uses some template pre-processing (via jinja2)[7], a blueprint and initial<br>
> > >> >> implementation will be posted soon, but overall the response was positive,<br>
> > >> >> and it may provide a workable path to fully composable roles that won't<br>
> > >> >> break upgrades of existing deployments.<br>
> > >> >><br>
> > >> >> ---------------------------------<br>
> > >> >> Work session (API and TripleO UI)<br>
> > >> >> ---------------------------------<br>
> > >> >><br>
> > >> >> In this session we disccussed the current status of the TripleO UI, and the<br>
> > >> >> Mistral based API implementation it depends on.<br>
> > >> >><br>
> > >> >> Overall it's clear there is a lot of good progress in this area, but there<br>
> > >> >> are some key areas which require focus and additional work to enable a<br>
> > >> >> fully functional upstream TripleO UI:<br>
> > >> >><br>
> > >> >> - The undercloud requires some configuration changes to enable the UI<br>
> > >> >> necessary access to the undercloud services<br>
> > >> >><br>
> > >> >> - The UI currently depends on the previous prototype API implementation,<br>
> > >> >> and must be converted to the new Mistral based API (in-progress)<br>
> > >> >><br>
> > >> >> - We need to improve velocity of the Mistral based implementation (need<br>
> > >> >> more testing and reviewing), such that we can land it and folks can start<br>
> > >> >> integrating with it.<br>
> > >> >><br>
> > >> >> - There was agreement that the previously proposed validation API can be<br>
> > >> >> implemented as another Mistral action, which will provide a way to run<br>
> > >> >> validation related to the undercloud configuration/state.<br>
> > >> >><br>
> > >> >> - There are some features we could add to Heat which would make<br>
> > >> >> implementation cleaner (description/metadata in environment files, enable<br>
> > >> >> multiple parameter groups.<br>
> > >> >><br>
> > >> >> The session concluded with some discussion around the requirements related<br>
> > >> >> to network configuration. Currently the templates offer considerable<br>
> > >> >> flexibility in this regard, and we need to decide how this is surfaced via<br>
> > >> >> the API such that it's easily consumable via TripleO Ux interfaces.<br>
> > >> >><br>
> > >> >> -----------------------------------<br>
> > >> >> Work session (Reducing the CI pain)<br>
> > >> >> -----------------------------------<br>
> > >> >><br>
> > >> >> This session covered a few topics, but mostly ended up focussed on the<br>
> > >> >> debate with infra regarding moving to 3rd party CI. There are arguments on<br>
> > >> >> both sides here, and I'll perhaps let derekh or dprince reply with a more<br>
> > >> >> detailed discussion of them, but suffice to say there wasn't a clear<br>
> > >> >> conclusion, and discussion is ongoing.<br>
> > >> >><br>
> > >> > It was mostly me pushing for tripleo to move to 3rd party CI. I still think it<br>
> > >> > is the right place for tripleo however after hearing dprince's concerns I think<br>
> > >> > we have a compromise for the moment. I've gone a head and done the work to<br>
> > >> > upgrade tripleo-ci jenkins slave from Fedora-22 to the centos-7 DIB[1] produced by<br>
> > >> > openstack-infra. Please take a moment to review the patch as it exposed 3<br>
> > >> > issues.<br>
> > >> ><br>
> > >> > 1) CentOS 7 does not support nbd out of the box, and we can't compile a new<br>
> > >> > kernel ATM. So, I've worked around the problem by converting the qcow2 image to<br>
> > >> > raw format, update instack and reconverted it back to qcow2. Ideally, if I can<br>
> > >> > find where the instack.qcow2 image is build, we also produce a raw format so we<br>
> > >> > don't have to do this every gate job.<br>
> > >><br>
> > >> The conversion should be ok for the moment to allow use to make<br>
> > >> progress, longer term<br>
> > >> we'll probably need to change the libvirt domain definitions on the<br>
> > >> testenvs in order to<br>
> > >> be able to just generate and use a raw format.<br>
> > >><br>
> > >> ><br>
> > >> > 2) Jenkins slave needs more HDD space. Using centos-7 we cache data to the slave<br>
> > >> > now, mostly packages and git repos. As a result the HDD starts at 7.5GB and<br>
> > >> > because the current slaves use 20GB we quickly run out of space. Ideally we<br>
> > >> > need 80GB[2] of space to be consistent with the other cloud provides we run<br>
> > >> > jenkins slaves on.<br>
> > >><br>
> > >> This is where we'll likely hit the biggest problems, In order to bump<br>
> > >> the disk space allocated to the jenkins slaves and to simultaneously<br>
> > >> take advantage of the SSD's we're going to have to look into using the<br>
> > >> SSD's as a cache for the spinning disks. I havn't done this before but<br>
> > >> I hope we can look into it soon.<br>
> > >><br>
> > >> ><br>
> > >> > 3) No AFS mirror in tripleo-ci[3]. To take advantage of the new centos-7 dib,<br>
> > >> > openstack-infra has an AFS mirroring infrastructure in place. As a result,<br>
> > >> > we'll also need to launch one in tripleo-ci. For the moment, I've disabled the<br>
> > >> > logic to configure the mirror. Mirrors include pypi, npm, wheel, ubuntu trusty,<br>
> > >> > ubuntu precise, ceph. We are bringing RPM mirrors online shortly.<br>
> > >><br>
> > >> I'm not sure we'll get as much a benefit from this as the devstack<br>
> > >> based jobs do, as is some of the mirrors you mention wouldn't be used<br>
> > >> at all while others we would only make very light use of. Is it<br>
> > >> possible to selectively add mirrors to the AFS mirror, or add<br>
> > >> additional things that tripleo would be interested in? e.g. image<br>
> > >> cache<br>
> > >><br>
> > > I think you'll actually benefit from this, mostly because you no longer have to<br>
> > > run your own mirror / squid servers in tripleo. The way AFS mirrors work is<br>
> > > more like a cache.<br>
> > ><br>
> > > Currently our AFS volumes in rax-dfw are over 1TB of data now, but since our<br>
> > > jobs only access a small fraction of the data, most mirror AFS servers are only<br>
> > > using about 5GB of data locally.<br>
> > ><br>
> > > In the case of tripleo, it will even be less since you are not running the full<br>
> > > suite of job in your cloud.<br>
> > ><br>
> > > Right now, nothing would need to chance to selectively use mirrors, because<br>
> > > AFS will only cache what is used. As for adding things specific to tripleo, it<br>
> > > could be possible, it is also possible other jobs will likely need the same bits<br>
> > > too.<br>
> > ><br>
> > > I strongly encourage us to setup an AFS mirror.<br>
> ><br>
> > Ok, I'm still a little skeptical because our biggest bandwidth hogs<br>
> > arn't mentioned in the list of things mirrored , but that's not a good<br>
> > reason to get in your way, if it proves to be a help then great, if<br>
> > not at least we tried, so what do you need from me to try it out? If I<br>
> > create a d1.medium trusty instance with a floating IP, will that work<br>
> > for you? This should allow you to test things for now, longer term<br>
> > were going to have the same problems we do with larger jenkins<br>
> > instance so until we solve this we wont be able to consider this a<br>
> > permanent part of the infrastructure.<br>
> ><br>
> I just need to know the flavor we are using, I'll be using our<br>
> opentack-infra/system-config launch-node script to provision the server. Since<br>
> we need to loop it into our ansible / puppet wheel.<br>
><br>
> If you are okay with d1.medium for now, I can start it.</p>
<p dir="ltr">I'm happy to go for that for now to allow you to test this out, we'll likely have to change something in future though.<br></p>
<p dir="ltr">><br>
> > ><br>
> > >> ><br>
> > >> > I'd really like to get some feedback on these 3 issue, I know they might not be<br>
> > >> > solved today because of the hardware move. However, I think we are pretty close<br>
> > >> > now to getting triplo-ci more inline with some of the openstack-infra tooling.<br>
> > >> ><br>
> > >> > [1] <a href="https://review.openstack.org/#/c/312725/">https://review.openstack.org/#/c/312725/</a><br>
> > >> > [2] <a href="https://review.openstack.org/#/c/312992/">https://review.openstack.org/#/c/312992/</a><br>
> > >> > [3] <a href="https://review.openstack.org/#/c/312058/">https://review.openstack.org/#/c/312058/</a><br>
> > >> ><br>
> > >> >> The other output from this session was agreement that we'd move our jobs to<br>
> > >> >> a different cloud (managed by the RDO community) ahead of a planned<br>
> > >> >> relocation of our current hardware. This has advantages in terms of<br>
> > >> >> maintenance overhead, and if it all goes well we can contribute our<br>
> > >> >> hardware to this cloud long term vs maintaining our own infrastructure.<br>
> > >> >><br>
> > >> >><br>
> > >> >> Overall it was an excellent week, and I thank all the session participants<br>
> > >> >> for their input and discussion. Further notes can be found in the<br>
> > >> >> etherpads linked from [1] but feel free to reply if specific items require<br>
> > >> >> clarification (and/or I've missed anything!)<br>
> > >> >><br>
> > >> >> Thanks,<br>
> > >> >><br>
> > >> >> Steve<br>
> > >> >><br>
> > >> >> [1] <a href="https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#TripleO">https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#TripleO</a><br>
> > >> >> [2] <a href="https://review.openstack.org/#/c/299628/">https://review.openstack.org/#/c/299628/</a><br>
> > >> >> [3] <a href="https://blueprints.launchpad.net/tripleo/+spec/refactor-puppet-manifests">https://blueprints.launchpad.net/tripleo/+spec/refactor-puppet-manifests</a><br>
> > >> >> [4] <a href="https://blueprints.launchpad.net/tripleo/+spec/composable-services-within-roles">https://blueprints.launchpad.net/tripleo/+spec/composable-services-within-roles</a><br>
> > >> >> [5] <a href="https://etherpad.openstack.org/p/tripleo-composable-roles-work">https://etherpad.openstack.org/p/tripleo-composable-roles-work</a><br>
> > >> >> [6] <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-April/093533.html">http://lists.openstack.org/pipermail/openstack-dev/2016-April/093533.html</a><br>
> > >> >> [7] <a href="http://paste.fedoraproject.org/360836/87416814/">http://paste.fedoraproject.org/360836/87416814/</a><br>
> > >> >><br>
> > >> >> __________________________________________________________________________<br>
> > >> >> OpenStack Development Mailing List (not for usage questions)<br>
> > >> >> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > >> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > >> ><br>
> > >> > __________________________________________________________________________<br>
> > >> > OpenStack Development Mailing List (not for usage questions)<br>
> > >> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > >> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > >><br>
> > >> __________________________________________________________________________<br>
> > >> OpenStack Development Mailing List (not for usage questions)<br>
> > >> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > ><br>
> > > __________________________________________________________________________<br>
> > > OpenStack Development Mailing List (not for usage questions)<br>
> > > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> > __________________________________________________________________________<br>
> > OpenStack Development Mailing List (not for usage questions)<br>
> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</p>