Open Stack

Hi Clark,

> There is more to it than that. This service is part of the CI system we operate.
> The way you consume it is through the use of Zuul jobs. If you want to inject
> data into our Logstash/Elasticsearch system you do that by configuring your jobs
> in Zuul to do so. We are not in the business of operating one off solutions to
> problems. We support a large variety of users and projects and using generic
> flexible systems like this one is how we make that viable.
> 
> Additionally these systems are community managed so that we can work
> together to solve these problems in a way that gives the infra team appropriate
> administrative access while still allowing you and others to get specific work
> done. Rather than avoid this tooling can we please attempt to use it when it has
> preexisting solutions to problems like this? We will happily do our best to make
> re-consumption of existing systems a success, but building one off solutions to
> solve problems that are already solved does not scale.
> 

Sure, OK, understood...

[snip]
> 
> I wasn't directly involved with the decision making at the time but back at the
> beginning of the year my understanding was that Jenkins was chosen over Zuul
> for expediency. This wasn't a bad choice as the Github support in Zuul was still
> quite new (though having more users would likely have pushed it along more
> quickly). It probably would be worthwhile to decide separately if Jenkins is the
> permanent solution to the Kata CI tooling problem, or if we should continue to
> push for Zuul. If we want to push for Zuul then I think we need to stop choosing
> Jenkins as a default and start implementing new stuff in Zuul then move the
> existing CI as Kata is able.
> 
> As for who has Zuul access, the Infra team has administrative access to the
> service. Zuul configuration for the existing Kata jobs is done through a repo
> managed by the infra team, but anyone can push and propose changes to this
> repo. The reason for this is Zuul wants to gate its config updates to prevent new
> configs from being merged without being tested. Bypassing this testing does
> allow you to break your Zuul configuration. Currently we aren't gating Kata with
> Zuul so the configs live in the Infra repo. If we started gating Kata changes with
> Zuul we could move the configs into Kata repos and Kata could self manage
> them.
> 
> Looking ahead Zuul is multitenant aware, and we could deploy a Kata tenant.
> This would give Kata a bit more freedom to configure its Zuul pipeline behavior
> as desired, though gating is still strongly recommended as that will prevent
> broken configs from merging.

I spoke with some of the other Kata folks - we agreed I'd try to move the Kata metrics CI into Zuul utilizing the packet.net hardware, and let's see how that pans out. I think that will help both sides understand the current state of kata/zuul so we can move things forwards there.

Wrt the packet.net slaves, I believe we can do that using some of the packet.net/zuul integration work done by JohnStudarus - John and I had some chats at the Summit in Berlin.
https://opensource.com/article/18/10/building-zuul-cicd-cloud

I'll do some Zuul readup, work out how I need to PR the additional ansible/yaml items to the infra repos to add the metrics build/runs (I see the repos and code, and a metrics run is very very similar to a normal kata CI run - and to begin with we  can do those runs in the VM builders to test out the flows before moving to the packet.net hardware).

[move this down to the end...]
> 
> No, we would inject the data through the existing test node -> Zuul -> Logstash -
> > Elasticsearch path.

This might be one bit we have to work out. The metrics generates raw JSON results. The best method I found for landing that directly into logstash was through the socket filebeat. It is not clear in my head how this ties in with Zuul - the best method I found previously for direct JSON injection into logstash, and thus elastic, was using the socket filebeat. Will that fit in with the infra?

Graham

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.