From jimmy at openstack.org Wed May 2 19:44:01 2018 From: jimmy at openstack.org (Jimmy McArthur) Date: Wed, 02 May 2018 14:44:01 -0500 Subject: [OpenStack-Infra] [openstack-dev] Zuul memory improvements In-Reply-To: <87wowo4tyz.fsf@meyer.lemoncheese.net> References: <87wowo4tyz.fsf@meyer.lemoncheese.net> Message-ID: <5AEA1501.3090809@openstack.org> Congrats on the improvements, Jim! Sounds like this is going to make a huge difference. Go Zuul! Cheers, Jimmy > James E. Blair > April 30, 2018 at 10:03 AM > Hi, > > We recently made some changes to Zuul which you may want to know about > if you interact with a large number of projects. > > Previously, each change to Zuul which updated Zuul's configuration > (e.g., a change to a project's zuul.yaml file) would consume a > significant amount of memory. If we had too many of these in the queue > at a time, the server would run out of RAM. To mitigate this, we asked > folks who regularly submit large numbers of configuration changes to > only submit a few at a time. > > We have updated Zuul so it now caches much more of its configuration, > and the cost in memory of an additional configuration change is very > small. An added bonus: they are computed more quickly as well. > > Of course, there's still a cost to every change pushed up to Gerrit -- > each one uses test nodes, for instance, so if you need to make a large > number of changes, please do consider the impact to the whole system and > other users. However, there's no longer a need to severely restrict > configuration changes as a class -- consider them as any other change. > > -Jim > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Mon May 7 21:41:19 2018 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 07 May 2018 14:41:19 -0700 Subject: [OpenStack-Infra] Team get together/dinner at Vancouver Summit In-Reply-To: <1524673975.838621.1350524704.5B4FDF90@webmail.messagingengine.com> References: <1524673975.838621.1350524704.5B4FDF90@webmail.messagingengine.com> Message-ID: <1525729279.3966752.1364019640.5E57B6BD@webmail.messagingengine.com> On Wed, Apr 25, 2018, at 9:32 AM, Clark Boylan wrote: > Hello everyone, > > Many of us will be at the Vancouver summit in just under a month and > thought we might try to organize a get together/dinner of some sort. > I've quickly thrown up https://ethercalc.openstack.org/7vm2xrsk1yju to > start collecting availability info. If you are interested please mark > down when you can join us. > > As for venue options I have yet to start looking and missed the last > Vancouver summit so if you have any suggestions lets me know. > It looks like Monday will be the best night. The Marketplace mixer starts at 6pm so maybe we say 7pm for dinner? As for a venue the Steamworks brewpub which is a ten minute walk from the convention center appears to have lots of space and the Internet says they handle groups well. I don't think I will try to set up a formal group meal as that is a lot of work, but I think worst case we'll be able to squeeze in there on a couple tables. EmilienM had other suggestions but they look small and one had a cover. If you have a better idea (because you've been there before or are local or the food doesn't look good) please suggest one now. More than happy to sort out other plans if I have help. Clark From cboylan at sapwetik.org Mon May 7 21:45:29 2018 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 07 May 2018 14:45:29 -0700 Subject: [OpenStack-Infra] PTG September 10-14 in Denver In-Reply-To: <1524246168.158201.1345189728.6F882F1F@webmail.messagingengine.com> References: <1524246168.158201.1345189728.6F882F1F@webmail.messagingengine.com> Message-ID: <1525729529.3967931.1364020048.3F01779E@webmail.messagingengine.com> On Fri, Apr 20, 2018, at 10:42 AM, Clark Boylan wrote: > Hello everyone, > > I've been asked if the Infra team plans to attend the next PTG in > Denver. My current position is that it would be good to attend as a team > as I think it will give us a good opportunity to work on modernizing > config management efforts. But before I go ahead and commit to that it > would be helpful to get a rough headcount of who intends to go (if it > will just be me then likely don't need to have team space). > > Don't worry if you don't have approval yet or have to sort out other > details. Mostly just interested in a "do we intend on being there or > not" type of answer. > > More details on the event can be found at > http://lists.openstack.org/pipermail/openstack-dev/2018-April/129564.html. > Feel free to ask questions if that will help you too. > > Let me know (doesn't have to be to the list if you aren't comfortable > with that) and thanks! As a followup to this I have asked for space for the Infra team for two days at the PTG. In Dublin the time breakdown was roughly two days of shared help room, one day of Zuul, and two days of Infra (if you ignore that snow happened). My understanding is that Zuul and helproom stuff would be scheduled separately (and I've explicitly asked for Infra time to not conflict with Zuul time) which is how I ended up with the two day number. As a reminder the early bird PTG registration price is available until May 17. Hope to see you all there, Clark From Remo at italy1.com Mon May 7 21:51:06 2018 From: Remo at italy1.com (Remo Mattei) Date: Mon, 7 May 2018 14:51:06 -0700 Subject: [OpenStack-Infra] Team get together/dinner at Vancouver Summit In-Reply-To: <1525729279.3966752.1364019640.5E57B6BD@webmail.messagingengine.com> References: <1524673975.838621.1350524704.5B4FDF90@webmail.messagingengine.com> <1525729279.3966752.1364019640.5E57B6BD@webmail.messagingengine.com> Message-ID: <11239E9C-D410-4151-A2CA-A742830A24A7@italy1.com> That sounds great Clark! Ciao > On May 7, 2018, at 2:41 PM, Clark Boylan wrote: > > On Wed, Apr 25, 2018, at 9:32 AM, Clark Boylan wrote: >> Hello everyone, >> >> Many of us will be at the Vancouver summit in just under a month and >> thought we might try to organize a get together/dinner of some sort. >> I've quickly thrown up https://ethercalc.openstack.org/7vm2xrsk1yju to >> start collecting availability info. If you are interested please mark >> down when you can join us. >> >> As for venue options I have yet to start looking and missed the last >> Vancouver summit so if you have any suggestions lets me know. >> > > It looks like Monday will be the best night. The Marketplace mixer starts at 6pm so maybe we say 7pm for dinner? > > As for a venue the Steamworks brewpub which is a ten minute walk from the convention center appears to have lots of space and the Internet says they handle groups well. I don't think I will try to set up a formal group meal as that is a lot of work, but I think worst case we'll be able to squeeze in there on a couple tables. EmilienM had other suggestions but they look small and one had a cover. > > If you have a better idea (because you've been there before or are local or the food doesn't look good) please suggest one now. More than happy to sort out other plans if I have help. > > Clark > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra From cboylan at sapwetik.org Tue May 8 22:00:52 2018 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 08 May 2018 15:00:52 -0700 Subject: [OpenStack-Infra] Followup on the future of Infra config management specs Message-ID: <1525816852.1695493.1365405672.640E0FF1@webmail.messagingengine.com> Hello everyone, Last week we got all three of the promised potential future config management system specs pushed to Gerrit. They can be found here: * https://review.openstack.org/449933 Puppet 4 Infra * https://review.openstack.org/469983 Ansible Infra * https://review.openstack.org/565550 Containerized Infra A good chunk of us appear to have reviewed them at this point. During today's Infra meeting I asked for some initial thoughts and the direction people thought they saw us going in. The general mood seems to be using a system that decouples applications from their host platforms (containers as packaging essentially) and config management to build the base platform(s) that doesn't require every server have specific versions of specific tools (Ansible) would be a helpful long term goal. That said any transition will take time and the puppet upgrade is long over due. With all of this considered the rough plan that I propose is: "life support puppet4 short/medium term, transition to ansible base + container application "packaging" longer term, eventually having zuul do deployments (but this last bit should be its own spec and is out of scope of current effort)". I think this gives us a good short term option that should be doable (upgrade puppetry to puppet 4). Then we can transition in the goodness of not tightly coupling our config management tooling and applications themselves to the platforms we run. Monty has volunteered to do the combining of the specs to reflect what this more concrete plan would look like. I know not everyone can attend the meetings so wanted to make sure everyone saw this and hence this thread. Please provide feedback if you feel strongly about this plan (think it is terrible or think it is great, info is useful in both cases). Thank you, Clark From cboylan at sapwetik.org Sun May 20 23:32:58 2018 From: cboylan at sapwetik.org (Clark Boylan) Date: Sun, 20 May 2018 16:32:58 -0700 Subject: [OpenStack-Infra] Team get together/dinner at Vancouver Summit In-Reply-To: <1525729279.3966752.1364019640.5E57B6BD@webmail.messagingengine.com> References: <1524673975.838621.1350524704.5B4FDF90@webmail.messagingengine.com> <1525729279.3966752.1364019640.5E57B6BD@webmail.messagingengine.com> Message-ID: <1526859178.1932039.1378741688.04D30447@webmail.messagingengine.com> On Mon, May 7, 2018, at 2:41 PM, Clark Boylan wrote: > On Wed, Apr 25, 2018, at 9:32 AM, Clark Boylan wrote: > > Hello everyone, > > > > Many of us will be at the Vancouver summit in just under a month and > > thought we might try to organize a get together/dinner of some sort. > > I've quickly thrown up https://ethercalc.openstack.org/7vm2xrsk1yju to > > start collecting availability info. If you are interested please mark > > down when you can join us. > > > > As for venue options I have yet to start looking and missed the last > > Vancouver summit so if you have any suggestions lets me know. > > > > It looks like Monday will be the best night. The Marketplace mixer > starts at 6pm so maybe we say 7pm for dinner? > > As for a venue the Steamworks brewpub which is a ten minute walk from > the convention center appears to have lots of space and the Internet > says they handle groups well. I don't think I will try to set up a > formal group meal as that is a lot of work, but I think worst case we'll > be able to squeeze in there on a couple tables. EmilienM had other > suggestions but they look small and one had a cover. > > If you have a better idea (because you've been there before or are local > or the food doesn't look good) please suggest one now. More than happy > to sort out other plans if I have help. > > Clark I've seen the event space now and I think we can all meet up under the giant globe hanging in the front entrance of the convention center. See you there. Clark From Remo at italy1.com Sun May 20 23:46:30 2018 From: Remo at italy1.com (Remo Mattei) Date: Sun, 20 May 2018 16:46:30 -0700 Subject: [OpenStack-Infra] {Disarmed} Re: Team get together/dinner at Vancouver Summit In-Reply-To: <1526859178.1932039.1378741688.04D30447@webmail.messagingengine.com> References: <1524673975.838621.1350524704.5B4FDF90@webmail.messagingengine.com> <1525729279.3966752.1364019640.5E57B6BD@webmail.messagingengine.com> <1526859178.1932039.1378741688.04D30447@webmail.messagingengine.com> Message-ID: <217F23C8-6F35-4C75-94C5-97D2FEA97124@italy1.com> Thanks sounds good. 7:30 should work for me..I can try to make it by 7 if that’s the call. Ciao > On May 20, 2018, at 4:32 PM, Clark Boylan wrote: > > On Mon, May 7, 2018, at 2:41 PM, Clark Boylan wrote: >> On Wed, Apr 25, 2018, at 9:32 AM, Clark Boylan wrote: >>> Hello everyone, >>> >>> Many of us will be at the Vancouver summit in just under a month and >>> thought we might try to organize a get together/dinner of some sort. >>> I've quickly thrown up https://ethercalc.openstack.org/7vm2xrsk1yju to >>> start collecting availability info. If you are interested please mark >>> down when you can join us. >>> >>> As for venue options I have yet to start looking and missed the last >>> Vancouver summit so if you have any suggestions lets me know. >>> >> >> It looks like Monday will be the best night. The Marketplace mixer >> starts at 6pm so maybe we say 7pm for dinner? >> >> As for a venue the Steamworks brewpub which is a ten minute walk from >> the convention center appears to have lots of space and the Internet >> says they handle groups well. I don't think I will try to set up a >> formal group meal as that is a lot of work, but I think worst case we'll >> be able to squeeze in there on a couple tables. EmilienM had other >> suggestions but they look small and one had a cover. >> >> If you have a better idea (because you've been there before or are local >> or the food doesn't look good) please suggest one now. More than happy >> to sort out other plans if I have help. >> >> Clark > > I've seen the event space now and I think we can all meet up under the giant globe hanging in the front entrance of the convention center. See you there. > > Clark > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -------------- next part -------------- An HTML attachment was scrubbed... URL: From iwienand at redhat.com Thu May 24 07:40:58 2018 From: iwienand at redhat.com (Ian Wienand) Date: Thu, 24 May 2018 17:40:58 +1000 Subject: [OpenStack-Infra] afs02 r/o volume mirrors - ongoing incident Message-ID: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> Hi, We were notified of an issue around 22:45GMT with the volumes backing the storage on afs02.dfw.o.o, which holds R/O mirrors for our AFS volumes. It seems that during this time there were a number of "vos release"s in flight, or started, that ended up with volumes in a range of unreliable states that made them un-releaseable (essentially halting mirror updates). Several of the volumes were recoverable with a manual "vos unlock" and re-releasing the volume. However, others were not. To keep it short, fairly extensive debugging took place [2], but we had corrupt volumes and deadlocked transactions between afs01 & afs02 with no reasonable solution. In an effort to resolve this, the afs01 & 02 servers were restarted to clear all old transactions, and for the affected mirrors I essentially removed their read-only copies and re-added them with: k5start -t -f /etc/afsadmin.keytab service/afsadmin -- vos unlock $MIRROR k5start -t -f /etc/afsadmin.keytab service/afsadmin -- vos remove -server afs02.dfw.openstack.org -partition a -id $MIRROR.readonly k5start -t -f /etc/afsadmin.keytab service/afsadmin -- vos release -v $MIRROR k5start -t -f /etc/afsadmin.keytab service/afsadmin -- vos addsite -server afs02.dfw.openstack.org -partition a -id $MIRROR The following volumes needed to be recovered mirror.fedora mirror.pypi mirror.ubuntu mirror.ubuntu-ports mirror.debian (these are the largest repositories, and maybe it's no surprise that's why they became corrupt?) I have placed mirror-update.o.o in the emergency file, and commented out all cron jobs on it. Right now, I am running a script in a screen as the root user on mirror-update.o.o to "vos release" these in sequence (/root/release.sh). Hopefully, this brings thing back into sync by recreating the volumes. If not, more debugging will be required :/ Please feel free to check in on this, otherwise I will update tomorrow .au time -i [1] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-05-23.log.html#t2018-05-23T22:43:46 [2] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-05-24.log.html#t2018-05-24T04:01:21 From iwienand at redhat.com Thu May 24 10:45:37 2018 From: iwienand at redhat.com (Ian Wienand) Date: Thu, 24 May 2018 20:45:37 +1000 Subject: [OpenStack-Infra] afs02 r/o volume mirrors - ongoing incident In-Reply-To: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> References: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> Message-ID: <4fea80bd-ebe6-8311-74c3-ac6f5805f7da@redhat.com> On 05/24/2018 05:40 PM, Ian Wienand wrote: > In an effort to resolve this, the afs01 & 02 servers were restarted to > clear all old transactions, and for the affected mirrors I essentially > removed their read-only copies and re-added them with: It seems this theory of removing the volumes and re-adding them is not sufficient to get things working; "vos release" is still failing. I have sent a message to the openafs-devel list [1] with details and logs. We should probably see if any help can be gained from there. If not, I'm starting to think that removing all R/O volumes, a "rm -rf /vicepa/*" on afs02 and then starting the R/O mirrors again might be an option? If we critically need the mirrors updated, we can "vos remove" the R/O volumes from any mirror and run an update just on afs01. However note that mirror-update.o.o is still in the emergency file and all cron jobs stopped. -i [1] https://lists.openafs.org/pipermail/openafs-devel/2018-May/020491.html From iwienand at redhat.com Thu May 24 13:36:41 2018 From: iwienand at redhat.com (Ian Wienand) Date: Thu, 24 May 2018 23:36:41 +1000 Subject: [OpenStack-Infra] afs02 r/o volume mirrors - ongoing incident In-Reply-To: <4fea80bd-ebe6-8311-74c3-ac6f5805f7da@redhat.com> References: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> <4fea80bd-ebe6-8311-74c3-ac6f5805f7da@redhat.com> Message-ID: <210346f9-99c7-02cc-6887-7d46553f2cda@redhat.com> On 05/24/2018 08:45 PM, Ian Wienand wrote: > On 05/24/2018 05:40 PM, Ian Wienand wrote: >> In an effort to resolve this, the afs01 & 02 servers were restarted to >> clear all old transactions, and for the affected mirrors I essentially >> removed their read-only copies and re-added them with: > > It seems this theory of removing the volumes and re-adding them is not > sufficient to get things working; "vos release" is still failing. I > have sent a message to the openafs-devel list [1] with details and > logs. Thanks to the help of Jeffrey Altman [1], we have managed to get mirror.pypi starting to resync again. This is running in the root screen on mirror-update.o.o (sorry, I forgot the "-v" on the command). For reference, you can look at the transaction and see it receiving data, e.g. root at afs02:/var/log/openafs# vos status -verbose -server localhost -localauth Total transactions: 1 -------------------------------------- transaction: 62 created: Thu May 24 12:58:23 2018 lastActiveTime: Thu May 24 12:58:23 2018 volumeStatus: volume: 536870932 partition: /vicepa procedure: Restore packetRead: 2044135 lastReceiveTime: Thu May 24 13:33:17 2018 packetSend: 1 lastSendTime: Thu May 24 13:33:17 2018 -------------------------------------- Assuming this goes OK over the next few hours, that leaves mirror.ubuntu and mirror.ubuntu-ports as the last two out-of-sync mirrors. As we do not want to run large releases in parallel, we can tackle this when pypi is back in sync. Thanks, -i [1] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-05-24.log.html#t2018-05-24T12:57:39 From iwienand at redhat.com Fri May 25 10:00:27 2018 From: iwienand at redhat.com (Ian Wienand) Date: Fri, 25 May 2018 20:00:27 +1000 Subject: [OpenStack-Infra] afs02 r/o volume mirrors - ongoing incident In-Reply-To: <210346f9-99c7-02cc-6887-7d46553f2cda@redhat.com> References: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> <4fea80bd-ebe6-8311-74c3-ac6f5805f7da@redhat.com> <210346f9-99c7-02cc-6887-7d46553f2cda@redhat.com> Message-ID: On 05/24/2018 11:36 PM, Ian Wienand wrote: > Thanks to the help of Jeffrey Altman [1], we have managed to get > mirror.pypi starting to resync again. And thanks to user error on my behalf, and identified by jeblair, in the rush of all this I ran this under k5start on mirror-update, instead of on one of the afs hosts with -localauth, so the ticket timed out and the release failed. --- root at mirror-update01:~# k5start -t -f /etc/afsadmin.keytab service/afsadmin -- vos release mirror.pypi Kerberos initialization for service/afsadmin at OPENSTACK.ORG Release failed: rxk: authentication expired Could not end transaction on a ro volume: rxk: authentication expired Could not update VLDB entry for volume 536870931 Failed to end transaction on the release clone 536870932 Could not release lock on the VLDB entry for volume 536870931 rxk: authentication expired Error in vos release command. rxk: authentication expired --- If it is any consolation, it's the type of mistake you only make once :) I am now re-running the sync in a root screen on afs02 with -localauth so it won't timeout. Expect it to finish about 20 hours from this mail :/ Thanks, -i From iwienand at redhat.com Sun May 27 03:32:09 2018 From: iwienand at redhat.com (Ian Wienand) Date: Sun, 27 May 2018 13:32:09 +1000 Subject: [OpenStack-Infra] afs02 r/o volume mirrors - resolved In-Reply-To: References: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> <4fea80bd-ebe6-8311-74c3-ac6f5805f7da@redhat.com> <210346f9-99c7-02cc-6887-7d46553f2cda@redhat.com> Message-ID: <54a46534-e6cf-7606-3d1f-4f07167d3bae@redhat.com> On 05/25/2018 08:00 PM, Ian Wienand wrote: > I am now re-running the sync in a root screen on afs02 with -localauth > so it won't timeout. I've now finished syncing back all R/O volumes on afs02, and the update cron jobs have been running successfully. Thanks, -i From mnaser at vexxhost.com Sun May 27 15:27:33 2018 From: mnaser at vexxhost.com (Mohammed Naser) Date: Sun, 27 May 2018 08:27:33 -0700 Subject: [OpenStack-Infra] afs02 r/o volume mirrors - resolved In-Reply-To: <54a46534-e6cf-7606-3d1f-4f07167d3bae@redhat.com> References: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> <4fea80bd-ebe6-8311-74c3-ac6f5805f7da@redhat.com> <210346f9-99c7-02cc-6887-7d46553f2cda@redhat.com> <54a46534-e6cf-7606-3d1f-4f07167d3bae@redhat.com> Message-ID: On Sat, May 26, 2018 at 8:32 PM, Ian Wienand wrote: > On 05/25/2018 08:00 PM, Ian Wienand wrote: >> >> I am now re-running the sync in a root screen on afs02 with -localauth >> so it won't timeout. > > > I've now finished syncing back all R/O volumes on afs02, and the update > cron jobs have been running successfully. > > Thanks, Thank you for your work in getting everything back together while most were busy at the summit! > -i > > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra From cboylan at sapwetik.org Tue May 29 17:57:09 2018 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 29 May 2018 10:57:09 -0700 Subject: [OpenStack-Infra] afs02 r/o volume mirrors - resolved In-Reply-To: References: <67d75a35-c06f-764d-297b-ec70994918f0@redhat.com> <4fea80bd-ebe6-8311-74c3-ac6f5805f7da@redhat.com> <210346f9-99c7-02cc-6887-7d46553f2cda@redhat.com> <54a46534-e6cf-7606-3d1f-4f07167d3bae@redhat.com> Message-ID: <1527616629.1086219.1389427688.726CF6AE@webmail.messagingengine.com> On Sun, May 27, 2018, at 8:27 AM, Mohammed Naser wrote: > On Sat, May 26, 2018 at 8:32 PM, Ian Wienand wrote: > > On 05/25/2018 08:00 PM, Ian Wienand wrote: > >> > >> I am now re-running the sync in a root screen on afs02 with -localauth > >> so it won't timeout. > > > > > > I've now finished syncing back all R/O volumes on afs02, and the update > > cron jobs have been running successfully. > > > > Thanks, > > Thank you for your work in getting everything back together while most > were busy at the summit! ++ From cboylan at sapwetik.org Tue May 29 21:05:44 2018 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 29 May 2018 14:05:44 -0700 Subject: [OpenStack-Infra] Vancouver Summit v2 Recap Message-ID: <1527627944.1798590.1389617336.5F06DCCB@webmail.messagingengine.com> Hello everyone, Not everyone was able to attend the summit and even for those of us that did we were not able to attend all the sessions related to Infra. I'm hoping this thread can help us all make sense of the busy week we just had and give us a general sense of the direction for the next 6 months or so of work. I'll try to give a high level overview of the topics I was involved in, feel free to ask for or add specifics if you think it will be helpful. The first day of the summit was largely Zuul's day in the spotlight. Corvus had an excellent keynote to talk about Zuul. After keynotes were a variety of forum sessions that related to Infra team work. Storyboard migrations came up. These migrations seem to be going reasonably well and we have identified some specific areas to improve like search performance and web client deployment improvements. In the machine learning for CI session we've identified that there are individuals interested in doing machine learning on the logs we produce. The big barrier there is getting the logs to a location with the appropriate hardware and tools for teaching machines what to look for. To start we've pointed people at grabbing the logs from our existing logs server to start understanding what we collect and doing testing, but we'll need to publish somewhere with a bit more hardware power (GPUs) for longer term work. Doug Hellmann proposed a plan for deprecating python2 within OpenStack. We'll need to make sure we can accommodate the needs of that deprecation process on the Infra testing platform. We should also keep in mind we ourselves will need to move off of python2 and head towards python3. Days two and three I spent all of my time in the OpenDev conference within a conference. Pleia2 did an awesome demo of deploying to "production" starting with nothing but a cloud. "Free software needs free tools" by Benjamin Mako Hill was probably my favorite talk of the entire event. Explains why the Infra team is determined to use free tools to build software. The OpenDev working sessions were great venues to interact between open source communities and other organizations. We share many of the same concerns and have to address many similar problems. A lot of time was spent feeling out a common vocabulary and constraining the problem space of CI and CD. The attendees were happy to use https://openci.io to continue these conversations longer term, hope to see you there. OpenDev also had hands on workshops. Shout out to pabelanger and mnaser who ran an excellent zuul workshop. The fourth and last day of the summit was a busy one for me. Dmsimard and I gave an Infra team onboarding session which had great attendance. After that I was on a security panel. Then we had a conversation with the Kata team to talk about how we can start running Zuul jobs against Kata. Plan there seems to be get it started on one of their lower volume repos and start running a job they can interact with. My summit largely ended with the Infra project update that I gave. The last sessions I participated in was somewhat of an extension of the Kata discussion earlier where we talked about cross community governance between OSF projects. Big outcome of this was that we need to communicate but at the same time don't be surprised if communicating is difficult. Many of us come from different places and don't have the shared OpenStack background to work off of. This email got long very fast and I feel like I've barely scratched the surface. Happy to dig in where people are interested and to hear from others on what their experiences and takeaways were. Thank you, Clark From corvus at inaugust.com Wed May 30 16:25:14 2018 From: corvus at inaugust.com (James E. Blair) Date: Wed, 30 May 2018 09:25:14 -0700 Subject: [OpenStack-Infra] Winterscale: a proposal regarding the project infrastructure Message-ID: <87o9gxdsb9.fsf@meyer.lemoncheese.net> Hi, With recent changes implemented by the OpenStack Foundation to include projects other than "OpenStack" under its umbrella, it has become clear that the "Project Infrastructure Team" needs to change. The infrastructure that is run for the OpenStack project is valued by other OpenStack Foundation projects (and beyond). Our community has not only produced an amazing cloud infrastructure system, but it has also pioneered new tools and techniques for software development and collaboration. For some time it's been apparent that we need to alter the way we run services in order to accommodate other Foundation projects. We've been talking about this informally for at least the last several months. One of the biggest sticking points has been a name for the effort. It seems very likely that we will want a new top-level domain for hosting multiple projects in a neutral environment (so that people don't have to say "hosted on OpenStack's infrastructure"). But finding such a name is difficult, and even before we do, we need to talk about it. I propose we call the overall effort "winterscale". In the best tradition of code names, it means nothing; look for no hidden meaning here. We won't use it for any actual services we provide. We'll use it to refer to the overall effort of restructuring our team and infrastructure to provide services to projects beyond OpenStack itself. And we'll stop using it when the restructuring effort is concluded. This is my first proposal: that we acknowledge this effort is underway and name it as such. My second proposal is an organizational structure for this effort. First, some goals: * The infrastructure should be collaboratively run as it is now, and the operational decisions should be made by the core reviewers as they are now. * Issues of service definition (i.e., what services we offer and how they are used) should be made via a collaborative process including the infrastructure operators and the projects which use it. To that end, I propose that we: * Work with the Foundation to create a new effort independent of the OpenStack project with the goal of operating infrastructure for the wider OpenStack Foundation community. * Work with the Foundation marketing team to help us with the branding and marketing of this effort. * Establish a "winterscale infrastructure team" (to be renamed) consisting of the current infra-core team members to operate this effort. * Move many of the git repos currently under the OpenStack project infrastructure team's governance to this new team. * Establish a "winterscale infrastructure council" (to be renamed) which will govern the services that the team provides by vote. The council will consist of the PTL of the winterscale infrastructure team and one member from each official OpenStack Foundation project. Currently, as I understand it, there's only one: OpenStack. But we expect kata, zuul, and others to be declared official in the not too distant future. The winterscale representative (the PTL) will have tiebreaking and veto power over council decisions. (This is structured loosely based on the current Infrastructure Council used by the OpenStack Project Infrastructure Team.) None of this is obviously final. My goal here is to give this effort a name and a starting point so that we can discuss it and make progress. -Jim From doug at doughellmann.com Wed May 30 16:43:50 2018 From: doug at doughellmann.com (Doug Hellmann) Date: Wed, 30 May 2018 12:43:50 -0400 Subject: [OpenStack-Infra] Winterscale: a proposal regarding the project infrastructure In-Reply-To: <87o9gxdsb9.fsf@meyer.lemoncheese.net> References: <87o9gxdsb9.fsf@meyer.lemoncheese.net> Message-ID: <1527698378-sup-1721@lrrr.local> Excerpts from corvus's message of 2018-05-30 09:25:14 -0700: > Hi, > > With recent changes implemented by the OpenStack Foundation to include > projects other than "OpenStack" under its umbrella, it has become clear > that the "Project Infrastructure Team" needs to change. > > The infrastructure that is run for the OpenStack project is valued by > other OpenStack Foundation projects (and beyond). Our community has not > only produced an amazing cloud infrastructure system, but it has also > pioneered new tools and techniques for software development and > collaboration. > > For some time it's been apparent that we need to alter the way we run > services in order to accommodate other Foundation projects. We've been > talking about this informally for at least the last several months. One > of the biggest sticking points has been a name for the effort. It seems > very likely that we will want a new top-level domain for hosting > multiple projects in a neutral environment (so that people don't have to > say "hosted on OpenStack's infrastructure"). But finding such a name is > difficult, and even before we do, we need to talk about it. > > I propose we call the overall effort "winterscale". In the best > tradition of code names, it means nothing; look for no hidden meaning > here. We won't use it for any actual services we provide. We'll use it > to refer to the overall effort of restructuring our team and > infrastructure to provide services to projects beyond OpenStack itself. > And we'll stop using it when the restructuring effort is concluded. > > This is my first proposal: that we acknowledge this effort is underway > and name it as such. > > My second proposal is an organizational structure for this effort. > First, some goals: > > * The infrastructure should be collaboratively run as it is now, and > the operational decisions should be made by the core reviewers as > they are now. > > * Issues of service definition (i.e., what services we offer and how > they are used) should be made via a collaborative process including > the infrastructure operators and the projects which use it. > > To that end, I propose that we: > > * Work with the Foundation to create a new effort independent of the > OpenStack project with the goal of operating infrastructure for the > wider OpenStack Foundation community. > > * Work with the Foundation marketing team to help us with the branding > and marketing of this effort. > > * Establish a "winterscale infrastructure team" (to be renamed) > consisting of the current infra-core team members to operate this > effort. > > * Move many of the git repos currently under the OpenStack project > infrastructure team's governance to this new team. I'm curious about the "many" in that sentence. Which do you anticipate not moving, and if this new team replaces the existing team then who would end up owning the ones that do not move? > > * Establish a "winterscale infrastructure council" (to be renamed) which > will govern the services that the team provides by vote. The council > will consist of the PTL of the winterscale infrastructure team and one > member from each official OpenStack Foundation project. Currently, as > I understand it, there's only one: OpenStack. But we expect kata, > zuul, and others to be declared official in the not too distant > future. The winterscale representative (the PTL) will have > tiebreaking and veto power over council decisions. That structure seems sound, although it means the council is going to be rather small (at least in the near term). What sorts of decisions do you anticipate needing to be addressed by this council? > > (This is structured loosely based on the current Infrastructure > Council used by the OpenStack Project Infrastructure Team.) > > None of this is obviously final. My goal here is to give this effort a > name and a starting point so that we can discuss it and make progress. > > -Jim > Thanks for starting this thread! I've replied to both mailing lists because I wasn't sure which was more appropriate. Please let me know if I should focus future replies on one list. Doug From corvus at inaugust.com Wed May 30 17:09:23 2018 From: corvus at inaugust.com (James E. Blair) Date: Wed, 30 May 2018 10:09:23 -0700 Subject: [OpenStack-Infra] Winterscale: a proposal regarding the project infrastructure In-Reply-To: <1527698378-sup-1721@lrrr.local> (Doug Hellmann's message of "Wed, 30 May 2018 12:43:50 -0400") References: <87o9gxdsb9.fsf@meyer.lemoncheese.net> <1527698378-sup-1721@lrrr.local> Message-ID: <87wovlcbp8.fsf@meyer.lemoncheese.net> Doug Hellmann writes: >> * Move many of the git repos currently under the OpenStack project >> infrastructure team's governance to this new team. > > I'm curious about the "many" in that sentence. Which do you anticipate > not moving, and if this new team replaces the existing team then who > would end up owning the ones that do not move? There are a lot. Generally speaking, I think most of the custom software, deployment tooling, and configuration would move. An example of something that probably shouldn't move is "openstack-zuul-jobs". We still need people that are concerned with how OpenStack uses the winterscale service. I'm not sure whether that should be its own team or should those functions get folded into other teams. >> * Establish a "winterscale infrastructure council" (to be renamed) which >> will govern the services that the team provides by vote. The council >> will consist of the PTL of the winterscale infrastructure team and one >> member from each official OpenStack Foundation project. Currently, as >> I understand it, there's only one: OpenStack. But we expect kata, >> zuul, and others to be declared official in the not too distant >> future. The winterscale representative (the PTL) will have >> tiebreaking and veto power over council decisions. > > That structure seems sound, although it means the council is going > to be rather small (at least in the near term). What sorts of > decisions do you anticipate needing to be addressed by this council? Yes, very small. Perhaps we need an interim structure until it gets larger? Or perhaps just discipline and agreement that the two people on it will consult with the necessary constituencies and represent them well? I expect the council not to have to vote very often. Perhaps only on substantial changes to services (bringing a new offering online, retiring a disused offering, establishing parameters of a service). As an example, the recent thread on "terms of service" would be a good topic for the council to settle. >> (This is structured loosely based on the current Infrastructure >> Council used by the OpenStack Project Infrastructure Team.) >> >> None of this is obviously final. My goal here is to give this effort a >> name and a starting point so that we can discuss it and make progress. >> >> -Jim >> > > Thanks for starting this thread! I've replied to both mailing lists > because I wasn't sure which was more appropriate. Please let me > know if I should focus future replies on one list. Indeed, perhaps we should steer this toward openstack-dev now. I'll drop openstack-infra from future replies. -Jim From Remo at italy1.com Wed May 30 21:25:34 2018 From: Remo at italy1.com (Remo Mattei) Date: Wed, 30 May 2018 14:25:34 -0700 Subject: [OpenStack-Infra] Puppet modules Message-ID: Hello all, I have talked to several people about this and I would love to get this finalized once and for all. I have checked the OpenStack puppet modules which are mostly developed by the Red Hat team, as of right now, TripleO is using a combo of Ansible and puppet to deploy but in the next couple of releases, the plan is to move away from the puppet option. So consequently, what will be the plan of TripleO and the puppet modules? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamamoto at midokura.com Thu May 31 05:52:52 2018 From: yamamoto at midokura.com (Takashi Yamamoto) Date: Thu, 31 May 2018 14:52:52 +0900 Subject: [OpenStack-Infra] How to take over a project? (tap-as-a-service) Message-ID: hi, i want to take over tap-as-a-service project. originally the project had two leaders. but one of them (vinay) has not been responding these days. another (anil) recently told us he might not work on it anymore. this email has cc: to them. i think gerrit is ok as i'm already a member of tap-as-a-service-core group. however, for launchpad, i'm not sure who can change the permissions. can the infra team help? the LP project is this one: https://launchpad.net/tap-as-a-service thank you. From thierry at openstack.org Thu May 31 08:07:26 2018 From: thierry at openstack.org (Thierry Carrez) Date: Thu, 31 May 2018 10:07:26 +0200 Subject: [OpenStack-Infra] How to take over a project? (tap-as-a-service) In-Reply-To: References: Message-ID: <4f4046b9-ca01-a420-278a-abee04d364a7@openstack.org> Takashi Yamamoto wrote: > i want to take over tap-as-a-service project. > > originally the project had two leaders. but one of them (vinay) has not been > responding these days. another (anil) recently told us he might not work > on it anymore. this email has cc: to them. > > i think gerrit is ok as i'm already a member of tap-as-a-service-core group. > > however, for launchpad, i'm not sure who can change the permissions. > can the infra team help? > the LP project is this one: https://launchpad.net/tap-as-a-service The infra team can't help much with that. The Launchpad project is owned by Anil Rao (although Vinay Yadhav might be able to change that as the original registrant), and the OpenStack Administrators was never added as a co-owner. Anil: any chance you could update the maintainer ? Thanks in advance, -- Thierry Carrez (ttx) From sorin.sbarnea at gmail.com Wed May 23 10:17:33 2018 From: sorin.sbarnea at gmail.com (=?utf-8?B?U29yaW4gSW9udcibIFNiw6JybmVh?=) Date: Wed, 23 May 2018 11:17:33 +0100 Subject: [OpenStack-Infra] Updating gerrit jenkins-job-builder-release group members Message-ID: <528CBCE5-E8F2-40ED-9143-78213F4340F8@gmail.com> Hi Can someone help updating the jenkins-job-builder-release group on gerrit to add two persons to it? Apparently there is no way to raise a CR to update group membership, or if it is I am unaware about it. https://review.openstack.org/#/admin/groups/321,members Sorin Sbarnea sorin.sbarnea at gmail.com Darragh Bailey daragh.bailey at gmail.com We are both already members of python-jenkins-release but not this one and it would be very useful to be able to release much easier. This change would bring the two gerrit groups in-sync, which makes sense to me. Thanks Sorin Sbarnea -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From zxiiro at gmail.com Wed May 23 20:57:20 2018 From: zxiiro at gmail.com (Thanh Ha) Date: Wed, 23 May 2018 13:57:20 -0700 Subject: [OpenStack-Infra] Updating gerrit jenkins-job-builder-release group members In-Reply-To: <528CBCE5-E8F2-40ED-9143-78213F4340F8@gmail.com> References: <528CBCE5-E8F2-40ED-9143-78213F4340F8@gmail.com> Message-ID: On 23 May 2018 at 03:17, Sorin Ionuț Sbârnea wrote: > Hi > > Can someone help updating the *jenkins-job-builder-release* group on > gerrit to add two persons to it? Apparently there is no way to raise a CR > to update group membership, or if it is I am unaware about it. > https://review.openstack.org/#/admin/groups/321,members > > Sorin Sbarnea sorin.sbarnea at gmail.com > Darragh Bailey daragh.bailey at gmail.com > > We are both already members of python-jenkins-release > but not this > one and it would be very useful to be able to release much easier. > > This change would bring the two gerrit groups in-sync, which makes sense > to me. > > Thanks > Sorin Sbarnea > I talked to Clark and we got this resolved. The both of you should have the ability to push tags now. Remember to be careful about what tags you push as it's policy that tags never get deleted. Regards, Thanh -------------- next part -------------- An HTML attachment was scrubbed... URL: From arao012 at gmail.com Thu May 31 14:12:28 2018 From: arao012 at gmail.com (Anil Rao) Date: Thu, 31 May 2018 07:12:28 -0700 Subject: [OpenStack-Infra] How to take over a project? (tap-as-a-service) In-Reply-To: <4f4046b9-ca01-a420-278a-abee04d364a7@openstack.org> References: <4f4046b9-ca01-a420-278a-abee04d364a7@openstack.org> Message-ID: <3CEC22DB-1343-4419-819E-220A5BE51626@gmail.com> Hi, Sure. I’ll give it a try and get back to you guys. Regards, Anil Sent from my iPhone > On May 31, 2018, at 1:07 AM, Thierry Carrez wrote: > > Delivered-To: arao012 at gmail.com > Received: by 2002:a19:e21a:0:0:0:0:0 with SMTP id z26-v6csp5901938lfg; Thu, 31 > May 2018 01:07:30 -0700 (PDT) > X-Google-Smtp-Source: ADUXVKKQOqQkg/OK3B0quxkdCcVZjBMNtHZCom1G9aIxUS5xa+BuOM817LVG2bYdhSq7h3dXhX3a > X-Received: by 2002:a0c:8b4b:: with SMTP id > d11-v6mr1529025qvc.241.1527754050657; Thu, 31 May 2018 01:07:30 -0700 (PDT) > ARC-Seal: i=1; a=rsa-sha256; t=1527754050; cv=none; d=google.com; > s=arc-20160816; > b=xO4W1y0PjqXxJDsVA80BzHl4UZllHdNeJEwEJlkCu7sCvO/fTQ/en0OfeUO5NxwMF1 > D7yOPvc3rGwfWtvg80lvrpDmweigmLhXwMbHH0J0wT3kjIkwbRwj6tH6nLYfeWgLvKDy > HksPO8upTUWDrneEuZjZ9YDLSBTiPcxu4bo4nyFEXmxQ+wyuNXPykB+teatCGGQNdh2K > 1A346RyPgHiWIlbYlI/9dd4wuLBpVUqcORtRDqJCQWwFFd6eY8RkE4/9NEFjpCkxNydy > yzzV1sONBzi4Gsp/e1XRI9O8P3Z1MR52SQZevGLpov2T4DKr+e+raJCT9oFdXBZem1k9 7aYA== > ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; > s=arc-20160816; > h=content-transfer-encoding:content-language:in-reply-to:mime-version > :user-agent:date:message-id:organization:from:references:cc:to > :subject:arc-authentication-results; > bh=/buaa2NmAYjG3+0fppj3fIUX8lWhDsIpVeBp7YZBZtw=; > b=ZMeGxd7gYz1o/BmdrMmryG18F0yAT98ZhXtRFqDkhmeLtBnmCpaAqIeYVbCBJd40c7 > FA2sYWm/3SYrbAWlY9XRGxIPuhbXzMV9JOgwrEL3U2Kd1Y/Vp7eqC6Qlzp4z+jE0+M0C > ePvp3eB4q8pMTgdvVt/vjHh8gCdxa7AYRH1MRh5dVrNP0gCrG23/SIYHnX9XDWE7I5pY > bNszkCu/ymZGIIXHS2LDBt18vLZvJ0sSfIoeg0OYISGn0cV/W+x1jSB8ESEmxK2p2sOY > bePq2U4tFZChMlLQSxBxh2ZHMGob+DB2IwfYrT89Nx4FNAJnNN/EqTK9fs9BCVARJkS0 yW4Q== > ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain > of thierry at openstack.org designates 173.203.187.79 as permitted sender) > smtp.mailfrom=thierry at openstack.org > Return-Path: > Received: from smtp79.iad3a.emailsrvr.com (smtp79.iad3a.emailsrvr.com. > [173.203.187.79]) by mx.google.com with ESMTPS id > r5-v6si360921qvc.200.2018.05.31.01.07.28 for > (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 May > 2018 01:07:29 -0700 (PDT) > Received-SPF: pass (google.com: domain of thierry at openstack.org designates > 173.203.187.79 as permitted sender) client-ip=173.203.187.79; > Authentication-Results: mx.google.com; spf=pass (google.com: domain of > thierry at openstack.org designates 173.203.187.79 as permitted sender) > smtp.mailfrom=thierry at openstack.org > Received: from smtp34.relay.iad3a.emailsrvr.com (localhost [127.0.0.1]) by > smtp34.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 6367924BC5; Thu, > 31 May 2018 04:07:27 -0400 (EDT) > X-Auth-ID: thierry.carrez at openstack.org > Received: by smtp34.relay.iad3a.emailsrvr.com (Authenticated sender: > thierry.carrez-AT-openstack.org) with ESMTPSA id D22C824BC6; Thu, 31 May 2018 > 04:07:26 -0400 (EDT) > X-Sender-Id: thierry.carrez at openstack.org > Received: from [192.168.0.113] (LFbn-1-13111-9.w109-209.abo.wanadoo.fr > [109.209.34.9]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA) by 0.0.0.0:465 > (trex/5.7.12); Thu, 31 May 2018 04:07:27 -0400 > Subject: Re: [OpenStack-Infra] How to take over a project? (tap-as-a-service) > To: openstack-infra at lists.openstack.org > Cc: Takashi Yamamoto , Anil Rao , > Vinay Yadhav > References: > From: Thierry Carrez > Organization: OpenStack > Message-ID: <4f4046b9-ca01-a420-278a-abee04d364a7 at openstack.org> > Date: Thu, 31 May 2018 10:07:26 +0200 > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 > Thunderbird/52.8.0 > MIME-Version: 1.0 > In-Reply-To: > Content-Type: text/plain; charset=utf-8; format=flowed > Content-Language: en-US > Content-Transfer-Encoding: 7bit > > Takashi Yamamoto wrote: >> i want to take over tap-as-a-service project. >> originally the project had two leaders. but one of them (vinay) has not been >> responding these days. another (anil) recently told us he might not work >> on it anymore. this email has cc: to them. >> i think gerrit is ok as i'm already a member of tap-as-a-service-core group. >> however, for launchpad, i'm not sure who can change the permissions. >> can the infra team help? >> the LP project is this one: https://launchpad.net/tap-as-a-service > > The infra team can't help much with that. The Launchpad project is owned by Anil Rao (although Vinay Yadhav might be able to change that as the original registrant), and the OpenStack Administrators was never added as a co-owner. > > Anil: any chance you could update the maintainer ? > Thanks in advance, > > -- > Thierry Carrez (ttx) > Takashi Yamamoto wrote: >> i want to take over tap-as-a-service project. >> originally the project had two leaders. but one of them (vinay) has not been >> responding these days. another (anil) recently told us he might not work >> on it anymore. this email has cc: to them. >> i think gerrit is ok as i'm already a member of tap-as-a-service-core group. >> however, for launchpad, i'm not sure who can change the permissions. >> can the infra team help? >> the LP project is this one: https://launchpad.net/tap-as-a-service > > The infra team can't help much with that. The Launchpad project is owned by Anil Rao (although Vinay Yadhav might be able to change that as the original registrant), and the OpenStack Administrators was never added as a co-owner. > > Anil: any chance you could update the maintainer ? > Thanks in advance, > > -- > Thierry Carrez (ttx)