From cboylan at sapwetik.org Wed May 1 17:12:19 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Wed, 01 May 2019 13:12:19 -0400 Subject: [OpenStack-Infra] =?utf-8?q?Move_packstack_from_openstack_to_redh?= =?utf-8?q?at-openstack=09organization_in_github?= In-Reply-To: References: Message-ID: <4dc7c8bc-f346-4bd4-be41-981c31cfc9ca@www.fastmail.com> On Tue, Apr 30, 2019, at 7:51 AM, Alfredo Moralejo Alonso wrote: > Hi, > > After migration to opendev an moving openstack/packstack to > x/packstack, mirroring to github has been lost as announced in [1]. > > Could we get the repository moved to redhat-openstack organization in > github?, we'd like to get redirection from old repo to the new one > what's require to transfer the repo using github iiuc. > > Best regards, > > Alfredo > > [1] > http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005629.html > Thank you for sending this out so that we have a record of why the move was made. I have no objections to making this move in GitHub. If dmsimard still has time to do the move I think we can go ahead with that. Otherwise, we'll likely get to it once we return to our daily lives after the summit and PTG. Clark From fungi at yuggoth.org Thu May 2 04:51:06 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 2 May 2019 04:51:06 +0000 Subject: [OpenStack-Infra] Train PTG for OpenDev/Infra in Denver Message-ID: <20190502045106.gs7nifdslynffkqk@yuggoth.org> For those joining the OpenDev/Infra team at the Train PTG in Denver this week, don't forget we start tomorrow (Thursday) morning at 9:00 AM MDT in room 105 of the Colorado Convention Center. If you haven't checked in to get your badge yet, there should be folks at the PTG registration desk just past the main entrance lobby to help you out as early as 8:00 AM. Quick links to important PTG information can be found at http://ptg.openstack.org/ should you need it. See you soon! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Thu May 2 16:52:44 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 2 May 2019 16:52:44 +0000 Subject: [OpenStack-Infra] Train PTG for OpenDev/Infra in Denver In-Reply-To: <20190502045106.gs7nifdslynffkqk@yuggoth.org> References: <20190502045106.gs7nifdslynffkqk@yuggoth.org> Message-ID: <20190502165243.jnliyszavbxawvd4@yuggoth.org> On 2019-05-02 04:51:06 +0000 (+0000), Jeremy Stanley wrote: > For those joining the OpenDev/Infra team at the Train PTG in Denver > this week, don't forget we start tomorrow (Thursday) morning at 9:00 > AM MDT in room 105 of the Colorado Convention Center. [...] Just to update for those arriving late, we've temporarily relocated to room 106 so as not to compete with QA team discussions. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From ryan.beisner at canonical.com Fri May 3 16:32:56 2019 From: ryan.beisner at canonical.com (Ryan Beisner) Date: Fri, 3 May 2019 10:32:56 -0600 Subject: [OpenStack-Infra] Post opendev git repo hosting status repo In-Reply-To: <0ac69311-1f9b-4265-96f5-06b6bb8adff7@www.fastmail.com> References: <8ad20ab9-96c8-41c6-888a-ae67402f8470@www.fastmail.com> <0ac69311-1f9b-4265-96f5-06b6bb8adff7@www.fastmail.com> Message-ID: Hi All, First: stellar job in this migration! It has been amazingly smooth, especially considering the amount of Earth shifting that just happened. I'm curious when the new repo/project freeze is anticipated to be lifted? Thanks, Ryan On Sat, Apr 20, 2019 at 3:26 PM Clark Boylan wrote: > On Fri, Apr 19, 2019, at 9:18 PM, Clark Boylan wrote: > > Hello infra! > > > > We've completed the initial work to do this migration. YAY. But there > > are a few things to keep in mind while we clean up after ourselves. > > > > This was a bit of an inception moment where we rely on Gerrit to drive > > our infrastructure but we renamed and moved Gerrit. This means that the > > ansible crons on bridge.o.o are disabled. Please DO NOT reenable the > > run_all.sh cron until we are ready. > > > > We should also avoid creating new projects until we are happy with > > things and have run_all.sh running again. config-core please DO NOT > > merge new project additions until we are ready. > > > > In general though bug fixes to zuul jobs should be fine (and I expect > > we'll have to start here in order to merge other code). And we'll fix > > more bugs from there. Thanks again to everyone that helped and for your > > patience otherwise. This was the work of many people. > > > > Also I'll respond to this when we think we are ready to go on the > > things I asked us to leave alone for now. > > We have reenabled the run_all.sh cron on bridge.openstack.org after > iterating through each playbook it runs and examining the results. Things > looked happy so we are back to operating with normal deployment procedures. > > That said we would like to keep a freeze on new project creations for a > little while longer. We found some bugs around creating and renaming > projects particularly if the new target org (or namespace) does not already > exist. This means config-core and infra-root please avoid creating new > projects until we are ready. I will followup to this message with a note > when we have reached that point. > > Thank you, > Clark > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Fri May 3 16:42:53 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Fri, 03 May 2019 12:42:53 -0400 Subject: [OpenStack-Infra] Post opendev git repo hosting status repo In-Reply-To: References: <8ad20ab9-96c8-41c6-888a-ae67402f8470@www.fastmail.com> <0ac69311-1f9b-4265-96f5-06b6bb8adff7@www.fastmail.com> Message-ID: <90895cec-7c1f-4c97-9725-f4e605fdc927@www.fastmail.com> On Fri, May 3, 2019, at 9:33 AM, Ryan Beisner wrote: > Hi All, > > First: stellar job in this migration! It has been amazingly smooth, > especially considering the amount of Earth shifting that just happened. > > I'm curious when the new repo/project freeze is anticipated to be lifted? > We've just discussed this now at the PTG to do a quick sanity check and we are happy to start creating new repos again in existing namespaces. We have a bit of work to do so that we can scale up the management of new namespaces. Details are on the PTG etherpad [0]. [0] https://etherpad.openstack.org/p/2019-denver-ptg-infra-planning Hope this helps, Clark From cboylan at sapwetik.org Mon May 6 21:13:05 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 06 May 2019 17:13:05 -0400 Subject: [OpenStack-Infra] Infra Meeting Agenda for May 7, 2019 Message-ID: <218c9afa-9c35-4352-a65e-c57af2ee2a5e@www.fastmail.com> == Agenda for next meeting == * Announcements ** Clarkb AFK off and on this week * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** PTG Recap (clarkb 20190507) *** https://etherpad.openstack.org/p/2019-denver-ptg-infra-planning ** LetsEncrypt Certs for mirror nodes (clarkb 20190507) ** Trusty Upgrade Progress (clarkb 20190507) * Open discussion From dmsimard at redhat.com Tue May 7 14:13:25 2019 From: dmsimard at redhat.com (David Moreau Simard) Date: Tue, 7 May 2019 10:13:25 -0400 Subject: [OpenStack-Infra] Move packstack from openstack to redhat-openstack organization in github In-Reply-To: <4dc7c8bc-f346-4bd4-be41-981c31cfc9ca@www.fastmail.com> References: <4dc7c8bc-f346-4bd4-be41-981c31cfc9ca@www.fastmail.com> Message-ID: Hi, github.com/openstack/packstack was moved successfully to github.com/redhat-openstack/packstack. Note that the replication needs to be set up to keep the repository up to date. This can be done by inheriting from the upload-git-mirror [1] job. [1]: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html David Moreau Simard dmsimard = [irc, github, twitter] On Wed, May 1, 2019 at 1:17 PM Clark Boylan wrote: > > On Tue, Apr 30, 2019, at 7:51 AM, Alfredo Moralejo Alonso wrote: > > Hi, > > > > After migration to opendev an moving openstack/packstack to > > x/packstack, mirroring to github has been lost as announced in [1]. > > > > Could we get the repository moved to redhat-openstack organization in > > github?, we'd like to get redirection from old repo to the new one > > what's require to transfer the repo using github iiuc. > > > > Best regards, > > > > Alfredo > > > > [1] > > http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005629.html > > > > Thank you for sending this out so that we have a record of why the move was made. I have no objections to making this move in GitHub. If dmsimard still has time to do the move I think we can go ahead with that. Otherwise, we'll likely get to it once we return to our daily lives after the summit and PTG. > > Clark > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra From alifshit at redhat.com Tue May 7 17:47:01 2019 From: alifshit at redhat.com (Artom Lifshitz) Date: Tue, 7 May 2019 13:47:01 -0400 Subject: [OpenStack-Infra] [nova][CI] GPUs in the gate Message-ID: Hey all, Following up on the CI session during the PTG [1], I wanted to get the ball rolling on getting GPU hardware into the gate somehow. Initially the plan was to do it through OpenLab and by convincing NVIDIA to donate the cards, but after a conversation with Sean McGinnis it appears Infra have access to machines with GPUs. >From Nova's POV, the requirements are: * The machines with GPUs should probably be Ironic baremetal nodes and not VMs [*]. * The GPUs need to support virtualization. It's hard to get a comprehensive list of GPUs that do, but Nova's own docs [2] mention two: Intel cards with GVT [3] and NVIDIA GRID [4]. So I think at this point the question is whether Infra can support those reqs. If yes, we can start concrete steps towards getting those machines used by a CI job. If not, we'll fall back to OpenLab and try to get them hardware. [*] Could we do double-passthrough? Could the card be passed through to the L1 guest via the PCI passthrough mechanism, and then into the L2 guest via the mdev mechanism? [1] https://etherpad.openstack.org/p/nova-ptg-train-ci [2] https://docs.openstack.org/nova/rocky/admin/virtual-gpu.html [3] https://01.org/igvt-g [4] https://docs.nvidia.com/grid/5.0/pdf/grid-vgpu-user-guide.pdf From ljia at redhat.com Thu May 2 17:32:30 2019 From: ljia at redhat.com (Leo Jia) Date: Thu, 2 May 2019 13:32:30 -0400 Subject: [OpenStack-Infra] Opendev/base-jobs questions Message-ID: Hi Openstack-infra team, May I have a question for the zuul-jobs in this repo: https://opendev.org/opendev/base-jobs. There is a Zuul job that uploads the image to docker hub by running this Ansible playbook base-jobs/playbooks/docker-image/upload.yaml. Inside the playbook, it uses an Ansible role called upload-docker-image, but I can't find this role in this repo or elsewhere. How does this playbook know where is the upload-docker-image role is and where can I find the detail implementation of this role? Because I would like to create a similar role that uploads the image to quay.io. Thanks, Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From shrews at redhat.com Wed May 8 15:43:26 2019 From: shrews at redhat.com (David Shrewsbury) Date: Wed, 8 May 2019 11:43:26 -0400 Subject: [OpenStack-Infra] Opendev/base-jobs questions In-Reply-To: References: Message-ID: Hi, On Wed, May 8, 2019 at 11:09 AM Leo Jia wrote: > Hi Openstack-infra team, > > May I have a question for the zuul-jobs in this repo: > https://opendev.org/opendev/base-jobs. There is a Zuul job that uploads > the image to docker hub by running this Ansible playbook > base-jobs/playbooks/docker-image/upload.yaml. Inside the playbook, it uses > an Ansible role called upload-docker-image, but I can't find this role in > this repo or elsewhere. How does this playbook know where is the > upload-docker-image role is and where can I find the detail implementation > of this role? Because I would like to create a similar role that uploads > the image to quay.io. > > The upload-docker-image role can be found in the zuul/zuul-jobs repo: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-docker-image That role is made available in the base job definition, specifically here: https://opendev.org/opendev/base-jobs/src/branch/master/zuul.yaml#L144 You can read more about the job.roles configuration directive here: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.roles -David Thanks, > Leo > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce.e.jones at intel.com Thu May 9 17:08:08 2019 From: bruce.e.jones at intel.com (Jones, Bruce E) Date: Thu, 9 May 2019 17:08:08 +0000 Subject: [OpenStack-Infra] Question: How to attach large files to LP bugs? Message-ID: <9A85D2917C58154C960D95352B22818BD072FCFC@fmsmsx123.amr.corp.intel.com> Greetings Infra team. The StarlingX project has a question - we have had some bug reports where the log files needed to debug the bug turn out to be quite large. Larger than can be attached to a LP entry. Is there a recommended way to handle this case? Thank you! brucej -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Mon May 13 16:48:53 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 13 May 2019 12:48:53 -0400 Subject: [OpenStack-Infra] Question: How to attach large files to LP bugs? In-Reply-To: <9A85D2917C58154C960D95352B22818BD072FCFC@fmsmsx123.amr.corp.intel.com> References: <9A85D2917C58154C960D95352B22818BD072FCFC@fmsmsx123.amr.corp.intel.com> Message-ID: On Fri, May 10, 2019, at 6:47 AM, Jones, Bruce E wrote: > > Greetings Infra team. The StarlingX project has a question – we have > had some bug reports where the log files needed to debug the bug turn > out to be quite large. Larger than can be attached to a LP entry. > > > Is there a recommended way to handle this case? I'm not even sure what the launchpad attachment size limit is, so I think this is the first time I have had to consider this problem. One option if possible would be to use an efficient compression algorithm like xz to compress the logs down to a size that LP will accept the upload. If that still doesn't get things small enough maybe compress multiple chunks and upload them separately? If we can't come up with a workaround like the above options, then the next best option is likely to host the content somewhere else and link LP to that location. Clark From cboylan at sapwetik.org Mon May 13 17:49:44 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 13 May 2019 13:49:44 -0400 Subject: [OpenStack-Infra] 2019 Denver Summit/Forum/PTG Recap Message-ID: Apologies for not getting this out earlier, but last week was a very odd week for. Slowly returning to normal now though. The long week started with a board meeting. In that board meeting Zuul was confirmed as a top level OpenStack Foundation Project. An exciting start to the week for our friends in Zuul land (which happens to be many of us too)! The first day of the summit we had Keynotes where corvus talked about the way we speculatively test changes with containers. ttx had a talk after keynotes on "Open Source is not enough" [0] which helps to point out why the infrastructure that we run is valuable. Later that day we had a Forum session on Storyboard painpoints. There was good feedback and we got a couple volunteers to help fix existing issues [1]. Tuesday started with a back to back Zuul BoF then Working Group session (in the same room) that ended up being a good discussion among current and potential Zuul users. We discussed how people were using Zuul and hangups upgrading from v2 to v3. There was a forum discussion on PTL tips and tricks [2] which I found interesting (as current PTL). In particular is seems that a number of projects use regular update emails (like Zuul) to keep people up to date on goings on rather than the weekly meeting. We recently discussed this as a team so I don't think we need to rehash it but was good to see other approaches. This day ended with a session on improving the Help Most Wanted list [3] which we are still on. General consensus seemed to be that the first pass at writing business cases addressed the wrong audience so a second pass would be made. In addition to that Allison Randall would reach out to institutions like OSUOSL to see if there is possibility to collaboration there. The last day of the summit included a forum session on OpenLab [4]. This was useful to help communicate where OpenDev can serve its users and where OpenLab is able to help too. At this point I think I was a zombie and we finished up the Summit with back to back Infra and Zuul project updates. But wait there is more! We had an Infra PTG room the next two days. Notes were taken at this etherpad [5] under the bolded headings. We kicked things off with a rundown of recent tech changes so that everyone had a rough idea of what those looked like. Jhesketh showed us that docker-compose can do things like get logs for your composed docker containers :) We discussed what the future of our deployment tooling looks like using Zuul as a CD engine and how we transition to that point. The current plan is that we'll split up the base.yaml playbook into other playbooks (but continue to have cron drive them). Then we can have Zuul take over the execution of each playbook over time (and possibly even use Zuul's periodic pipeline to do convergence if things are skipped for some reason). Initial work on that has started here [6]. We also talked about what new repo and namespace creation looks like in OpenDev. We ended up with a plan that roughly comes down to individual namespaces will be self managed as much as possible. This means that the Infra team won't be approving what goes into openstack/ or airship/ or zuul/ instead appropriate representatives from those groups will make those approvals. To make this happen we'll likely end up with a metadata repo per namespaces that tracks who can approve these things then rely on some combo of Gerrit config and Zuul jobs to make that happen. We also discussed how the namespaces might manage info needed to create valid zuul main.yaml then we can have tooling aggregate that together and deploy it. This way we don't have to manage each of those additions for zuul directly. Finally we discussed if OpenDev would be an appropriate location for throwaway repos and unfortunately we don't think we are there yet. Gerrit currently won't scale to that use case for us. Last major item we poked at was improving the reliability of our docker based jobs, particularly those that run on ipv6 clouds. The plan there is to use the in region mirror nodes for dockerhub proxying rather than local buildset registry proxies as the in region mirror has a floating IP to do 1:1 ipv4 NAT but the test nodes are many:1 ipv4 NAT. We expect this will improve reliability significantly. Before we can do that though we need to enable TLS on those mirror nodes as we'll be publishing the resulting artifacts. We can do this via letsencrypting new opendev.org mirror endpoints. The StarlingX folks came over to talk to us about their testing needs. They would like to do larger test scenarios on multiple nodes and ideally on baremetal. We suggested that they start by scaling up the testing on VMs (as many issues fall out when you go from one to two to three nodes) and getting that tested even on VMs would be valuable. For the baremetal problem we pointed out that we can support baremetal testing, we just need someone to give us access to baremetal test nodes. They were going to look into how feasible giving opendev access to some baremetal test nodes is from their side. The last day of the PTG I ended up floating between rooms and helping people with various issues. Restored a nova etherpad and dug into multinode test networking with the octavia team (for ipv6 testing). [0] https://www.openstack.org/videos/summits/denver-2019/open-source-is-not-enough [1] https://etherpad.openstack.org/p/storyboard-pain-points [2] https://etherpad.openstack.org/p/DEN-ptl-tips-and-tricks [3] https://etherpad.openstack.org/p/Den-forum-help-most-needed [4] https://etherpad.openstack.org/p/DEN-openlab-whats-next [5] https://etherpad.openstack.org/p/2019-denver-ptg-infra-planning [6] https://review.opendev.org/#/c/656871/ I hope others find this useful. Feel free to ask questions or for clarification on any of this. Clark From fungi at yuggoth.org Mon May 13 18:14:15 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 13 May 2019 18:14:15 +0000 Subject: [OpenStack-Infra] Question: How to attach large files to LP bugs? In-Reply-To: References: <9A85D2917C58154C960D95352B22818BD072FCFC@fmsmsx123.amr.corp.intel.com> Message-ID: <20190513181415.6tdxjo6ee3v7imy7@yuggoth.org> On 2019-05-13 12:48:53 -0400 (-0400), Clark Boylan wrote: [...] > compress multiple chunks and upload them separately? [...] I would go the other direction: compress as thoroughly as possible and then use the `split` utility specifying whatever the upload limit for LP is (I too have no idea what they limit attachment sizes to). Reassembly can be done with `cat` to stuff the chunks back together and then decompress the resulting file as usual. This is starting to remind me of splitting uuencoded data to overcome posting length limits on Usenet. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Mon May 13 22:30:51 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 13 May 2019 18:30:51 -0400 Subject: [OpenStack-Infra] Infra Meeting Agenda for May 14, 2019 Message-ID: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190507) ** https mirror update (ianw 20190514) * Open discussion From graham.whaley at intel.com Mon May 13 15:04:55 2019 From: graham.whaley at intel.com (Whaley, Graham) Date: Mon, 13 May 2019 15:04:55 +0000 Subject: [OpenStack-Infra] Feedback on git/repos on opendev.org Message-ID: <01D2D3EEA433C0419A12152B3A36557E2059D9EE@IRSMSX101.ger.corp.intel.com> Hi, (please reply-all if you want me to see it - I am not subscribed to this list) I was heading to read the code for the kata zuul tenant, and had some niggles when trying to locate/navigate. I thought I'd at least drop the feedback here (thanks ttx). Some of it will be specific to my 'flow'.... Sooo, as I don't store the path to the repos in my head, I normally start out by going to git.openstack.org: That used to get me to (iirc) the top level cgit type interface, where I could then navigate down to the kata repos. It now takes me to the top level of opendev.org, but not to the git area, and it is not immediately obvious to me where the git area is. I found the git area under 'Explore' :-). The 'search' is a bit odd. It took me a moment to work out that it is alphabetical, if you ignore all the subdir prefix's (project names?). I'm not sure that is the most useful sort order tbh? Confused me for some time. And then, if you search for 'kata' in the default search box, you get **no matching repos**. Which, also, is not quite what I'd expect. Eventually I found the kata repos by sorting reverse-alphabetically and scrolling down :-) I know, the search is on 'repos', and not 'organisations' - but, if you just land on that page and search for something that happens to only have its name active at the project level (like kata), then you end up with no results. Maybe the search/results can be smarter, and provide the list of what it was searching for first (so in this case 'no repositories found'), and then some 'suggestions', like 'you may have been looking for organisation: kata'. Just thinking out loud. Thanks, Graham --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. From smooney at redhat.com Mon May 13 18:01:47 2019 From: smooney at redhat.com (Sean Mooney) Date: Mon, 13 May 2019 19:01:47 +0100 Subject: [OpenStack-Infra] [nova][CI] GPUs in the gate In-Reply-To: References: Message-ID: <17dbb752e66f914087f19d65b03fd83c747c5a35.camel@redhat.com> On Tue, 2019-05-07 at 13:47 -0400, Artom Lifshitz wrote: > Hey all, > > Following up on the CI session during the PTG [1], I wanted to get the > ball rolling on getting GPU hardware into the gate somehow. Initially > the plan was to do it through OpenLab and by convincing NVIDIA to > donate the cards, but after a conversation with Sean McGinnis it > appears Infra have access to machines with GPUs. > > From Nova's POV, the requirements are: > * The machines with GPUs should probably be Ironic baremetal nodes and > not VMs [*]. > * The GPUs need to support virtualization. It's hard to get a > comprehensive list of GPUs that do, but Nova's own docs [2] mention > two: Intel cards with GVT [3] and NVIDIA GRID [4]. Intel cards is a misnomer GVT is currently only supported by the integrated gpu on intel cpus which was removed form xeon cpus when GVT support was added. in the future with the descrete gpus from intel slated for release sometime in 2020 we should hopefully have intel cards that actully support GVT assuming that is on there gpu product roadmap but i can see how it would not be given they developed the tech for there integrated gpu. it would also be intersting to test amd gpus using there sriov approach but i think NVIDA tesla gpus would be the shortest path forword. > > So I think at this point the question is whether Infra can support > those reqs. If yes, we can start concrete steps towards getting those > machines used by a CI job. If not, we'll fall back to OpenLab and try > to get them hardware. > > [*] Could we do double-passthrough? Could the card be passed through > to the L1 guest via the PCI passthrough mechanism, and then into the > L2 guest via the mdev mechanism? i have a theory about how this "migth" be possible but openstack is missing the features requried to pull it off. i may test it locally with libvirt but the only why i think this could work would be to do a full passthough of the PF to an l1 guest using the q35 chipset with a viommu(not supported in nova) with hypervior hiding enabled and then run the grid driver in the l1 guest to expose a mdev to the l2 guest. ironic would be much simpler > > [1] https://etherpad.openstack.org/p/nova-ptg-train-ci > [2] https://docs.openstack.org/nova/rocky/admin/virtual-gpu.html > [3] https://01.org/igvt-g > [4] https://docs.nvidia.com/grid/5.0/pdf/grid-vgpu-user-guide.pdf > From cboylan at sapwetik.org Tue May 14 18:33:20 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 14 May 2019 11:33:20 -0700 Subject: [OpenStack-Infra] Feedback on git/repos on opendev.org In-Reply-To: <01D2D3EEA433C0419A12152B3A36557E2059D9EE@IRSMSX101.ger.corp.intel.com> References: <01D2D3EEA433C0419A12152B3A36557E2059D9EE@IRSMSX101.ger.corp.intel.com> Message-ID: On Tue, May 14, 2019, at 11:18 AM, Whaley, Graham wrote: > Hi, > (please reply-all if you want me to see it - I am not subscribed to this list) > > I was heading to read the code for the kata zuul tenant, and had some > niggles when trying to locate/navigate. I thought I'd at least drop the > feedback here (thanks ttx). Some of it will be specific to my 'flow'.... > > Sooo, as I don't store the path to the repos in my head, I normally > start out by going to git.openstack.org: > That used to get me to (iirc) the top level cgit type interface, where > I could then navigate down to the kata repos. It now takes me to the > top level of opendev.org, but not to the git area, and it is not > immediately obvious to me where the git area is. The old hosting system was cgit and we've moved to gitea as it renders code more nicely (for example can link to sections of code) and renders markdown and rst nicely. One thing we can probably do is redirect top level https://git.foo.org/ urls to https://opendev.org/explore so that you don't have to discover that tab yourself. > > I found the git area under 'Explore' :-). The 'search' is a bit odd. It > took me a moment to work out that it is alphabetical, if you ignore all > the subdir prefix's (project names?). I'm not sure that is the most > useful sort order tbh? Confused me for some time. > And then, if you search for 'kata' in the default search box, you get > **no matching repos**. Which, also, is not quite what I'd expect. Note that you can select among a number of sort orders. > > Eventually I found the kata repos by sorting reverse-alphabetically and > scrolling down :-) I know, the search is on 'repos', and not > 'organisations' - but, if you just land on that page and search for > something that happens to only have its name active at the project > level (like kata), then you end up with no results. Maybe the > search/results can be smarter, and provide the list of what it was > searching for first (so in this case 'no repositories found'), and then > some 'suggestions', like 'you may have been looking for organisation: > kata'. Just thinking out loud. Yup, I think gitea expects users to select the organizations tab and search on that if they are looking for an org and not a repo. However, I agree that the distinction is often confusing as a repo is fully qualified by both organization and repo name. I think this feedback is worth taking to gitea. We'll be happy to do that for you unless you would like to report it upstream. Let us know. (I think https://github.com/go-gitea/gitea/issues is the best venue for that) > > Thanks, > Graham > From fungi at yuggoth.org Tue May 14 18:41:06 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 14 May 2019 18:41:06 +0000 Subject: [OpenStack-Infra] 2019 Denver Summit/Forum/PTG Recap In-Reply-To: References: Message-ID: <20190514184105.ibw2radc7yqokxmg@yuggoth.org> On 2019-05-13 13:49:44 -0400 (-0400), Clark Boylan wrote: [...] > we discussed if OpenDev would be an appropriate location for > throwaway repos and unfortunately we don't think we are there yet. > Gerrit currently won't scale to that use case for us. [...] Thinking about this a little more, what if we allowed personal sandboxes in Gerrit and just didn't wrap them in all the other replication, CI and general formality we do for normal repositories? Gerrit will allow you to grant access to paths with "$user" in them[*], of which WMF's[**] and some other deployments take advantage to that end. [*] https://gerrit-review.googlesource.com/Documentation/access-control.html#_project_access_control_lists [**] https://www.mediawiki.org/wiki/Gerrit/personal_sandbox -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Tue May 14 21:23:51 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 14 May 2019 14:23:51 -0700 Subject: [OpenStack-Infra] 2019 Denver Summit/Forum/PTG Recap In-Reply-To: <20190514184105.ibw2radc7yqokxmg@yuggoth.org> References: <20190514184105.ibw2radc7yqokxmg@yuggoth.org> Message-ID: On Tue, May 14, 2019, at 11:41 AM, Jeremy Stanley wrote: > On 2019-05-13 13:49:44 -0400 (-0400), Clark Boylan wrote: > [...] > > we discussed if OpenDev would be an appropriate location for > > throwaway repos and unfortunately we don't think we are there yet. > > Gerrit currently won't scale to that use case for us. > [...] > > Thinking about this a little more, what if we allowed personal > sandboxes in Gerrit and just didn't wrap them in all the other > replication, CI and general formality we do for normal repositories? > Gerrit will allow you to grant access to paths with "$user" in > them[*], of which WMF's[**] and some other deployments take > advantage to that end. These are ref paths not repo paths. Are you thinking every user could have a single scratch space repo then have arbitrary refs within that repo? > > [*] > https://gerrit-review.googlesource.com/Documentation/access-control.html#_project_access_control_lists > [**] https://www.mediawiki.org/wiki/Gerrit/personal_sandbox > -- > Jeremy Stanley From fungi at yuggoth.org Tue May 14 21:43:03 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 14 May 2019 21:43:03 +0000 Subject: [OpenStack-Infra] 2019 Denver Summit/Forum/PTG Recap In-Reply-To: References: <20190514184105.ibw2radc7yqokxmg@yuggoth.org> Message-ID: <20190514214303.m5kkv26ockawjsjx@yuggoth.org> On 2019-05-14 14:23:51 -0700 (-0700), Clark Boylan wrote: > On Tue, May 14, 2019, at 11:41 AM, Jeremy Stanley wrote: > > On 2019-05-13 13:49:44 -0400 (-0400), Clark Boylan wrote: > > [...] > > > we discussed if OpenDev would be an appropriate location for > > > throwaway repos and unfortunately we don't think we are there yet. > > > Gerrit currently won't scale to that use case for us. > > [...] > > > > Thinking about this a little more, what if we allowed personal > > sandboxes in Gerrit and just didn't wrap them in all the other > > replication, CI and general formality we do for normal repositories? > > Gerrit will allow you to grant access to paths with "$user" in > > them[*], of which WMF's[**] and some other deployments take > > advantage to that end. > > These are ref paths not repo paths. Are you thinking every user > could have a single scratch space repo then have arbitrary refs > within that repo? [...] It looks like the way they're doing it is granting users the ability to create (via push) their own personal branches in a common Git repository... so not quite the solution to throwaway repositories, more like user-namespaced throwaway branches in a single existing repository (e.g. opendev/sandbox). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From ks3019 at att.com Wed May 15 20:35:06 2019 From: ks3019 at att.com (SKELS, KASPARS) Date: Wed, 15 May 2019 20:35:06 +0000 Subject: [OpenStack-Infra] Airship project replication/sync to GitHub Message-ID: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Hi OpenStack infra team, we would like to set up replication >From https://opendev.org/airship -> https://github.com/airshipit The GitHub space airshipit should already be available to OpenStack Foundation. Let us know how to proceed with this. Kindly, Kaspars & Airship team -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Thu May 16 15:06:33 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Thu, 16 May 2019 08:06:33 -0700 Subject: [OpenStack-Infra] Airship project replication/sync to GitHub In-Reply-To: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: On Wed, May 15, 2019, at 4:17 PM, SKELS, KASPARS wrote: > > Hi OpenStack infra team, > > > we would like to set up replication > > From https://opendev.org/airship -> https://github.com/airshipit > > > The GitHub space airshipit should already be available to OpenStack Foundation. > > Let us know how to proceed with this. > > > Kindly, Kaspars & Airship team The first step is for us to migrate the Airship repos in github under https://github.com/openstack to https://github.com/airshipit. To do this you will need to add the "OpenStack Infrastructure" openstackadmin account as admin on the airshipit org (this is a necessary permission to do transfer between orgs). Then provide us with a list of the repos that should be moved and we'll run our script over them to perform the org transfer. This org transfer ensures that github redirects are created so that people using the older urls will find the new urls. Once that is done you can remove the openstackadmin account from the airshipit org. Then the next step is to set up the Zuul git replication jobs as described at http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html for each of these repos (this is something you can do in your repos without infra involvement). To get things going maybe respond to this email thread with the list of repos to transfer on github once the openstackadmin account is added to the airshipit org? Clark From jonathan at openstack.org Thu May 16 15:10:46 2019 From: jonathan at openstack.org (Jonathan Bryce) Date: Thu, 16 May 2019 08:10:46 -0700 Subject: [OpenStack-Infra] Airship project replication/sync to GitHub In-Reply-To: References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: <16ac13214f0.27a5.724bfb4e02fbbdf80530460149ac8b11@openstack.org> On May 16, 2019 08:07:25 "Clark Boylan" wrote: > On Wed, May 15, 2019, at 4:17 PM, SKELS, KASPARS wrote: >> >> >> Hi OpenStack infra team, >> >> >> >> >> we would like to set up replication >> >> >> From https://opendev.org/airship -> https://github.com/airshipit >> >> >> >> >> The GitHub space airshipit should already be available to OpenStack Foundation. >> >> >> Let us know how to proceed with this. >> >> >> >> >> Kindly, Kaspars & Airship team > > The first step is for us to migrate the Airship repos in github under > https://github.com/openstack to https://github.com/airshipit. To do this > you will need to add the "OpenStack Infrastructure" openstackadmin account > as admin on the airshipit org (this is a necessary permission to do > transfer between orgs). I think I might have set up this org, so I need to find those credentials and share with the team to do step 1. Jonathan > Then provide us with a list of the repos that should be moved and we'll run > our script over them to perform the org transfer. > > > This org transfer ensures that github redirects are created so that people > using the older urls will find the new urls. Once that is done you can > remove the openstackadmin account from the airshipit org. > > > Then the next step is to set up the Zuul git replication jobs as described > at > http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html > for each of these repos (this is something you can do in your repos without > infra involvement). > > > To get things going maybe respond to this email thread with the list of > repos to transfer on github once the openstackadmin account is added to the > airshipit org? > > > Clark > > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra From dtroyer at gmail.com Thu May 16 16:44:05 2019 From: dtroyer at gmail.com (Dean Troyer) Date: Thu, 16 May 2019 11:44:05 -0500 Subject: [OpenStack-Infra] Airship project replication/sync to GitHub In-Reply-To: <16ac13214f0.27a5.724bfb4e02fbbdf80530460149ac8b11@openstack.org> References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> <16ac13214f0.27a5.724bfb4e02fbbdf80530460149ac8b11@openstack.org> Message-ID: On Thu, May 16, 2019 at 10:16 AM Jonathan Bryce wrote: > I think I might have set up this org, so I need to find those credentials > and share with the team to do step 1. Jonathan, if you have that for the StarlingX org can we set it up too? We do not have plans to replicate at this time but it would be good to have this sorted. Thanks dt -- Dean Troyer dtroyer at gmail.com From cboylan at sapwetik.org Fri May 17 20:09:40 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Fri, 17 May 2019 13:09:40 -0700 Subject: [OpenStack-Infra] Airship project replication/sync to GitHub In-Reply-To: References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: <43f5a2e8-026f-473d-82a1-73c16960d950@www.fastmail.com> On Thu, May 16, 2019, at 8:06 AM, Clark Boylan wrote: > On Wed, May 15, 2019, at 4:17 PM, SKELS, KASPARS wrote: > > > > Hi OpenStack infra team, > > > > > > we would like to set up replication > > > > From https://opendev.org/airship -> https://github.com/airshipit > > > > > > The GitHub space airshipit should already be available to OpenStack Foundation. > > > > Let us know how to proceed with this. > > > > > > Kindly, Kaspars & Airship team > > The first step is for us to migrate the Airship repos in github under > https://github.com/openstack to https://github.com/airshipit. To do > this you will need to add the "OpenStack Infrastructure" openstackadmin > account as admin on the airshipit org (this is a necessary permission > to do transfer between orgs). Then provide us with a list of the repos > that should be moved and we'll run our script over them to perform the > org transfer. > > This org transfer ensures that github redirects are created so that > people using the older urls will find the new urls. Once that is done > you can remove the openstackadmin account from the airshipit org. > > Then the next step is to set up the Zuul git replication jobs as > described at > http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html for each of these repos (this is something you can do in your repos without infra involvement). > > To get things going maybe respond to this email thread with the list of > repos to transfer on github once the openstackadmin account is added to > the airshipit org? As a followup we successfully added the openstackadmin account to the github org, transferred 16 repos, and then removed openstackadmin from the org. Airship is now in a position to self manage the replication from opendev (via zuul jobs) as well as self manage the Github org. Clark From ks3019 at att.com Fri May 17 18:19:23 2019 From: ks3019 at att.com (SKELS, KASPARS) Date: Fri, 17 May 2019 18:19:23 +0000 Subject: [OpenStack-Infra] Airship project replication/sync to GitHub In-Reply-To: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E74F6E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: <2ADBF0C373B7E84E944B1E06D3CDDFC91E759FCA@MOKSCY3MSGUSRGI.ITServices.sbc.com> Hi all, please let me know how to proceed on this. Kindly, Kaspars From: SKELS, KASPARS Sent: Wednesday, May 15, 2019 3:35 PM To: 'openstack-infra at lists.openstack.org' Cc: MCEUEN, MATT ; HU, BIN ; 'Chris Hoge' Subject: Airship project replication/sync to GitHub Hi OpenStack infra team, we would like to set up replication >From https://opendev.org/airship -> https://github.com/airshipit The GitHub space airshipit should already be available to OpenStack Foundation. Let us know how to proceed with this. Kindly, Kaspars & Airship team From cboylan at sapwetik.org Mon May 20 19:02:41 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 20 May 2019 12:02:41 -0700 Subject: [OpenStack-Infra] Meeting Agenda for 1900UTC May 21, 2019 Message-ID: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** We now install puppet-4 directly using the puppet-agent package in puppet inc's archive as they deleted the proper apt repo. *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Project renames to clean things up after the great bit OpenDev rename (clarkb 20190521) *** Friday May 31, 2019 *** https://review.opendev.org/#/c/655476/ Fixes for rename playbook *** Need changes for infra projects that were not renamed properly *** Need changes for openstack-dev projects that were not renamed properly. ** Trusty Upgrade Progress (clarkb 20190521) *** Progress with askbot thanks to ianw ** https mirror update (clarkb 20190521) *** https://review.opendev.org/658281 -- the actual change to implement opendev.org mirrors * Open discussion From amoralej at redhat.com Fri May 24 17:20:45 2019 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Fri, 24 May 2019 19:20:45 +0200 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 Message-ID: Hi, Since last hours puppet-openstack-integrating jobs in RDO are failing to deploy puppet modules from https://opendev.org/openstack/puppet- in CentOS7. The sympthom is jobs get stuck randomly when installing modules using r10k. Investigating it, I've been able to reproduce the issue running: git clone --mirror https://github.com/openstack/puppet- About 50% of the times, commands get stuck for any project in opendev.org/openstack. I've checked that it works fine if i don't use "--mirror". As it seems to work fine with fedora, I've compiled and installed latest version of git from upstream and in a CentOS7 and then, git clone --mirror works fine, but in this case r10k also gets stuck when running: git --git-dir /home/centos/.r10k/git/https---opendev.org-openstack-puppet-keystone fetch origin --prune In this case, it only happens with puppet-keystone. Note that it always works fine using github.com/openstack mirrors instead of opendev.org so the problem seems to be some combination of git repos from opendev with CentOS7 git. Has there been any change that may be affecting git repos behavior somehow? Best regards, Alfredo -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Fri May 24 17:31:56 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Fri, 24 May 2019 10:31:56 -0700 Subject: [OpenStack-Infra] =?utf-8?q?Problems_cloning_repos_from_opendev?= =?utf-8?q?=2Eorg_in_CentOS7?= In-Reply-To: References: Message-ID: <9d81c196-ddfe-47b2-b3de-42cee78cd75e@www.fastmail.com> On Fri, May 24, 2019, at 10:23 AM, Alfredo Moralejo Alonso wrote: > Hi, > > Since last hours puppet-openstack-integrating jobs in RDO are failing > to deploy puppet modules from > https://opendev.org/openstack/puppet- in CentOS7. > > The sympthom is jobs get stuck randomly when installing modules using > r10k. Investigating it, I've been able to reproduce the issue running: > > git clone --mirror https://github.com/openstack/puppet- > > About 50% of the times, commands get stuck for any project in > opendev.org/openstack. I've checked that it works fine if i don't use > "--mirror". > > As it seems to work fine with fedora, I've compiled and installed > latest version of git from upstream and in a CentOS7 and then, git > clone --mirror works fine, but in this case r10k also gets stuck when > running: > > git --git-dir > /home/centos/.r10k/git/https---opendev.org-openstack-puppet-keystone > fetch origin --prune > > In this case, it only happens with puppet-keystone. > > Note that it always works fine using github.com/openstack mirrors > instead of opendev.org so the problem seems to be some combination of > git repos from opendev with CentOS7 git. Has there been any change that > may be affecting git repos behavior somehow? Yesterday we replicated all of refs/changes and refs/notes from Gerrit to Gitea. We had these replicated on the old Cgit farm, but gitea didn't support this replication until wednesday when we upgraded Gitea. The manpage for git clone --mirror says it will "map all refs (including remote-tracking branches, notes etc.)" so this is possibly related. When you say it gets stuck randomly does that mean the git process never completes the clone? or it stops with failure? In either case stracing the clone process might help to figure out if it is these new refs that it is having problems with. Clark From fungi at yuggoth.org Fri May 24 17:49:04 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 24 May 2019 17:49:04 +0000 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 In-Reply-To: References: Message-ID: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> On 2019-05-24 19:20:45 +0200 (+0200), Alfredo Moralejo Alonso wrote: > Since last hours puppet-openstack-integrating jobs in RDO are failing to > deploy puppet modules from https://opendev.org/openstack/puppet- > in CentOS7. [...] Can you be a little more clear on the timeframe? We've switched from the smart Git HTTP backend with Apache to Gitea as of April 20, 2019. We've upgraded the version of Gitea in use several times since then, most recently from 1.8.0 to a newer commit from their master branch as of 01:45 UTC on May 23. We also spent some hours pushing a lot of additional change and note refs into the Gitea service between 16:00 UTC on May 23 and 04:00 UTC on on May 24. Getting a tighter scope for when you saw problems arise may help us nail down which changes on our end might be involved. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From amoralej at redhat.com Fri May 24 18:33:40 2019 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Fri, 24 May 2019 20:33:40 +0200 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 In-Reply-To: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> References: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> Message-ID: On Fri, May 24, 2019 at 7:53 PM Jeremy Stanley wrote: > On 2019-05-24 19:20:45 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > Since last hours puppet-openstack-integrating jobs in RDO are failing to > > deploy puppet modules from https://opendev.org/openstack/puppet- > > > in CentOS7. > [...] > > Can you be a little more clear on the timeframe? We've switched from > the smart Git HTTP backend with Apache to Gitea as of April 20, > 2019. We've upgraded the version of Gitea in use several times since > then, most recently from 1.8.0 to a newer commit from their master > branch as of 01:45 UTC on May 23. We also spent some hours pushing a > lot of additional change and note refs into the Gitea service > between 16:00 UTC on May 23 and 04:00 UTC on on May 24. Getting a > tighter scope for when you saw problems arise may help us nail down > which changes on our end might be involved. > For periodic jobs we run every 4 hours i see: - Last successful jobs started at 23-May-2019 21:06:42 UTC - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC I'll try to find out if i can make the window smaller. > -- > Jeremy Stanley > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Fri May 24 18:41:58 2019 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Fri, 24 May 2019 20:41:58 +0200 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 In-Reply-To: References: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> Message-ID: On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso wrote: > > > On Fri, May 24, 2019 at 7:53 PM Jeremy Stanley wrote: > >> On 2019-05-24 19:20:45 +0200 (+0200), Alfredo Moralejo Alonso wrote: >> > Since last hours puppet-openstack-integrating jobs in RDO are failing to >> > deploy puppet modules from https://opendev.org/openstack/puppet- >> >> > in CentOS7. >> [...] >> >> Can you be a little more clear on the timeframe? We've switched from >> the smart Git HTTP backend with Apache to Gitea as of April 20, >> 2019. We've upgraded the version of Gitea in use several times since >> then, most recently from 1.8.0 to a newer commit from their master >> branch as of 01:45 UTC on May 23. We also spent some hours pushing a >> lot of additional change and note refs into the Gitea service >> between 16:00 UTC on May 23 and 04:00 UTC on on May 24. Getting a >> tighter scope for when you saw problems arise may help us nail down >> which changes on our end might be involved. >> > > > For periodic jobs we run every 4 hours i see: > - Last successful jobs started at 23-May-2019 21:06:42 UTC > - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC > > I'll try to find out if i can make the window smaller. > > >> -- >> Jeremy Stanley >> _______________________________________________ >> OpenStack-Infra mailing list >> OpenStack-Infra at lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri May 24 18:54:58 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 24 May 2019 18:54:58 +0000 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 In-Reply-To: References: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> Message-ID: <20190524185457.n5t43teceab6xtwv@yuggoth.org> On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote: > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso > wrote: [...] > > For periodic jobs we run every 4 hours i see: > > - Last successful jobs started at 23-May-2019 21:06:42 UTC > > - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC > > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC [...] Thanks, between the timing and symptoms, we're starting to suspect this could be related to how older Git clients' requests are being distributed between our load-balanced backends with inconsistent packing. We're weighing some ideas for how to improve things there, so stay tuned. The good news is that within the next 24 hours the backends will be mostly in sync most of the time, so the symptoms you're observing may subside for the most part once that's the case (but to be entirely honest, we're still not quite sure just yet). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From amoralej at redhat.com Fri May 24 18:55:38 2019 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Fri, 24 May 2019 20:55:38 +0200 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 In-Reply-To: <9d81c196-ddfe-47b2-b3de-42cee78cd75e@www.fastmail.com> References: <9d81c196-ddfe-47b2-b3de-42cee78cd75e@www.fastmail.com> Message-ID: On Fri, May 24, 2019 at 7:37 PM Clark Boylan wrote: > On Fri, May 24, 2019, at 10:23 AM, Alfredo Moralejo Alonso wrote: > > Hi, > > > > Since last hours puppet-openstack-integrating jobs in RDO are failing > > to deploy puppet modules from > > https://opendev.org/openstack/puppet- in CentOS7. > > > > The sympthom is jobs get stuck randomly when installing modules using > > r10k. Investigating it, I've been able to reproduce the issue running: > > > > git clone --mirror https://github.com/openstack/puppet- > > > > About 50% of the times, commands get stuck for any project in > > opendev.org/openstack. I've checked that it works fine if i don't use > > "--mirror". > > > > As it seems to work fine with fedora, I've compiled and installed > > latest version of git from upstream and in a CentOS7 and then, git > > clone --mirror works fine, but in this case r10k also gets stuck when > > running: > > > > git --git-dir > > /home/centos/.r10k/git/https---opendev.org-openstack-puppet-keystone > > fetch origin --prune > > > > In this case, it only happens with puppet-keystone. > > > > Note that it always works fine using github.com/openstack mirrors > > instead of opendev.org so the problem seems to be some combination of > > git repos from opendev with CentOS7 git. Has there been any change that > > may be affecting git repos behavior somehow? > > Yesterday we replicated all of refs/changes and refs/notes from Gerrit to > Gitea. We had these replicated on the old Cgit farm, but gitea didn't > support this replication until wednesday when we upgraded Gitea. The > manpage for git clone --mirror says it will "map all refs (including > remote-tracking branches, notes etc.)" so this is possibly related. > > When you say it gets stuck randomly does that mean the git process never > completes the clone? or it stops with failure? In either case stracing the > clone process might help to figure out if it is these new refs that it is > having problems with. > > When fails, it never completes, example output with GIT_TRACE enabled: $ git clone --mirror https://opendev.org/openstack/puppet-neutron trace: built-in: git 'clone' '--mirror' ' https://opendev.org/openstack/puppet-neutron' Cloning into bare repository 'puppet-neutron.git'... trace: run_command: 'git-remote-https' 'origin' ' https://opendev.org/openstack/puppet-neutron' trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' 'https://opendev.org/openstack/puppet-neutron/' trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' 'https://opendev.org/openstack/puppet-neutron/' trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' 'https://opendev.org/openstack/puppet-neutron/' It always get stuck in: trace: built-in: git 'fetch-pack' ... I have a 2MB strace output, but it's probably more useful for you to just replicate it in a centos:latest container?, simply: " docker run -it docker.io/centos:latest /bin/bash yum install -y git strace git clone --mirror https://opendev.org/openstack/puppet-nova " you can run the git clone several times if it finishes fine. > Clark > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Fri May 24 19:17:04 2019 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Fri, 24 May 2019 21:17:04 +0200 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 In-Reply-To: <20190524185457.n5t43teceab6xtwv@yuggoth.org> References: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> <20190524185457.n5t43teceab6xtwv@yuggoth.org> Message-ID: On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley wrote: > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso < > amoralej at redhat.com> > > wrote: > [...] > > > For periodic jobs we run every 4 hours i see: > > > - Last successful jobs started at 23-May-2019 21:06:42 UTC > > > - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC > > > > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC > [...] > > Thanks, between the timing and symptoms, we're starting to suspect > this could be related to how older Git clients' requests are being > distributed between our load-balanced backends with inconsistent > packing. We're weighing some ideas for how to improve things there, > so stay tuned. The good news is that within the next 24 hours the > backends will be mostly in sync most of the time, so the symptoms > you're observing may subside for the most part once that's the case > (but to be entirely honest, we're still not quite sure just yet). > ok, thanks for the update. I'll switch some jobs to use github mirrors as workaround until we find it more stable. > -- > Jeremy Stanley > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Tue May 28 15:37:03 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 28 May 2019 08:37:03 -0700 Subject: [OpenStack-Infra] Infra meeting agenda for May 28, 2019 Message-ID: Sorry this is getting out late this week. Yesterday was a holiday in my corner of the world and I didn't end up getting in front of the keyboard. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Puppet 4 effectively done. **** arm64 hosts running puppet 3 due to lack of packages for puppet 4 on arm64. Affects mirror and builder node. **** Our backup host which is in the disabled group is also puppet 3 but not running config mgmt. *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Project renames to clean things up after the great bit OpenDev rename (clarkb 20190528) *** Friday May 31, 2019 *** https://review.opendev.org/#/q/topic:project-rename+status:open ** Trusty Upgrade Progress (clarkb 20190528) ** https mirror update (clarkb 20190528) *** https://review.opendev.org/#/c/661187/ switch rax dfw mirror to new opendev mirror. * Open discussion From cboylan at sapwetik.org Tue May 28 20:54:49 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 28 May 2019 13:54:49 -0700 Subject: [OpenStack-Infra] =?utf-8?q?Problems_cloning_repos_from_opendev?= =?utf-8?q?=2Eorg_in=09CentOS7?= In-Reply-To: References: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> <20190524185457.n5t43teceab6xtwv@yuggoth.org> Message-ID: On Fri, May 24, 2019, at 12:17 PM, Alfredo Moralejo Alonso wrote: > > > On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley wrote: > > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso > > > wrote: > > [...] > > > > For periodic jobs we run every 4 hours i see: > > > > - Last successful jobs started at 23-May-2019 21:06:42 UTC > > > > - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC > > > > > > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC > > [...] > > > > Thanks, between the timing and symptoms, we're starting to suspect > > this could be related to how older Git clients' requests are being > > distributed between our load-balanced backends with inconsistent > > packing. We're weighing some ideas for how to improve things there, > > so stay tuned. The good news is that within the next 24 hours the > > backends will be mostly in sync most of the time, so the symptoms > > you're observing may subside for the most part once that's the case > > (but to be entirely honest, we're still not quite sure just yet). > > ok, thanks for the update. I'll switch some jobs to use github mirrors > as workaround until we find it more stable. Can you test opendev.org again? We've updated the load balancer method from least connections to a source IP hash. We think this will address the state mismatch problems that you noticed previously. I have since been able to run a git clone --mirror and git fetch origin --prune myself (when it didn't work reliably in the past). Please let us know if it works now or if it doesn't. That data will be useful either way. Clark From amoralej at redhat.com Wed May 29 08:48:50 2019 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Wed, 29 May 2019 10:48:50 +0200 Subject: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7 In-Reply-To: References: <20190524174904.fan4jiuhnh5ynehp@yuggoth.org> <20190524185457.n5t43teceab6xtwv@yuggoth.org> Message-ID: On Tue, May 28, 2019 at 11:01 PM Clark Boylan wrote: > On Fri, May 24, 2019, at 12:17 PM, Alfredo Moralejo Alonso wrote: > > > > > > On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley > wrote: > > > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > > > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso < > amoralej at redhat.com> > > > > wrote: > > > [...] > > > > > For periodic jobs we run every 4 hours i see: > > > > > - Last successful jobs started at 23-May-2019 21:06:42 UTC > > > > > - First failed jobs with this issue started at 24-May-2019 > 03:06:37 UTC > > > > > > > > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC > > > [...] > > > > > > Thanks, between the timing and symptoms, we're starting to suspect > > > this could be related to how older Git clients' requests are being > > > distributed between our load-balanced backends with inconsistent > > > packing. We're weighing some ideas for how to improve things there, > > > so stay tuned. The good news is that within the next 24 hours the > > > backends will be mostly in sync most of the time, so the symptoms > > > you're observing may subside for the most part once that's the case > > > (but to be entirely honest, we're still not quite sure just yet). > > > > ok, thanks for the update. I'll switch some jobs to use github mirrors > > as workaround until we find it more stable. > > Can you test opendev.org again? We've updated the load balancer method > from least connections to a source IP hash. We think this will address the > state mismatch problems that you noticed previously. I have since been able > to run a git clone --mirror and git fetch origin --prune myself (when it > didn't work reliably in the past). > > Please let us know if it works now or if it doesn't. That data will be > useful either way. > > I've tested it both locally and in CI jobs and it's working fine in all cases. Thanks for your support, Alfredo > Clark > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra at lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri May 31 20:02:12 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 31 May 2019 20:02:12 +0000 Subject: [OpenStack-Infra] [Starlingx-discuss] Question: How to attach large files to LP bugs? In-Reply-To: <9A85D2917C58154C960D95352B22818BD074CCFC@fmsmsx123.amr.corp.intel.com> References: <9A85D2917C58154C960D95352B22818BD072FCFC@fmsmsx123.amr.corp.intel.com> <9A85D2917C58154C960D95352B22818BD074CCFC@fmsmsx123.amr.corp.intel.com> Message-ID: <20190531200211.gqehlmzuzrbracty@yuggoth.org> On 2019-05-31 19:56:30 +0000 (+0000), Jones, Bruce E wrote: > Resending this query. Any thoughts or ideas? Thanks! [...] I'm intentionally keeping the cross-post between mailing lists on this reply (even though it's generally a bad idea), on the assumption your follow-up indicates you didn't see the original replies: http://lists.openstack.org/pipermail/openstack-infra/2019-May/006366.html http://lists.openstack.org/pipermail/openstack-infra/2019-May/006368.html That said, the OpenDev/OpenStack Infrastructure sysadmins don't maintain Launchpad so don't really have any inside information on it. For real answers you probably need to hit up https://askubuntu.com/questions/tagged/launchpad as Canonical's canonical (heh) place to ask questions about their service. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From bruce.e.jones at intel.com Fri May 31 19:56:30 2019 From: bruce.e.jones at intel.com (Jones, Bruce E) Date: Fri, 31 May 2019 19:56:30 +0000 Subject: [OpenStack-Infra] Question: How to attach large files to LP bugs? In-Reply-To: <9A85D2917C58154C960D95352B22818BD072FCFC@fmsmsx123.amr.corp.intel.com> References: <9A85D2917C58154C960D95352B22818BD072FCFC@fmsmsx123.amr.corp.intel.com> Message-ID: <9A85D2917C58154C960D95352B22818BD074CCFC@fmsmsx123.amr.corp.intel.com> Resending this query. Any thoughts or ideas? Thanks! brucej From: Jones, Bruce E Sent: Thursday, May 9, 2019 10:08 AM To: openstack-infra at lists.openstack.org Cc: starlingx-discuss at lists.starlingx.io Subject: Question: How to attach large files to LP bugs? Greetings Infra team. The StarlingX project has a question - we have had some bug reports where the log files needed to debug the bug turn out to be quite large. Larger than can be attached to a LP entry. Is there a recommended way to handle this case? Thank you! brucej -------------- next part -------------- An HTML attachment was scrubbed... URL: