From cboylan at sapwetik.org Mon Jul 1 19:05:41 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 01 Jul 2019 12:05:41 -0700 Subject: [OpenStack-Infra] Meeting Agenda for July 2, 2019 Message-ID: == Agenda for next meeting == * Announcements ** July 4 is major US holiday. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190701) ** https mirror update (clarkb 20190701) *** OpenAFS 1.8.3 (via infra PPA) is stable on bionic *** New mirrors should be built with Bionic and OpenAFS 1.8.3 via PPA ** Cloud status update (clarkb 20190702) *** FortNebula addition *** Limestone networking *** Inap upgrades * Open discussion From cboylan at sapwetik.org Mon Jul 8 21:36:11 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 08 Jul 2019 14:36:11 -0700 Subject: [OpenStack-Infra] Meeting Agenda for July 9, 2019 Message-ID: <04a01e26-b103-4eed-aa19-cb4e9db3efa8@www.fastmail.com> == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190709) ** Mirror setup updates (clarkb 20190709) *** Do we replace existing mirrors with new opendev mirrors running openafs 1.8.3? ** Cloud status update (clarkb 20190709) *** FortNebula addition * Open discussion From iwienand at redhat.com Tue Jul 9 07:43:06 2019 From: iwienand at redhat.com (Ian Wienand) Date: Tue, 9 Jul 2019 17:43:06 +1000 Subject: [OpenStack-Infra] Meeting Agenda for July 9, 2019 In-Reply-To: <04a01e26-b103-4eed-aa19-cb4e9db3efa8@www.fastmail.com> References: <04a01e26-b103-4eed-aa19-cb4e9db3efa8@www.fastmail.com> Message-ID: <20190709074306.GA6844@fedora19.localdomain> On Mon, Jul 08, 2019 at 02:36:11PM -0700, Clark Boylan wrote: > ** Mirror setup updates (clarkb 20190709) > *** Do we replace existing mirrors with new opendev mirrors running openafs 1.8.3? I won't make it to the meeting tomorrow sorry, but here's the current status, which is largely reflected in https://etherpad.openstack.org/p/opendev-mirror-afs The kafs based servers have been paused for now due the hard crashes in fscache, which requires us to monitor it very closely, which wasn't happening over holiday breaks. https://review.opendev.org/#/c/669231/ dhowells is on vacation till at least the 15th, and there is no real prospect of those issues being looked at until after then. There are some changes in the afs-next kernel branch for us to try, which should help with the "volume offline" issues we saw being reported when a "vos release" was happening (basically making kafs switch to the other server better). I believe that capturing logs from our AFS servers helped debug these issues. I can take an action item to build a kernel with them and switch it back in for testing late next week when I am back (or someone else can, if they like). This will give enough for a Tested-By: flag for sending those changes to Linus upstream. Once everyone is back, we can look more closely at the fscache issues, which are currently a blocker for future work. I'm not aware of any issues with openafs 1.8.3 based mirrors. If we need any new mirrors, or feel the need to replace those in production, we should be fine bringing them up with that. Thanks, -i From them0419 at naver.com Tue Jul 9 05:56:21 2019 From: them0419 at naver.com (=?utf-8?B?6rmA66+87ISd?=) Date: Tue, 09 Jul 2019 14:56:21 +0900 Subject: [OpenStack-Infra] =?utf-8?q?May_I_get_your_help=3F_I=27m_struggli?= =?utf-8?q?ng_with_devstack_installation?= Message-ID: <305dd85f5d7e78dac13e646c22e58da@cweb016.nm.nfra.io> I have nowhere to ask question about this So finally I got here. Pleas Please, give me some advice I'm trying to install openstack rocky by devstack with ceilometer and gnocchi-api. ( I also tried with 'enable_service gnocchi-api,gnocchi-metricd' ) I checked endpoint of metric then post below It gave an internal error message. I could get metric data with openstack CLI, but gnocchi CLI doesn't work Did I missed some command at local.conf? or is this gnocchi's probleam...? How should I configure local.conf to use ceilometer and gnocchi-api Thanks for reading -------------- next part -------------- An HTML attachment was scrubbed... URL: From lei12zhang12 at gmail.com Wed Jul 10 06:43:50 2019 From: lei12zhang12 at gmail.com (Lei Zhang) Date: Wed, 10 Jul 2019 14:43:50 +0800 Subject: [OpenStack-Infra] [rsd-lib] how to sync code from opendev to github Message-ID: Hi guys, We have a project named rsd-lib originally was under openstack and CI takes the charge to push new changes to github repo. Now the project has been changed from openstack/rsd-lib to x/rsd-lib(https://opendev.org/x/rsd-lib). The github repo(https://github.com/openstack/rsd-lib) stops getting newly merged code. My question is that is there a way for rsd-lib to continue syncing code to https://github.com/openstack/rsd-lib, or we have to create another github repo to mirror the code? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at suse.com Wed Jul 10 07:05:30 2019 From: aj at suse.com (Andreas Jaeger) Date: Wed, 10 Jul 2019 09:05:30 +0200 Subject: [OpenStack-Infra] [rsd-lib] how to sync code from opendev to github In-Reply-To: References: Message-ID: On 10/07/2019 08.43, Lei Zhang wrote: > Hi guys, > > We have a project named rsd-lib originally was under openstack and CI > takes the charge to push new changes to github repo. Now the project has > been changed from openstack/rsd-lib to > x/rsd-lib(https://opendev.org/x/rsd-lib). The github > repo(https://github.com/openstack/rsd-lib) stops getting newly merged code. > > My question is that is there a way for rsd-lib to continue syncing code > to https://github.com/openstack/rsd-lib, or we have to create another > github repo to mirror the code? You can set this up on your own - but not to openstack/rsd-lib, that name space is only for official OpenStack projects, you need to create your own namespace. For details, read the Infra Manual: https://docs.openstack.org/infra/manual/creators.html#mirroring-projects-to-git-mirrors Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From lei12zhang12 at gmail.com Wed Jul 10 07:41:39 2019 From: lei12zhang12 at gmail.com (Lei Zhang) Date: Wed, 10 Jul 2019 15:41:39 +0800 Subject: [OpenStack-Infra] [rsd-lib] how to sync code from opendev to github In-Reply-To: References: Message-ID: Got it, thanks! On Wed, Jul 10, 2019 at 3:05 PM Andreas Jaeger wrote: > On 10/07/2019 08.43, Lei Zhang wrote: > > Hi guys, > > > > We have a project named rsd-lib originally was under openstack and CI > > takes the charge to push new changes to github repo. Now the project has > > been changed from openstack/rsd-lib to > > x/rsd-lib(https://opendev.org/x/rsd-lib). The github > > repo(https://github.com/openstack/rsd-lib) stops getting newly merged > code. > > > > My question is that is there a way for rsd-lib to continue syncing code > > to https://github.com/openstack/rsd-lib, or we have to create another > > github repo to mirror the code? > > You can set this up on your own - but not to openstack/rsd-lib, that > name space is only for official OpenStack projects, you need to create > your own namespace. > > > For details, read the Infra Manual: > > > https://docs.openstack.org/infra/manual/creators.html#mirroring-projects-to-git-mirrors > > Andreas > -- > Andreas Jaeger aj at suse.com Twitter: jaegerandi > SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah > HRB 21284 (AG Nürnberg) > GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmsimard at redhat.com Mon Jul 15 19:03:36 2019 From: dmsimard at redhat.com (David Moreau Simard) Date: Mon, 15 Jul 2019 15:03:36 -0400 Subject: [OpenStack-Infra] ARA 1.0 deployment plans In-Reply-To: <874l4vae1y.fsf@meyer.lemoncheese.net> References: <20190611080001.GA13504@fedora19.localdomain> <874l4vae1y.fsf@meyer.lemoncheese.net> Message-ID: On Tue, Jun 11, 2019 at 7:45 PM James E. Blair wrote: > Given that our direction is to remove logs.o.o from the system along > with all of its proxy services and only serve static files from swift, I > think we will have to continue to use the old version of ARA until the > story with static generation is worked out for 1.0. We're currently experimenting with a proof of concept [1] to generate static reports in 1.x -- you can see what it looks like in the uploaded logs here [2]. In a nutshell, it uses the offline API client to retrieve data and then it's rendered using simple html templates. No added dependencies, no javascript (yet?) and the only css is included from a CDN. The challenge with static reports remains the same, though: if you run 100 tasks against 10 hosts, it's 1000 task results. If we keep results in their own pages to keep the playbook pages lightweight, we end up with 1000+ small files. Inlcuding those task results in the same pages as the playbooks reduces the amount of files to generate but it also results in heavier pages to load in users' browsers. Another option is to have arguments or parameters to omit data from the report which would result in less bloat. For example, there could be a flag to omit (detailed) results for "OK" and "SKIPPED" tasks or there could be a flag to only include failed playbooks in the report. Doing something like this isn't complicated since the API already supports filtering by playbook and task status. I'll think about it a little bit but I'm open to suggestions if anyone has an idea. In the meantime, we've landed an implementation of the sqlite middleware for the API [3]. I know it wouldn't work with Swift (unless we use something like s3ql [4]?) but I think that an API to browse and query any job's results could be useful. [1]: https://review.opendev.org/#/c/670597/ [2]: http://logs.openstack.org/97/670597/7/check/ansible-role-ara-api-fedora-devel/1dcf203/logs/static/ [3]: https://ara.readthedocs.io/en/latest/distributed-sqlite-backend.html [4]: https://github.com/s3ql/s3ql David Moreau Simard dmsimard = [irc, github, twitter] From cboylan at sapwetik.org Mon Jul 15 19:43:28 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 15 Jul 2019 12:43:28 -0700 Subject: [OpenStack-Infra] Meeting Agenda for July 16, 2019 at 1900UTC Message-ID: We will meet tomorrow, July 16, at 1900UTC for our weekly infra team meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190716) ** Cloud status update (clarkb 20190716) *** FortNebula status update *** Linaro and MOC updates ** PPA Management Process (clarkb 20190716) ** PTG Planning (clarkb 20190716) *** Do we want to attend? *** Space for Gitea collaboration * Open discussion From lei12zhang12 at gmail.com Tue Jul 16 09:01:15 2019 From: lei12zhang12 at gmail.com (Lei Zhang) Date: Tue, 16 Jul 2019 17:01:15 +0800 Subject: [OpenStack-Infra] [rsd-lib] how to sync code from opendev to github In-Reply-To: References: Message-ID: Hi Andreas Is there any action to take on github repos outside OpenStack governance but still under openstack namespace? This is somewhat confusing for repo consumers as they may not notice this and still use outdated code. On Wed, Jul 10, 2019 at 3:05 PM Andreas Jaeger wrote: > On 10/07/2019 08.43, Lei Zhang wrote: > > Hi guys, > > > > We have a project named rsd-lib originally was under openstack and CI > > takes the charge to push new changes to github repo. Now the project has > > been changed from openstack/rsd-lib to > > x/rsd-lib(https://opendev.org/x/rsd-lib). The github > > repo(https://github.com/openstack/rsd-lib) stops getting newly merged > code. > > > > My question is that is there a way for rsd-lib to continue syncing code > > to https://github.com/openstack/rsd-lib, or we have to create another > > github repo to mirror the code? > > You can set this up on your own - but not to openstack/rsd-lib, that > name space is only for official OpenStack projects, you need to create > your own namespace. > > > For details, read the Infra Manual: > > > https://docs.openstack.org/infra/manual/creators.html#mirroring-projects-to-git-mirrors > > Andreas > -- > Andreas Jaeger aj at suse.com Twitter: jaegerandi > SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah > HRB 21284 (AG Nürnberg) > GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lei12zhang12 at gmail.com Tue Jul 16 09:43:20 2019 From: lei12zhang12 at gmail.com (Lei Zhang) Date: Tue, 16 Jul 2019 17:43:20 +0800 Subject: [OpenStack-Infra] [rsd-lib] how to sync code from opendev to github In-Reply-To: <5734bbf6-acb4-c1ae-4d95-2b11633f2869@suse.com> References: <5734bbf6-acb4-c1ae-4d95-2b11633f2869@suse.com> Message-ID: For rsd-lib, if we mirror the project to a new namespace in github, could we make openstack/rsd-lib redirects to that new repo? Thanks On Tue, Jul 16, 2019 at 5:07 PM Andreas Jaeger wrote: > On 7/16/19 11:01 AM, Lei Zhang wrote: > > Hi Andreas > > > > Is there any action to take on github repos outside OpenStack > > governance but still under openstack namespace? This is somewhat > > confusing for repo consumers as they may not notice this and still use > > outdated code. > > You can work together with the infra team to transfer the repo for that > so that redirects exist, > > Andreas > -- > Andreas Jaeger aj at suse.com Twitter: jaegerandi > SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah > HRB 21284 (AG Nürnberg) > GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at suse.com Tue Jul 16 10:14:14 2019 From: aj at suse.com (Andreas Jaeger) Date: Tue, 16 Jul 2019 12:14:14 +0200 Subject: [OpenStack-Infra] [rsd-lib] how to sync code from opendev to github In-Reply-To: References: Message-ID: On 7/16/19 11:01 AM, Lei Zhang wrote: > Hi Andreas > > Is there any action to take on  github repos outside OpenStack > governance but still under openstack namespace? This is somewhat > confusing for repo consumers as they may not notice this and still use > outdated code. You can work together with the infra team to transfer the repo for that so that redirects exist, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From vasanth3g at gmail.com Wed Jul 17 02:06:34 2019 From: vasanth3g at gmail.com (Vasanth M.Vasanth) Date: Wed, 17 Jul 2019 07:36:34 +0530 Subject: [OpenStack-Infra] Issue on instance launching Message-ID: Hi all, When I try to start the nova compute service as conflict the ports to metadata service. I have killed/stop the process but again listen the port and throw the error . Unable to start the both service . Could you please suggest me. openstack-nova-os-compute-api.service openstack-nova-metadata-api.service Nova Api logs ; 2019-07-16 18:40:04.236 31644 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports. 2019-07-16 18:40:04.238 31644 DEBUG nova.wsgi [-] Loading app metadata from /etc/nova/api-paste.ini load_app /usr/lib/python2.7/site-packages/nova/wsgi.py:496 2019-07-16 18:40:04.257 31644 ERROR nova.wsgi [-] Could not bind to 0.0.0.0:8775 2019-07-16 18:40:04.258 31644 CRITICAL nova [-] error: [Errno 98] Address already in use 2019-07-16 18:40:04.258 31644 ERROR nova Traceback (most recent call last): 2019-07-16 18:40:04.258 31644 ERROR nova File "/usr/bin/nova-api-metadata", line 10, in 2019-07-16 18:40:04.258 31644 ERROR nova sys.exit(main()) 2019-07-16 18:40:04.258 31644 ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/api_metadata.py", line 50, in main 2019-07-16 18:40:04.258 31644 ERROR nova server = service.WSGIService('metadata', use_ssl=should_use_ssl) 2019-07-16 18:40:04.258 31644 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 311, in __init__ 2019-07-16 18:40:04.258 31644 ERROR nova max_url_len=max_url_len) 2019-07-16 18:40:04.258 31644 ERROR nova File "/usr/lib/python2.7/site-packages/nova/wsgi.py", line 101, in __init__ 2019-07-16 18:40:04.258 31644 ERROR nova self._socket = eventlet.listen(bind_addr, family, backlog=backlog) 2019-07-16 18:40:04.258 31644 ERROR nova File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in listen 2019-07-16 18:40:04.258 31644 ERROR nova sock.bind(addr) 2019-07-16 18:40:04.258 31644 ERROR nova File "/usr/lib64/python2.7/socket.py", line 224, in meth 2019-07-16 18:40:04.258 31644 ERROR nova return getattr(self._sock,name)(*args) 2019-07-16 18:40:04.258 31644 ERROR nova error: [Errno 98] Address already in use 2019-07-16 18:40:04.258 31644 ERROR Thanks __Vasanth Sent from my iPhone From lennyb at mellanox.com Thu Jul 18 07:42:58 2019 From: lennyb at mellanox.com (Lenny Verkhovsky) Date: Thu, 18 Jul 2019 07:42:58 +0000 Subject: [OpenStack-Infra] [github] Please delete stole github networking-mlnx repo Message-ID: Hi Infra team, As discussed in irc chat[1] Could you please, delete stale project of networking-mlnx[2]. The updated code location is in Opendev now[3] Thanks in advance. Lenny (aka lennyb) p.s. relevant admin group[4] is updated. [1] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-07-18.log.html [2] https://github.com/openstack/networking-mlnx [3] https://opendev.org/x/networking-mlnx [4] https://review.opendev.org/#/admin/groups/559,members -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Thu Jul 18 16:41:23 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Thu, 18 Jul 2019 09:41:23 -0700 Subject: [OpenStack-Infra] Issue on instance launching In-Reply-To: References: Message-ID: <1d4ae922-f9cf-4a13-88dc-034d18759333@www.fastmail.com> On Tue, Jul 16, 2019, at 7:07 PM, Vasanth M.Vasanth wrote: > Hi all, > > When I try to start the nova compute service as conflict the ports to > metadata service. I have killed/stop the process but again listen the > port and throw the error . Unable to start the both service . > > Could you please suggest me. We are the team that manages the developer infrastructure for building openstack the software, but don't have much experience running openstack directly ourselves. You may have better luck with this request sending it to openstack-discuss at lists.openstack.org which is the general openstack list and has developers, operators, and users subscribed to it. > > openstack-nova-os-compute-api.service > openstack-nova-metadata-api.service > > Nova Api logs ; > > > 2019-07-16 18:40:04.236 31644 WARNING > oslo_reports.guru_meditation_report [-] Guru meditation now registers > SIGUSR1 and SIGUSR2 by default for > backward compatibility. SIGUSR1 will no longer be registered in a > future release, so please use SIGUSR2 to generate reports. > 2019-07-16 18:40:04.238 31644 DEBUG nova.wsgi [-] Loading app metadata > from /etc/nova/api-paste.ini load_app > /usr/lib/python2.7/site-packages/nova/wsgi.py:496 > 2019-07-16 18:40:04.257 31644 ERROR nova.wsgi [-] Could not bind to > 0.0.0.0:8775 > 2019-07-16 18:40:04.258 31644 CRITICAL nova [-] error: [Errno 98] > Address already in use > 2019-07-16 18:40:04.258 31644 ERROR nova Traceback (most recent call > last): > 2019-07-16 18:40:04.258 31644 ERROR nova File > "/usr/bin/nova-api-metadata", line 10, in > 2019-07-16 18:40:04.258 31644 ERROR nova sys.exit(main()) > 2019-07-16 18:40:04.258 31644 ERROR nova File > "/usr/lib/python2.7/site-packages/nova/cmd/api_metadata.py", line 50, > in main > 2019-07-16 18:40:04.258 31644 ERROR nova server = > service.WSGIService('metadata', use_ssl=should_use_ssl) > 2019-07-16 18:40:04.258 31644 ERROR nova File > "/usr/lib/python2.7/site-packages/nova/service.py", line 311, in > __init__ > 2019-07-16 18:40:04.258 31644 ERROR nova max_url_len=max_url_len) > 2019-07-16 18:40:04.258 31644 ERROR nova File > "/usr/lib/python2.7/site-packages/nova/wsgi.py", line 101, in __init__ > 2019-07-16 18:40:04.258 31644 ERROR nova self._socket = > eventlet.listen(bind_addr, family, backlog=backlog) > 2019-07-16 18:40:04.258 31644 ERROR nova File > "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in > listen > 2019-07-16 18:40:04.258 31644 ERROR nova sock.bind(addr) > 2019-07-16 18:40:04.258 31644 ERROR nova File > "/usr/lib64/python2.7/socket.py", line 224, in meth > 2019-07-16 18:40:04.258 31644 ERROR nova return > getattr(self._sock,name)(*args) > 2019-07-16 18:40:04.258 31644 ERROR nova error: [Errno 98] Address > already in use > 2019-07-16 18:40:04.258 31644 ERROR > The above having been said the problem appears to be that the address is already in use. You should check netstat or ss to see what is listening on tcp port 8775 and kill that before starting the service again. > > Thanks > __Vasanth From fungi at yuggoth.org Thu Jul 18 17:56:22 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 18 Jul 2019 17:56:22 +0000 Subject: [OpenStack-Infra] [github] Please delete stole github networking-mlnx repo In-Reply-To: References: Message-ID: <20190718175622.hc4d2ih5meaidpyb@yuggoth.org> On 2019-07-18 07:42:58 +0000 (+0000), Lenny Verkhovsky wrote: [...] > Could you please, delete stale project of networking-mlnx [...] At your request, as a core reviewer of the project in OpenDev, I have removed the stale networking-mlnx repository mirror from the openstack organization on GitHub. Please let us know if you need any further assistance! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Mon Jul 22 21:17:45 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 22 Jul 2019 14:17:45 -0700 Subject: [OpenStack-Infra] Team meeting agenda for July 23, 2019 Message-ID: Our next meeting will be July 23, 2019 at 19:00UTC. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Rebuilding Gitea servers to add disk space and swap. * General topics ** Trusty Upgrade Progress (clarkb 20190723) ** Cloud status update (clarkb 20190723) *** FortNebula status update *** Linaro and MOC updates ** PTG Planning (clarkb 20190716) *** Surveys submitted for OpenDev and Gitea *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 * Open discussion From iwienand at redhat.com Thu Jul 25 07:57:44 2019 From: iwienand at redhat.com (Ian Wienand) Date: Thu, 25 Jul 2019 17:57:44 +1000 Subject: [OpenStack-Infra] opendev.org downtime Thu Jul 25 07:00 UTC 2019 Message-ID: <20190725075744.GA13003@fedora19.localdomain> Hello, We received reports of connectivity issues to opendev.org at about 06:30 [1]. After some initial investigation, I could not contact gitea-lb01.opendev.org via ipv4 or 6. Upon checking it's console I saw a range of kernel errors that suggest the host was probably having issues with it's disk [2]. I attempted to hard-reboot it, and it went into an error state. The initial error in the server status was {'message': 'Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainCreateWithFlags)', 'code': 500, 'created': '2019-07-25T07:25:25Z'} After a short period, I tried again and got a different error state {'message': "internal error: process exited while connecting to monitor: lc=,keyid=masterKey0,iv=jHURYcYDkXqGBu4pC24bew==,format=base64 -drive 'file=rbd:volumes/volume-41553c15-6b12-4137-a318-7caf6a9eb44c:id=cinder:auth_supported=cephx\\;none:mon_host=172.24.0.56\\:6789", 'code': 500, 'created': '2019-07-25T07:27:21Z'} The vexxhost status page [3] is currently not showing any outages in the sjc1 region where this resides. I think this probably requires vexxhost to confirm the status of load-balancer VM. I tried to launch a new node, at least to have it ready in case of bigger issues. This failed with errors about the image service [4]. This further suggets there might be some storage issues on the backend. I then checked on the gitea* backend servers, and they have similar messages in their kernel logs referring to storage too (I should have done this first, probably). So this again suggests it is a region-wide issue. I have reached out to mnaser on IRC. I think he is GMT-4 usually so that gives a few hours to expect a response. This will also mean more experienced gitea admins will be around too. Given it appears to be a backend provider issue, I will not take further at this point. Thanks, -i [1] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-07-25.log.html#t2019-07-25T06:36:51 [2] http://paste.openstack.org/show/754834/ [3] https://status.vexxhost.com/ [4] http://paste.openstack.org/show/754835/ From xmzheng at cn.ibm.com Thu Jul 25 08:18:35 2019 From: xmzheng at cn.ibm.com (Xiao Mei GZ Zheng) Date: Thu, 25 Jul 2019 08:18:35 +0000 Subject: [OpenStack-Infra] Can not access the devstack-gate repo. Message-ID: An HTML attachment was scrubbed... URL: From aj at suse.com Thu Jul 25 08:37:38 2019 From: aj at suse.com (Andreas Jaeger) Date: Thu, 25 Jul 2019 10:37:38 +0200 Subject: [OpenStack-Infra] Can not access the devstack-gate repo. In-Reply-To: References: Message-ID: <4f59bb86-8971-ab06-68f6-9bf5483678bf@suse.com> On 7/25/19 10:18 AM, Xiao Mei GZ Zheng wrote: > Hi Infra, >   > Is there any change for the devstack-gate repo? I found this error when > trying to clone the project. >   > > + git clone https://git.openstack.org/openstack-infra/devstack-gateCloning into 'devstack-gate'... > fatal: unable to access 'https://opendev.org/openstack-infra/devstack-gate/': GnuTLS recv error (-110): The TLS connection was non-properly terminated. > This should fixed now, there was a problem in the cloud provider we run the service, Andreas -- Andreas Jaeger aj at suse.com Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 From lei.a.zhang at intel.com Mon Jul 29 15:32:54 2019 From: lei.a.zhang at intel.com (Zhang, Lei A) Date: Mon, 29 Jul 2019 15:32:54 +0000 Subject: [OpenStack-Infra] [github] Request for deleting rsd-lib repo Message-ID: <423D97463F7DE141A44922197628026E4829F9F2@shsmsx102.ccr.corp.intel.com> Hi Infra team, As mentioned in a mail thread[1], infra team could help us to remove stole rsd-lib github repo. We now use Opendev to host code , could you please delete the copy on github[2]. I'm in the admin-list of rsd-release[3] [1] https://www.mail-archive.com/openstack-infra at lists.openstack.org/msg06382.html [2] https://github.com/openstack/rsd-lib [3] https://review.opendev.org/#/admin/groups/1827,members BR Lei Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Mon Jul 29 17:35:47 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 29 Jul 2019 17:35:47 +0000 Subject: [OpenStack-Infra] [github] Request for deleting rsd-lib repo In-Reply-To: <423D97463F7DE141A44922197628026E4829F9F2@shsmsx102.ccr.corp.intel.com> References: <423D97463F7DE141A44922197628026E4829F9F2@shsmsx102.ccr.corp.intel.com> Message-ID: <20190729173547.b6qhsjngfjry5lal@yuggoth.org> On 2019-07-29 15:32:54 +0000 (+0000), Zhang, Lei A wrote: > As mentioned in a mail thread[1], infra team could help us to > remove stole rsd-lib github repo. > > We now use Opendev to host code , could you please delete the copy > on github[2]. > > I'm in the admin-list of rsd-release[3] > > [1] https://www.mail-archive.com/openstack-infra at lists.openstack.org/msg06382.html > [2] https://github.com/openstack/rsd-lib > [3] https://review.opendev.org/#/admin/groups/1827,members I have deleted the old copy of openstack/rsd-lib from GitHub at your request. Let us know if you need anything else. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Mon Jul 29 20:42:27 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 29 Jul 2019 13:42:27 -0700 Subject: [OpenStack-Infra] Agenda for Team Meeting on July 30, 2019 at 19:00UTC Message-ID: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Rebuilding Gitea servers to add disk space and swap. * General topics ** Trusty Upgrade Progress (clarkb 20190730) *** wiki-dev02 puppetry now works? *** Zuul log rendering now ready for review: https://review.opendev.org/#/c/673105/ and parents **** Would allow us to switch to swift hosted logs and delete the log server. ** Cloud status update (clarkb 20190730) *** FortNebula status update *** Linaro and MOC updates ** PTG Planning (clarkb 20190730) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 * Open discussion From corvus at inaugust.com Mon Jul 29 20:52:20 2019 From: corvus at inaugust.com (James E. Blair) Date: Mon, 29 Jul 2019 13:52:20 -0700 Subject: [OpenStack-Infra] [release][infra] Supporting rget in our release process Message-ID: <8736io7ed7.fsf@meyer.lemoncheese.net> Hi, A colleague at Red Hat is working on an effort to record signatures of release artifacts. Essentially it's a way to help users verify release artifacts (or determine if they have been changed) independent of PGP signatures. You can read about it here: https://github.com/merklecounty/rget#rget It sounds like an interesting and useful effort, and I think we can support it at little cost. If we wanted to do so, I think we would need to do the following things: 1) Generate SHA256SUMS of our release artifacts. These could even include the GPG signature files. 2) Run "rget submit" on the resulting files after publication. That's it. Both of those would be changes to the release publication jobs, and wouldn't require any other changes to our processes. As mentioned in the README this is very early stages and the author, Brandon Philips, welcomes both further testing and feedback on the process in general. Thoughts? -Jim From cboylan at sapwetik.org Mon Jul 29 21:52:25 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 29 Jul 2019 14:52:25 -0700 Subject: [OpenStack-Infra] [release][infra] Supporting rget in our release process In-Reply-To: <8736io7ed7.fsf@meyer.lemoncheese.net> References: <8736io7ed7.fsf@meyer.lemoncheese.net> Message-ID: <74d85430-9197-465a-bfcf-a6651b057d2d@www.fastmail.com> On Mon, Jul 29, 2019, at 1:52 PM, James E. Blair wrote: > Hi, > > A colleague at Red Hat is working on an effort to record signatures of > release artifacts. Essentially it's a way to help users verify release > artifacts (or determine if they have been changed) independent of PGP > signatures. You can read about it here: > https://github.com/merklecounty/rget#rget > > It sounds like an interesting and useful effort, and I think we can > support it at little cost. If we wanted to do so, I think we would need > to do the following things: > > 1) Generate SHA256SUMS of our release artifacts. These could even > include the GPG signature files. We'll also need to publish the sha256sums file. We are already publishing the other files so this should be easy. > > 2) Run "rget submit" on the resulting files after publication. > > That's it. There is also a step 0) install rget. Unfortunately their docs don't mention how to verify the installation of rget (self bootstrapping) though I think you would download rget, hash it before running it, then run it to get the hashes of rget and then compare? Though a modified rget could just tell you it was the unmodified version. Maybe that is why they don't bother telling you what to do there. We may also want to set up some sort of periodic audit of our "certs" in the certificate transparency logs. Just to ensure there are no unexpected changes. > > Both of those would be changes to the release publication jobs, and > wouldn't require any other changes to our processes. > > As mentioned in the README this is very early stages and the author, > Brandon Philips, welcomes both further testing and feedback on the > process in general. > > Thoughts? Overall seems like a good way for people (including ourselves) to sanity check that our releases are not changing unexpectedly. > > -Jim > > From fungi at yuggoth.org Mon Jul 29 22:03:03 2019 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 29 Jul 2019 22:03:03 +0000 Subject: [OpenStack-Infra] [release][infra] Supporting rget in our release process In-Reply-To: <8736io7ed7.fsf@meyer.lemoncheese.net> References: <8736io7ed7.fsf@meyer.lemoncheese.net> Message-ID: <20190729220303.ukvhvhzppezy7ufo@yuggoth.org> On 2019-07-29 13:52:20 -0700 (-0700), James E. Blair wrote: > A colleague at Red Hat is working on an effort to record signatures of > release artifacts. Essentially it's a way to help users verify release > artifacts (or determine if they have been changed) independent of PGP > signatures. You can read about it here: > https://github.com/merklecounty/rget#rget [...] > As mentioned in the README this is very early stages and the author, > Brandon Philips, welcomes both further testing and feedback on the > process in general. > > Thoughts? I really like the way it leverages RFC 6962 Certificate Transparency logs for checksum distribution; the decision not to fall into the blockchain-all-things trap lends a lot of additional credibility to the idea. I agree this would be fairly trivial to integrate into our release publication jobs, and even to backfill with our existing archives. It would grant consumers of our release artifacts the ability to validate them against an open third-party registry, separately from checking the cryptographic release signatures we currently provide alongside them... it could even be used to detect tampering with the signatures themselves in the event of a signing key compromise. This seems like a great idea for URLs of artifacts we host, and I'm happy to hack on the implementation in OpenDev's CI system, likely via a new role in Zuul's standard library of job components. For artifacts we upload to third-party services like PyPI and Docker Hub on the other hand, assuming I've digested (pun intended) the relevant literature correctly, it might make more sense for the maintainers of those services to do something similar as they tend to perform a fair amount of URL indirection and so trying to keep up historical data for those URLs ourselves could be tricky. On the other hand if those third-party services were to integrate rget updating as part of their infrastructure it would be a lot more seamless (especially if they similarly integrated CT checks into the corresponding client-side tooling). Another challenge I see is that, due to the fact that most of what we host is source code, and most consumers of our source code are obtaining it via Git rather than release artifacts, rget wouldn't really do much for them as far as I can see... though once Git completes its planned transition to SHA2-256 in the coming years, I could see a call for some solution to publish known object hashes to a CT log in a similar fashion. I suppose it could be done now by re-checksumming all content over a Git object and submitting a certificate for that, but it seems a bit heavy-weight and I'll admit to not having thought through it completely so there are likely hidden gotchas with that idea. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From thierry at openstack.org Tue Jul 30 08:28:59 2019 From: thierry at openstack.org (Thierry Carrez) Date: Tue, 30 Jul 2019 10:28:59 +0200 Subject: [OpenStack-Infra] [release][infra] Supporting rget in our release process In-Reply-To: <20190729220303.ukvhvhzppezy7ufo@yuggoth.org> References: <8736io7ed7.fsf@meyer.lemoncheese.net> <20190729220303.ukvhvhzppezy7ufo@yuggoth.org> Message-ID: <949b37fd-29df-0bc0-b1c7-ba548cf69cef@openstack.org> Jeremy Stanley wrote: > [...] > For artifacts we upload to third-party services like PyPI and Docker > Hub on the other hand, assuming I've digested (pun intended) the > relevant literature correctly, it might make more sense for the > maintainers of those services to do something similar as they tend > to perform a fair amount of URL indirection and so trying to keep up > historical data for those URLs ourselves could be tricky. On the > other hand if those third-party services were to integrate rget > updating as part of their infrastructure it would be a lot more > seamless (especially if they similarly integrated CT checks into the > corresponding client-side tooling). > > Another challenge I see is that, due to the fact that most of what > we host is source code, and most consumers of our source code are > obtaining it via Git rather than release artifacts, rget wouldn't > really do much for them as far as I can see... though once Git > completes its planned transition to SHA2-256 in the coming years, I > could see a call for some solution to publish known object hashes to > a CT log in a similar fashion. I suppose it could be done now by > re-checksumming all content over a Git object and submitting a > certificate for that, but it seems a bit heavy-weight and I'll admit > to not having thought through it completely so there are likely > hidden gotchas with that idea. I agree with Jeremy, it seems to cover a limited amount of use cases (people who download tarball source releases from releases.o.o). But covering only a few use cases is not a reason not to do it: we should support it for the same reason we provide signatures for released artifacts right now. Furthermore it is an initiative I'm fine being early adopters of this idea, if only so that one day we may find it covering other ways to retrieve our deliverables (pypi, git). -- Thierry Carrez (ttx) From mordred at inaugust.com Tue Jul 30 14:03:03 2019 From: mordred at inaugust.com (Monty Taylor) Date: Tue, 30 Jul 2019 10:03:03 -0400 Subject: [OpenStack-Infra] [release][infra] Supporting rget in our release process In-Reply-To: <74d85430-9197-465a-bfcf-a6651b057d2d@www.fastmail.com> References: <8736io7ed7.fsf@meyer.lemoncheese.net> <74d85430-9197-465a-bfcf-a6651b057d2d@www.fastmail.com> Message-ID: <2880293e-dab3-6730-396c-3fc383d80d44@inaugust.com> On 7/29/19 5:52 PM, Clark Boylan wrote: > On Mon, Jul 29, 2019, at 1:52 PM, James E. Blair wrote: >> Hi, >> >> A colleague at Red Hat is working on an effort to record signatures of >> release artifacts. Essentially it's a way to help users verify release >> artifacts (or determine if they have been changed) independent of PGP >> signatures. You can read about it here: >> https://github.com/merklecounty/rget#rget >> >> It sounds like an interesting and useful effort, and I think we can >> support it at little cost. If we wanted to do so, I think we would need >> to do the following things: >> >> 1) Generate SHA256SUMS of our release artifacts. These could even >> include the GPG signature files. > > We'll also need to publish the sha256sums file. We are already publishing the other files so this should be easy. > >> >> 2) Run "rget submit" on the resulting files after publication. >> >> That's it. > > There is also a step 0) install rget. Unfortunately their docs don't mention how to verify the installation of rget (self bootstrapping) though I think you would download rget, hash it before running it, then run it to get the hashes of rget and then compare? Though a modified rget could just tell you it was the unmodified version. Maybe that is why they don't bother telling you what to do there. Actually - it turns out this is just doing a single submit to a URL, so it will likely be much easier to just use curl or an ansible URI call to do the submission step. (it's great that rget has this for folks, but in our case I think we don't need to use rget just to do the submission) > We may also want to set up some sort of periodic audit of our "certs" in the certificate transparency logs. Just to ensure there are no unexpected changes. ++ >> >> Both of those would be changes to the release publication jobs, and >> wouldn't require any other changes to our processes. >> >> As mentioned in the README this is very early stages and the author, >> Brandon Philips, welcomes both further testing and feedback on the >> process in general. >> >> Thoughts? > > Overall seems like a good way for people (including ourselves) to sanity check that our releases are not changing unexpectedly. > >> >> -Jim >> >> > >