- openstack-discuss - lists.openstack.org

[magnum] Gerrit breach and commit audit
by feilong 21 Oct '20

21 Oct '20

Hi everyone, As per the announcement of the Gerrit breach [1], I have audited all repos under Magnum (magnum, python-magnumclient, magnum-ui and magnum-tempest-plugin) for 1. Suspicious commit 2. Core group list Everything looks OK for me, see https://etherpad.opendev.org/p/magnum-gerrit-breach-audit [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html -- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------

1 0

Keystone auth method v3oidcpassword throws RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
by Robert Duncan 21 Oct '20

21 Oct '20

I'm trying to use openID with openstack --versionopenstack 5.2.0 keystoneauth1==3.17.3 keystonemiddleware==7.0.1 python-keystoneclient==3.21.0 I have created an RC file like so : export OS_INTERFACE="public" export OS_AUTH_URL="https://openstack:5000/v3" export OS_IDENTITY_PROVIDER="<my_idp>" export OS_PROTOCOL="openid" export OS_CLIENT_ID="<the -id >" export OS_CLIENT_SECRET="<the secret>" export OS_DISCOVERY_ENDPOINT="https://login.microsoftonline.com/<the-tenant-id>/v2.0/.well-known/openid-configuration" export OS_IDENTITY_API_VERSION="3" export OS_AUTH_TYPE="v3oidcpassword" export OS_USERNAME="<AzureAD user>" # this is the local openstack project id export OS_PROJECT_ID="<The project ID>" # set password by querying user export OS_PASSWORD="" echo "Please enter your O365 Password: " read -sr OS_PASSWORD_INPUT export OS_PASSWORD=$OS_PASSWORD_INPUT all is working on Horizon WebSSO- but I cannot access the API the openstack --debug output shows that I get a token from MS debug info (redacted) command: token issue -> openstackclient.identity.v3.token.IssueToken (auth=True) Auth plugin v3oidcpassword selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'project_id': '516706052ebd4f9a89c0b7d7e075754d'}, 'additional_user_agent': [('osc-lib', '2.0.0')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'auth_url': 'https://openstack.*********:5000/v3', 'username': 'r********', 'password': '***', 'identity_provider': '*****', 'protocol': 'openid', 'client_id': '*******', 'client_secret': '***', 'discovery_endpoint': 'https://login.microsoftonline.com/*******/v2.0/.well-known/openid-configura…', 'beta_command': False, 'identity_api_version': '3', 'data_processing_api_version': '1.1', 'container_infra_api_version': '1', 'region_name': '', 'auth_type': 'v3oidcpassword', 'networks': []} Using auth plugin: v3oidcpassword Using parameters {'auth_url': 'https://openstack.******:5000/v3', 'project_id': '516706052ebd4f9a89c0b7d7e07575, 'identity_provider': '*****', 'protocol': 'openid', 'client_id': '********', 'client_secret': '***', 'discovery_endpoint': 'https://login.microsoftonline.com/*******/v2.0/.well-known/openid-configura…', 'username': '*****', 'password': '***'} Get auth_ref REQ: curl -g -i -X GET https://login.microsoftonline.com/*****/v2.0/.well-known/openid-configurati… -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.2" Starting new HTTPS connection (1): login.microsoftonline.com:443 https://login.microsoftonline.com:443 "GET /******/v2.0/.well-known/openid-configuration HTTP/1.1" 200 1651 RESP: [200] Access-Control-Allow-Methods: GET, OPTIONS Access-Control-Allow-Origin: * Cache-Control: max-age=86400, private Content-Length: 1651 Content-Type: application/json; charset=utf-8 Date: Wed, 21 Oct 2020 09:37:13 GMT P3P: CP="DSP CUR OTPi IND OTRi ONL FIN" Set-Cookie: fpc=AhupGfLmUQlEsKNlR-m0ATg; expires=Fri, 20-Nov-2020 09:37:14 GMT; path=/; secure; HttpOnly; SameSite=None, esctx=*****: domain=.login.microsoftonline.com; path=/; secure; HttpOnly; SameSite=None, x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly, stsservicecookie=estsfd; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff x-ms-ests-server: 2.1.11154.9 - NEULR2 ProdSlices x-ms-request-id: 0819033b-dcb3-409a-8115-54dc06a31900 RESP BODY: {"token_endpoint":"https://login.microsoftonline.com/*****/oauth2/v2.0/token","token_endpoint_auth_methods_supported":["client_secret_post","private_key_jwt","client_secret_basic"],"jwks_uri":"https://login.microsoftonline.com/****/discovery/v2.0/keys","response_modes_supported":["query","fragment","form_post"],"subject_types_supported":["pairwise"],"id_token_signing_alg_values_supported":["RS256"],"response_types_supported":["code","id_token","code id_token","id_token token"],"scopes_supported":["openid","profile","email","offline_access"],"issuer":"https://login.microsoftonline.com/****/v2.0","request_uri_parameter_supported":false,"userinfo_endpoint":"https://graph.microsoft.com/oidc/userinfo","authorization_endpoint":"https://login.microsoftonline.com/****/oauth2/v2.0/authorize","device_authorization_endpoint":"https://login.microsoftonline.com/*****/oauth2/v2.0/devicecode","http_logout_supported":true,"frontchannel_logout_supported":true,"end_session_endpoint":"https://login.microsoftonline.com/****/oauth2/v2.0/logout","claims_supported":["sub","iss","cloud_instance_name","cloud_instance_host_name","cloud_graph_host_name","msgraph_host","aud","exp","iat","auth_time","acr","nonce","preferred_username","name","tid","ver","at_hash","c_hash","email"],"tenant_region_scope":"EU","cloud_instance_name":"microsoftonline.com","cloud_graph_host_name":"graph.windows.net","msgraph_host":"graph.microsoft.com","rbac_url":"https://pas.windows.net"} REQ: curl -g -i -X POST https://login.microsoftonline.com/******/oauth2/v2.0/token -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.2" -d '{'username': '*****', 'password': '*****', 'scope': 'openid profile', 'grant_type': 'password'}' https://login.microsoftonline.com:443 "POST /*****/oauth2/v2.0/token HTTP/1.1" 200 3764 RESP: [200] Cache-Control: no-store, no-cache Content-Length: 3764 Content-Type: application/json; charset=utf-8 Date: Wed, 21 Oct 2020 09:37:14 GMT Expires: -1 P3P: CP="DSP CUR OTPi IND OTRi ONL FIN" Pragma: no-cache Set-Cookie: fpc=****; expires=Fri, 20-Nov-2020 09:37:14 GMT; path=/; secure; HttpOnly; SameSite=None, x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly, stsservicecookie=estsfd; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff x-ms-ests-server: 2.1.11154.9 - WEULR1 ProdSlices x-ms-request-id: 92d66fde-6763-4be2-b1cf-00fd18bc1800 RESP BODY: {"token_type":"Bearer","scope":"email openid profile 00000003-0000-0000-c000-000000000000/User.Read","expires_in":3599,"ext_expires_in":3599,"access_token":"<the access token>","id_token":"<the id token>"} REQ: curl -g -i -X POST https://openstack***:5000/v3/OS-FEDERATION/identity_providers/***/protocols… -H "Authorization: {SHA256}aff774c8663ee647a08aefcea698f8c30e1d1cc8d85a3d55a17cae53defd8955" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.2" Starting new HTTPS connection (1): openstack.****:5000 https://openstack.***:5000 "POST /v3/OS-FEDERATION/identity_providers/****/protocols/openid/auth HTTP/1.1" 200 541 RESP: [200] Content-Length: 541 Content-Type: text/html Date: Wed, 21 Oct 2020 09:37:14 GMT Server: Apache RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged. Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cliff/app.py", line 393, in run_subcommand self.prepare_to_run_command(cmd) File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 493, in prepare_to_run_command self.client_manager.auth_ref File "/usr/lib/python3/dist-packages/osc_lib/clientmanager.py", line 202, in auth_ref self._auth_ref = self.auth.get_auth_ref(self.session) File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/federation.py", line 65, in get_auth_ref auth_ref = self.get_unscoped_auth_ref(session) File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/oidc.py", line 265, in get_unscoped_auth_ref return access.create(resp=response) File "/usr/lib/python3/dist-packages/keystoneauth1/access/access.py", line 36, in create body = resp.json() File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) what's gone wrong? thanks, Rob. ________________________________ The information contained and transmitted in this e-mail is confidential information, and is intended only for the named recipient to which it is addressed. The content of this e-mail may not have been sent with the authority of National College of Ireland. Any views or opinions presented are solely those of the author and do not necessarily represent those of National College of Ireland. If the reader of this message is not the named recipient or a person responsible for delivering it to the named recipient, you are notified that the review, dissemination, distribution, transmission, printing or copying, forwarding, or any other use of this message or any part of it, including any attachments, is strictly prohibited. If you have received this communication in error, please delete the e-mail and destroy all record of this communication. Thank you for your assistance. ________________________________

2 6

[nova] Gerrit breach and commit audit
by Ghanshyam Mann 21 Oct '20

21 Oct '20

Hello Everyone, As per the announcement of the Gerrit breach [1], I have audited all repos under Nova for 1. Suspicious commit 2. Core group list and everything is fine. - https://etherpad.opendev.org/p/nova-gerrit-breach-audit [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[designate] Gerrit breach and commit audit
by Nicolas Bock 21 Oct '20

21 Oct '20

Hi, Herve's script [1] identified the following commits in the designate repository. I second Michael's assessment from earlier that all of those commits are legitimate. Best, Nick Commit to analyze: https://review.opendev.org/#/q/8459ed9460ab492adfd305875c717ebe31bf68f3 Files to analyze: - lower-constraints.txt ------------------------------- Commit to analyze: https://review.opendev.org/#/q/568b559b67b0d9a663172d681fe887bf06feea0a Files to analyze: - tox.ini ------------------------------- Commit to analyze: https://review.opendev.org/#/q/0d277e9cbb42404b1b1082afdfa5bcfc49691e6f Files to analyze: - lower-constraints.txt ------------------------------- Commit to analyze: https://review.opendev.org/#/q/53dcbe8584434c21b1a340763e9ee419dc58117f Files to analyze: - .zuul.yaml ------------------------------- Commit to analyze: https://review.opendev.org/#/q/348771951fa6e9740b629e81227cc0311751a471 Files to analyze: - lower-constraints.txt ------------------------------- Commit to analyze: https://review.opendev.org/#/q/1831a5b1016563681df6905114f0dc95efd24391 Files to analyze: - designate/locale/ko_KR/LC_MESSAGES/designate.po ------------------------------- Commit to analyze: https://review.opendev.org/#/q/19da5d79d5d24dfda31773f70971f640856b10b1 Files to analyze: - releasenotes/source/locale/en_GB/LC_MESSAGES/releasenotes.po ------------------------------- Commit to analyze: https://review.opendev.org/#/q/a0d20514c6ebbfe6b42b989b6192ae77312d3ba7 Files to analyze: - lower-constraints.txt ------------------------------- [1] https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra…

1 0

[qa] Gerrit breach and commit audit
by Ghanshyam Mann 21 Oct '20

21 Oct '20

Hello Everyone, As per the announcement of the Gerrit breach [1], I have audited all repos under QA for 1. Suspicious commit 2. Core group list and everything is fine. - https://etherpad.opendev.org/p/qa-gerrit-breach-audit [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html -gmann

1 0

[manila] Gerrit breach and commit audit
by Goutham Pacha Ravi 21 Oct '20

21 Oct '20

Hello Zorillas, I join the list of contributors to thank the infra team for their work in isolating, protecting and fixing the recent vulnerability. I looked through the code changes committed to repositories we maintain and didn't find any malicious code submissions or approvals. If you would like to take a look, here is an Ethercalc with the changes: https://ethercalc.openstack.org/uscp1x7i8ewn Please do respond here in case you notice anything amiss. Thank you! Goutham ---------- Forwarded message --------- From: Ian Wienand <iwienand(a)redhat.com> Date: Tue, Oct 20, 2020 at 5:33 PM Subject: [service-announce] October 20 Gerrit Outage Update To: <service-announce(a)lists.opendev.org> As of this mail, Gerrit access has been restored. Please read on for important information, especially around change verification. Background ----------- On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval to a change that they were not expected to have access to. At 02:06 UTC an alert was raised via IRC. Administrators found the account had added themselves to a core group and made the +W vote. The account was disabled, and removed from the groups it had added itself to by 02:55 UTC. Administrators began to analyse the situation and Gerrit was taken offline at 04:02 UTC to preserve state and allow for analysis. >From this time, administrators were working on log collection and analysis, along with restoring backups for comparison purposes. By around 08:45 UTC it was clear that the privilege escalation had been achieved by gaining control of a Launchpad SSO account with Gerrit administrator privileges. By this time, we had ruled out software vulnerabilities. Logs showed the first unauthorized access of the administrator account in Gerrit on 2020-10-06. Communication with Launchpad admins agrees with this analysis. We saw one session opened as the administrator user to StoryBoard on this same day, but logs show no data was modified or hidden stories viewed. Analysis has been performed on the Gerrit database and git trees from October 1st, pre-dating any known unauthorized access. Access was restored at around 2020-10-21 00:00 UTC Outcomes ----------- The following has been verified: The administrator account used has been disabled and credentials updated We have verified that all group and user addition/removals since Oct 1 are valid. The only invalid additions were made by the compromised administrator account to add a single user account to the Administrators group; and then that account added itself to another known group. The account given administrator privilege has been removed from the groups it added itself to and is disabled. There is no evidence of any unauthorized access via methods other than Gerrit HTTP and Gerrit SSH access. No commits have been pushed to git trees bypassing code review. Every git tree has been compared to the Oct 1 version and all commits have been correctly inserted via Gerrit changes. The version of Gerrit we use stores HTTP API passwords in plain-text. We know that a limited number of passwords were gathered via the HTTP API and it is possible passwords were gathered via the database. We thus have assumed that all HTTP API passwords have been disclosed. This password needs to be explicitly enabled by users, and many users do not have it enabled. Remediation ----------- This leaves us with the following remediation actions: Users should double-check their Launchpad recent activity at https://login.launchpad.net/activity for any suspicious logins. If found, please notify the OpenDev admins in Freenode #opendev and Launchpad admins in #launchpad immediately. All HTTP API passwords have been cleared. If you push changes via HTTPS (instead of typical SSH), are a gertty user, or run a CI system or something else that communicates with the Gerrit HTTP API, you will need to regenerate a password. Any SSH keys added to accounts since 2020-10-01 have been removed. This affects only a limited number of accounts. This is done in an abundance of caution, and we do not believe any accounts had unauthorized SSH keys added We should audit all changes for projects since 2020-10-01. We have no evidence that any account had its ssh keys compromised, thus we can rule out any unauthorized changes being uploaded via SSH. However we can not conclusively rule out that compromised HTTP API passwords were used to push a change through Gerrit. For example, a change could be uploaded that looks like it came from a user, or the API key of a core team member may have been used to approve a change without authorization. Given our extensive analysis we consider it exceedingly unlikely that this vector was used. We have had no notifications of users seeing unexpected changes either uploaded by them, or approved by them in projects they work on. This said, we believe it is important to inform the community of this very unlikely, but still possible, vulnerability of the source code. To this end, we have prepared a list of all changes from the known affected period which should be audited for correctness. These are available at https://static.opendev.org/project/opendev.org/gerrit-diffs/ Team members should browse these changes and make sure they were correctly approved in Gerrit. If any change looks suspicious you should notify OpenDev administrators in Freenode #opendev immediately. Further actions ---------------- We are planning the following for the short term future: The Opendev administrators will be looking at alternative models for Gerrit admin account management. We are already well into planning and testing a coming upgrade to a version of Gerrit which does not store plain-text API keys. Longer term, we've written a spec for replacing Launchpad SSO as our authentication provider. We thank you for your patience during this trying time, and we look forward to returning to supporting the community doing what it does best -- working together to create great things. _______________________________________________ service-announce mailing list service-announce(a)lists.opendev.org http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

1 0

[neutron] Same physical NIC for SR-IOV and other mechanism driver
by Gabriel Omar Gamero Montenegro 21 Oct '20

21 Oct '20

Dear all, Is it possible to use the same physical NIC with the SR-IOV mechanism driver together with other mechanism driver like Open vSwitch or Linux Bridge? If this is the case, how should I map the physical NIC for each agent? I only known how to map the NIC separately for each driver; for example: - for the Linux bridge agent: physnet1: eth3, - for the OvS agent: physnet1: br-provider (eth3 is a port of br-provider) - for the SR-IOV agent: physnet1: eth3 Any guide or sample reference about implementing multiple mechanism drivers would be highly appreciated. Thanks in advance, Gabriel Gamero

2 1

Octavia amphora missing network interface
by Dibbo, Alexander (STFC,RAL,SC) 21 Oct '20

21 Oct '20

Hi All, I'm having an issue with my Octavia deployment in Train. When a load balancer is created I can see the amphora get created with an interface on the management network. After a short while an interface is added for our external network. The external network interface never becomes responsive. If I ssh in to the amphora I can only see the management interface present. If I the reboot the amphora, the external interface appears and becomes responsive. Given this it seems like some kind of race condition. I have tried using all of the test amphora images that are published but the centos8 one seems the closest to working. Is anyone familiar with this issue or a fix? Thanks Alex Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support(a)gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

3 3

[designate][octavia] Gerrit breach and commit audit - OK
by Michael Johnson 21 Oct '20

21 Oct '20

I did not see any unusual commits on the lists for either the Octavia or Designate repositories. Thanks, Michael

1 0

[charms] Gerrit breach and commit audit
by Frode Nordahl 21 Oct '20

21 Oct '20

Hello all, On the back of the service announcement [0] from OpenDev admins this morning the OpenStack Charms team have performed an audit of the provided commit blobs. We have found everything to be in order for the repositories we manage. The sheet used for tracking has been made available for review [1]. I would like to thank everyone on the team for the prompt response in dealing with this issue, and we want to thank OpenDev for due diligence in handling the situation. Cheers! 0: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html 1: https://docs.google.com/spreadsheets/d/1jxswC09mPCrtbAchNQBH5_RO95CTEOJ6VUr… -- Frode Nordahl

1 0

Fwd: [ops] [cinder] __DEFAULT__ volume type
by Rajat Dhasmana 21 Oct '20

21 Oct '20

---------- Forwarded message --------- From: Rajat Dhasmana <rdhasman(a)redhat.com> Date: Sat, Oct 17, 2020 at 12:31 AM Subject: Re: [ops] [cinder] __DEFAULT__ volume type To: Massimo Sgaravatto <massimo.sgaravatto(a)gmail.com> Hi Massimo, On Fri, Oct 16, 2020 at 7:42 PM Massimo Sgaravatto < massimo.sgaravatto(a)gmail.com> wrote: > I have recently updated my Cloud from Rocky to Train (I am running Cinder > v. 15.4.0) > I have a question concerning the __DEFAULT__ volume type, that I don't > remember to have seen before. > > Since Train, cinder has decided to discourage untyped volumes (volumes with None volume type) as it doesn't add any value to the volume attributes. To achieve this, we created the __DEFAULT__ type and migrated all untyped volumes to have the __DEFAULT__ type. > Since: > - I have no volumes using this volume type > - I defined in in the [DEFAULT] section of cinder.conf the attribute > "default_volume_type" to a value different than "__DEFAULT___" > > I assume that I can safely delete the __DEFAULT__ volume type > > Is this correct? > > Yes, as long as there are no volumes using it and there is a valid volume type defined with ``default_volume_type`` config in cinder.conf, you can safely delete it. NOTE: ``default_volume_type`` has a default value of ``__DEFAULT__`` to not allow users to further create untyped volumes, if you are deleting this, make sure there is always a valid volume type defined with ``default_volume_type`` in cinder.conf > Thanks, Massimo > Thanks and Regards Rajat Dhasmana

3 5

[glance] Gerrit breach and commit audit
by Abhishek Kekane 21 Oct '20

21 Oct '20

Hi All, As suggested by the recent announcement of gerrit breach [1], I have audited all commits for glance repo (glance, glance_store, glanceclient) and confirmed that everything looks good. [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html Thanks & Best Regards, Abhishek Kekane

1 0

[cinder] Gerrit breach and commit audit
by Brian Rosmaita 21 Oct '20

21 Oct '20

tl;dr The cinder deliverables repos look OK. Thanks to jungleboyj and smcginnis for help in looking them over. As you may have heard, there was a Gerrit breach in the 1 October to yesterday time frame. Here's a link to the analysis of the breach: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html The infra team has cleared all Gerrit HTTP passwords and removed any ssh keys that were added after the breach. If you are super-paranoid, feel free to look over the commits to any of the cinder deliverables during the time frame of the breach. You can find them here: https://static.opendev.org/project/opendev.org/gerrit-diffs/ Like I said, some of us have looked them over and not noticed anything suspect. What you'd be looking for is that an approval you've made or your +2 vote on an approved patch may have been put there by someone other than you. So if you see your +2 or +W on a patch that you have no memory of actually looking at, that would be considered suspicious activity. (On many levels.) If you notice such a situation, please notify us immediately. cheers, brian

1 0

[Telemetry] Gerrit breach and commit audit
by Matthias Runge 21 Oct '20

21 Oct '20

Hi there, you may have seen the announcement of the gerrit breach[1]. I went through the changes we had in telemetry and I'm confident we were not affected. In any case, if you happen to see something suspicious, please let me know. Matthias [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[nova][live-migration] Nova live-migration fails due to duplicate records in ml2_port_bindings table
by Zhi CZ Chang 21 Oct '20

21 Oct '20

2 1

[horizon] No meetings during the Summit and PTG
by Ivan Kolodyazhny 21 Oct '20

21 Oct '20

He team, As agreed on the last weekly meeting, we'll skip the next two meetings to allow everybody attend OpenInfra Summit and PTG. The next meeting will be on November, 4th Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

1 0

[Ussuri] [horizon] Cannot Create Port with Fixed IP Address
by Lazuardi Nasution 21 Oct '20

21 Oct '20

Hi, With Ussuri on Ubuntu 20.04, I can create port with fixed IP address by CLI but I cannot do the same by Horizon GUI. I find some error like following on /var/log/apache2/error.log openstack_dashboard.dashboards.project.networks.ports.workflows Failed to create a port for network 91f04dfb-7f69-4050-8b3b-142ee555ae55: dictionary keys changed during iteration By more inspection, I can find that horizon never send that create port request to neutron. So I think it is horizon problem. Is this expected result or is this horizon bug? Is this related to policy? Best regards,

2 2

Re: packstack fails every time - pymysql.err.OperationalError: (1040, 'Too many connections')
by lejeczek 21 Oct '20

21 Oct '20

On 20/10/2020 11:02, Javier Pena wrote: > > Hi guys. > I'm trying to deploy with "packstack --allinone" and > it always (tried ~5 times) fails with: > ... > ERROR : Error appeared during Puppet > run:A.B.C.D_compute.pp > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > pymysql.err.OperationalError: (1040, 'Too many > connections') > You will find full trace in log > /var/tmp/packstack/20201018-183415-ik3ijyw7/manifests/172.24.154.202_compute.pp.log > Please check log file > /var/tmp/packstack/20201018-183415-ik3ijyw7/openstack-setup.log > for more information > > > If some expert might be interested then in > /var/tmp/packstack/20201018-183415-ik3ijyw7/manifests/172.24.154.202_compute.pp.log > .... > Notice: > /Stage[main]/Nova::Compute::Libvirt/Nova_config[DEFAULT/compute_driver]/ensure: > created > Notice: > /Stage[main]/Nova::Compute::Libvirt/Nova_config[vnc/server_listen]/ensure: > created > Notice: > /Stage[main]/Nova::Compute::Libvirt/Nova_config[libvirt/virt_type]/ensure: > created > Notice: > /Stage[main]/Nova::Compute::Libvirt/Nova_config[libvirt/cpu_mode]/ensure: > created > Notice: > /Stage[main]/Nova::Compute::Libvirt/Nova_config[libvirt/inject_password]/ensure: > created > Notice: > /Stage[main]/Nova::Compute::Libvirt/Nova_config[libvirt/inject_key]/ensure: > created > Notice: > /Stage[main]/Nova::Compute::Libvirt/Nova_config[libvirt/inject_partition]/ensure: > created > Notice: > /Stage[main]/Nova::Migration::Libvirt/Nova_config[libvirt/live_migration_uri]/ensure: > created > Notice: > /Stage[main]/Nova::Compute::Libvirt::Services/Service[libvirt]/ensure: > ensure changed 'stopped' to 'running' > Notice: > /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: > Triggered 'refresh' from 2 events > Notice: > /Stage[main]/Nova::Deps/Anchor[nova::config::end]: > Triggered 'refresh' from 22 events > Notice: > /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: > Triggered 'refresh' from 1 event > Notice: > /Stage[main]/Nova::Compute/Nova::Generic_service[compute]/Service[nova-compute]/ensure: > ensure changed 'stopped' to 'running' > Notice: > /Stage[main]/Nova::Deps/Anchor[nova::service::end]: > Triggered 'refresh' from 1 event > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > An error has occurred: > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > Traceback (most recent call last): > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", > line 2262, in _wrap_pool_connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return fn() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 294, in unique_connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return _ConnectionFairy._checkout(self) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 751, in _checkout > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > fairy = _ConnectionRecord.checkout(pool) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 483, in checkout > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > rec = pool._do_get() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", > line 138, in _do_get > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self._dec_overflow() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", > line 68, in __exit__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > compat.reraise(exc_type, exc_value, exc_tb) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", > line 129, in reraise > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > raise value > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", > line 135, in _do_get > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return self._create_connection() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 299, in _create_connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return _ConnectionRecord(self) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 428, in __init__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self.__connect(first_connect_check=True) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 630, in __connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > connection = pool._invoke_creator(self) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", > line 114, in connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return dialect.connect(*cargs, **cparams) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", > line 453, in connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return self.dbapi.connect(*cargs, **cparams) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/__init__.py", > line 94, in Connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return Connection(*args, **kwargs) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 325, in __init__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self.connect() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 599, in connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self._request_authentication() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 861, in _request_authentication > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > auth_packet = self._read_packet() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 684, in _read_packet > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > packet.check_error() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/protocol.py", > line 220, in check_error > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > err.raise_mysql_exception(self._data) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/err.py", > line 109, in raise_mysql_exception > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > raise errorclass(errno, errval) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > pymysql.err.OperationalError: (1040, 'Too many > connections') > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > The above exception was the direct cause of the > following exception: > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > Traceback (most recent call last): > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/nova/cmd/manage.py", > line 2777, in main > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > ret = fn(*fn_args, **fn_kwargs) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/nova/cmd/manage.py", > line 1144, in discover_hosts > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > by_service) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/nova/objects/host_mapping.py", > line 256, in discover_hosts > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > cell_mappings = objects.CellMappingList.get_all(ctxt) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", > line 184, in wrapper > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > result = fn(cls, context, *args, **kwargs) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/nova/objects/cell_mapping.py", > line 256, in get_all > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > db_mappings = cls._get_all_from_db(context) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", > line 1013, in wrapper > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > with self._transaction_scope(context): > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File "/usr/lib64/python3.6/contextlib.py", line 81, in > __enter__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return next(self.gen) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", > line 1063, in _transaction_scope > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > context=context) as resource: > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File "/usr/lib64/python3.6/contextlib.py", line 81, in > __enter__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return next(self.gen) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", > line 658, in _session > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > bind=self.connection, mode=self.mode) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", > line 418, in _create_session > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self._start() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", > line 510, in _start > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > engine_args, maker_args) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", > line 534, in _setup_for_connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > sql_connection=sql_connection, **engine_kwargs) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/debtcollector/renames.py", > line 43, in decorator > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return wrapped(*args, **kwargs) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", > line 201, in create_engine > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > test_conn = _test_connection(engine, max_retries, > retry_interval) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", > line 376, in _test_connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return engine.connect() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", > line 2193, in connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return self._connection_cls(self, **kwargs) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", > line 103, in __init__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > else engine.raw_connection() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", > line 2293, in raw_connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self.pool.unique_connection, _connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", > line 2266, in _wrap_pool_connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > e, dialect, self > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", > line 1534, in _handle_dbapi_exception_noconnection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > util.raise_from_cause(newraise, exc_info) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", > line 383, in raise_from_cause > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > reraise(type(exception), exception, tb=exc_tb, > cause=cause) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", > line 128, in reraise > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > raise value.with_traceback(tb) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", > line 2262, in _wrap_pool_connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return fn() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 294, in unique_connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return _ConnectionFairy._checkout(self) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 751, in _checkout > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > fairy = _ConnectionRecord.checkout(pool) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 483, in checkout > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > rec = pool._do_get() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", > line 138, in _do_get > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self._dec_overflow() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", > line 68, in __exit__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > compat.reraise(exc_type, exc_value, exc_tb) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", > line 129, in reraise > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > raise value > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", > line 135, in _do_get > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return self._create_connection() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 299, in _create_connection > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return _ConnectionRecord(self) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 428, in __init__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self.__connect(first_connect_check=True) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", > line 630, in __connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > connection = pool._invoke_creator(self) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", > line 114, in connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return dialect.connect(*cargs, **cparams) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", > line 453, in connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return self.dbapi.connect(*cargs, **cparams) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/__init__.py", > line 94, in Connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > return Connection(*args, **kwargs) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 325, in __init__ > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self.connect() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 599, in connect > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > self._request_authentication() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 861, in _request_authentication > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > auth_packet = self._read_packet() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/connections.py", > line 684, in _read_packet > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > packet.check_error() > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/protocol.py", > line 220, in check_error > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > err.raise_mysql_exception(self._data) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > File > "/usr/lib/python3.6/site-packages/pymysql/err.py", > line 109, in raise_mysql_exception > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > raise errorclass(errno, errval) > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > sqlalchemy.exc.OperationalError: > (pymysql.err.OperationalError) (1040, 'Too many > connections') > Notice: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: > (Background on this error at: http://sqlalche.me/e/e3q8) > Error: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]: > Failed to call refresh: 'nova-manage cell_v2 > discover_hosts' returned 255 instead of one of [0] > Error: > /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]: > 'nova-manage cell_v2 discover_hosts' returned 255 > instead of one of [0] > > Hi, > > This might happen if you have a lot of CPUs in your > machine, since Packstack will spawn by default 1 worker > per CPU on every service. In some cases, that is too much > for the default MariaDB open file parameters. > > Can you try adding "--service-workers=2" to your packstack > command-line? Alternatively, you can set the > CONFIG_SERVICE_WORKERS option in your answer file. > Yes! that seems to be the fix for my. Lots of cores in the box is my case. > Regards, > Javier > > Bit discouraging to fail at start but I trust you can > share some good advice. > I'm on Centos8 with ussuri. > Many thanks, L. > >

1 0

[all][tc] Re: divisive language discussion
by Brian Rosmaita 21 Oct '20

21 Oct '20

Hi Ryan, Thank you for your thoughtful email. Apologies for top posting, but I simply want to make a general observation. Hegel postulated a master/slave, not a master/apprentice, dialectic. I think Western culture, at least, has been tainted to the point where on a word association test, the most common association to "master" would be "slave", even if you took the test in a recording studio. (But that's an empirical question and I haven't done any research, it's just a guess.) But what it comes down to for me, is that "master" in "master tape", "headmaster", "master of my domain", etc., is being used metaphorically and not accurately. A "master tape" isn't really a master of all the other tapes, etc. I think the "master" metaphor encourages sloppy thinking, and here I do have some empirical evidence. Take a look at the discussion on this OpenStack Glance review from 2017: https://review.opendev.org/#/c/424752 The original code was using master/slave, and everybody sort of understood what was going on, but it was not in fact accurate, and in addition to purging Hegelianism from Glance, the patch does improve the code and make the relationship more clear. So what's not to like? I don't have a problem removing language whose related metaphors are offensive or exclusionary if it results in a better technical solution. I don't think anyone is proposing that we put up general patches simply replacing 'master' with some other term; the ask is that we do this in a thoughtful way (which will be fought out in gerrit in the usual manner). At least that's how I look at this issue. One more thing and then I will shut up. OpenStack is written in Python partially because it was seen as having a lower barrier to entry than other languages, and would encourage participation and make the community more inclusive. Thus the TC or other appropriate body saying that it's preferred not to use wording identified as oppressive, racist, and sexist by members of our communities doesn't strike me as significantly different from core contributors asking for revisions on patches that write Python code in a non-Pythonic way (that is, using code patterns common in some other programming language, but which are non-idiomatic for Python). I don't mean to minimize your censorship concerns, but I do think they are misplaced. cheers, brian On 10/21/20 12:01 AM, Ryan Taylor wrote: > Hello, > > I was unable to join the session on divisive language today, and there > did not seem to be a recording of the video session, but I wanted to > share some thoughts. I hope you will forgive me for contacting this > email address from the etherpad page. > > Fundamentally, all meaning is symbolic and relational. A word or concept > only has meaning, and can only be defined, in terms of other words and > concepts. That is to say, the meaning of words is highly context-dependent. > > To illustrate my point, please consider the word "owner", and ownership > as a concept: > - generally inoffensive and innocuous > - it is an important meaning that we must be able to express in a > functional language in order to communicate with each other > - has different connotations in different contexts (like most words), > most of which are unrelated to slavery > - not realistically censorable (more to the point, it would be misguided > and futile to attempt to remove it from use) > > However, a slave master and a slave owner are the same thing. Every > logical argument for or against censorship of the word "master" applies > equally to the word "owner", but a reasonable rational person would not > consider "owner" to be an objectionable word. Please genuinely > contemplate this. You may get that "ick" feeling thinking about > ownership of a human being while simultaneously recognizing that this > does not irrevocably taint the word "owner" or the concept of ownership > in general. Careful consideration of this point will reveal that it is > not the words "owner" or "master", nor the concepts of ownership or > mastery, that are problematic - it is the /context/ of these concepts in > relation to slavery that is problematic, when what is owned or mastered > is a human being. > > To further illustrate this point, please consider e.g. master tape, > master copy, digital masters > (https://en.wikipedia.org/wiki/Digital_master > <https://en.wikipedia.org/wiki/Digital_master>), audio mastering, master > of ceremonies, master key, master of arts or science, postmaster, > headmaster/schoolmaster, mastery of a language or musical instrument, > etc. These terms use the general meaning of "master" or "mastery", > which is to have control, authority, importance, or skill, and are not > related to slavery. > > We work in technical and knowledge-based fields. We need concise, rich > technical language to be able to express a wide range of relationships > and interactions in highly complex systems in order to be able to > communicate effectively, without further overloading terms or creating > ambiguity. And our work relies on intelligent rational thought and > decision making. > > Students of history know that anything resembling censorship > (self-censorship or otherwise) or expunging a word from use must be > considered, if at all, with great care and deliberation. > It would be a fallacy to believe that because slavery involved slave > masters, all uses and connotations of the term "master" involve slavery. > Similarly, considering and discussing the terms "master" and "slave" > together on the same basis and in the same context of slavery brings an > implicit bias to the discussion; they should be considered separately. > > I believe the best course of action is to remove the term "slave" and > the context of slavery from code and documentation, while carefully > assessing the context of usage of other words and evaluating them on a > case by case basis. For example, "master node" or "master branch", when > there are no slave nodes or slave branches and no context or implication > of slavery, constitute a valid and accurate non-slavery-related use of > the English word "master" and must be accepted as legitimate uses of > language. > > I hope this may clarify where to draw the line between a genuine effort > to avoid using terms with negative historical connotations that may > cause offense, and censorship in the form of designating words as "bad" > or "problematic" regardless of intent, context or meaning. > > Sincerely, > -rt >

1 0

[tc] weekly update
by Mohammed Naser 21 Oct '20

21 Oct '20

Hi everyone, Here's an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. # Patches ## Open Reviews - Add gmann liaison preference https://review.opendev.org/758602 - Resetting projects' TC liaisons empty https://review.opendev.org/758600 - Adopting the DPL governance model for oslo https://review.opendev.org/757906 - Implement distributed leadership in tools and schema https://review.opendev.org/757966 - Update TC size https://review.opendev.org/758168 - Add nomination for TC chair https://review.opendev.org/758181 - Appoint Adam Harwell as Octavia PTL https://review.opendev.org/758121 - Appoint XueFeng Liu as Senlin PTL https://review.opendev.org/758122 - Appoint Rafael Weingartner as CloudKitty PTL https://review.opendev.org/758120 - Appoint Frode Nordahl as OpenStack Charms PTL https://review.opendev.org/758119 - Add election schedule exceptions in charter https://review.opendev.org/751941 - Clarify impact on releases for SIGs https://review.opendev.org/752699 - Add assert:supports-standalone https://review.opendev.org/722399 - Select Prvisep as the Wallaby Goal https://review.opendev.org/755590 - Add danms liaison preferences https://review.opendev.org/758822 ## General Changes - Record Wallaby Cycle PTL and TC election results https://review.opendev.org/757971 # Other Reminders - PTG Brainstorming: https://etherpad.opendev.org/p/tc-wallaby-ptg - PTG Registration: https://october2020ptg.eventbrite.com Thanks for reading! Mohammed & Kendall -- Mohammed Naser VEXXHOST, Inc.

1 0

[kuryr] Gerrit breach and commit audit
by Maysa De Macedo Souza 21 Oct '20

21 Oct '20

Hello, Following the request on the announcement of the gerrit breah [1], the Kuryr team have audited all the Kuryr repositories. Nothing suspicious was seen. Thank you. Best regards, Maysa Macedo. [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

Re: [service-announce] October 20 Gerrit Outage Update
by Slawek Kaplonski 21 Oct '20

21 Oct '20

Hi, Hi, Both Lajos and me checked today all neutron and neutron stadium changes there. And all looks ok for us. I didn't notice anything wrong or suspicious there. Thx Lajos for help with that :) Dnia środa, 21 października 2020 09:49:49 CEST Slawek Kaplonski pisze: > On Wed, Oct 21, 2020 at 11:33:14AM +1100, Ian Wienand wrote: > > As of this mail, Gerrit access has been restored. Please read on for > > important information, especially around change verification. > > > > Background > > ----------- > > > > On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval > > to a change that they were not expected to have access to. At 02:06 > > UTC an alert was raised via IRC. Administrators found the account had > > added themselves to a core group and made the +W vote. The account > > was disabled, and removed from the groups it had added itself to by > > 02:55 UTC. Administrators began to analyse the situation and Gerrit > > was taken offline at 04:02 UTC to preserve state and allow for > > analysis. > > > > From this time, administrators were working on log collection and > > analysis, along with restoring backups for comparison purposes. > > > > By around 08:45 UTC it was clear that the privilege escalation had > > been achieved by gaining control of a Launchpad SSO account with > > Gerrit administrator privileges. By this time, we had ruled out > > software vulnerabilities. Logs showed the first unauthorized access > > of the administrator account in Gerrit on 2020-10-06. Communication > > with Launchpad admins agrees with this analysis. We saw one session > > opened as the administrator user to StoryBoard on this same day, but > > logs show no data was modified or hidden stories viewed. > > > > Analysis has been performed on the Gerrit database and git trees from > > October 1st, pre-dating any known unauthorized access. > > > > Access was restored at around 2020-10-21 00:00 UTC > > > > Outcomes > > ----------- > > > > The following has been verified: > > The administrator account used has been disabled and credentials > > updated > > > > We have verified that all group and user addition/removals since > > Oct 1 are valid. The only invalid additions were made by the > > compromised administrator account to add a single user account to > > the Administrators group; and then that account added itself to > > another known group. > > > > The account given administrator privilege has been removed from > > the groups it added itself to and is disabled. > > > > There is no evidence of any unauthorized access via methods other > > than Gerrit HTTP and Gerrit SSH access. > > > > No commits have been pushed to git trees bypassing code review. > > Every git tree has been compared to the Oct 1 version and all > > commits have been correctly inserted via Gerrit changes. > > > > The version of Gerrit we use stores HTTP API passwords in > > plain-text. We know that a limited number of passwords were > > gathered via the HTTP API and it is possible passwords were > > gathered via the database. We thus have assumed that all HTTP API > > passwords have been disclosed. This password needs to be > > explicitly enabled by users, and many users do not have it > > enabled. > > > > Remediation > > ----------- > > > > This leaves us with the following remediation actions: > > Users should double-check their Launchpad recent activity at > > https://login.launchpad.net/activity for any suspicious logins. If > > found, please notify the OpenDev admins in Freenode #opendev and > > Launchpad admins in #launchpad immediately. > > > > All HTTP API passwords have been cleared. If you push changes via > > HTTPS (instead of typical SSH), are a gertty user, or run a CI > > system or something else that communicates with the Gerrit HTTP > > API, you will need to regenerate a password. > > > > Any SSH keys added to accounts since 2020-10-01 have been removed. > > This affects only a limited number of accounts. This is done in > > an abundance of caution, and we do not believe any accounts had > > unauthorized SSH keys added > > > > We should audit all changes for projects since 2020-10-01. > > > > We have no evidence that any account had its ssh keys compromised, > > thus we can rule out any unauthorized changes being uploaded via SSH. > > However we can not conclusively rule out that compromised HTTP API > > passwords were used to push a change through Gerrit. For example, a > > change could be uploaded that looks like it came from a user, or the > > API key of a core team member may have been used to approve a change > > without authorization. > > > > Given our extensive analysis we consider it exceedingly unlikely that > > this vector was used. We have had no notifications of users seeing > > unexpected changes either uploaded by them, or approved by them in > > projects they work on. This said, we believe it is important to > > inform the community of this very unlikely, but still possible, > > vulnerability of the source code. > > > > To this end, we have prepared a list of all changes from the known > > affected period which should be audited for correctness. These are > > available at > > > > https://static.opendev.org/project/opendev.org/gerrit-diffs/ > > > > Team members should browse these changes and make sure they were > > correctly approved in Gerrit. If any change looks suspicious you > > should notify OpenDev administrators in Freenode #opendev immediately. > > > > Further actions > > ---------------- > > > > We are planning the following for the short term future: > > The Opendev administrators will be looking at alternative models > > for Gerrit admin account management. > > > > We are already well into planning and testing a coming upgrade to > > a version of Gerrit which does not store plain-text API keys. > > > > Longer term, we've written a spec for replacing Launchpad SSO as > > our authentication provider. > > > > We thank you for your patience during this trying time, and we look > > forward to returning to supporting the community doing what it does > > best -- working together to create great things. > > > > > > _______________________________________________ > > service-announce mailing list > > service-announce(a)lists.opendev.org > > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[storlets] Gerrit breach and commit audit
by Takashi Kajinami 21 Oct '20

21 Oct '20

Hi, As suggested by the recent announcement[1], I have audited all commits (actually, 1 commit) in storlets repository, and confirmed that everything looks good now. [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html Thank you, Takashi

1 0

[masakari] Branches reviewed post-gerrit-breach
by Radosław Piliszek 21 Oct '20

21 Oct '20

Hi Folks, I have reviewed Masakari repos and no suspicious commits were found. Thank you, OpenDev team, for handling the situation and providing patches list. -yoctozepto

1 0

[cloudkitty] Gerrit breach and commit audit
by Pierre Riteau 21 Oct '20

21 Oct '20

Hello, Following the announcement of the Gerrit breach [1], I have audited all commits in repositories of the CloudKitty project. Everything looks good. Thank you to the OpenDev admins for dealing with this issue efficiently. Pierre [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[blazar] Gerrit breach and commit audit
by Pierre Riteau 21 Oct '20

21 Oct '20

Hello, Following the announcement of the Gerrit breach [1], I have audited all commits in repositories of the Blazar project. Everything looks good. Thank you to the OpenDev admins for dealing with this issue efficiently. Pierre [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

Openstack client can't load modules
by Cristina Mayo 21 Oct '20

21 Oct '20

Hi everyone, Openstack client installed on an Ubuntu 20.04 is showing these errors when I launch as non root user; indeed it is fine with sudo: Could not load 'metric_metric_create': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricCreate' Could not load 'metric_metric_delete': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricDelete' Could not load 'metric_metric_list': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricList' Could not load 'metric_metric_show': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricShow' I don't have noticed any bad behaviour using it, but I'm not sure. Any help? Thanks

3 2

[barbican] gerrit breach and auditing
by Moises Guimaraes de Medeiros 21 Oct '20

21 Oct '20

Hi all, I'm putting up a review spreadsheet[0] for the barbican changes since October 1st based on Julia's spreadsheet for Ironic[1]. [0]: https://docs.google.com/spreadsheets/d/1WdhxDsAZCHlzhRpcH0jKfyY2kzO2h2jRzni… [1]: http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018148.… -- Moisés Guimarães Software Engineer Red Hat <https://www.redhat.com> <https://red.ht/sig>

1 0

[glance] Wallaby PTG schedule
by Abhishek Kekane 21 Oct '20

21 Oct '20

Hello All, Greetings!!! Wallaby PTG is around the corner and if you haven't already registered, please do so as soon as possible [1]. I have created a Virtual PTG planning etherpad [2] and also added day wise topics along with timings we are going to discuss. Kindly let me know if you have any concerns with allotted time slots. We also have some slots open on Thursday and Friday for unplanned discussions. So please feel free to add your topics if you still haven't added yet. As a reminder, these are the time slots for our discussion. Tuesday 27 October 2020 1400 UTC to 1700 UTC Folsom Wednesday 28 October 2020 1400 UTC to 1700 UTC Folsom Thursday 29 October 2020 1400 UTC to 1700 UTC Folsom Friday 30 October 2020 1400 UTC to 1700 UTC Folsom We will be using bluejeans for our discussion, kindly try to use it once before the actual discussion. The meeting URL is mentioned in etherpad [2] and will be the same throughout the PTG. [1] https://october2020ptg.eventbrite.com/ [2] https://etherpad.opendev.org/p/glance-wallaby-ptg Thank you, Abhishek

1 0

[kolla] Branches reviewed post-gerrit-breach
by Radosław Piliszek 21 Oct '20

21 Oct '20

Hi Folks, The Kolla team has reviewed its projects and no suspicious commits were found. Thank you, OpenDev team, for handling the situation and providing patches list. Thank you, Mark, Michał, Marcin, Pierre and myself for reviewing. Let's roll with development. :-) -yoctozepto

1 0

[release][core] gerrit breach and auditing all releases deliverables [1] since Oct 01
by Herve Beraud 21 Oct '20

21 Oct '20

Hello, As every team we are also concerned by the gerrit breach and we must take a look at our changes during this time frame on all our deliverables [1]. The majority of our changes during this period are changes related to releasing new deliverables, that mean changes on yaml files. Due to the criticality of the problem I propose the following action plan: - first, focus us on changes against our scripts and executable files; - second, inspect yaml files if teams identify possible issues in their code base. I don't think we need to inspect our `deliverable/*/*.yaml` files for now, it could be done later; - third, inspect documentation changes. They could be easily ignored for now and analyzed later too. I wrote a script [2][3] to help us to extract relevant changes (*.py, *.sh), all the rest ((deliverable|data)*.yaml, *.rst) have been ignored for now. I ran this script against the repos that we own, here is the result: - openstack/release-test - No changes merged since more than 3 months - openstack/releases - https://review.opendev.org/#/q/9e527f9e0e0cac29699171813b45c65b487010dc - https://review.opendev.org/#/q/9c4f3c43039fa686cee6d03fb6d9894de05c89a6 - https://review.opendev.org/#/q/a8ab91eebc7a0b9d0be863772ea286627e6b57d9 - openstack/reno - No changes merged since more than 2 months - openstack/specs-cookiecutter - No changes merged since more than 4 months Example of script usage against our openstack/releases repos: ``` $ cd releases $ curl https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra… | sh ``` I'll take a look at the identified commits but double checking would be greatly appreciated. Please ensure to double check your account activity [4] and make sure nothing is off. Special congrats to Julia Kreger and for her excellent job [5]. Thank you in advance for your help, [1] https://governance.openstack.org/tc/reference/projects/release-management.h… [2] https://gist.github.com/4383/511359cc2080e06295944c5f40bd1033 [3] https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra… [4] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018148.… -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

[nova][placement] Reviewing changes after gerrit admin account compromise
by Balázs Gibizer 21 Oct '20

21 Oct '20

Hi, As per [0] we are requested to check commits merged since 1st of October to nova and related repos. Lee and I went through nova master and stable branches and they seem to be clean. I also went through nova-specs, python-novaclient, os-vif, placement, osc-placement changes and those are seems to clean too. Still, if you check and see any suspicious things please contact OpenDev administrators in Freenode #opendev Cheers, gibi [0] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[openstack][magnum] No meeting for this week
by feilong 21 Oct '20

21 Oct '20

Hi team, Due to the Summit this week, we won't hold the weekly Magnum meeting on 21 October. -- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------

1 0

Implementation of project Trove in our Openstack-based Environment
by Bekir Fajkovic 20 Oct '20

20 Oct '20

Hello! My name is Bekir Fajkovic, i am resided in Sweden and i work as a member of a DBA team at City Network company that provides cloud-based services and solutions based mostly on OpenStack. We are about to evaluate the OpenStack project Trove in order to try to provide a reasonable and robust kind of DBaaS solution to our customers. As a first step i created a test environment based on DevStack (based in turn on latest release of OpenStack) including Trove (version 14.1.0.dev18) and i am currently evaluating the things. I see many promising features, like Docker-based deployments of database instances on dedicated images possible to be created and configured through Trove utilities, also some important security improvements of RabbitMQ message handling ensuring encryption key usage in communication with the db instances etc.. I do not see MSSQL datastore type provided in the initial setup but i guess that is a subject of extra configuration that can be carried out by us. So, i wonder about the status of the project in general, to begin with. According to some rumours going around, the project is to be abandoned soon by the developers and i would like to get that confirmed or not, just in order for us to make further decisions in regards to DBaaS development. I know that many DBaaS projects are nowadays based on Kubernetes and development of specific operators that provide certain kind of automated database services, but we are still interested about Trove as a native DBaaS project that well integrates into OpenStack and provides multitenancy, redundancy, and so on. I would be very pleased if we could get more in contact with OpenStack Trove Team somehow, to establish some sort of dialogue and to see more what we could build and provide based on that project. Best Regards Bekir Fajkovic Senior DBA Mobile: +46 70 019 48 47 www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED

2 1

[charms] HA, haproxy https backend checks
by Xav Paice 20 Oct '20

20 Oct '20

Hi, Looking at https://review.opendev.org/#/c/750224/, we've got a change that switches haproxy from looking at tcp socket connection, to http. I'd like to update that change to enable the same for when apache is providing https, and use the /healthcheck endpoint. See https://github.com/juju/charm-helpers/pull/521 for a change to the haproxy template to enable this, and I'd submit a fresh patchset for the change but can't right now with opendev in maintenance. I'm wondering re zaza tests for this, should I be adding bundles that have 3 units of Glance (to prove HA) and likely Vault so that we can check https? That would increase the bundle size a chunk, but I think it would improve the overall testing of the charm. Thoughts?

1 0

How to integrate Designate with Nova/Horizon for internal domains
by Daniel Heitepriem 20 Oct '20

20 Oct '20

Dear list, I'm in the process of setting up an Openstack installation for internal use. The environment consists of these nodes: - 1 controller node - 2 compute nodes My goal is now to get Designate working for internal domain resolution paired with automatic record creation from Nova/Neutron. The steps I did so far: - Installed and configured Designate (https://docs.openstack.org/designate/ussuri/install/install-ubuntu.html) and Designate dashboard for Horizon (https://docs.openstack.org/designate-dashboard/ussuri/install/index.html) - Adjusted handler "neutron_floatingip" [handler:neutron_floatingip] notification_topics = notifications_designate control_exchange = neutron formatv4 = '%(octet0)s-%(octet1)s-%(octet2)s-%(octet3)s.%(zone)s' formatv4 = '%(hostname)s.%(project)s.%(zone)s' formatv4 = '%(hostname)s.%(zone)s' - Adjusted handler "nova_fixed" [handler:nova_fixed] notification_topics = notifications_designate control_exchange = nova formatv4 = '%(octet0)s-%(octet1)s-%(octet2)s-%(octet3)s.%(zone)s' formatv4 = '%(hostname)s.%(project)s.%(zone)s' formatv4 = '%(hostname)s.%(zone)s' - Installed Designate-Sink and configured notification_handlers [service:sink] workers = 5 enabled_notification_handlers = "nova_fixed, neutron_floatingip" - Added notification topic "notifications_designate" to nova.conf and neutron.conf Let's say I have one additional domain configured in Openstack next to the default domain: "domain1.internal". So I logged in to the new domain in Horizon, created a few networks, created the new zone record "domain1.internal." and started a new instance called "test-vm". Expectations: Nova creates the instance, Neutron creates the required network port and Designate Sink receives the "Port created" notification via the "notifications_designate" topic Reality: Nova creates the instance, Neutron creates the required network port but Designate Sink does nothing. In the designate-api.log I can see the creation of the "domain1.internal." zone but the designate-sink.log stays empty 2020-10-20 08:29:19.573 3524893 INFO designate.api.v2.controllers.zones [req-31b7fa68-6621-4044-985e-f90c87dd8cae - - - - -] Created <Zone id:'d7c015ec-5079-450b-866d-f04042221d80' type:'PRIMARY' name:'domain1.internal.' pool_id:'794ccc2c-d751-44fe-b57f-8894c9f5c842' serial:'1603182558' action:'CREATE' status:'PENDING'> My question is: What piece of configuration did I miss? Is my desired configuration even possible at all? What I basically want to achieve is: - Create an instance using nova - The instance name automatically get's published in all(?) DNS zones of the project/Openstack domain - Instances can resolve other instance names out of this DNS zone Thank you very much in advance and regards, Daniel Heitepriem P.S.: Can you please keep my in CC as I'm not subscribed to the list

2 1

[Ironic] Teaching virtualbmc how to talk to Ironic?
by Lars Kellogg-Stedman 20 Oct '20

20 Oct '20

In the work that we're doing with the Mass Open Cloud [1], we're looking at using Ironic (and the multi-tenant support we contributed) to manage access to a shared pool of hardware while still permitting people to use their own provisioning tools. We don't want to expose the hardware BMC directly to consumers; we want Ironic to act as the access control mechanism for all activities involving the hardware. The missing part of this scenario is that at the moment this would require provisioning tools to know how to talk to the Ironic API if they want to perform BMC actions on the host, such as controlling power. While talking with Mainn the other day, it occurred to me that maybe we could teach virtualbmc [2] how to talk to Ironic, so that we could provide a virtual IPMI interface to provisioning tools. There are some obvious questions here around credentials (I think we'd probably generate them randomly when assigning control of a piece of hardware to someone, but that's more of an implementation detail). I wanted to sanity check this idea: does this seem reasonable? Are there alternatives you would suggest? Thanks! [1] https://github.com/CCI-MOC/esi [2] https://github.com/openstack/virtualbmc -- Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {irc,twitter,github} http://blog.oddbit.com/ | N1LKS

6 12

[cinder] no weekly meeting this week (or next)
by Brian Rosmaita 20 Oct '20

20 Oct '20

Due to the Summit this week, we won't hold the weekly cinder meeting on 21 October (tomorrow). Due to the PTG next week, we won't need to hold the weekly meeting on 28 October because we'll be discussing cinder stuff all week. The PTG planning etherpad is here: https://etherpad.opendev.org/p/wallaby-ptg-cinder-planning The next Cinder meeting will be at the usual place and time: Wednesday 4 November in #openstack-meeting-alt at 1400 UTC. (People who have been observing Daylight Savings Time, please check whether the meeting time may have shifted an hour earlier for you.) The cinder meeting agenda is available here: https://etherpad.opendev.org/p/cinder-wallaby-meetings Looking forward to seeing everyone virtually at the PTG! brian

1 0

Problem with octavia LBaaS and nova availability zones
by bjoernputtmann＠netprojects.de 20 Oct '20

20 Oct '20

Hi one and all! Currently, we have an ussuri openstack installation via kolla-ansible. The installation consists of three control nodes and two availability zones (az-1, az-2), each with three compute nodes: - os-cpt-10[1-3] > az-1 - os-cpt-20[1-3] > az-2 az-1 is the default compute availability zone. The compute nodes are hardwarewise equipped exactly the same. As storage system a ceph cluster is used. We wanted to also use Octavia LBaaS and got it up and running after some experimentation. We also wanted to be able to choose the availibility zones when starting a loadbalancer. We added the az info with: openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-1 --provider amphora --availability-zone-data '{"compute_zone": "az-1"}' openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-2 --provider amphora --availability-zone-data '{"compute_zone": "az-2"}' openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-1 --availabilityzoneprofile az-1 openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-2 --availabilityzoneprofile az-2 Creating a loadbalancer via cli: openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR --name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ works, as long as $LB_AZ == az-1. If we want to start the loadbalancer in az-2, this fails with an error in octavia-worker.log: ... 2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z', 'message': 'No valid host was found. There are not enough hosts available.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule\n claimed_instance_uuids)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts\n raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. There are not enough hosts available.\n\n'} ... We enabled debug logging for nova and found: ... 2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.003s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304 2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep", "sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep", "abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1", "rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2", "clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8", "umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36", "mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht", "pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2", "movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic", "wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm", "cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt", "rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm", "sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1373295770000000}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 386827280000000}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 139696145240000000}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["mempages", "cpu_usage", "memory", "memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata", "siblings", "cpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 27, 'uuid': '62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['c88b77f7-2c59-497b-9633-139e62ba2bb1', '635fbd95-dd64-4e0a-9767-a0e0a6958610', 'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.007s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch", "fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic", "adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de", "movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov", "perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd", "cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr", "clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec", "mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero", "pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht", "rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse", "abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep", "fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36", "clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 139958595980000000}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 388866090000000}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1118743070000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory", "cpuset", "siblings", "mempages", "network_metadata", "memory_usage", "pinned_cpus"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 21, 'uuid': 'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['96f4c101-d75b-44b6-8af9-dd75068d207d', '121c5df5-2d8b-48e0-a270-9f6f79193dd6', 'e235063d-f7a8-4f38-8294-d7a289167348', '425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm", "clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx", "pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca", "xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec", "invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext", "ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat", "lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht", "smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni", "tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp", "cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core", "fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq", "perfctr_nb", "smep", "de", "3dnowprefetch", "xsave", "tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 161246250000000}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 140551238730000000}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 366274340000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3829690000000}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages", "memory", "siblings", "network_metadata", "cpuset", "memory_usage", "id", "pcpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 18, 'uuid': '342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: [] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.266 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Starting with 3 host(s) get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:70 2020-10-13 13:50:49.266 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-101, os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-102, os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-103, os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.268 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filter AvailabilityZoneFilter returned 0 hosts 2020-10-13 13:50:49.269 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: [('AvailabilityZoneFilter', None)] get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:115 2020-10-13 13:50:49.269 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: ['AvailabilityZoneFilter: (start: 3, end: 0)'] 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtered [] _get_sorted_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] There are 0 hosts available but 1 instances requested to build. _ensure_sufficient_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300 ... So nova seems to only consider compute nodes in az-1. The amphora image is available in both availability zones. Starting an instance from the image directly in az-2 works. Does anybody have any idea on how to debug this any further? Any hints, tips or ideas would be greatly appreciated! With kind regards, Björn Puttmann

2 3

How to change nova db in Train ?
by Yingji Sun 20 Oct '20

20 Oct '20

Buddies, I have a question about updating nova db. I would like to change the length of internal_access_path in console_auth_tokens, starting from Train. I want to modify nova/db/sqlalchemy/models.py and the migration file under nova/db/sqlalchemy/migrate_repo/versions. However I did not find a placeholder file for Train. 402_add_resources.py is the last one in Train and 403_ is "Add reserved schema migrations for Ussuri". So how can I do it in Train ? Yingji.

3 4

packstack fails every time - pymysql.err.OperationalError: (1040, 'Too many connections')
by lejeczek 20 Oct '20

20 Oct '20

2 1

Ubuntu Focal Ussuri can't create a snapshot from running instance
by Cristina Mayo Sarmiento 20 Oct '20

20 Oct '20

Hi all, I don't have a lot experience with Openstack. I've installed Ussuri on my cloud. I have 1 controller node and 2 compute nodes. I'm not able to create a snapshot from a running instance, but it works perfectly if the instance is turned off. I checked the logs on controller node about glance-api and POST method executes fine, but suddenly appears a DELETE that seems remove the image: "POST /v2/images HTTP/1.1" 201 1254 0.045379 "GET /v2/images/c647160b-416d-4c98-bbe2-6302a509f 941 HTTP/1.1" 200 1097 0.014154 WARNING glance.api.v2.images [req-e29704f9-e147-42df-914c-6217c60ada30 c63d05ad0e9d43ebb0fd31f442156a2a c2 b7d6242f3140f09d283e8fbb88732a - default default] After upload to backend, deletion of staged image data has failed because it cannot be found at /tmp/staging//c647160b-416d-4c98-bbe2-6302a509f941 "DELETE /v2/images/c647160b-416d-4c98-bbe2-6302a5 09f941 HTTP/1.1" 204 213 0.140731 I have these versions installed: - python3-glance-store/focal,now 2.0.0-0ubuntu1 all [instalado, automático] - python3-glance/focal-updates,now 2:20.0.1-0ubuntu1 all [instalado, automático] Could anyone help me? Thanks in advance, Regards,

3 3

[largescale-sig] "Share your OpenStack scaling story" forum session
by Thierry Carrez 20 Oct '20

20 Oct '20

Hi everyone, In a few hours at 14:30UTC we'll have a Forum session where everyone is invited to share their scaling story. As you add nodes to an OpenStack cluster, at one point you hit scaling limitations that force you to scale out to multiple clusters, cells or zones. What breaks first? At which scale? The Large Scale SIG is working on pushing back the scaling limits within a single cluster. To that effect, it is collecting scaling stories from OpenStack deployers. This should lead to identifying common cases, usual suspects that the SIG can focus on, in order to raise scaling limitations within one cluster and reduce the need to scale out to multiple clusters too early. Join fellow OpenStack operators and share your experience and horror stories, adding nodes to OpenStack clusters! In case the event platform plays games again, you can access the session directly at: https://zoom.us/j/94011749848?pwd=Qmo5c1ZMK3RJWnVRSjZzZWFkTng3QT09 And the etherpad is at: https://etherpad.opendev.org/p/vSummit2020_OpenStackScalingStory See you all there! -- Thierry Carrez (ttx)

1 1

[neutron] bug deputy report (week of Oct 12)
by Akihiro Motoki 20 Oct '20

20 Oct '20

Hi, I was a bug deputy in neutron last week. Here is a summary. It was a quiet week and fixes have been proposed for medium or higher priority bugs. I see no bug to be raised from POV of bug triaging. [In Progress] https://bugs.launchpad.net/neutron/+bug/1899967 L3HA Router in primary state on all nodes Critical, In Progress, slaweq proposed a fix https://bugs.launchpad.net/neutron/+bug/1899917 neutron-tempest-plugin-scenario-linuxbridge jobs are broken on rocky/queens branches Critical, In Progress, slaweq proposed a fix https://bugs.launchpad.net/neutron/+bug/1899775 Ovs agent import error on Windows High, In Progress, a fix was proposed for review https://bugs.launchpad.net/neutron/+bug/1899864 Smartnic port's MTU is not set correctly Medium, In Progress, a fix was proposed for review [Confirmed] https://bugs.launchpad.net/neutron/+bug/1899502 router_centralized_snat ports do not have project_id Low, Confirmed, Assigned to the bug author Looks like we agree it is a valid bug. [Wishlist] These are bugs for tracking refactoring. https://bugs.launchpad.net/neutron/+bug/1900270 Implement "find_child_pids" natively Wishlist, In Progress, Assigned to ralonsoh https://bugs.launchpad.net/neutron/+bug/1900268 Implement "find_parent_pid" natively Wishlist, In Progress, Assigned to ralonsoh https://bugs.launchpad.net/neutron/+bug/1900269 Implement "get_process_count_by_name" natively Wishlist, In Progress, Assigned to ralonsoh Thanks, Akihiro Motoki (irc: amotoki)

1 0

[openstack][stein][cinder] python2-os-brick issue orphaned device
by Ignazio Cassano 20 Oct '20

20 Oct '20

Hello Guys, I am using cinder on centos7 stein with Dell unity iscsi driver. When I migrate vm from a kvm node to another, orphaned devices are left on source kvm node. I initialized debug on nova compute and i thing the issue si in the following lines (2020-10-20 09:52:33.078 132171 DEBUG os_brick.initiator.connectors.iscsi [-] Disconnecting from: []): connectors/iscsi.py:1111 2020-10-20 09:52:33.078 132171 DEBUG os_brick.initiator.connectors.iscsi [-] Resulting device map defaultdict(<function <lambda> at 0x7f4f1b1f7cf8>, {(u'10.138.215.17:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.a2'): (set([]), set([u'sdg', u'sdi'])), (u'10.138.209.47:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.b3'): (set([]), set([u'sdo', u'sdq'])), (u'10.138.209.48:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.a3'): (set([]), set([u'sdd', u'sdb'])), (u'10.138.215.18:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.b2'): (set([]), set([u'sdm', u'sdk']))}) _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:844 2020-10-20 09:52:33.078 132171 DEBUG os_brick.initiator.connectors.iscsi [-] Disconnecting from: [] _disconnect_connection /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1099 The version of package is: python2-os-brick-2.8.7-1.el7.noarch Please, any help ? Ignazio

1 0

[Automation SIG] BoF today! feel free to join
by Rico Lin 20 Oct '20

20 Oct '20

Hi all, Automation SIG is hosting a BoF on Tue 20, 1:00pm - 1:45pm (18:00 - 18:45 UTC) - Virtual - Virtual Floor - Forum Room 2 https://summit.openinfra.dev/a/event/24753 Please join if you're interested. -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 0

[Multi-arch SIG][OpenInfra Summit] BoF today! Feel free to join us
by Rico Lin 20 Oct '20

20 Oct '20

Hi all As a forming SIG, we hope to have more hands or resources to keep pushing things forward. In this summit, we will host a BoF on Tuesday this week and hope to have people to join. Multi-arch SIG Tue 20, 12:15pm - 1:00pm (17:15 - 18:00 UTC) - Virtual - Virtual Floor - Forum Room Multi-arch SIG BoF https://summit.openinfra.dev/a/event/24752 Also if you would like to see some related sessions here's some you can consider: Multi-Arch Support in OpenStack https://summit.openinfra.dev/a/event/24708 The Progress for Cloud Computing on Arm architecture https://summit.openinfra.dev/a/event/24719 Cloud-hypervisor: A new choice for virtual machine monitor https://summit.openinfra.dev/a/event/24684 -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 0

[nova] NUMA scheduling
by Eric K. Miller 19 Oct '20

19 Oct '20

Hi, I'm at a loss for finding good information about how a VM's vCPUs and Memory are assigned to NUMA nodes within a scheduled physical host. I think Libvirt does this, and the Nova Scheduler simply finds the right physical host to run the VM, and thus Nova has no input on which NUMA node to choose. So this might be a Libvirt question. We are running Stein and have the issue where VMs launch on NUMA Node 0, and not on NUMA Node 1, in physical hosts with two processors, and are simply looking for a way to tell Libvirt to consider NUMA Node 1 when scheduling a VM, since there is nearly all of the memory available on NUMA Node 1. Our flavors are defined with hw:numa_nodes='1' since we want all vCPUs+Memory to land on a single NUMA Node, and so the guest OS has visibility that a single NUMA Node is being used. We are "not" looking for a way to pin a VM to a specific NUMA node (such as for SR-IOV purposes). Any suggestions where to look for the solution? Thanks! Eric

5 17

[qa] Weekly meeting cancelled because of Summit and PTG
by Masayuki Igawa 19 Oct '20

19 Oct '20

Hi, We'll skip the meetings both this and next week because of the summit and PTG. See you there :) -- Masayuki Igawa

1 0

[openstack-helm] No meeting this week and next week
by Gage Hugo 19 Oct '20

19 Oct '20

Hello everyone, due to the summit and PTG there will be no meeting this week or next week. We will meet again on November 3rd at the usual time, thanks!

1 0