October 2020 - openstack-discuss - lists.openstack.org

[magnum] Gerrit breach and commit audit
by feilong 22 Oct '20

22 Oct '20

Hi everyone, As per the announcement of the Gerrit breach [1], I have audited all repos under Magnum (magnum, python-magnumclient, magnum-ui and magnum-tempest-plugin) for 1. Suspicious commit 2. Core group list Everything looks OK for me, see https://etherpad.opendev.org/p/magnum-gerrit-breach-audit [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html -- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------

1 0

Keystone auth method v3oidcpassword throws RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
by Robert Duncan 22 Oct '20

22 Oct '20

I'm trying to use openID with openstack --versionopenstack 5.2.0 keystoneauth1==3.17.3 keystonemiddleware==7.0.1 python-keystoneclient==3.21.0 I have created an RC file like so : export OS_INTERFACE="public" export OS_AUTH_URL="https://openstack:5000/v3" export OS_IDENTITY_PROVIDER="<my_idp>" export OS_PROTOCOL="openid" export OS_CLIENT_ID="<the -id >" export OS_CLIENT_SECRET="<the secret>" export OS_DISCOVERY_ENDPOINT="https://login.microsoftonline.com/<the-tenant-id>/v2.0/.well-known/openid-configuration" export OS_IDENTITY_API_VERSION="3" export OS_AUTH_TYPE="v3oidcpassword" export OS_USERNAME="<AzureAD user>" # this is the local openstack project id export OS_PROJECT_ID="<The project ID>" # set password by querying user export OS_PASSWORD="" echo "Please enter your O365 Password: " read -sr OS_PASSWORD_INPUT export OS_PASSWORD=$OS_PASSWORD_INPUT all is working on Horizon WebSSO- but I cannot access the API the openstack --debug output shows that I get a token from MS debug info (redacted) command: token issue -> openstackclient.identity.v3.token.IssueToken (auth=True) Auth plugin v3oidcpassword selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'project_id': '516706052ebd4f9a89c0b7d7e075754d'}, 'additional_user_agent': [('osc-lib', '2.0.0')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'auth_url': 'https://openstack.*********:5000/v3', 'username': 'r********', 'password': '***', 'identity_provider': '*****', 'protocol': 'openid', 'client_id': '*******', 'client_secret': '***', 'discovery_endpoint': 'https://login.microsoftonline.com/*******/v2.0/.well-known/openid-configura…', 'beta_command': False, 'identity_api_version': '3', 'data_processing_api_version': '1.1', 'container_infra_api_version': '1', 'region_name': '', 'auth_type': 'v3oidcpassword', 'networks': []} Using auth plugin: v3oidcpassword Using parameters {'auth_url': 'https://openstack.******:5000/v3', 'project_id': '516706052ebd4f9a89c0b7d7e07575, 'identity_provider': '*****', 'protocol': 'openid', 'client_id': '********', 'client_secret': '***', 'discovery_endpoint': 'https://login.microsoftonline.com/*******/v2.0/.well-known/openid-configura…', 'username': '*****', 'password': '***'} Get auth_ref REQ: curl -g -i -X GET https://login.microsoftonline.com/*****/v2.0/.well-known/openid-configurati… -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.2" Starting new HTTPS connection (1): login.microsoftonline.com:443 https://login.microsoftonline.com:443 "GET /******/v2.0/.well-known/openid-configuration HTTP/1.1" 200 1651 RESP: [200] Access-Control-Allow-Methods: GET, OPTIONS Access-Control-Allow-Origin: * Cache-Control: max-age=86400, private Content-Length: 1651 Content-Type: application/json; charset=utf-8 Date: Wed, 21 Oct 2020 09:37:13 GMT P3P: CP="DSP CUR OTPi IND OTRi ONL FIN" Set-Cookie: fpc=AhupGfLmUQlEsKNlR-m0ATg; expires=Fri, 20-Nov-2020 09:37:14 GMT; path=/; secure; HttpOnly; SameSite=None, esctx=*****: domain=.login.microsoftonline.com; path=/; secure; HttpOnly; SameSite=None, x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly, stsservicecookie=estsfd; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff x-ms-ests-server: 2.1.11154.9 - NEULR2 ProdSlices x-ms-request-id: 0819033b-dcb3-409a-8115-54dc06a31900 RESP BODY: {"token_endpoint":"https://login.microsoftonline.com/*****/oauth2/v2.0/token","token_endpoint_auth_methods_supported":["client_secret_post","private_key_jwt","client_secret_basic"],"jwks_uri":"https://login.microsoftonline.com/****/discovery/v2.0/keys","response_modes_supported":["query","fragment","form_post"],"subject_types_supported":["pairwise"],"id_token_signing_alg_values_supported":["RS256"],"response_types_supported":["code","id_token","code id_token","id_token token"],"scopes_supported":["openid","profile","email","offline_access"],"issuer":"https://login.microsoftonline.com/****/v2.0","request_uri_parameter_supported":false,"userinfo_endpoint":"https://graph.microsoft.com/oidc/userinfo","authorization_endpoint":"https://login.microsoftonline.com/****/oauth2/v2.0/authorize","device_authorization_endpoint":"https://login.microsoftonline.com/*****/oauth2/v2.0/devicecode","http_logout_supported":true,"frontchannel_logout_supported":true,"end_session_endpoint":"https://login.microsoftonline.com/****/oauth2/v2.0/logout","claims_supported":["sub","iss","cloud_instance_name","cloud_instance_host_name","cloud_graph_host_name","msgraph_host","aud","exp","iat","auth_time","acr","nonce","preferred_username","name","tid","ver","at_hash","c_hash","email"],"tenant_region_scope":"EU","cloud_instance_name":"microsoftonline.com","cloud_graph_host_name":"graph.windows.net","msgraph_host":"graph.microsoft.com","rbac_url":"https://pas.windows.net"} REQ: curl -g -i -X POST https://login.microsoftonline.com/******/oauth2/v2.0/token -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.2" -d '{'username': '*****', 'password': '*****', 'scope': 'openid profile', 'grant_type': 'password'}' https://login.microsoftonline.com:443 "POST /*****/oauth2/v2.0/token HTTP/1.1" 200 3764 RESP: [200] Cache-Control: no-store, no-cache Content-Length: 3764 Content-Type: application/json; charset=utf-8 Date: Wed, 21 Oct 2020 09:37:14 GMT Expires: -1 P3P: CP="DSP CUR OTPi IND OTRi ONL FIN" Pragma: no-cache Set-Cookie: fpc=****; expires=Fri, 20-Nov-2020 09:37:14 GMT; path=/; secure; HttpOnly; SameSite=None, x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly, stsservicecookie=estsfd; path=/; secure; samesite=none; httponly Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff x-ms-ests-server: 2.1.11154.9 - WEULR1 ProdSlices x-ms-request-id: 92d66fde-6763-4be2-b1cf-00fd18bc1800 RESP BODY: {"token_type":"Bearer","scope":"email openid profile 00000003-0000-0000-c000-000000000000/User.Read","expires_in":3599,"ext_expires_in":3599,"access_token":"<the access token>","id_token":"<the id token>"} REQ: curl -g -i -X POST https://openstack***:5000/v3/OS-FEDERATION/identity_providers/***/protocols… -H "Authorization: {SHA256}aff774c8663ee647a08aefcea698f8c30e1d1cc8d85a3d55a17cae53defd8955" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.2" Starting new HTTPS connection (1): openstack.****:5000 https://openstack.***:5000 "POST /v3/OS-FEDERATION/identity_providers/****/protocols/openid/auth HTTP/1.1" 200 541 RESP: [200] Content-Length: 541 Content-Type: text/html Date: Wed, 21 Oct 2020 09:37:14 GMT Server: Apache RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged. Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cliff/app.py", line 393, in run_subcommand self.prepare_to_run_command(cmd) File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 493, in prepare_to_run_command self.client_manager.auth_ref File "/usr/lib/python3/dist-packages/osc_lib/clientmanager.py", line 202, in auth_ref self._auth_ref = self.auth.get_auth_ref(self.session) File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/federation.py", line 65, in get_auth_ref auth_ref = self.get_unscoped_auth_ref(session) File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/oidc.py", line 265, in get_unscoped_auth_ref return access.create(resp=response) File "/usr/lib/python3/dist-packages/keystoneauth1/access/access.py", line 36, in create body = resp.json() File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) what's gone wrong? thanks, Rob. ________________________________ The information contained and transmitted in this e-mail is confidential information, and is intended only for the named recipient to which it is addressed. The content of this e-mail may not have been sent with the authority of National College of Ireland. Any views or opinions presented are solely those of the author and do not necessarily represent those of National College of Ireland. If the reader of this message is not the named recipient or a person responsible for delivering it to the named recipient, you are notified that the review, dissemination, distribution, transmission, printing or copying, forwarding, or any other use of this message or any part of it, including any attachments, is strictly prohibited. If you have received this communication in error, please delete the e-mail and destroy all record of this communication. Thank you for your assistance. ________________________________

2 6

[nova] Gerrit breach and commit audit
by Ghanshyam Mann 22 Oct '20

22 Oct '20

Hello Everyone, As per the announcement of the Gerrit breach [1], I have audited all repos under Nova for 1. Suspicious commit 2. Core group list and everything is fine. - https://etherpad.opendev.org/p/nova-gerrit-breach-audit [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[designate] Gerrit breach and commit audit
by Nicolas Bock 22 Oct '20

22 Oct '20

Hi, Herve's script [1] identified the following commits in the designate repository. I second Michael's assessment from earlier that all of those commits are legitimate. Best, Nick Commit to analyze: https://review.opendev.org/#/q/8459ed9460ab492adfd305875c717ebe31bf68f3 Files to analyze: - lower-constraints.txt ------------------------------- Commit to analyze: https://review.opendev.org/#/q/568b559b67b0d9a663172d681fe887bf06feea0a Files to analyze: - tox.ini ------------------------------- Commit to analyze: https://review.opendev.org/#/q/0d277e9cbb42404b1b1082afdfa5bcfc49691e6f Files to analyze: - lower-constraints.txt ------------------------------- Commit to analyze: https://review.opendev.org/#/q/53dcbe8584434c21b1a340763e9ee419dc58117f Files to analyze: - .zuul.yaml ------------------------------- Commit to analyze: https://review.opendev.org/#/q/348771951fa6e9740b629e81227cc0311751a471 Files to analyze: - lower-constraints.txt ------------------------------- Commit to analyze: https://review.opendev.org/#/q/1831a5b1016563681df6905114f0dc95efd24391 Files to analyze: - designate/locale/ko_KR/LC_MESSAGES/designate.po ------------------------------- Commit to analyze: https://review.opendev.org/#/q/19da5d79d5d24dfda31773f70971f640856b10b1 Files to analyze: - releasenotes/source/locale/en_GB/LC_MESSAGES/releasenotes.po ------------------------------- Commit to analyze: https://review.opendev.org/#/q/a0d20514c6ebbfe6b42b989b6192ae77312d3ba7 Files to analyze: - lower-constraints.txt ------------------------------- [1] https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra…

1 0

[qa] Gerrit breach and commit audit
by Ghanshyam Mann 22 Oct '20

22 Oct '20

Hello Everyone, As per the announcement of the Gerrit breach [1], I have audited all repos under QA for 1. Suspicious commit 2. Core group list and everything is fine. - https://etherpad.opendev.org/p/qa-gerrit-breach-audit [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html -gmann

1 0

[manila] Gerrit breach and commit audit
by Goutham Pacha Ravi 22 Oct '20

22 Oct '20

Hello Zorillas, I join the list of contributors to thank the infra team for their work in isolating, protecting and fixing the recent vulnerability. I looked through the code changes committed to repositories we maintain and didn't find any malicious code submissions or approvals. If you would like to take a look, here is an Ethercalc with the changes: https://ethercalc.openstack.org/uscp1x7i8ewn Please do respond here in case you notice anything amiss. Thank you! Goutham ---------- Forwarded message --------- From: Ian Wienand <iwienand(a)redhat.com> Date: Tue, Oct 20, 2020 at 5:33 PM Subject: [service-announce] October 20 Gerrit Outage Update To: <service-announce(a)lists.opendev.org> As of this mail, Gerrit access has been restored. Please read on for important information, especially around change verification. Background ----------- On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval to a change that they were not expected to have access to. At 02:06 UTC an alert was raised via IRC. Administrators found the account had added themselves to a core group and made the +W vote. The account was disabled, and removed from the groups it had added itself to by 02:55 UTC. Administrators began to analyse the situation and Gerrit was taken offline at 04:02 UTC to preserve state and allow for analysis. >From this time, administrators were working on log collection and analysis, along with restoring backups for comparison purposes. By around 08:45 UTC it was clear that the privilege escalation had been achieved by gaining control of a Launchpad SSO account with Gerrit administrator privileges. By this time, we had ruled out software vulnerabilities. Logs showed the first unauthorized access of the administrator account in Gerrit on 2020-10-06. Communication with Launchpad admins agrees with this analysis. We saw one session opened as the administrator user to StoryBoard on this same day, but logs show no data was modified or hidden stories viewed. Analysis has been performed on the Gerrit database and git trees from October 1st, pre-dating any known unauthorized access. Access was restored at around 2020-10-21 00:00 UTC Outcomes ----------- The following has been verified: The administrator account used has been disabled and credentials updated We have verified that all group and user addition/removals since Oct 1 are valid. The only invalid additions were made by the compromised administrator account to add a single user account to the Administrators group; and then that account added itself to another known group. The account given administrator privilege has been removed from the groups it added itself to and is disabled. There is no evidence of any unauthorized access via methods other than Gerrit HTTP and Gerrit SSH access. No commits have been pushed to git trees bypassing code review. Every git tree has been compared to the Oct 1 version and all commits have been correctly inserted via Gerrit changes. The version of Gerrit we use stores HTTP API passwords in plain-text. We know that a limited number of passwords were gathered via the HTTP API and it is possible passwords were gathered via the database. We thus have assumed that all HTTP API passwords have been disclosed. This password needs to be explicitly enabled by users, and many users do not have it enabled. Remediation ----------- This leaves us with the following remediation actions: Users should double-check their Launchpad recent activity at https://login.launchpad.net/activity for any suspicious logins. If found, please notify the OpenDev admins in Freenode #opendev and Launchpad admins in #launchpad immediately. All HTTP API passwords have been cleared. If you push changes via HTTPS (instead of typical SSH), are a gertty user, or run a CI system or something else that communicates with the Gerrit HTTP API, you will need to regenerate a password. Any SSH keys added to accounts since 2020-10-01 have been removed. This affects only a limited number of accounts. This is done in an abundance of caution, and we do not believe any accounts had unauthorized SSH keys added We should audit all changes for projects since 2020-10-01. We have no evidence that any account had its ssh keys compromised, thus we can rule out any unauthorized changes being uploaded via SSH. However we can not conclusively rule out that compromised HTTP API passwords were used to push a change through Gerrit. For example, a change could be uploaded that looks like it came from a user, or the API key of a core team member may have been used to approve a change without authorization. Given our extensive analysis we consider it exceedingly unlikely that this vector was used. We have had no notifications of users seeing unexpected changes either uploaded by them, or approved by them in projects they work on. This said, we believe it is important to inform the community of this very unlikely, but still possible, vulnerability of the source code. To this end, we have prepared a list of all changes from the known affected period which should be audited for correctness. These are available at https://static.opendev.org/project/opendev.org/gerrit-diffs/ Team members should browse these changes and make sure they were correctly approved in Gerrit. If any change looks suspicious you should notify OpenDev administrators in Freenode #opendev immediately. Further actions ---------------- We are planning the following for the short term future: The Opendev administrators will be looking at alternative models for Gerrit admin account management. We are already well into planning and testing a coming upgrade to a version of Gerrit which does not store plain-text API keys. Longer term, we've written a spec for replacing Launchpad SSO as our authentication provider. We thank you for your patience during this trying time, and we look forward to returning to supporting the community doing what it does best -- working together to create great things. _______________________________________________ service-announce mailing list service-announce(a)lists.opendev.org http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

1 0

[neutron] Same physical NIC for SR-IOV and other mechanism driver
by Gabriel Omar Gamero Montenegro 21 Oct '20

21 Oct '20

Dear all, Is it possible to use the same physical NIC with the SR-IOV mechanism driver together with other mechanism driver like Open vSwitch or Linux Bridge? If this is the case, how should I map the physical NIC for each agent? I only known how to map the NIC separately for each driver; for example: - for the Linux bridge agent: physnet1: eth3, - for the OvS agent: physnet1: br-provider (eth3 is a port of br-provider) - for the SR-IOV agent: physnet1: eth3 Any guide or sample reference about implementing multiple mechanism drivers would be highly appreciated. Thanks in advance, Gabriel Gamero

2 1

Octavia amphora missing network interface
by Dibbo, Alexander (STFC,RAL,SC) 21 Oct '20

21 Oct '20

Hi All, I'm having an issue with my Octavia deployment in Train. When a load balancer is created I can see the amphora get created with an interface on the management network. After a short while an interface is added for our external network. The external network interface never becomes responsive. If I ssh in to the amphora I can only see the management interface present. If I the reboot the amphora, the external interface appears and becomes responsive. Given this it seems like some kind of race condition. I have tried using all of the test amphora images that are published but the centos8 one seems the closest to working. Is anyone familiar with this issue or a fix? Thanks Alex Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support(a)gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

3 3

[designate][octavia] Gerrit breach and commit audit - OK
by Michael Johnson 21 Oct '20

21 Oct '20

I did not see any unusual commits on the lists for either the Octavia or Designate repositories. Thanks, Michael

1 0

[charms] Gerrit breach and commit audit
by Frode Nordahl 21 Oct '20

21 Oct '20

Hello all, On the back of the service announcement [0] from OpenDev admins this morning the OpenStack Charms team have performed an audit of the provided commit blobs. We have found everything to be in order for the repositories we manage. The sheet used for tracking has been made available for review [1]. I would like to thank everyone on the team for the prompt response in dealing with this issue, and we want to thank OpenDev for due diligence in handling the situation. Cheers! 0: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html 1: https://docs.google.com/spreadsheets/d/1jxswC09mPCrtbAchNQBH5_RO95CTEOJ6VUr… -- Frode Nordahl

1 0