October 2020 - openstack-discuss - lists.openstack.org

[kuryr] Gerrit breach and commit audit
by Maysa De Macedo Souza 21 Oct '20

21 Oct '20

Hello, Following the request on the announcement of the gerrit breah [1], the Kuryr team have audited all the Kuryr repositories. Nothing suspicious was seen. Thank you. Best regards, Maysa Macedo. [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

Re: [service-announce] October 20 Gerrit Outage Update
by Slawek Kaplonski 21 Oct '20

21 Oct '20

Hi, Hi, Both Lajos and me checked today all neutron and neutron stadium changes there. And all looks ok for us. I didn't notice anything wrong or suspicious there. Thx Lajos for help with that :) Dnia środa, 21 października 2020 09:49:49 CEST Slawek Kaplonski pisze: > On Wed, Oct 21, 2020 at 11:33:14AM +1100, Ian Wienand wrote: > > As of this mail, Gerrit access has been restored. Please read on for > > important information, especially around change verification. > > > > Background > > ----------- > > > > On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval > > to a change that they were not expected to have access to. At 02:06 > > UTC an alert was raised via IRC. Administrators found the account had > > added themselves to a core group and made the +W vote. The account > > was disabled, and removed from the groups it had added itself to by > > 02:55 UTC. Administrators began to analyse the situation and Gerrit > > was taken offline at 04:02 UTC to preserve state and allow for > > analysis. > > > > From this time, administrators were working on log collection and > > analysis, along with restoring backups for comparison purposes. > > > > By around 08:45 UTC it was clear that the privilege escalation had > > been achieved by gaining control of a Launchpad SSO account with > > Gerrit administrator privileges. By this time, we had ruled out > > software vulnerabilities. Logs showed the first unauthorized access > > of the administrator account in Gerrit on 2020-10-06. Communication > > with Launchpad admins agrees with this analysis. We saw one session > > opened as the administrator user to StoryBoard on this same day, but > > logs show no data was modified or hidden stories viewed. > > > > Analysis has been performed on the Gerrit database and git trees from > > October 1st, pre-dating any known unauthorized access. > > > > Access was restored at around 2020-10-21 00:00 UTC > > > > Outcomes > > ----------- > > > > The following has been verified: > > The administrator account used has been disabled and credentials > > updated > > > > We have verified that all group and user addition/removals since > > Oct 1 are valid. The only invalid additions were made by the > > compromised administrator account to add a single user account to > > the Administrators group; and then that account added itself to > > another known group. > > > > The account given administrator privilege has been removed from > > the groups it added itself to and is disabled. > > > > There is no evidence of any unauthorized access via methods other > > than Gerrit HTTP and Gerrit SSH access. > > > > No commits have been pushed to git trees bypassing code review. > > Every git tree has been compared to the Oct 1 version and all > > commits have been correctly inserted via Gerrit changes. > > > > The version of Gerrit we use stores HTTP API passwords in > > plain-text. We know that a limited number of passwords were > > gathered via the HTTP API and it is possible passwords were > > gathered via the database. We thus have assumed that all HTTP API > > passwords have been disclosed. This password needs to be > > explicitly enabled by users, and many users do not have it > > enabled. > > > > Remediation > > ----------- > > > > This leaves us with the following remediation actions: > > Users should double-check their Launchpad recent activity at > > https://login.launchpad.net/activity for any suspicious logins. If > > found, please notify the OpenDev admins in Freenode #opendev and > > Launchpad admins in #launchpad immediately. > > > > All HTTP API passwords have been cleared. If you push changes via > > HTTPS (instead of typical SSH), are a gertty user, or run a CI > > system or something else that communicates with the Gerrit HTTP > > API, you will need to regenerate a password. > > > > Any SSH keys added to accounts since 2020-10-01 have been removed. > > This affects only a limited number of accounts. This is done in > > an abundance of caution, and we do not believe any accounts had > > unauthorized SSH keys added > > > > We should audit all changes for projects since 2020-10-01. > > > > We have no evidence that any account had its ssh keys compromised, > > thus we can rule out any unauthorized changes being uploaded via SSH. > > However we can not conclusively rule out that compromised HTTP API > > passwords were used to push a change through Gerrit. For example, a > > change could be uploaded that looks like it came from a user, or the > > API key of a core team member may have been used to approve a change > > without authorization. > > > > Given our extensive analysis we consider it exceedingly unlikely that > > this vector was used. We have had no notifications of users seeing > > unexpected changes either uploaded by them, or approved by them in > > projects they work on. This said, we believe it is important to > > inform the community of this very unlikely, but still possible, > > vulnerability of the source code. > > > > To this end, we have prepared a list of all changes from the known > > affected period which should be audited for correctness. These are > > available at > > > > https://static.opendev.org/project/opendev.org/gerrit-diffs/ > > > > Team members should browse these changes and make sure they were > > correctly approved in Gerrit. If any change looks suspicious you > > should notify OpenDev administrators in Freenode #opendev immediately. > > > > Further actions > > ---------------- > > > > We are planning the following for the short term future: > > The Opendev administrators will be looking at alternative models > > for Gerrit admin account management. > > > > We are already well into planning and testing a coming upgrade to > > a version of Gerrit which does not store plain-text API keys. > > > > Longer term, we've written a spec for replacing Launchpad SSO as > > our authentication provider. > > > > We thank you for your patience during this trying time, and we look > > forward to returning to supporting the community doing what it does > > best -- working together to create great things. > > > > > > _______________________________________________ > > service-announce mailing list > > service-announce(a)lists.opendev.org > > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[storlets] Gerrit breach and commit audit
by Takashi Kajinami 21 Oct '20

21 Oct '20

Hi, As suggested by the recent announcement[1], I have audited all commits (actually, 1 commit) in storlets repository, and confirmed that everything looks good now. [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html Thank you, Takashi

1 0

[masakari] Branches reviewed post-gerrit-breach
by Radosław Piliszek 21 Oct '20

21 Oct '20

Hi Folks, I have reviewed Masakari repos and no suspicious commits were found. Thank you, OpenDev team, for handling the situation and providing patches list. -yoctozepto

1 0

[cloudkitty] Gerrit breach and commit audit
by Pierre Riteau 21 Oct '20

21 Oct '20

Hello, Following the announcement of the Gerrit breach [1], I have audited all commits in repositories of the CloudKitty project. Everything looks good. Thank you to the OpenDev admins for dealing with this issue efficiently. Pierre [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[blazar] Gerrit breach and commit audit
by Pierre Riteau 21 Oct '20

21 Oct '20

Hello, Following the announcement of the Gerrit breach [1], I have audited all commits in repositories of the Blazar project. Everything looks good. Thank you to the OpenDev admins for dealing with this issue efficiently. Pierre [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

Openstack client can't load modules
by Cristina Mayo 21 Oct '20

21 Oct '20

Hi everyone, Openstack client installed on an Ubuntu 20.04 is showing these errors when I launch as non root user; indeed it is fine with sudo: Could not load 'metric_metric_create': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricCreate' Could not load 'metric_metric_delete': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricDelete' Could not load 'metric_metric_list': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricList' Could not load 'metric_metric_show': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricShow' I don't have noticed any bad behaviour using it, but I'm not sure. Any help? Thanks

3 2

[barbican] gerrit breach and auditing
by Moises Guimaraes de Medeiros 21 Oct '20

21 Oct '20

Hi all, I'm putting up a review spreadsheet[0] for the barbican changes since October 1st based on Julia's spreadsheet for Ironic[1]. [0]: https://docs.google.com/spreadsheets/d/1WdhxDsAZCHlzhRpcH0jKfyY2kzO2h2jRzni… [1]: http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018148.… -- Moisés Guimarães Software Engineer Red Hat <https://www.redhat.com> <https://red.ht/sig>

1 0

[glance] Wallaby PTG schedule
by Abhishek Kekane 21 Oct '20

21 Oct '20

Hello All, Greetings!!! Wallaby PTG is around the corner and if you haven't already registered, please do so as soon as possible [1]. I have created a Virtual PTG planning etherpad [2] and also added day wise topics along with timings we are going to discuss. Kindly let me know if you have any concerns with allotted time slots. We also have some slots open on Thursday and Friday for unplanned discussions. So please feel free to add your topics if you still haven't added yet. As a reminder, these are the time slots for our discussion. Tuesday 27 October 2020 1400 UTC to 1700 UTC Folsom Wednesday 28 October 2020 1400 UTC to 1700 UTC Folsom Thursday 29 October 2020 1400 UTC to 1700 UTC Folsom Friday 30 October 2020 1400 UTC to 1700 UTC Folsom We will be using bluejeans for our discussion, kindly try to use it once before the actual discussion. The meeting URL is mentioned in etherpad [2] and will be the same throughout the PTG. [1] https://october2020ptg.eventbrite.com/ [2] https://etherpad.opendev.org/p/glance-wallaby-ptg Thank you, Abhishek

1 0

[kolla] Branches reviewed post-gerrit-breach
by Radosław Piliszek 21 Oct '20

21 Oct '20

Hi Folks, The Kolla team has reviewed its projects and no suspicious commits were found. Thank you, OpenDev team, for handling the situation and providing patches list. Thank you, Mark, Michał, Marcin, Pierre and myself for reviewing. Let's roll with development. :-) -yoctozepto

1 0

[release][core] gerrit breach and auditing all releases deliverables [1] since Oct 01
by Herve Beraud 21 Oct '20

21 Oct '20

Hello, As every team we are also concerned by the gerrit breach and we must take a look at our changes during this time frame on all our deliverables [1]. The majority of our changes during this period are changes related to releasing new deliverables, that mean changes on yaml files. Due to the criticality of the problem I propose the following action plan: - first, focus us on changes against our scripts and executable files; - second, inspect yaml files if teams identify possible issues in their code base. I don't think we need to inspect our `deliverable/*/*.yaml` files for now, it could be done later; - third, inspect documentation changes. They could be easily ignored for now and analyzed later too. I wrote a script [2][3] to help us to extract relevant changes (*.py, *.sh), all the rest ((deliverable|data)*.yaml, *.rst) have been ignored for now. I ran this script against the repos that we own, here is the result: - openstack/release-test - No changes merged since more than 3 months - openstack/releases - https://review.opendev.org/#/q/9e527f9e0e0cac29699171813b45c65b487010dc - https://review.opendev.org/#/q/9c4f3c43039fa686cee6d03fb6d9894de05c89a6 - https://review.opendev.org/#/q/a8ab91eebc7a0b9d0be863772ea286627e6b57d9 - openstack/reno - No changes merged since more than 2 months - openstack/specs-cookiecutter - No changes merged since more than 4 months Example of script usage against our openstack/releases repos: ``` $ cd releases $ curl https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra… | sh ``` I'll take a look at the identified commits but double checking would be greatly appreciated. Please ensure to double check your account activity [4] and make sure nothing is off. Special congrats to Julia Kreger and for her excellent job [5]. Thank you in advance for your help, [1] https://governance.openstack.org/tc/reference/projects/release-management.h… [2] https://gist.github.com/4383/511359cc2080e06295944c5f40bd1033 [3] https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra… [4] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018148.… -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1

[nova][placement] Reviewing changes after gerrit admin account compromise
by Balázs Gibizer 21 Oct '20

21 Oct '20

Hi, As per [0] we are requested to check commits merged since 1st of October to nova and related repos. Lee and I went through nova master and stable branches and they seem to be clean. I also went through nova-specs, python-novaclient, os-vif, placement, osc-placement changes and those are seems to clean too. Still, if you check and see any suspicious things please contact OpenDev administrators in Freenode #opendev Cheers, gibi [0] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[openstack][magnum] No meeting for this week
by feilong 21 Oct '20

21 Oct '20

Hi team, Due to the Summit this week, we won't hold the weekly Magnum meeting on 21 October. -- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------

1 0

Implementation of project Trove in our Openstack-based Environment
by Bekir Fajkovic 20 Oct '20

20 Oct '20

Hello! My name is Bekir Fajkovic, i am resided in Sweden and i work as a member of a DBA team at City Network company that provides cloud-based services and solutions based mostly on OpenStack. We are about to evaluate the OpenStack project Trove in order to try to provide a reasonable and robust kind of DBaaS solution to our customers. As a first step i created a test environment based on DevStack (based in turn on latest release of OpenStack) including Trove (version 14.1.0.dev18) and i am currently evaluating the things. I see many promising features, like Docker-based deployments of database instances on dedicated images possible to be created and configured through Trove utilities, also some important security improvements of RabbitMQ message handling ensuring encryption key usage in communication with the db instances etc.. I do not see MSSQL datastore type provided in the initial setup but i guess that is a subject of extra configuration that can be carried out by us. So, i wonder about the status of the project in general, to begin with. According to some rumours going around, the project is to be abandoned soon by the developers and i would like to get that confirmed or not, just in order for us to make further decisions in regards to DBaaS development. I know that many DBaaS projects are nowadays based on Kubernetes and development of specific operators that provide certain kind of automated database services, but we are still interested about Trove as a native DBaaS project that well integrates into OpenStack and provides multitenancy, redundancy, and so on. I would be very pleased if we could get more in contact with OpenStack Trove Team somehow, to establish some sort of dialogue and to see more what we could build and provide based on that project. Best Regards Bekir Fajkovic Senior DBA Mobile: +46 70 019 48 47 www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED

2 1

[charms] HA, haproxy https backend checks
by Xav Paice 20 Oct '20

20 Oct '20

Hi, Looking at https://review.opendev.org/#/c/750224/, we've got a change that switches haproxy from looking at tcp socket connection, to http. I'd like to update that change to enable the same for when apache is providing https, and use the /healthcheck endpoint. See https://github.com/juju/charm-helpers/pull/521 for a change to the haproxy template to enable this, and I'd submit a fresh patchset for the change but can't right now with opendev in maintenance. I'm wondering re zaza tests for this, should I be adding bundles that have 3 units of Glance (to prove HA) and likely Vault so that we can check https? That would increase the bundle size a chunk, but I think it would improve the overall testing of the charm. Thoughts?

1 0

How to integrate Designate with Nova/Horizon for internal domains
by Daniel Heitepriem 20 Oct '20

20 Oct '20

Dear list, I'm in the process of setting up an Openstack installation for internal use. The environment consists of these nodes: - 1 controller node - 2 compute nodes My goal is now to get Designate working for internal domain resolution paired with automatic record creation from Nova/Neutron. The steps I did so far: - Installed and configured Designate (https://docs.openstack.org/designate/ussuri/install/install-ubuntu.html) and Designate dashboard for Horizon (https://docs.openstack.org/designate-dashboard/ussuri/install/index.html) - Adjusted handler "neutron_floatingip" [handler:neutron_floatingip] notification_topics = notifications_designate control_exchange = neutron formatv4 = '%(octet0)s-%(octet1)s-%(octet2)s-%(octet3)s.%(zone)s' formatv4 = '%(hostname)s.%(project)s.%(zone)s' formatv4 = '%(hostname)s.%(zone)s' - Adjusted handler "nova_fixed" [handler:nova_fixed] notification_topics = notifications_designate control_exchange = nova formatv4 = '%(octet0)s-%(octet1)s-%(octet2)s-%(octet3)s.%(zone)s' formatv4 = '%(hostname)s.%(project)s.%(zone)s' formatv4 = '%(hostname)s.%(zone)s' - Installed Designate-Sink and configured notification_handlers [service:sink] workers = 5 enabled_notification_handlers = "nova_fixed, neutron_floatingip" - Added notification topic "notifications_designate" to nova.conf and neutron.conf Let's say I have one additional domain configured in Openstack next to the default domain: "domain1.internal". So I logged in to the new domain in Horizon, created a few networks, created the new zone record "domain1.internal." and started a new instance called "test-vm". Expectations: Nova creates the instance, Neutron creates the required network port and Designate Sink receives the "Port created" notification via the "notifications_designate" topic Reality: Nova creates the instance, Neutron creates the required network port but Designate Sink does nothing. In the designate-api.log I can see the creation of the "domain1.internal." zone but the designate-sink.log stays empty 2020-10-20 08:29:19.573 3524893 INFO designate.api.v2.controllers.zones [req-31b7fa68-6621-4044-985e-f90c87dd8cae - - - - -] Created <Zone id:'d7c015ec-5079-450b-866d-f04042221d80' type:'PRIMARY' name:'domain1.internal.' pool_id:'794ccc2c-d751-44fe-b57f-8894c9f5c842' serial:'1603182558' action:'CREATE' status:'PENDING'> My question is: What piece of configuration did I miss? Is my desired configuration even possible at all? What I basically want to achieve is: - Create an instance using nova - The instance name automatically get's published in all(?) DNS zones of the project/Openstack domain - Instances can resolve other instance names out of this DNS zone Thank you very much in advance and regards, Daniel Heitepriem P.S.: Can you please keep my in CC as I'm not subscribed to the list

2 1

[Ironic] Teaching virtualbmc how to talk to Ironic?
by Lars Kellogg-Stedman 20 Oct '20

20 Oct '20

In the work that we're doing with the Mass Open Cloud [1], we're looking at using Ironic (and the multi-tenant support we contributed) to manage access to a shared pool of hardware while still permitting people to use their own provisioning tools. We don't want to expose the hardware BMC directly to consumers; we want Ironic to act as the access control mechanism for all activities involving the hardware. The missing part of this scenario is that at the moment this would require provisioning tools to know how to talk to the Ironic API if they want to perform BMC actions on the host, such as controlling power. While talking with Mainn the other day, it occurred to me that maybe we could teach virtualbmc [2] how to talk to Ironic, so that we could provide a virtual IPMI interface to provisioning tools. There are some obvious questions here around credentials (I think we'd probably generate them randomly when assigning control of a piece of hardware to someone, but that's more of an implementation detail). I wanted to sanity check this idea: does this seem reasonable? Are there alternatives you would suggest? Thanks! [1] https://github.com/CCI-MOC/esi [2] https://github.com/openstack/virtualbmc -- Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {irc,twitter,github} http://blog.oddbit.com/ | N1LKS

6 12

[cinder] no weekly meeting this week (or next)
by Brian Rosmaita 20 Oct '20

20 Oct '20

Due to the Summit this week, we won't hold the weekly cinder meeting on 21 October (tomorrow). Due to the PTG next week, we won't need to hold the weekly meeting on 28 October because we'll be discussing cinder stuff all week. The PTG planning etherpad is here: https://etherpad.opendev.org/p/wallaby-ptg-cinder-planning The next Cinder meeting will be at the usual place and time: Wednesday 4 November in #openstack-meeting-alt at 1400 UTC. (People who have been observing Daylight Savings Time, please check whether the meeting time may have shifted an hour earlier for you.) The cinder meeting agenda is available here: https://etherpad.opendev.org/p/cinder-wallaby-meetings Looking forward to seeing everyone virtually at the PTG! brian

1 0

Problem with octavia LBaaS and nova availability zones
by bjoernputtmann＠netprojects.de 20 Oct '20

20 Oct '20

Hi one and all! Currently, we have an ussuri openstack installation via kolla-ansible. The installation consists of three control nodes and two availability zones (az-1, az-2), each with three compute nodes: - os-cpt-10[1-3] > az-1 - os-cpt-20[1-3] > az-2 az-1 is the default compute availability zone. The compute nodes are hardwarewise equipped exactly the same. As storage system a ceph cluster is used. We wanted to also use Octavia LBaaS and got it up and running after some experimentation. We also wanted to be able to choose the availibility zones when starting a loadbalancer. We added the az info with: openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-1 --provider amphora --availability-zone-data '{"compute_zone": "az-1"}' openstack --os-cloud service_octavia loadbalancer availabilityzoneprofile create --name az-2 --provider amphora --availability-zone-data '{"compute_zone": "az-2"}' openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-1 --availabilityzoneprofile az-1 openstack --os-cloud service_octavia loadbalancer availabilityzone create --name az-2 --availabilityzoneprofile az-2 Creating a loadbalancer via cli: openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR --name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ works, as long as $LB_AZ == az-1. If we want to start the loadbalancer in az-2, this fails with an error in octavia-worker.log: ... 2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z', 'message': 'No valid host was found. There are not enough hosts available.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 96, in select_destinations\n allocation_request_version, return_alternates)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 265, in _schedule\n claimed_instance_uuids)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", line 302, in _ensure_sufficient_hosts\n raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. There are not enough hosts available.\n\n'} ... We enabled debug logging for nova and found: ... 2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by "nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" :: held 0.003s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304 2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep", "sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep", "abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1", "rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2", "clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8", "umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36", "mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht", "pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2", "movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic", "wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm", "cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt", "rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm", "sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1373295770000000}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 386827280000000}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 139696145240000000}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:07.846401", "source": "libvirt.LibvirtDriver", "value": 3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", "reserved", "size_kb", "total"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["mempages", "cpu_usage", "memory", "memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata", "siblings", "cpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 27, 'uuid': '62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['c88b77f7-2c59-497b-9633-139e62ba2bb1', '635fbd95-dd64-4e0a-9767-a0e0a6958610', 'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-103', 'os-cpt-103')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.007s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch", "fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic", "adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de", "movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov", "perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd", "cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr", "clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec", "mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero", "pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht", "rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse", "abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep", "fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36", "clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 139958595980000000}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 388866090000000}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 1118743070000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source": "libvirt.LibvirtDriver", "value": 3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["reserved", "used", "total", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory", "cpuset", "siblings", "mempages", "network_metadata", "memory_usage", "pinned_cpus"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 21, 'uuid': 'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: ['96f4c101-d75b-44b6-8af9-dd75068d207d', '121c5df5-2d8b-48e0-a270-9f6f79193dd6', 'e235063d-f7a8-4f38-8294-d7a289167348', '425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-101', 'os-cpt-101')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: waited 0.000s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358 2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state from compute node: ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", "model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, "sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm", "clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx", "pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca", "xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec", "invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext", "ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat", "lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht", "smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni", "tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp", "cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core", "fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq", "perfctr_nb", "smep", "de", "3dnowprefetch", "xsave", "tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name": "cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 161246250000000}, {"name": "cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 140551238730000000}, {"name": "cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 366274340000000}, {"name": "cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 3829690000000}, {"name": "cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": "libvirt.LibvirtDriver", "value": 0.0}]',numa_topology='{"nova_object.name": "NUMATopology", "nova_object.namespace": "nova", "nova_object.version": "1.2", "nova_object.data": {"cells": [{"nova_object.name": "NUMACell", "nova_object.namespace": "nova", "nova_object.version": "1.4", "nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], "siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], [7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, "total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}, {"nova_object.name": "NUMAPagesTopology", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, "total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", "used", "reserved", "size_kb"]}], "network_metadata": {"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", "nova_object.version": "1.0", "nova_object.data": {"physnets": [], "tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, "nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages", "memory", "siblings", "network_metadata", "cpuset", "memory_usage", "id", "pcpuset"]}]}, "nova_object.changes": ["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2) _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172 2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with aggregates: [Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with service dict: {'id': 18, 'uuid': '342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary': 'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled': False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False, 'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178 2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Update host state with instances: [] _locked_update /usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182 2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Lock "('os-cpt-102', 'os-cpt-102')" released by "nova.scheduler.host_manager.HostState.update.<locals>._locked_update" :: held 0.005s inner /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370 2020-10-13 13:50:49.266 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Starting with 3 host(s) get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:70 2020-10-13 13:50:49.266 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-101, os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-102, os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.267 33 DEBUG nova.scheduler.filters.availability_zone_filter [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Availability Zone 'az-2' requested. (os-cpt-103, os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs: {'az-1'} host_passes /usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61 2020-10-13 13:50:49.268 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filter AvailabilityZoneFilter returned 0 hosts 2020-10-13 13:50:49.269 33 DEBUG nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: [('AvailabilityZoneFilter', None)] get_filtered_objects /usr/lib/python3.6/site-packages/nova/filters.py:115 2020-10-13 13:50:49.269 33 INFO nova.filters [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtering removed all hosts for the request with instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: ['AvailabilityZoneFilter: (start: 3, end: 0)'] 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] Filtered [] _get_sorted_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443 2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler [req-390e67a0-5078-4b5d-a2b5-0111e8a1527b f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - default default] There are 0 hosts available but 1 instances requested to build. _ensure_sufficient_hosts /usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300 ... So nova seems to only consider compute nodes in az-1. The amphora image is available in both availability zones. Starting an instance from the image directly in az-2 works. Does anybody have any idea on how to debug this any further? Any hints, tips or ideas would be greatly appreciated! With kind regards, Björn Puttmann

2 3

How to change nova db in Train ?
by Yingji Sun 20 Oct '20

20 Oct '20

Buddies, I have a question about updating nova db. I would like to change the length of internal_access_path in console_auth_tokens, starting from Train. I want to modify nova/db/sqlalchemy/models.py and the migration file under nova/db/sqlalchemy/migrate_repo/versions. However I did not find a placeholder file for Train. 402_add_resources.py is the last one in Train and 403_ is "Add reserved schema migrations for Ussuri". So how can I do it in Train ? Yingji.

3 4

packstack fails every time - pymysql.err.OperationalError: (1040, 'Too many connections')
by lejeczek 20 Oct '20

20 Oct '20

2 1

Ubuntu Focal Ussuri can't create a snapshot from running instance
by Cristina Mayo Sarmiento 20 Oct '20

20 Oct '20

Hi all, I don't have a lot experience with Openstack. I've installed Ussuri on my cloud. I have 1 controller node and 2 compute nodes. I'm not able to create a snapshot from a running instance, but it works perfectly if the instance is turned off. I checked the logs on controller node about glance-api and POST method executes fine, but suddenly appears a DELETE that seems remove the image: "POST /v2/images HTTP/1.1" 201 1254 0.045379 "GET /v2/images/c647160b-416d-4c98-bbe2-6302a509f 941 HTTP/1.1" 200 1097 0.014154 WARNING glance.api.v2.images [req-e29704f9-e147-42df-914c-6217c60ada30 c63d05ad0e9d43ebb0fd31f442156a2a c2 b7d6242f3140f09d283e8fbb88732a - default default] After upload to backend, deletion of staged image data has failed because it cannot be found at /tmp/staging//c647160b-416d-4c98-bbe2-6302a509f941 "DELETE /v2/images/c647160b-416d-4c98-bbe2-6302a5 09f941 HTTP/1.1" 204 213 0.140731 I have these versions installed: - python3-glance-store/focal,now 2.0.0-0ubuntu1 all [instalado, automático] - python3-glance/focal-updates,now 2:20.0.1-0ubuntu1 all [instalado, automático] Could anyone help me? Thanks in advance, Regards,

3 3

[largescale-sig] "Share your OpenStack scaling story" forum session
by Thierry Carrez 20 Oct '20

20 Oct '20

Hi everyone, In a few hours at 14:30UTC we'll have a Forum session where everyone is invited to share their scaling story. As you add nodes to an OpenStack cluster, at one point you hit scaling limitations that force you to scale out to multiple clusters, cells or zones. What breaks first? At which scale? The Large Scale SIG is working on pushing back the scaling limits within a single cluster. To that effect, it is collecting scaling stories from OpenStack deployers. This should lead to identifying common cases, usual suspects that the SIG can focus on, in order to raise scaling limitations within one cluster and reduce the need to scale out to multiple clusters too early. Join fellow OpenStack operators and share your experience and horror stories, adding nodes to OpenStack clusters! In case the event platform plays games again, you can access the session directly at: https://zoom.us/j/94011749848?pwd=Qmo5c1ZMK3RJWnVRSjZzZWFkTng3QT09 And the etherpad is at: https://etherpad.opendev.org/p/vSummit2020_OpenStackScalingStory See you all there! -- Thierry Carrez (ttx)

1 1

[neutron] bug deputy report (week of Oct 12)
by Akihiro Motoki 20 Oct '20

20 Oct '20

Hi, I was a bug deputy in neutron last week. Here is a summary. It was a quiet week and fixes have been proposed for medium or higher priority bugs. I see no bug to be raised from POV of bug triaging. [In Progress] https://bugs.launchpad.net/neutron/+bug/1899967 L3HA Router in primary state on all nodes Critical, In Progress, slaweq proposed a fix https://bugs.launchpad.net/neutron/+bug/1899917 neutron-tempest-plugin-scenario-linuxbridge jobs are broken on rocky/queens branches Critical, In Progress, slaweq proposed a fix https://bugs.launchpad.net/neutron/+bug/1899775 Ovs agent import error on Windows High, In Progress, a fix was proposed for review https://bugs.launchpad.net/neutron/+bug/1899864 Smartnic port's MTU is not set correctly Medium, In Progress, a fix was proposed for review [Confirmed] https://bugs.launchpad.net/neutron/+bug/1899502 router_centralized_snat ports do not have project_id Low, Confirmed, Assigned to the bug author Looks like we agree it is a valid bug. [Wishlist] These are bugs for tracking refactoring. https://bugs.launchpad.net/neutron/+bug/1900270 Implement "find_child_pids" natively Wishlist, In Progress, Assigned to ralonsoh https://bugs.launchpad.net/neutron/+bug/1900268 Implement "find_parent_pid" natively Wishlist, In Progress, Assigned to ralonsoh https://bugs.launchpad.net/neutron/+bug/1900269 Implement "get_process_count_by_name" natively Wishlist, In Progress, Assigned to ralonsoh Thanks, Akihiro Motoki (irc: amotoki)

1 0

[openstack][stein][cinder] python2-os-brick issue orphaned device
by Ignazio Cassano 20 Oct '20

20 Oct '20

Hello Guys, I am using cinder on centos7 stein with Dell unity iscsi driver. When I migrate vm from a kvm node to another, orphaned devices are left on source kvm node. I initialized debug on nova compute and i thing the issue si in the following lines (2020-10-20 09:52:33.078 132171 DEBUG os_brick.initiator.connectors.iscsi [-] Disconnecting from: []): connectors/iscsi.py:1111 2020-10-20 09:52:33.078 132171 DEBUG os_brick.initiator.connectors.iscsi [-] Resulting device map defaultdict(<function <lambda> at 0x7f4f1b1f7cf8>, {(u'10.138.215.17:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.a2'): (set([]), set([u'sdg', u'sdi'])), (u'10.138.209.47:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.b3'): (set([]), set([u'sdo', u'sdq'])), (u'10.138.209.48:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.a3'): (set([]), set([u'sdd', u'sdb'])), (u'10.138.215.18:3260', u'iqn.1992-04.com.emc:cx.ckm00184400687.b2'): (set([]), set([u'sdm', u'sdk']))}) _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:844 2020-10-20 09:52:33.078 132171 DEBUG os_brick.initiator.connectors.iscsi [-] Disconnecting from: [] _disconnect_connection /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1099 The version of package is: python2-os-brick-2.8.7-1.el7.noarch Please, any help ? Ignazio

1 0

[Automation SIG] BoF today! feel free to join
by Rico Lin 19 Oct '20

19 Oct '20

Hi all, Automation SIG is hosting a BoF on Tue 20, 1:00pm - 1:45pm (18:00 - 18:45 UTC) - Virtual - Virtual Floor - Forum Room 2 https://summit.openinfra.dev/a/event/24753 Please join if you're interested. -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 0

[Multi-arch SIG][OpenInfra Summit] BoF today! Feel free to join us
by Rico Lin 19 Oct '20

19 Oct '20

Hi all As a forming SIG, we hope to have more hands or resources to keep pushing things forward. In this summit, we will host a BoF on Tuesday this week and hope to have people to join. Multi-arch SIG Tue 20, 12:15pm - 1:00pm (17:15 - 18:00 UTC) - Virtual - Virtual Floor - Forum Room Multi-arch SIG BoF https://summit.openinfra.dev/a/event/24752 Also if you would like to see some related sessions here's some you can consider: Multi-Arch Support in OpenStack https://summit.openinfra.dev/a/event/24708 The Progress for Cloud Computing on Arm architecture https://summit.openinfra.dev/a/event/24719 Cloud-hypervisor: A new choice for virtual machine monitor https://summit.openinfra.dev/a/event/24684 -- May The Force of OpenStack Be With You, *Rico Lin*irc: ricolin

1 0

[nova] NUMA scheduling
by Eric K. Miller 19 Oct '20

19 Oct '20

Hi, I'm at a loss for finding good information about how a VM's vCPUs and Memory are assigned to NUMA nodes within a scheduled physical host. I think Libvirt does this, and the Nova Scheduler simply finds the right physical host to run the VM, and thus Nova has no input on which NUMA node to choose. So this might be a Libvirt question. We are running Stein and have the issue where VMs launch on NUMA Node 0, and not on NUMA Node 1, in physical hosts with two processors, and are simply looking for a way to tell Libvirt to consider NUMA Node 1 when scheduling a VM, since there is nearly all of the memory available on NUMA Node 1. Our flavors are defined with hw:numa_nodes='1' since we want all vCPUs+Memory to land on a single NUMA Node, and so the guest OS has visibility that a single NUMA Node is being used. We are "not" looking for a way to pin a VM to a specific NUMA node (such as for SR-IOV purposes). Any suggestions where to look for the solution? Thanks! Eric

5 17

[qa] Weekly meeting cancelled because of Summit and PTG
by Masayuki Igawa 19 Oct '20

19 Oct '20

Hi, We'll skip the meetings both this and next week because of the summit and PTG. See you there :) -- Masayuki Igawa

1 0

[openstack-helm] No meeting this week and next week
by Gage Hugo 19 Oct '20

19 Oct '20

Hello everyone, due to the summit and PTG there will be no meeting this week or next week. We will meet again on November 3rd at the usual time, thanks!

1 0

[Security SIG] No meeting this week or next week
by Gage Hugo 19 Oct '20

19 Oct '20

Hello everyone, due to the summit and PTG there will be no meeting this week or next week. We will meet again on November 5th at the usual time, thanks!

1 0

Forum Etherpads
by Kendall Nelson 19 Oct '20

19 Oct '20

Hello All! I wanted to encourage you to add your etherpads to the wiki for the Forum[1] like we normally do. Sorry I didn't sooner! -Kendall Nelson [1] https://wiki.openstack.org/wiki/Forum/Virtual2020

1 0

[announce][debian][victoria] general availability of OpenStack Victoria in Debian
by Thomas Goirand 19 Oct '20

19 Oct '20

Hi! It's my pleasure to announce the general availability of OpenStack Victoria in Debian. I've just finished uploading everything to Debian Unstable today, and the Buster backports are available the usual way, for example using extrepo (which is in the official Debian backports): apt-get install extrepo extrepo enable openstack_victoria apt-get update ... or directly setting-up the buster-victoria.debian.net repository the usual way. New in this release =================== As all projects are deprecating the policy.json files, all of the services are now using a /etc/<PROJECT>/policy.d/00_default_policy.yaml. The normal /etc/<PROJECT>/policy.json will be automatically deleted, as it can be a problem if it remains. Looking backward, a sensible rename to get the file away would have been nicer to Debian users, so this will be implemented soonish. Implementation in openstack-cluster-installer ============================================= Victoria can already be installed by OCI. The only issue that remains to be fixed, is puppet-openstack setting-up a /etc/keystone/puppet.conf wrongly in the controllers. Once the file contains the correct things, deployment can go on. Also, it looks like LVM volume nodes still need some fixes. Apart from these early glitches, everything seems working smoothly so far, and of course, these will be addressed soon. I'll be testing upgrades using the usual oci-cluster-upgrade-openstack-release script. Testing Victoria over Bullseye and Python 3.9 ============================================= The day OpenStack was released, Python 3.9 was uploaded to Debian Unstable. I've already sent a few patches to address some problems, but I haven't yet fully deployed OpenStack on top of Bullseye. Since Victoria will be the release for Bullseye, I will spend time polishing all of this before the forthcoming freeze in February. Hopefully, I'll also be able to finish scripting for upgrading a cluster from Buster to Bullseye before the freeze, just like one was shipped in OCI for Stretch to Buster upgrades. Cheers, Thomas Goirand (zigo)

1 0

Re: CentOS 8 Ussuri can't launch instance /usr/libexec/qemu-kvm: Permission denied
by Oliver Weinmann 19 Oct '20

19 Oct '20

Ok, I will try to disable selinux and deploy one more compute node. I just stumbled across another issue, not sure if it is related. The instance seems to be deployed just fine but now I looked on the console and neither cirros nor centos 7 seem to be booting up correctly. on cirros i see an error: [ 0.846019] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- and on centos7: error: not a correct XFS inode. I tried to create with ephemeral and volume. Cheers, Oliver Am 19. Oktober 2020 um 16:09 schrieb Alex Schultz <aschultz(a)redhat.com>: On Mon, Oct 19, 2020 at 7:59 AM Oliver Weinmann <oliver.weinmann(a)me.com> wrote: First of all thanks a lot for the quick reply. I just checked and it seems that the package is really not available for centos8 from the upstream repo: https://centos.pkgs.org/8/centos-appstream-x86_64/podman-1.6.4-15.module_el… When you say it should be available via rdo, does this mean I have to add or use a different repo when deploying undercloud / overcloud? I have followed the tripleo guide to deploy it: I thought we shipped it, maybe we don't because we run with selinux disabled so it doesn't show up in CI. https://docs.openstack.org/tripleo-docs/latest/ And is there a way to disable selinux on all overcloud nodes by default? I guess it is the default to disable it? Set the following in an environment file as part of the deployment: parameter_defaults: SELinuxMode: permissive Cheers, Oliver Am 19. Oktober 2020 um 15:29 schrieb Alex Schultz <aschultz(a)redhat.com>: On Mon, Oct 19, 2020 at 7:09 AM Oliver Weinmann <oliver.weinmann(a)me.com> wrote: Hi all, I have successfully deployed the overcloud many many times, but this time I have a strange behaviour. Whenever I try to launch an instance it fails. I checked the logs on the compute node and saw this error: Failed to build and run instance: libvirt.libvirtError: internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied googling led me to the solution to disable selinux: setenforce 0 I have not made this change persistent yet, as I would like to know why I'm facing this issue right now. What is actually the default for the overcloud nodes SeLinux? Enforcing, permissive or disabled? I build the ipa and overcloud image myself as I had to include drivers. Is this maybe the reason why SeLinux is now enabled, but is actually disabled when using the default ipa images? From a TripleO perspective, we do not officially support selinux enabled when running with CentOS. In theory it should work, however it is very dependent on versions. I think you're likely running into an issue with the correct version of podman which is likely causing this. We've had some issues as of late which require a very specific version of podman in order to work correctly with nova compute when running with selinux enabled. You need 1.6.4-15 or higher which I don't think is available with centos8. It should be available via RDO. Related: https://review.opendev.org/#/c/736173/ Thanks and Best Regards, Oliver

1 0

Introducing the Open Infrastructure Foundation
by Kendall Nelson 19 Oct '20

19 Oct '20

Hello Everyone, If you’re not watching the Open Infrastructure Summit keynotes right now, you just missed some big news (and it’s not too late to tune in <https://www.youtube.com/watch?v=wNNOFfDBB0o&ab_channel=OpenStackFoundation>!). Jonathan Bryce just announced the Open Infrastructure Foundation (OIF) <http://openinfra.dev> as the successor to the OpenStack Foundation (OSF). The name change reflects all of our community’s work building open infrastructure with dozens of open source components, and we could not have done it without your help. We look forward to continuing to collaborate with you over the next decade to build open source communities that write software for production. The Summit keynotes <https://www.youtube.com/watch?v=wNNOFfDBB0o&ab_channel=OpenStackFoundation> are still going on the YouTube live stream, so login now to hear more about this announcement and how OpenStack and other open source projects fit into the next decade of open infrastructure. You can also watch Imtiaz present an update today on what the community is doing and support them in the chat as they encourage Summit attendees to participate. Tomorrow Dawson Coleman and myself will also be presenting on the community and the last two releases. And if you haven’t already, join as an individual member <http://openinfra.dev/join> so you can get involved! Let’s get started! -Kendall Nelson (diablo_rojo) Upstream Developer Advocate The Open Infrastructure Foundation

1 0

Summit issues
by Kanevsky, Arkady 19 Oct '20

19 Oct '20

I was able to join summit. But I am perpetually stuck in "checking credentials..." Any recommendations? Thanks, Arkady

7 6

Re: CentOS 8 Ussuri can't launch instance /usr/libexec/qemu-kvm: Permission denied
by Oliver Weinmann 19 Oct '20

19 Oct '20

First of all thanks a lot for the quick reply. I just checked and it seems that the package is really not available for centos8 from the upstream repo: https://centos.pkgs.org/8/centos-appstream-x86_64/podman-1.6.4-15.module_el… When you say it should be available via rdo, does this mean I have to add or use a different repo when deploying undercloud / overcloud? I have followed the tripleo guide to deploy it: https://docs.openstack.org/tripleo-docs/latest/ And is there a way to disable selinux on all overcloud nodes by default? I guess it is the default to disable it? Cheers, Oliver Am 19. Oktober 2020 um 15:29 schrieb Alex Schultz <aschultz(a)redhat.com>: On Mon, Oct 19, 2020 at 7:09 AM Oliver Weinmann <oliver.weinmann(a)me.com> wrote: Hi all, I have successfully deployed the overcloud many many times, but this time I have a strange behaviour. Whenever I try to launch an instance it fails. I checked the logs on the compute node and saw this error: Failed to build and run instance: libvirt.libvirtError: internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied googling led me to the solution to disable selinux: setenforce 0 I have not made this change persistent yet, as I would like to know why I'm facing this issue right now. What is actually the default for the overcloud nodes SeLinux? Enforcing, permissive or disabled? I build the ipa and overcloud image myself as I had to include drivers. Is this maybe the reason why SeLinux is now enabled, but is actually disabled when using the default ipa images? From a TripleO perspective, we do not officially support selinux enabled when running with CentOS. In theory it should work, however it is very dependent on versions. I think you're likely running into an issue with the correct version of podman which is likely causing this. We've had some issues as of late which require a very specific version of podman in order to work correctly with nova compute when running with selinux enabled. You need 1.6.4-15 or higher which I don't think is available with centos8. It should be available via RDO. Related: https://review.opendev.org/#/c/736173/ Thanks and Best Regards, Oliver

2 1

Divisive Language stance final(?) draft and Forum session
by Amy Marrich 19 Oct '20

19 Oct '20

Hi everyone, Below is what we hope is the final draft of the OSF's stance on Divisive Language. If you have any comments please place them on the etherpad[0] and/or join us this Tuesday during the forum[1]. Please note that decisions related to what words should be replaced with are being left with the project's technical leadership with the knowledge that context is important to determine the best alternative. That said, the Diversity and Inclusion WG will be available to assist these groups. Thanks, Amy Marrich (spotz) 0 - https://etherpad.opendev.org/p/divisivelanguage 1- https://www.openstack.org/summit/2020/summit-schedule/events/24778/divisive… 1a - https://zoom.us/j/94295083896?pwd=VmZCTFN3eERDK1ltRHRyWDl0eG1hZz09 1b - https://etherpad.opendev.org/p/vSummit2020__DivisiveLanguage ------------------ The OpenStack Foundation (OSF) Board of Directors supports removal of wording identified as oppressive, racist and sexist by members of our communities from the software and documentation they produce. While we know there will be challenges, both technical and non-technical, this is an action we feel is important. These efforts are the responsibility of the various technical leadership groups within our communities, and we trust them to make appropriate decisions about applicability, timelines, minimizing impact to users and operators, and choosing the changes that make the most sense for their projects. Contributors will take care to make changes to software in the least disruptive way possible. While standardized wording is a laudable goal, we recognize that different implementation contexts might require different solutions. In many cases the work is also complicated by external dependencies for these projects, and their capacity to make necessary changes before we can implement ours. Terminology to which special attention should be paid includes: 1. The use of "slave," or "master" in reference to slavery-oriented relationships, as is currently found in databases, the domain name system, etc. 1. The terms "blacklist" and "whitelist" in various contexts (which might require a variety of different replacements to make sense for those contexts) 1. The use of "master" in non-slavery-related contexts such as revision control branches and documentation builds (pending feedback from our community members) We shall continue to be vigilant for other language areas that cause challenges and work with the community to evolve this policy.

2 1

CentOS 8 Ussuri can't launch instance /usr/libexec/qemu-kvm: Permission denied
by Oliver Weinmann 19 Oct '20

19 Oct '20

Hi all, I have successfully deployed the overcloud many many times, but this time I have a strange behaviour. Whenever I try to launch an instance it fails. I checked the logs on the compute node and saw this error: Failed to build and run instance: libvirt.libvirtError: internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied googling led me to the solution to disable selinux: setenforce 0 I have not made this change persistent yet, as I would like to know why I'm facing this issue right now. What is actually the default for the overcloud nodes SeLinux? Enforcing, permissive or disabled? I build the ipa and overcloud image myself as I had to include drivers. Is this maybe the reason why SeLinux is now enabled, but is actually disabled when using the default ipa images? Thanks and Best Regards, Oliver

2 1

[tripleo][ironic] fails to introspect: my fsm encountered an exception
by Ruslanas Gžibovskis 19 Oct '20

19 Oct '20

Hi all, I have a clean undercloud deployment. but when I add a node, it fails to introspect. in ironic logs I see interesting lines about FSM (what is it?) [1] and in the same paste, I have provided/pasted, how openstack overcloud node import instack.json looks like. [1] http://paste.openstack.org/show/uPwWVYlO3UQbrF0WzDFH/ -- Ruslanas Gžibovskis +370 6030 7030

3 14

[cyborg] Weekly meeting cancelled due to overlapping with Summit and PTG
by yumeng bao 19 Oct '20

19 Oct '20

Hi cyborg members! This is to announce that we will cancel our weekly meetings for the next two weeks(Oct 22 and Oct 29) and resume Nov 05 due to schedule overlapping with the OpenInfra China Hackathon, OpenInfra Summit, and PTG. If you have any questions, please let us know! Regards， Yumeng

1 0

[magnum] error when trying to overwrite master_lb_enabled in cluster create
by Ionut Biru 19 Oct '20

19 Oct '20

Hi guys, In template i have master_lb_disabled and i want to overwrite this option using: » openstack coe cluster create cluster002 --cluster-template b7e0be01-bf3b-4c32-af60-98b9edefa0be --keypair ioni --master-count 2 --node-count 2 --master-lb-enabled master_count must be 1 when master_lb_enabled is False (HTTP 400) (Request-ID: req-608b8634-d98e-4efd-b0e4-543ebc9eb315 Looking at the code https://opendev.org/openstack/magnum/src/branch/master/magnum/api/controlle… https://opendev.org/openstack/magnum/src/branch/master/magnum/api/attr_vali… It seems that the validator doesn't take in consideration the new cluster dictionary. I've modified the validator with: cluster.get('master_lb_enabled', cluster_template['master_lb_enabled']): and it seems to work fine Not sure if it is the right fix. -- Ionut Biru - https://fleio.com

1 0

approach to deploying rhosp 16.1 with octavia on standalone
by Chen Sagi 19 Oct '20

19 Oct '20

Hi, I am trying to figure out if I am able to deploy Red Hat Openstack platform with one physical server available, included with the Octavia solution. For now I have been trying to use the "openstack triplo deploy -standalone" command (with various environment files) and it Is unable to install Octavia without an undercloud available. Is it even possible? If so, I would love to hear some suggestions. Thanks, Chen Sagi.

2 3

Fwd: [ops] [cinder] __DEFAULT__ volume type
by Rajat Dhasmana 19 Oct '20

19 Oct '20

Sorry i forgot to include openstack discuss mailing list with my reply. ---------- Forwarded message --------- From: Rajat Dhasmana <rdhasman(a)redhat.com> Date: Sun, Oct 18, 2020 at 1:06 PM Subject: Re: [ops] [cinder] __DEFAULT__ volume type To: Eugen Block <eblock(a)nde.ag> Hi Eugen, On Sun, Oct 18, 2020 at 12:54 PM Eugen Block <eblock(a)nde.ag> wrote: > Hi, > > I also was confused about that during our upgrade to Train. I noticed > that many volumes in our cloud didn't have any volume type defined at > all, so I had to update the respective table before I could continue > the upgrade process. What do you mean by "updating the respective table"? The optimal way to update a volume with a volume type is: 1) create a volume type pointing to the backend in which the volume currently is 2) retype the volume to that volume type > I have also defined a different volume type in > cinder.conf but I don't think you can just delete the __DEFAULT__ > type, The functionality to delete the __DEFAULT__ type is only available since cinder train >= 15.4.0 (See the releasenotes for more info regarding the __DEFAULT__ type[1]) Also you need to have a valid value defined with ``default_volume_type`` in cinder.conf Let me know if you have respective settings and still not able to delete the __DEFAULT__ type. > I'm not sure anymore but I think I also tried that. But it > doesn't really hurt if you have one of your own types as default. > > Regards, > Eugen > > [1] https://docs.openstack.org/releasenotes/cinder/train.html#relnotes-15-4-0-s… Regards Rajat Dhasmana

2 1

[Image Encryption] No meeting this week and next week
by Josephine Seifert 19 Oct '20

19 Oct '20

Hi, due to the Summit and the PTG, there will be no meeting today and no meeting next week. Next meeting will be at November 2nd. greeting Josephine (Luzi)

1 0

Sorry for the stupid question... it’s been a while
by Florian Rommel 18 Oct '20

18 Oct '20

Hey everyone. So, I have developed openstack, wrote installers and fixed parts in the raw phytonutrients.. but... so I am , for fun and refreshing my memory, building an openstack deployment by hand, the hood old fashioned way. I have a few servers and I get everything working perfectly EXCEPT neutron ovs with dvr integration. I am using Ubuntu focal with ussuri release. When I deploy neutron i get the agents to show up and register with UP. The neutron-openvswitch-agent crashes with physical network floating to br-public not found. I am wondering how in need to configure the bridge in order to make this work. I have a empty interface (internet facing) which I added to br-public but where do I define the floating network name? Yes it’s stupid but I been banging my head against the wall:) Thanks already for help and sorry to anyone else for the dumb question:) //Florian

3 2

[ops] [cinder] __DEFAULT__ volume type
by Massimo Sgaravatto 18 Oct '20

18 Oct '20

I have recently updated my Cloud from Rocky to Train (I am running Cinder v. 15.4.0) I have a question concerning the __DEFAULT__ volume type, that I don't remember to have seen before. Since: - I have no volumes using this volume type - I defined in in the [DEFAULT] section of cinder.conf the attribute "default_volume_type" to a value different than "__DEFAULT___" I assume that I can safely delete the __DEFAULT__ volume type Is this correct? Thanks, Massimo

2 1

[glance][devstack] Problem with stack.sh in devsatck install
by James 17 Oct '20

17 Oct '20

hi guys：       I met some problems in installing devstack . the details please refers to "Errors" listed as below.     The environment: 1) Win 10 ; 2) VMware workstation ;3) ubuntu-18.04.5-desktop-amd64;4)devstack          The installation processes refer to the devstack quick start (https://docs.openstack.org/devstack/latest/) Could you give me the solution to solve the problem. Error : Installing collected packages: oslo.rootwrap, retrying, os-win, oslo.privsep, os-brick, simplejson, python-cinderclient, glance-store Attempting uninstall: simplejson Found existing installation: simplejson 3.13.2 ERROR: Cannot uninstall 'simplejson'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. +inc/python:pip_install:1 exit_trap +./stack.sh:exit_trap:489 local r=1 ++./stack.sh:exit_trap:490 jobs -p +./stack.sh:exit_trap:490 jobs= +./stack.sh:exit_trap:493 [[ -n '' ]] +./stack.sh:exit_trap:499 '[' -f '' ']' +./stack.sh:exit_trap:504 kill_spinner +./stack.sh:kill_spinner:399 '[' '!' -z '' ']' +./stack.sh:exit_trap:506 [[ 1 -ne 0 ]] +./stack.sh:exit_trap:507 echo 'Error on exit' Error on exit +./stack.sh:exit_trap:509 type -p generate-subunit +./stack.sh:exit_trap:510 generate-subunit 1589476087 780 fail +./stack.sh:exit_trap:512 [[ -z /opt/stack/logs ]] +./stack.sh:exit_trap ...

2 1

Your Virtual PTG Checklist
by Kendall Waters 16 Oct '20

16 Oct '20

REGISTRATION If you haven't done so, please register for the PTG! This is how we will be able to provide you with the tooling information and passwords. Register here: https://october2020ptg.eventbrite.com <https://october2020ptg.eventbrite.com/> FINAL SCHEDULE The final schedule [1] for the event is set here and in the PTGBot [2]. IRC The main form of synchronous communication between attendees during the PTG is on IRC. If you are not on IRC, learn how to get started here [3]. The main PTG IRC channel is #openstack-ptg on Freenode. It's used to interact with the PTGbot, and Foundation staff will be present to help answer questions. PTGBOT The PTGbot [3] is an open source tool that PTG track moderators use to surface what's currently happening at the event. Track moderators will send messages to the bot via IRC, and from that information, the bot publishes a webpage with several sections of information: - The discussion topics currently discussed in the room ("now") - An indicative set of discussion topics coming up next ("next") - The schedule for the day with available extra slots you can book Learn more about the ptgbot via the documentation here [4]. HELP DESK We are here to help! If you have any questions during the event week, we encourage you to join the #openstack-ptg IRC channel and ask them there. You can ping Kendall Waters (wendallkaters) and Kendall Nelson(diablo_rojo) directly on IRC. We will also have a dedicated Zoom room, but ONLY FOR MONDAY, October 26 where an OSF staff member will be available to answer your event related questions. You can also always reach someone at ptg(a)openstack.org if you are unable to connect to IRC. FEEDBACK We have preemptively created an etherpad[5] to collect all of your feedback throughout the event. Please add your thoughts as the week goes on. GAME TIME! Since we all know that coming together as a community to hang out and play games is part of what makes the PTG great, we are going to try something new this time around. This PTG, we are going to have two 'game nights'- obviously time zones are hard and it might not be night for everyone, but the times are as follows. - Thursday, October 29 at 19:00 UTC - Friday, October 30 at 10:00 UTC - Friday, October 30 at 23:00 UTC If you are interested in participating, write your name in the etherpad [6]. We will use it to coordinate games for each of the timeslots. [1] PTG schedule: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c0… <https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c0…> [2] PTGbot: http://ptg.openstack.org/ptg.html <http://ptg.openstack.org/ptg.html> [3] How to get started on IRC: https://docs.openstack.org/contributors/common/irc.html <https://docs.openstack.org/contributors/common/irc.html> [4] PTGbot documentation: https://github.com/openstack/ptgbot/blob/master/README.rst <https://github.com/openstack/ptgbot/blob/master/README.rst> [5] Feedback Etherpad: https://etherpad.opendev.org/p/October2020-PTG-Feedback <https://etherpad.opendev.org/p/October2020-PTG-Feedback> [6] Game Time Etherpad: https://etherpad.opendev.org/p/October2020-PTG-Games <https://etherpad.opendev.org/p/October2020-PTG-Games>

1 0

[OSSA-2020-007] Blazar: Remote code execution in blazar-dashboard (CVE-2020-26943)
by Pierre Riteau 16 Oct '20

16 Oct '20

======================================================== OSSA-2020-007: Remote code execution in blazar-dashboard ======================================================== :Date: October 12, 2020 :CVE: CVE-2020-26943 Affects ~~~~~~~ - Blazar-dashboard: <1.3.1, ==2.0.0, ==3.0.0 Description ~~~~~~~~~~~ Lukas Euler (Positive Security) reported a vulnerability in blazar-dashboard. A user allowed to access the Blazar dashboard in Horizon may trigger code execution on the Horizon host as the user the Horizon service runs under. This may result in Horizon host unauthorized access and further compromise of the Horizon service. All setups using the Horizon dashboard with the blazar-dashboard plugin are affected. Patches ~~~~~~~ - https://review.opendev.org/755814 (Stein) - https://review.opendev.org/755813 (Train) - https://review.opendev.org/755812 (Ussuri) - https://review.opendev.org/756064 (Victoria) - https://review.opendev.org/755810 (Wallaby) Credits ~~~~~~~ - Lukas Euler from Positive Security (CVE-2020-26943) References ~~~~~~~~~~ - https://launchpad.net/bugs/1895688 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-26943

1 0