November 2024 - openstack-discuss

[neutron] Drivers meeting canceled Friday, November 8th
by Brian Haley 08 Nov '24

08 Nov '24

Hello Neutrinos: I'm going to cancel tomorrow's drivers meeting as there is no agenda and I will be offline. I will triage incoming ones next week so plan to meet next Friday. Have a nice weekend! -Brian

1 0

[Cinder][Ceph] Volumetria errada no cluster ceph
by lfsilva＠binario.cloud 07 Nov '24

07 Nov '24

Hello, I'm experiencing an issue with the volume usage of the Ceph cluster, which is currently used in OpenStack for volumes. I am working with Ceph version Octopus (15.2.17). When I run the ceph df command, I get this output: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 120 TiB 33 TiB 87 TiB 87 TiB 72.72 TOTAL 120 TiB 33 TiB 87 TiB 87 TiB 72.72 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 32 69 MiB 101 137 MiB 0 11 TiB images 2 256 1.3 TiB 166.90k 2.5 TiB 10.72 11 TiB vms 3 64 574 KiB 21 2.5 MiB 0 11 TiB volumes 4 2048 41 TiB 3.94M 82 TiB 79.42 11 TiB backups 5 1024 407 GiB 111.85k 818 GiB 3.63 11 TiB And when I run the rbd du -p volumes command, I get this total: NAME PROVISIONED USED <TOTAL> 14 TiB 8.3 TiB This "volumes" pool is currently set to replica 2, and mirroring is not enabled. I have checked for any locked or deleted snapshots but found none. I also ran a ceph osd pool deep-scrub volumes, but it didn't resolve the issue. Has anyone encountered this problem before? Could someone provide assistance?

3 21

[kolla] Vacation - kevko
by Michal Arbet 07 Nov '24

07 Nov '24

Hello, I would like to inform you that I will be on vacation from November 7th, 2024, to November 23rd, 2024. Thank you, Michal Arbet Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet(a)ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

1 0

Pointers to FOSDEM devrooms CfPs
by Felix Kronlage-Dammers 07 Nov '24

07 Nov '24

Hi, in the last couple weeks the Call for Participations for the various FOSDEM[1] devrooms have been published. In case not everyone knows FOSDEM: FOSDEM is _the_ yearly European free and open source software event where people from around the globe meet and discuss. Aside from the main track FOSDEM consists mainly of many devrooms which focus on certain topics. I wanted to quickly highlight a few of the relevant devrooms for the OpenStack audience and would like to encourage to submit talks :) An overview of the devrooms can be seen here: https://fosdem.org/2025/schedule/ The CfP for the room is linked from the details page of each devroom. Virtualization and Cloud Infrastructure: https://lists.fosdem.org/pipermail/fosdem/2024q4/003574.html Confidential Computing: https://falder.org/fosdem25-cfp Software Defined Storage: https://lists.fosdem.org/pipermail/fosdem/2024q4/003563.html Monitoring and Observability: https://docs.google.com/document/d/1KQmNqzN3cO29ZZUSpd83REvJY_QhbvxIYr4IRBr… The SCS projects organizes, together with the german ZenDiS and the french DINUM, a room with the topic ‘Government Collaboration’. As part of the room we would love to receive proposals that cover: - Standardization and certification of digital sovereign solutions and infrastructure - Automating deployment and operations of such solutions on sovereign infrastructure - Examples and success stories of digital sovereign cloud computing by the public sector The CfP for this room is here: https://lists.fosdem.org/pipermail/fosdem/2024q4/003608.html Am sure there are more relevant devrooms :) An easy overview is also the mailing list archive: https://lists.fosdem.org/pipermail/fosdem/2024q4/thread.html regards felix [1] <url: https://fosdem.org/2025/>

1 0

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 06 Nov '24

06 Nov '24

Hello OpenStack Community, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base image includes a number of packages (such as Apache) that don’t appear to be necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though they aren’t used by all containers. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

[publiccloud-sig] Reminder: next meeting November 6th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 05 Nov '24

05 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 6th) the next meeting of the publiccloud-sig is going to happen. We’re going to do a quick re-cap from PTG as well as the ALASCA Summit that happened last week. Furthermore I’d like to see wether we can spark some activity around the discoverability service that we’ve been thinking and discussing about in the past. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you tomorrow! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

debugging eventlet services in vscode.
by Sean Mooney 05 Nov '24

05 Nov '24

hi folks i recently had to debug a problematic interaction between eventlet and a native thread in watcher. while i probably could have printed my way though that eventually i decided to try and get debugging working in vscode instead. in this case i happened to be looking at watcher but this should apply to any eventlet service installed with devstack. in my .vscode directoy i created a launch.json that looks like this. { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python Debugger: watcher-applier", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-applier", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } }, { "name": "Python Debugger: watcher-decision-engine", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-decision-engine", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } } ] } there are a few odd things about this config. first the program i am launching is the console script run in the devstack@* systemd service file. you can find what that is using systemctl show devstack@<name> to make this work without conflict do `sudo systemctl stop devstack@<name>` first. after your done debugging you can just start the systemd service again. second when using vscode to view the source of a project select the devstack global virtual env as the project virtual env. third im telling vscode to enable gevent support. now this will raise a warning/error as gevent is not installed but you can ignore that. for those of you not familiar with gevent its a fork of eventlet that actually has much better integration with debuggers due to its wider usage outside of openstack. eventlet and gevent both use greenlet and the eventlet implementation is close enough that when you enable gevent debugging it mostly works. forth to fix the "mostly" part, specifically the fact that dajngo will explode on import, we set "PYTEST_CURRENT_TEST": "true" in the terminal env. PYTEST_CURRENT_TEST is an undocumented environment variable intended only for unit testing that is check by eventlet https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… when we set that to true we skip this `isinstance(o, rlock_type)` check https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… which crashes the debugpy debugger because it tries to load setting form the danjgo module which get loaded into the python interpreter as a side effect of debugpy. i have not tried this on other project yet but it enables me to single step debug, run to breakpoints and many of the normal debug action like using the debug console to inspect the current function scope. all of the above worked without any code change to watcher or running in a specal debug mode in the service code. we occasionally get questions on this list about how to debug openstack services and i usually respond with "its possible to do, is often fragile and need a lot of context to do right" in this case it seam to work reasonable ok in the limited usage i have made of it so far. given the current incantation is pretty simple and non invasive i decided to share this in case others find it useful. keep in mind however most service are distributed systems so if you leave it stopped on a break point for 5 mins while you inspect things, what ever you were originally doing will probably time out and it might fail in odd ways but for local development that may still be fine. please don't try this in production. i have not tried this with a wsgi app. that probably will work fine with the eventlet wsgi web-server via the console scripts i.e. nova-api binary but probably wont work for uwsig/Apache. regards sean

2 1

[cinder][PTG] 2025.1 Epoxy PTG summary
by Jon Bernard 05 Nov '24

05 Nov '24

Hello everyone, thanks for a very productive PTG for the Epoxy cycle. Below is a brief summary of the highlights. The raw etherpad notes are here [1] and both the summary and transcribed etherpad notes are captured in our wiki here [2]. # Retrospective We spent the first few minutes discussing the state the Dalmatian release, what went well, not so well, and how we can improve in the Epoxy release cycle. The most notable point raised is that we have two less cores this cycle. This adds additional strain to an already sizable review queue. We will be looking to cultivate and elevate any qualified contributors if the opportunity arises. # NetApp We covered 6 different topics related to NetApp's feature development plans for the Epoxy cycle in an effort to make sure we're all aligned. Below is a summary of the full set: * Certificate-based authentication Right now only basic auth is implemented, which requires a username and password. There is a strong desire to add support for certificate-based auth. Discussions covered possible approaches and a few helpful links to existing implementations that can be used as a reference. * HA Active-Active support for NVMe driver This will allow users to configure NVMe driver backends in cinder clustered environments. We covered the failover process and which methods needed to be implemented. The related use cases were discussed as well as testing scenarios. We do not have replication and failover coverage in our upstream gate, so special care is needed in testing and verification on the developer side at the moment. Mention of Rally for testing as a possibility. * SVM scoping support for NVMe Desire to add support to the NVMe driver to SVM scoping. At the moment the driver fails to initialize. Work is ongoing to add full support for this feature. Nothing concerning was raised, looking forward to this for Epoxy. * ASA r2 support ASA r2 systems use storage units and consistency groups to simplify storage management and data protection. The REST API is different, and some features are restricted. We'll see how this unfolds in this cycle, good to have this on the radar as we go forward. * Replication - Snapmirror Sync Support Synchronous mirroring achieves a recovery point objective (RPO) of zero lost data by having a copy of important data available if a disaster happens on one of the two storage arrays. The copy is identical to production data at every moment because each time a write is done to the primary volume, a write is done to the secondary volume. The host does not receive an acknowledgment that the write was successful until the secondary volume is successfully updated with the changes that were made on the primary volume. * Metrocluster Support Metro cluster is a clustering solution with synchronous replication to deliver continuous availability, immediately duplicating all of the mission-critical data on a transaction-by-transaction basis. MetroCluster configurations enhance the built-in high availability and non disruptive operations of NetApp hardware and ONTAP storage software, providing an additional layer of protection for the entire storage and host environment. A few changes beyond driver-specific updates were discussed, no blockers. # Handling MYPY Updates We support mypy in our gate jobs, but having the version unpinned causes occasional breakages. We pin hacking and similar tools, and we agreed that mypy should be included. The caveat being we need to remember to periodically unpin and update these version numbers to stay current. # New volume driver LVM+Clone Jan Horstmann proposed a derivative of our LVM driver to add support for using device mapper clone target for transparently migrating volumes. This would all an admin to leverage local storage for performance and still be able to live migrate instances across compute hosts. A POC/Demo is hopefully coming soon, no fundamental blockers were identified, very interested to see how this comes together. # Reviews for backup service It is very difficult to get reviews for cinder-backup related changes. Tobias asked if any cores can focus some time on cinder-backup specifically (an potential backports). We agreed that mention of the backup review queue and review dashboard at the weekly meeting may help raise awareness. # New Location API adoption Rajat proposed a patch that adds the call for the `add_image_location` method which triggers the new location API workflow that is more secure and robust than the old location workflow. This call is made when glance is using cinder as a backend and we want to perform an optimized "upload volume to image" operation. Additional patches are up that add support for testing this feature. We may still need some input from the Glance team, to be followed up on in our weekly meeting. # Adding `service` query parameter to AZ API Tobias proposed an API change, adding a query parameter to an existing API. Specifically, adding `service` query parameter to os-availability-zone API. The patch is thought to be complete and ready for review. # Adding a backup summary API Tobias proposed adding a backup summary API (same way that is available for volumes). He also found a volume summary API bug when implementing this feature. The patch is ready for review. # Horizon Feature Gap We had the privilege to meet with the Horizon team and discuss their efforts on closing existing feature gaps. Several specifics were discussed and clarity was achieved. Concerns were raised over possible incompatibility issues between the openstack client and the SDK. Further, Horizon updates microversion only when necessary and value was identified in reviewing microversions to see if any additional functionality can be leveraged. # Optimize RBD update from Cinder to Glance Currently the path to upload a cinder volume to glance image is not optimized when cinder and glance are both using RBD. We can leverage RBD layering (or copy-on-write cloning) to complete the operation in seconds irrespective of the volume size (since no actual copy is happening). There are 3 features implemented that were needed to support this feature: 1. RBD clone dependency deletion: We have landed changes in glance and cinder that handle the RBD dependency chains so dependency from volumes to images pools shouldn't be an issue 2. Service role: Currently the attachment delete API already uses service role + service token and we can use similar changes for this feature 3. New location APIs: This new glance feature eliminates the security risk as we will be calculating the image hash even in this optimized path (in background so our operation performance won't be affected) * Previously several blockers, appear to be all resolved now * Significant performance increase * Glance -> Cinder already supported * Fallback to unoptimized codepath * Related, recent patches to improve RBD cinder/glance, need to be considered/reviewed Eric has committed to help with the reviews. # State of volume replication Some time was spent discussing the current state of replication. We do not have testing coverage in tempest, so the current state of any particular driver implementation is difficult to know. An idea was raised to poll vendors for use cases and testing results. There is a general lack of clarity for this feature across projects and what our overall disaster recovery solution is. # Image Encryption (Glance Cross-Project) The goal with this effort is to standardize encrypted images between Glance, Cinder and Nova, in an effort to make them consistent and accessible to users. Here is a recap: 1. We merged a patchset in os-brick so far that extends key handling to support more types of Barbican keys for image encryption in addition to Cinder's existing implementation (which strictly uses binascii.hexlify() conversion). 2. Patchset for Glance has been written to add Secret Consumer API from Barbican for encrypted images and to handle the new standardized format. 3. Patchset for Cinder has been written, making it able to produce and use the new standardized format + additionally support qcow2+LUKS in addition to raw LUKS format as well as non-hexlified keys from Barbican. After the initial discussion with Glance, we agreed on *not* introducing any new container_format or disk_format and instead handle this with `os_encrypt_*` properties instead (e.g. os_encrypt_format=luks); this is how the patchsets are currently implemented. However, due to the recent CVEs around image format parsing and conversion, valid concerns were raised that this approach might lead to new vulnerabilities due to potential conflicts and ambiguities between container_format, disk_format and the encryption attributes. We need to agree on a new way of handling the classification of encrypted images in the metadata. We have at least two cases to consider: 1. Raw LUKS (as currently produced by Cinder from encrypted volumes) 2. qcow2+LUKS (as produced by Nova for Ephemeral Storage); both could also be produced by the user (see the gist link in the reference section). # Volume type metadata progress We received a progress update on the volume type metadata effort. Consensus that the direction is sound. A new API endpoint in addition to changes to the response of volume type show / create will require a new microversion. The tempest test still need to be adjusted. Patches need review. # Status of OpenAPI effort Very little traction on initial patches, needs help getting reviews for the first set of patches. Patches also cleanup technical debt and fix several bugs, would be very good to have. The next set of patches is more meaningful, but being held until initial reviews are available. # Removal of uWSGI scripts (stephenfin) The patch to remove uWSGI scripts has been up and needs review. Relatively small patch. Update: this has been reviewed and merged. # Eventlet A dedicated etherpad was created to help document current status. Cinder is still in the investigatory phase. Not only do we use eventlet directly, some drivers such as RBD use C libraries that spawn native threads. We must take care in how we handle these cases. Our goal is only preparatory changes in Epoxy (because it is a SLURP release), and hold big changes for F (which most people won't upgrade to, so will give us more time to detect & fix issues) [1] https://etherpad.opendev.org/p/epoxy-ptg-cinder [2] https://wiki.openstack.org/wiki/CinderEpoxyPTGSummary -- Jon

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-21)
by Goutham Pacha Ravi 05 Nov '24

05 Nov '24

Hello Stackers, Thanks to all moderators and volunteers for compiling virtual PTG notes and sharing summaries through this mailing list. A lot of exciting work seems to be tracked through the action items from these summaries. I hope you were able to review the TC's PTG summary [1]. Next week is milestone-1 of the 2025.1 "Epoxy" release cycle [2]. This is typically a bug-targeting milestone for project teams. The following is a summary of this past week's happenings within the OpenStack Technical Committee. === Weekly Meeting === During the past week, the OpenStack Technical Committee met over IRC [3]. We followed up on action items from weeks preceding the Project Teams Gathering. We finalized PTL and DPL appointments for the "mistral" and "watcher" project teams. We spent some time reviewing action items related to translating documentation and user strings within OpenStack services. Automated jobs posting translation updates from Zanata are broken due to incompatibility with newer Python versions. Migration to Weblate remains a priority and the i18n team may require more support to complete this migration. We're collaborating over IRC to identify the needs and next steps. The TC also discussed progress on the community goal to migrate integration testing to Python 3.12 and to the latest LTS version of the operating system used in DevStack test jobs. Like me, you may have wondered why we’re combining these efforts. The reason is that it allows us to have a simpler test matrix despite the number of Python versions and OS distributions that OpenStack software is tested with. While we maintain unit test jobs for each supported Python version, it is impractical to duplicate each integration job for every OS distribution available. Therefore, we choose the latest LTS (e.g., Ubuntu 24.04 LTS / "Noble Numbat"), which includes the upper bound of the supported Python versions (Python 3.12). Additionally, each project will have at least one test job for the minimum supported Python version (Python 3.9), so you'll see a test job for the previous LTS release of the OS that will begin in this release and continue running as it transitions to a "stable" release branch. The next meeting of the OpenStack Technical Committee is on 2024-11-05 (today) at 1800 UTC. This is our monthly video meeting too! This meeting will occur simultaneously on Video (Zoom) and IRC (OFTC's #openstack-tc channel). Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenStack VMware Migration Working Group [4] will be joining us to discuss the working group's efforts. Meeting information, including other agenda items, is on the meeting wiki [5]. I hope you'll be able to join us. Please remember that you can always propose items to the TC's weekly meeting agenda. === Governance Proposals === ==== Merged ==== Add Watcher DPL for Epoxy | https://review.opendev.org/c/openstack/governance/+/933018 Adding Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 === How to Contact the TC === You can reach the TC in several ways: 1. **Email:** Send an email with the tag [tc] on this mailing list. 2. **IRC:** Use the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. 3. **Weekly Meeting:** Join us every Tuesday at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" TC PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [3] TC Meeting IRC Log 2024-10-29: https://meetings.opendev.org/meetings/tc/2024/tc.2024-10-29-18.00.log.html [4] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [5] TC Meeting Agenda, 2024-11-05: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

Re: GPU PCI passthrough woes.
by Andy Speagle 05 Nov '24

05 Nov '24

Aye, that seems to have done it. Strange that this wasn't necessary for my old 1080 cards. Thanks! On Wed, 2024-10-30 at 08:12 +0000, Alexander Dibbo - STFC UKRI wrote: > ⚠️CAUTION⚠️ > This email originated outside of Toyon's network. Do not download > attachments or click links unless you know the content is safe. > > Hi Andy, > > We've seen many issues with pci passthrough and I suspect that the > l40s will have a seen similar issues with the a100s > > Here is an alias string that works for our a100s: > `alias = {"vendor_id": "10de", "product_id": "20f1", "device_type": > "type-PF", "name": "nvidia-tesla-a100-pcie-vga"}` > > I believe it is the `device_type` that will be your issue > > Regards > > Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader > For STFC Cloud Documentation visit > https://urldefense.us/v3/__https://stfc.atlassian.net/wiki/spaces/CLOUDKB/o… > To raise a support ticket with the cloud team please email > cloud-support(a)stfc.ac.uk > To receive notifications about the service please subscribe to our > mailing list at: > https://urldefense.us/v3/__https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=S… > To receive fast notifications or to discuss usage of the cloud please > join our Slack: > https://urldefense.us/v3/__https://stfc-cloud.slack.com/__;!!A4KLn-w15EM4BR… > > -----Original Message----- > From: Andy Speagle <aspeagle(a)toyon.com> > Sent: 30 October 2024 02:15 > To: openstack-discuss(a)lists.openstack.org > Subject: GPU PCI passthrough woes. > > Hey Folks, > > I could use a little assistance getting GPU passthrough working. I > had > this working already for one flavor of nvidia gpu... and I've added > some hosts with a much newer gpu... > > I've updated the pci_alias and pci_passthrough variables and those > seem > to be getting set properly in nova.conf > > passthrough_whitelist = [{"vendor_id":"10de", > "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] > > alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} > alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": > "10de"} > > I believe I have all of the iommu stuff configured and have the pci- > stub module entries... dmesg output shows that the GPUs are being > claimed by the stub module. > > $ openstack flavor show t1.small_gpu_l40s > +----------------------------+--------------------------------------+ > > Field | Value | > +----------------------------+--------------------------------------+ > > OS-FLV-DISABLED:disabled | False | > > OS-FLV-EXT-DATA:ephemeral | 0 | > > access_project_ids | None | > > description | None | > > disk | 0 | > > id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | > > name | t1.small_gpu_l40s | > > os-flavor-access:is_public | True | > > properties | pci_passthrough:alias='gpu-l40s:1' | > > ram | 2048 | > > rxtx_factor | 1.0 | > > swap | 0 | > > vcpus | 1 | > +----------------------------+--------------------------------------+ > > Yet... I can't seem to get an instance to run using that new > flavor... > keeps complaining about there not being enough hosts available. > > Fault: > code: 500 > created: 2024-10-30T02:04:33Z > message: "No valid host was found. There are not enough hosts > available." > details: | > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 1580, in schedule_and_build_instances > host_lists = self._schedule_instances(context, > request_specs[0], > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 940, in _schedule_instances > host_lists = self.query_client.select_destinations( > File "/usr/lib/python3/dist- > packages/nova/scheduler/client/query.py", line 41, in > select_destinations > return self.scheduler_rpcapi.select_destinations(context, > spec_obj, > File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", > line 160, in select_destinations > return cctxt.call(ctxt, 'select_destinations', **msg_args) > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/client.py", line 189, in call > result = self.transport._send( > File "/usr/lib/python3/dist- > packages/oslo_messaging/transport.py", line 123, in _send > return self._driver.send(target, ctxt, message, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send > return self._send(target, ctxt, message, wait_for_reply, > timeout, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send > raise result > nova.exception_Remote.NoValidHost_Remote: No valid host was > found. > There are not enough hosts available. > > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/server.py", line 241, in inner > return func(*args, **kwargs) > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 223, in select_destinations > selections = self._select_destinations( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 250, in _select_destinations > selections = self._schedule( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 416, in _schedule > self._ensure_sufficient_hosts( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 455, in _ensure_sufficient_hosts > raise exception.NoValidHost(reason=reason) > > nova.exception.NoValidHost: No valid host was found. There are > not > enough hosts available. > > Any clues on how/where to dig into this more to see what might be > missing? Thanks. > > -- > Andy Speagle > -- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431

3 2