October 2022 - openstack-discuss - lists.openstack.org

[nova][placement] 2023.1 Antelope PTG summary
by Sylvain Bauza 24 Oct '22

24 Oct '22

We're on a wetty Monday here and I guess this is time for me to take my pen again and write a PTG recap email for the current release, which is 2023.1 Antelope. Yet again, I *beg* you to *not* use any translation tool when reading any etherpad, including Google Chrome embedded translation or it would fully translate the whole etherpad content for *all* readers. In order to prevent any accidental misusage, here is a readonly copy of the Nova etherpad we had for the week https://etherpad.opendev.org/p/r.8ae4e0ef997aebfe626b2b272ff23f1b Again, I'm human and while I can write bugs (actually I write a lot of them), I can also write wrong things in this email and I can misinterpretate something precise. Apologies if so, and please don't refrain them yourself to correct me by replying to this email. Last but not the least, I'm more than open to any questions or remarks that would come from the reading of this long email. Good luck with this very long thread, you should maybe grab a coffee before starting to read it (and I promise to keep it as short as I can). ### Operator hours ### After the success of the productive Nova meet-and-greet session we held in Berlin (packed room w/ lots of feedback), we were eager to again have a discussion with operators, this time virtually which would have hopefully lowered the entry barrier by not requiring an in-person event in order to attend the session. Consequently, we allocated three timeslots of one hour, two back-to-back on Tuesday and one on Wednesday (on different times to arrange time differences between operators). Unfortunately, I have to write here that we only had a very small attendance on Tuesday with only three operators joining (but providing great feedback, thanks btw. !) and *none* on the 1-hour Wednesday slot. As a result, I won't provide statistics about release and project usage here as numbers are too low to be representative, but here is what we discussed : (again, a readonly etherpad is available here https://etherpad.opendev.org/p/r.aa8b12b385297b455138d35172698d48 for further context) # Upcoming features and general discussions - the possibility to mount Manila shares in Nova seems promising. - As nova doesn't support multiple physnets per neutron network today, this prevents the use of routed networks in some cases. - modifying our mod_wsgi usage to be allowed to pass some arguments is requested - operators seem interested in having PCI devices tracked in Placement # Pain points - getting inventories or used/availables resources in Placement becomes costly as it requires N calls, with N be the number of Resource Providers (instead of a single HTTP call). Problem has been acknowledged by the team and some design discussion occurred. - We should paginate on the flavors API and we need to fix the private > public flavor bug. - mediated devices disappearing at reboot is a problem. This has been discussed during the contributor session later, see below. - routing metrics are hard to manage when you have a long list of multiple ports attached. We eventually agreed on the fact the proposal fix could be an interesting feature for Neutron team to develop. That's it for the operators discussions. Let's now discuss about what we discussed in the contributors PTG agenda : ### Cross-project discussions Two projects this time were discussing with the Nova community : # Ironic-Nova The whole session was about the inconsistencies that were happening if a nova-compute was failing down with Ironic rebalancing the nodes. - we agreed on the fact this wasn't an easy problem to solve - a consensus was around the fact Ironic should support a sharding key so the Nova compute could use it instead of using hash rings. - JayF and johnthetubayguy agreed on codifying this idea into a spec - we'd hear feedback from operators about what they feel of the above migration (sharding their node cloud into sharding pieces and providing sharding details to nova) - in parallel of the above, documentation has to be amended in order to recommend for *existing deployments* to setup active/passive failover mode for Nova computes (instead of relying on hashring rebalances) # Neutron-Nova (I'll briefly cover the details here, as it will also be covered by the Neutron PTG recap) - while Nova is finishing up tracking of PCI devices in Placement this cycle, we agreed on defering the modeling of Neutron-based PCI requests until next cycle. This won't have any impact in terms of existing features that will continue to operate seamlessly. - the way we define switchdev capabilities as of now is incorrect. We agreed on modfying Nova to allow it to report such capabilities to Neutron. This may be a specless blueprint or just a bugfix (and then potential backportability), to be determined later. - MTUs are unfortunately immutable. If you change the MTU value by Neutron, you need to restart the instance( (or reattach the port). We consequently agreed on documenting this caveat, which is due to libvirt/qemu not able to change the MTU while the guest is running. - We eventually agreed on letting --hostname (parameter of an instance creation request) to represent a FQDN thru a nova microversion. This value won't be sanitized by nova and directly passed to Neutron as the hostname value. We also agreed on the fact Nova WON'T ever touch the port domain value in metadata, as this is not in the project scope to manage name domains. ### Procedural discussions (skip it if you're not interacting with the Nova contributors) ### # Zed retrospective - We had a productive Zed release and we made a good job on reducing bugs and bug reports. Kudos to the team again. - The microversions etherpad we had was nice but planning microversions usage is hard. WE agreed on rather providing a document to contributors explaining them how to write an API change that adds a new microversion and how to easily rebase this change if a merge conflict occurs (due to the micoversion being taken by another patch that merged) - We agreed on keeping an etherpad for tracking all API changes during milestone-3 - We agreed on filing blueprints for TC goals that impact Nova - bauzas will promote again the use of review-priority label in Gerrit during weekly meetings in order for cores to show their interest in reviewing a particular patch. # Promoting backlog features to new contributors and enhance mentoring - Starting this cycle, we'll put our list of small and easily actionable blueprints into Launchpad bug reports that have a "Wishlist" status and both a 'low-hanging-fruit' and a 'rfe' tag. New contributors or people wanting to mentor a new upstream member are more than welcome to consume that list of 'rfe' bugs and identify the ones they're willing to work on. A detailed process will be documented for helping newcomers to join. - We'll also draft our pile of work we defer into 'backlog specs' if they require further design (and are less actionable from a newcomer perspective) # Other procedural and release discussions - we'll stop using Storyboard for tracking Placement features and bugs and we'll pivot back to Launchpad for the 'placement' sub-project. - we agreed on a *feature* review day possibly 4 weeks before the feature freeze in order to catch up any late design issue we could have missed when reviewing the spec. - we will EOL stein and older nova branches with elodilles proposing the deletion patch. We agred on discussing EOL'ing train at next PTG. - gibi will work on providing a static requirements file for nova that would use minimum versions of our dependencies (non transitively) and modify unit and functional test jobs to rather use this capped requirements file for testing. - we discussed the User Survey and we agreed on discussing any question we may want to add in the next survey during the next weekly meetings. - "2023.1 Antelope" naming is a bit confusing, but we agreed on we should continue to either use "2023.1" or "2023.1 Antelope" for naming our docs. We also wait for guidelines in order to consistently name our next stable branch (2023.1 possibly). (That's the end of procedural and release discussions, please resume here if you skipped the above) ### Technical bits ### # VMware driver status - As currently no job runs happened since April, we agreed on communicating in the nova-compute logs at periodic times that the code isn't tested so operators running it would know its upstream status. - we'll update the supported matrix documentation to refiect this 'UNTESTED' state # TC goals this cycle - for the FIPS goal, we agreed on the fact the current FIPS job (running on centos 9 stream) shouldn't be running on gate and check pipelines. As the current job is busted (all periodic runs go into TIMEOUT state), we also want to modify the job timeout to allow 30 mins more time for running (due to a reboot in the job definition) - for the oslo.privsep goal, no effort is required from a nova perspective (all deliverables are already using privsep). sean-k-mooney can propose a 'rfe' bug (see the note above on new contributors) for modifying our privsep usage in nova (using different privsep context by callers) - for the Ubunutu 2022.4 goal, gmann already provided changes. We need to review them. - for the RBAC goal, let's discuss it in a proper section just below # Next steps on RBAC - we need to test new policy defaults in a full integrated tempest testsuite (meaning with other projects), ideally before the first milestone. - once we check everything works fine as expected, we can flip the default (enabling new policies) before milestone-2 - we'd like to start drafting the new service role usage by two new specs (one for nova, the other one for placement) # Power management in Nova A pile of work I'm proud we gonna start this cycle. This is about disabling/enabling cores on purpose, so power savings occur. - we agreed on supporting it directly in Nova (and to not design and support an external tool which would suppose to draft some heave interface for an easy quickwin). This would just be a config flag to turn on that would enable CPU cores on demand. - a potential follow-up *may* be to use a different CPU governor depending on flavors or images but this won't be scoped for this release. # Power monitoring in Nova I don't really like this name, as monitoring isn't part of the Nova mission statement so I'll clarify. Here, this is about providing an internal readonly interface for power monitoring tools running on guests that would be able to capture host consumption metrics. One example of such monitoring tools is Scaphandre, demonstrated during the last OpenInfraSummit at a keynote. - we agreed on reusing virtiofs support we're gonna introduce in this release for the Manila share attachment usecase - this share would be readonly and would be unique per instance (ie. we wouldn't be supporting multiple guest agents reading different directories) - this share would be enabled thru a configuration flag per compute, and would only be mounted per instance by specific flavor or image extraspec/metadata. # Database soft-deleted records and the archive/purge commands - We don't want to deprecate DB soft-deleted records as some APIs continue to rely on those records. - We rather prefer to add a new parameter to the purge command that will directly delete the 'soft-deleted' records from the main DBs and not only the shadow tables (skipping the need to archive, if the operator wants to use this parameter) # Nova-compute failing at reboot due to vGPU missing devices Talking of bug #1900800, the problem is that mediated devices aren't persistent so nova-compute isn't able to respawn the instances after a reboot if those are vGPU-flavored. - we agreed on the fact that, like any other device, Nova shouldn't try to create them and we should rather ask the operator to pre-create the mediated devices, exaclty like we do for SR-IOV VFs. - we'll accordingly deprecate the mdev creation in the libvirt driver (but we'll continue to support it) and we'll log a warning if Nova has to create one mdev. - we'll change the nova-compute init script to raise a better exception explaining which mdev is missing - we'll document a procedure for explaining how to get existing mdev information and persist them by udev rules (for upgrade cases) # Robustify our compute hostname changes To be clear, at first step, we will continue to *NOT* support nova-compute hostname changes but we'll better detect the hostname change to prevent later issues. - first step to persist the compute UUID on disk seems a good candidate so a spec is targeted this cycle. - next steps could be to robustify our relationships between instance, compute node and service object records but this design will be deferred for later in a backlog spec. # Move to OpenStackClient and SDK OSC is already fully supported in Zed but it continues to rely on novaclient python bindings for calling the Nova API. - we agreed on modifying OSC to rather use openstacksdk (instead of novaclient) for communicating to the Nova APIs. Contributors welcome on this move. - we agreed on stopping to use project client libraries in nova services (eg. cinderclient used by nova-compute) and rather use openstacksdk directly. A 'rfe' bug per project client will be issued, anyone willing to work on it is welcome. - we also agreed on continuing to support novaclient for a couple of releases, as operators or other consumers of this package could require substantial efforts to move to the sdk. - we agreed on changing the release model for novaclient to be independent so we can release anytime we need. # Evacuate to target state We understand the usecase (evacuate an instance shouldn't always turn the evacuated instance on) - that said, we don't want to amend the API for passing a parameter as this would carry a tech debt indefinitely) - we prefer to propose a new microversion that would stop the instance eventually instead of starting it - operators wanting to keep the original behaviour would need to negociate an older microversion to the Nova API as we don't intend to make spawning optionally chosen by API on evacuate. (aaaaaaaaaaaaaaaand that's it for the PTG recap) Kudos, you were brave, you reached that point. Hope your coffee was good and now you feel rejuvenated. Anyway, time for me now to rest my fingers and to enjoy a deserved time off. As said, I'm all up for any questions or remarks that would come from the reading of this enormous thread. -Sylvain

2 1

[nova] Proposing EOL of nova project's old stable branches (Stein, Rocky, Queens)
by Elõd Illés 24 Oct '22

24 Oct '22

Hi, At the 2023.1 Antelope PTG, Nova team discussed that currently open Queens, Rocky and Stein branches for Nova teams' repositories will be moved to End of Life. There are multiple reasons behind this decision: - gate of these branches are broken for a couple of months now - zuul job definitions for these branches use the pre-"zuul-v3" [1] style, which makes the maintenance harder of these jobs - CI on Rocky and Queens use Ubuntu Xenial, which is also beyond its public maintenance window date And last but not least bugfix backports to these branches don't really get enough reviews anymore. If anyone has any interest to keep any of the above branches open then this is the last opportunity to step up, otherwise, as Nova team decided on the PTG, these branches will transition to End of Life. Please let us know before November 7th, we will proceed with the transition in case no responses will come till that day. [1] https://docs.openstack.org/project-team-guide/zuulv3.html Thanks in advance, Előd Illés irc:elodilles @ #openstack-nova

1 0

[telemetry][cloudkitty][ceilometer] Billing windows instances
by Thomas Goirand 24 Oct '22

24 Oct '22

Hi there! We're using telemetry+cloudkitty for our rating. We're happy of it, though we're having the issue that we would like to bill instances running Windows. The example given in the Cloudkitty doc shows how to bill more when an instance runs Windows. That works in theory, though in practice, Microsoft has a license model based on how many vCPU the instannce runs. So the model that the Cloudkitty documentation shows simply doesn't work with the SPLA thingy. We've looked at options. One way would of course writing a new custom pollster, but we don't like the idea: this would mean polling for all of the thousands of instances that are running in our deployments. So we would very much prefer having ceilometer running on compute node (with polling_namespaces=compute) to do the work, as this scales a way better. However, Ceilometer polls libvirt, which only has the information about the image ID, not the metadata associated with the image (like the property os_type=windows, for example). So, is there a better way than a dynamic pollster? Can this be done with ceilometer on the compute nodes? Cheers, Thomas Goirand (zigo)

2 3

Openstack cluster cannot create instances when 1 of 3 rabbitmq cluster node down
by Nguyễn Hữu Khôi 24 Oct '22

24 Oct '22

Description =========== I set up 3 controllers and 3 compute nodes. My system cannot work well when 1 rabbit node in cluster rabbitmq is down, cannot launch instances. It stucked at scheduling. Steps to reproduce =========== Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672// * Reboot 1 of 3 rabbitmq node. * Create instances then it stucked at scheduling. Workaround =========== Point to rabbitmq VIP address. But We cannot share the load with this solution. Please give me some suggestions. Thank you very much. I did google and enabled system log's debug but I still cannot understand why. Nguyen Huu Khoi

5 9

Openstack Cinder and Nova Services cannot work when rabbitmq cluter node down
by Nguyễn Hữu Khôi 24 Oct '22

24 Oct '22

Title: Openstack cluster cannot create when 1 of 3 rabbitmq cluster node down Bug description: Description =========== I set up 3 controllers and 3 compute nodes. My system cannot work when 1 rabbit node in cluster rabbitmq is down, cannot create volume or launch instance. It stucked at creating and scheduling respectively. Steps to reproduce =========== Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672// * Reboot 1 of 3 rabbitmq node. * Create volume or launch instance then it stucked at creating and scheduling respectively. Workaround =========== I need reboot cinder and nova services to create volume and launch instance . More Info: I see in cinder_scheduler, it looks like cinder cannot change to another rabbitmq node. I hope we have ideas for that.. 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last): 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._queues[msg_id].get(block=True, timeout=timeout) 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return waiter.wait() 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return get_hub().switch() 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self.greenlet.switch() 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task _queue.Empty 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task During handling of the above exception, another exception occurred: 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last): 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task task(self, context) 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 9716, in _sync_power_states 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task db_instances = objects.InstanceList.get_by_host(context, self.host, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = cls.indirection_api.object_class_action_versions( 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return cctxt.call(context, 'object_class_action_versions', 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self.transport._send( 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._driver.send(target, ctxt, message, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._send(target, ctxt, message, wait_for_reply, timeout, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout) 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task raise oslo_messaging.MessagingTimeout( 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c8a676a9709242908dcff97046d7976d Nguyen Huu Khoi

1 0

[large-scale][oslo.messaging] RPC workers and db connection
by Arnaud Morin 24 Oct '22

24 Oct '22

Hey all, TLDR: How can I fine tune the number of DB connection on OpenStack services? Long story, with some inline questions: I am trying to figure out the maximum number of db connection we should allow on our db cluster. For this short speech, I will use neutron RPC as example service, but I think nova is acting similar. So, to do so, I identified few parameters that I can tweak: rpc_workers [1] max_pool_size [2] max_overflow [3] executor_thread_pool_size [4] rpc_worker default is half CPU threads available (result of nproc) max_pool_size default is 5 max_overflow default is 50 executor_thread_pool_size is 64 Now imagine I have a server with 40 cores, So rpc_worker will be 20. Each worker will have a DB pool with 5+50 connections available. Each worker will use up to 64 "green" thread. The theorical max connection that I should set on my database is then: rpc_workers*(max_pool_size+max_overflow) = 20*(5+50) = 1100 Q1: am I right here? I have the feeling that this is huge. Now, let's assume each thread is consuming 1 connection from the DB pool. Under heavy load, I am affraid that the 64 threads could exceed the number of max_pool_size+max_overflow. Also, I noticed that some green threads were consuming more than 1 connection from the pool, so I can reach the max even sooner! Another thing, I notice that I have 21 RPC workers, not 20. Is it normal? [1] https://docs.openstack.org/neutron/latest/configuration/neutron.html#DEFAUL… [2] https://docs.openstack.org/neutron/latest/configuration/neutron.html#databa… [3] https://docs.openstack.org/neutron/latest/configuration/neutron.html#databa… [4] https://docs.openstack.org/neutron/latest/configuration/neutron.html#DEFAUL… Cheers, Arnaud.

3 4

RE: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem)
by Felix Hüttner 24 Oct '22

24 Oct '22

Sorry, no idea about that, for us the group also has write permissions -- Felix Huettner From: 박경원 <park0kyung0won(a)dgist.ac.kr> Sent: Monday, October 24, 2022 9:24 AM To: Felix Hüttner <felix.huettner(a)mail.schwarz>; openstack-discuss(a)lists.openstack.org Subject: RE: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem) Hello Felix Thank you very much for kind reply Do I also need to change permission setting of volume file in /var/lib/nova/mnt/... ? By default its: drwxr-x--- 2 64061 64061 11 Oct 24 04:19 99c4f7e8b15983b65e20cb7d37db899f group has only read and execute permission, no write permission ---------- 원본 메일 ---------- 보낸사람: "Felix Hüttner" <felix.huettner(a)mail.schwarz<mailto:felix.huettner@mail.schwarz>> 받는사람: "park0kyung0won(a)dgist.ac.kr<mailto:park0kyung0won@dgist.ac.kr>" <park0kyung0won(a)dgist.ac.kr<mailto:park0kyung0won@dgist.ac.kr>>, "openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>" <openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> 날짜: 2022-10-24 (월) 16:12:48 제목: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem) Hi, we are solving this issue for us by creating a “cinder” group on all hypervisors with the same gid (64061 in your case). Then we add the nova user to the cinder group and we are fine afterwards. You might need set “dynamic_ownership = 0" In your libvirt qemu.conf -- Felix Huettner From: 박경원 <park0kyung0won(a)dgist.ac.kr<mailto:park0kyung0won@dgist.ac.kr>> Sent: Monday, October 24, 2022 7:21 AM To: openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> Subject: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem) Hi I'm trying to setup cinder-volume service with NFS backend When I create a new VM instance with a volume from web UI, cinder-volume service on storage node creates volume file just fine But I get the following error on compute node and instance fails to spawn. 2022-10-24 02:14:25.347 402789 ERROR nova.compute.manager [req-47ec9fb1-9daa-4c24-8673-538797a217cc 8769cfaf608349bd9fbb36f92b188fe3 e1e8e8397cde49899b00d09dec76b29e - default default] [instance: 5acb1dc3-0685-4980-977b-b6dfff6dfb45] Instance failed to spawn: libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-10-24T02:14:24.819644Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/nova/mnt/99c4f7e8b15983b65e20cb7d37db899f/volume-8f478992-dde3-4c20-9005-61cd34eacf30","aio":"native","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/nova/mnt/99c4f7e8b15983b65e20cb7d37db899f/volume-8f478992-dde3-4c20-9005-61cd34eacf30': Permission denied I've added appropriate configs to apparmor profile. (Using Ubuntu 22.04) Apparmor isn't blocking this access. While the instance is spawning, I've checked ownership of the volume file on compute node: root@compute-node:/var/lib/nova/mnt$ ls -al total 17 drwxr-xr-x 3 nova nova 4096 Oct 24 04:19 . drwxr-xr-x 12 nova nova 4096 Oct 24 02:14 .. drwxr-x--- 2 64061 64061 11 Oct 24 04:19 99c4f7e8b15983b65e20cb7d37db899f It seems like cinder user on storage node creates volume file with UID/GID of 64061 (cinder user's UID/GID) But nova user on compute node has UID/GID of 64060, therefore cannot open volume file(/var/lib/nova/mnt/99c4f7e8b15983b65e20cb7d37db899f/volume-8f478992-dde3-4c20-9005-61cd34eacf30) Should I manually set the UID/GID of nova user on compute node to 64061, so both nova user on compute node and cinder user on storage node would have the same UID/GID? Feels like this duct taping isn't a proper solution. Did I miss something? Thank you Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.daten…>. Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz>.

1 0

RE: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem)
by 박경원 24 Oct '22

24 Oct '22

1 0

[yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem)
by 박경원 24 Oct '22

24 Oct '22

2 1

[largescale-sig] No meeting this week
by Thierry Carrez 23 Oct '22

23 Oct '22

Hi everyone, With the PTG just over, the Large Scale SIG will not be meeting this week. Our next regular IRC meeting will be November 9, at 1500utc on #openstack-operators on OFTC. Feel free to add topics to the agenda: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 0