From laurentfdumont at gmail.com Wed Jun 1 01:45:41 2022 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Tue, 31 May 2022 21:45:41 -0400 Subject: error creating image from volume In-Reply-To: References: Message-ID: I am not familiar with the error. 1. What client version are you using? openstack --version? 2. What Openstack version are you running? 3. Any specific properties on the volume? Laurent On Tue, May 31, 2022 at 8:10 AM Russell Stather < Russell.Stather at ignitiongroup.co.za> wrote: > Hi > > Trying to create an image from a volume. > > Getting this very unhelpful error message. > > igadmin at ig-umh-maas:~$ openstack image create --volume > 57bc3efa-6cdc-4e2d-a0e2-262340cc6180 commvaultimage > upload_to_image() got an unexpected keyword argument 'visibility' > > Anyone seen this before? > > Thanks > > Russell > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Wed Jun 1 07:01:07 2022 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 01 Jun 2022 09:01:07 +0200 Subject: [neutron] CI meeting 7.06.2022 cancelled Message-ID: <3190548.KDQWzQIof1@p1> Hi, Next week there is OpenInfra summit in Berlin and some of us will be busy there, so we decided yesterday to cancel next week's Neutron CI meeting. Please ping me on IRC or email if there would be anything urgent related to our CI. See You all on the CI meeting on 14.06.2022 (and in Berlin if You will be there :)) -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From katonalala at gmail.com Wed Jun 1 07:31:19 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 09:31:19 +0200 Subject: [neutron] Drivers meeting - Friday 2.6.2022 & 9. 6. 2022. - cancelled Message-ID: Hi Neutron Drivers! This Friday I will be at a conference and next Friday I will be travelling back from Berlin, so I can't drive these drivers' meetings, let's cancel them. I hope I can meet a few of you in Berlin, and see you online the week after the summit. Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 07:36:58 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 09:36:58 +0200 Subject: [neutron] team meeting 7. 06. 2022 cancelled Message-ID: Hi, Next week I will attend the Openinfra summit in Berlin, so I can't drive the team meeting. See you online on the week after the summit, or I hope we can meet in Berlin. Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Wed Jun 1 08:06:34 2022 From: zigo at debian.org (Thomas Goirand) Date: Wed, 1 Jun 2022 10:06:34 +0200 Subject: Fwd: [Forum] Meet the Projects Session! In-Reply-To: References: Message-ID: <932f56ec-a9f9-eea0-c3d5-69be9e185397@debian.org> On 5/31/22 17:53, Kendall Nelson wrote: > Hello :) > > I wanted to take a moment to invite project maintainers, core > contributors, PTLs, governance officials, etc to a meet the projects > (projects being the OIF top level ones- Kata, Zuul, OpenStack, > StarlingX, and Airship) session during the Forum at the upcoming Summit > in Berlin. It will take place Tue, June 7, 11:20am - 12:30pm local in? A > - A06. > > The idea is to gather in a single place so that newer members of the > community or people unfamiliar with the project can come and meet some > of the active participants and ask questions. It's all really informal > so there is no need to prepare anything ahead of time. > > I realize it is pretty late in everyone's planning their schedules for > the summit, but if you have a few minutes to spare to come hang out and > meet some people and network, please do! We would love to see you there. > > -Kendall Hi, Thanks a lot for this proposal, I very much like the idea of meeting (new) project members in real. I definitively will join and hope to match faces with IRC nicks after the meeting. Cheers, Thomas Goirand (zigo) From skaplons at redhat.com Wed Jun 1 08:39:30 2022 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 01 Jun 2022 10:39:30 +0200 Subject: Fwd: [Forum] Meet the Projects Session! In-Reply-To: <932f56ec-a9f9-eea0-c3d5-69be9e185397@debian.org> References: <932f56ec-a9f9-eea0-c3d5-69be9e185397@debian.org> Message-ID: <4286047.ni3Bx2OuKx@p1> Hi, Dnia ?roda, 1 czerwca 2022 10:06:34 CEST Thomas Goirand pisze: > On 5/31/22 17:53, Kendall Nelson wrote: > > Hello :) > > > > I wanted to take a moment to invite project maintainers, core > > contributors, PTLs, governance officials, etc to a meet the projects > > (projects being the OIF top level ones- Kata, Zuul, OpenStack, > > StarlingX, and Airship) session during the Forum at the upcoming Summit > > in Berlin. It will take place Tue, June 7, 11:20am - 12:30pm local in A > > - A06. > > > > The idea is to gather in a single place so that newer members of the > > community or people unfamiliar with the project can come and meet some > > of the active participants and ask questions. It's all really informal > > so there is no need to prepare anything ahead of time. > > > > I realize it is pretty late in everyone's planning their schedules for > > the summit, but if you have a few minutes to spare to come hang out and > > meet some people and network, please do! We would love to see you there. > > > > -Kendall > > Hi, > > Thanks a lot for this proposal, I very much like the idea of meeting > (new) project members in real. I definitively will join and hope to > match faces with IRC nicks after the meeting. > > Cheers, > > Thomas Goirand (zigo) > > Yeah. Thx for organizing this Kendal. I already added it to my calendar and will be there too :) -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From rdhasman at redhat.com Wed Jun 1 09:36:44 2022 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Wed, 1 Jun 2022 15:06:44 +0530 Subject: error creating image from volume In-Reply-To: References: Message-ID: Hi Russell, Thanks for finding the issue. On Tue, May 31, 2022 at 5:46 PM Russell Stather < Russell.Stather at ignitiongroup.co.za> wrote: > Hi > > Trying to create an image from a volume. > > Getting this very unhelpful error message. > > igadmin at ig-umh-maas:~$ openstack image create --volume > 57bc3efa-6cdc-4e2d-a0e2-262340cc6180 commvaultimage > upload_to_image() got an unexpected keyword argument 'visibility' > > Anyone seen this before? > > This is indeed a valid bug for which I've proposed the fix here[1]. To summarize, this is caused because OSC is calling cinderclient with *visibility* and *protected* arguments[2] without checking the microversion and the support for passing *visibility* and *protected* fields is available only since microversion *3.1* or greater[3][4]. For a quick workaround, pass the* ``--os-volume-api-version 3.1``* parameter along with the command or set the environment variable *``OS_VOLUME_API_VERSION``* to *3.1* or greater value. [1] https://review.opendev.org/c/openstack/python-openstackclient/+/844268 [2] https://opendev.org/openstack/python-openstackclient/src/branch/master/openstackclient/image/v2/image.py#L492-L493 [3] https://opendev.org/openstack/cinder/src/branch/master/cinder/api/openstack/api_version_request.py#L49 [4] https://opendev.org/openstack/python-cinderclient/src/branch/master/cinderclient/v3/volumes.py#L197-L211 Thanks and regards Rajat Dhasmana Thanks > > Russell > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkopec at redhat.com Wed Jun 1 12:06:15 2022 From: mkopec at redhat.com (Martin Kopec) Date: Wed, 1 Jun 2022 14:06:15 +0200 Subject: [qa] Cancelling office hour on June 7th Message-ID: Hello everyone, QA team is cancelling the next office hour Tuesday June 7th, I'm gonna attend the OpenStack Summit. https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA IM: kopecmartin -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Wed Jun 1 12:12:43 2022 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 1 Jun 2022 09:12:43 -0300 Subject: [cinder] Bug deputy report for week of 06-01-2022 Message-ID: This is a bug report from 05-25-2022 to 06-01-2022. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Low - https://bugs.launchpad.net/cinder/+bug/1976400 "[Storwize] Fix multiple SVC CLI calls for rc-relationship operations." Fix proposed to master. Wishlist - https://bugs.launchpad.net/cinder/+bug/1975794 "How to configure Cinder to use as much as possible, (post-provision)." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1976331 "Check attachment_specs may lead to poor performance." Unassigned. Cheers, Sofi -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 12:24:39 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 14:24:39 +0200 Subject: [neutron][networking-odl] Propose to EOL stable/queens, stable/rocky and stable/stein branches of networking-odl Message-ID: Hi, The Queens, Rocky and Stein branches of networking-odl see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/queens, stable/rocky and stable/stein branches of networking-odl End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-odl. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 12:27:57 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 14:27:57 +0200 Subject: [neutron][networking-bagpipe][networking-bgpvpn] Propose to EOL stable/rocky and stable/stein branches of networking-odl Message-ID: Hi, The Rocky and Stein branches of networking-bagpipe and networking-bgpvpn see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/rocky and stable/stein branches of networking-bagpipe and networking-bgpvpn End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-bagpipe and networking-bgpvpn. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 12:30:56 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 14:30:56 +0200 Subject: Fwd: [neutron][networking-bagpipe][networking-bgpvpn] Propose to EOL stable/rocky and stable/stein branches of networking-odl In-Reply-To: References: Message-ID: Sorry: of course the subject is networking-bagpipe & networking-bgpvpn ---------- Forwarded message --------- Felad?: Lajos Katona Date: 2022. j?n. 1., Sze, 14:27 Subject: [neutron][networking-bagpipe][networking-bgpvpn] Propose to EOL stable/rocky and stable/stein branches of networking-bagpipe & bgpvpn To: Openstack Discuss List Hi, The Rocky and Stein branches of networking-bagpipe and networking-bgpvpn see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/rocky and stable/stein branches of networking-bagpipe and networking-bgpvpn End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-bagpipe and networking-bgpvpn. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 12:33:19 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 14:33:19 +0200 Subject: Fwd: [neutron][networking-bagpipe][networking-bgpvpn] Propose to EOL stable/rocky and stable/stein branches of networking-bagpipe & bgpvpn In-Reply-To: References: Message-ID: ---------- Forwarded message --------- Felad?: Lajos Katona Date: 2022. j?n. 1., Sze, 14:27 Subject: [neutron][networking-bagpipe][networking-bgpvpn] Propose to EOL stable/rocky and stable/stein branches of networking-odl To: Openstack Discuss List Hi, The Rocky and Stein branches of networking-bagpipe and networking-bgpvpn see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/rocky and stable/stein branches of networking-bagpipe and networking-bgpvpn End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-bagpipe and networking-bgpvpn. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 12:35:27 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 14:35:27 +0200 Subject: [neutron][networking-sfc] Propose to EOL stable/queens, stable/rocky and stable/stein branches of networking-sfc Message-ID: Hi, The Queens, Rocky and Stein branches of networking-sfc see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/queens, stable/rocky and stable/stein branches of networking-sfc End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-sfc. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 13:17:30 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 15:17:30 +0200 Subject: [neutron][neutron-fwaas] Propose to EOL stable/queens, stable/rocky and stable/stein branches of neutron-fwaas Message-ID: Hi, The Queens, Rocky and Stein branches of neutron-fwaas see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/queens, stable/rocky and stable/stein branches of neutron-fwaas End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in neutron-fwaas. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 13:36:48 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 15:36:48 +0200 Subject: [neutron][tap-as-a-service] Propose to EOL stable/queens, stable/rocky and stable/stein branches of tap-as-a-service Message-ID: Hi, The Queens, Rocky and Stein branches of tap-as-a-service see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/queens, stable/rocky and stable/stein branches of tap-as-a-service End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in tap-as-a-service. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Wed Jun 1 13:40:39 2022 From: katonalala at gmail.com (Lajos Katona) Date: Wed, 1 Jun 2022 15:40:39 +0200 Subject: [neutron][networking-ovn] Propose to EOL stable/queens, stable/rocky and stable/stein branches of networking-ovn Message-ID: Hi, The Queens, Rocky and Stein branches of networking-ovn see little activity and maintaining them and keeping their jobs running is a big effort. We discussed this topic during the team meeting (see [1]), and agreed to propose stable/queens, stable/rocky and stable/stein branches of networking-ovn End Of Life. I will wait until the end of next week for anyone who would like to step up as maintainer of those branches and who would at least try to fix CI of them. If there will be no volunteers for that, I will EOL those branches in networking-ovn. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-05-31-14.00.log.html#l-25 Lajos Katona (lajoskatona) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim+openstack.org at coote.org Wed Jun 1 13:43:36 2022 From: tim+openstack.org at coote.org (tim+openstack.org at coote.org) Date: Wed, 1 Jun 2022 14:43:36 +0100 Subject: Novice question In-Reply-To: References: <83EE0BBE-7FE2-4815-912F-A6383A0FFFA8@coote.org> Message-ID: <93811472-C43D-4A13-9B21-A3E53F953B51@coote.org> Thanks, Sean. Unfortunately, this is a small (16GB x86). I did respond with more error messages, but I think they?re probably too obtuse. > On 30 May 2022, at 12:53, Sean Mooney wrote: > > On Mon, 2022-05-30 at 12:06 +0100, tim+openstack.org at coote.org wrote: >> Thanks, Michael. Very reassuring. I?ll have a look and comment back. > > if you have an m1 mac i woudl suggest using utm with the ubuntu 20.04 image > https://mac.getutm.app/gallery/ubuntu-20-04 > to create a qemu vm whihc will use macos's hypervirio api to hardware acclearte > the l1 vm. the l2 vms will still use qemu but by using arm based images for the host os > you can get pretty good perforamce in teh vm and spin up arm based cirrous iamge with nova and get ok performance. > it will be slower then nested virt but the apple silicon macs dont support that a the hardware level. > > > i have mostly for my own use being developing > https://github.com/SeanMooney/ansible_role_devstack > im probably goign to rename that to "ard" by the way in the future if that link does nto work later. > > this repo currently has ansible playbooks that will use the upstream devstack roles we use in ci to deploy multi node devstack > it can create vms using molecule/vagrant and then run the devstack install. > > so on a linux laptop you can do > > bootstrap-repo.sh > molecule create > molecule converge > and that will create a 2 node openstack based on centos 9 stream with master devstack installed and deployed. > > currently the molecule role creates two 8gb vms but i have an example of using it to deploy onto externally provisioned host > as a pr https://github.com/SeanMooney/ansible_role_devstack/pull/4/files > > if you continue to have troble deploying devstack by hand perhaps that will be of use to you. > > it not really ready for primetime/general use but if people find it intersting ye are welcome to fork and use as ye see fit. > the molecule template seamed to set the licence to "BSD" by default. i was plaaing ot make it apache but i guess bsd > is just as good so i shoudl fix that. > anyway i just wanted to point out that UTM is proably beter the virtual box form a performace point of view. > >> >>> On 27 May 2022, at 20:11, Michael Johnson wrote: >>> >>> Hi Tim, >>> >>> This should work fine. You will want a localrc/local.conf file to >>> configure devstack. I didn't see that mentioned in your steps. >>> See this section in the docs: >>> https://docs.openstack.org/devstack/latest/#create-a-local-conf >>> >>> The only caveat I would mention is the VM instances in Nova will run >>> super slow on virtualbox as it lacks the required "nested >>> virtualization" support and will run them all in software emulation. >>> >>> To find the root cause of the issue in nova, I would look through the >>> devstack at n-cpu log file (journal -u devstack at n-cpui) and the >>> devstack at n-sch logs. >>> >>> Also, you might have a look at one of the nova test localrc file as an example: >>> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da7/843127/6/check/tempest-integrated-compute-centos-9-stream/da7bebc/controller/logs/local_conf.txt >>> >>> Michael >>> >>> >>> On Fri, May 27, 2022 at 4:16 AM wrote: >>>> >>>> Hullo >>>> >>>> Na?ve question follows. Sorry. >>>> >>>> I?m trying a minimal OS install on a Virtualbox VM on a mac. I?d like to get to the point where I can launch a compute node. I?ve failed with `packstack` and I?m trying `devstack`. >>>> >>>> Should this work out of the box: ie >>>> >>>> Spin up a vm with vagrant: I?m using Centos Stream 9 to ensure that I get a current(ish) version of Python. It has 9GB RAM >>>> Ensure that SELinux and firewalls are turned off >>>> Clone devstack, cd to the directory and run `stack.sh` as user `vagrant` (this fails 1/3 of the time as some repo or other resets a connection. `stack.sh` doesn?t seem to be idempotent as reinvoking it may or may not install and run the OS environment >>>> Upload a ssh keypair through the web interface >>>> Use the web interface to launch the m1.nano flavor with Cirros image (I think that this flavor is quite new as some of the documentation refers to creating such a flavor with 64MB, whereas this one has 128MB. I did try the 64MB route [with `packstack`] and concluded that at least 96MB was needed and the documentation was wrong. I couldn?t log into launchpad.net to report this ? >>>> At this point the launch process fails with the error message: ?Build of instance 157bfa1d-7f8c-4a6c-ba3a-b02fb4f4b6a9 aborted: Failed to allocate the network(s), not rescheduling.? In the web ui >>>> >>>> >>>> Afaict, the vm has enough memory (just: it?s using a bit of swap, but more cache, so it could reclaim that). I?d expected the instance to launch, and I can well believe that I?ve missed something, but the documentation seems to point all over the place for various logs. >>>> >>>> Should this approach work? Is there an alternative that?s better (e.g. use Ubuntu: I?m not keen on apt/dpkg/.deb based distros as I?ve been tripped up in the past over the dependency handling and systemd integration, so I?ve avoided this, but I can see that Canonical is spending money on OS. But so is IBM/Redhat)? Where can I find info on how to trouble shoot the failing process? >>>> >>>> tia >>>> Tim >> >> > From radoslaw.piliszek at gmail.com Wed Jun 1 15:53:14 2022 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Wed, 1 Jun 2022 17:53:14 +0200 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? Message-ID: Hello Fellow OpenStackers, I am wondering if and how OpenStackers are approaching the subject of VDI (Virtual Desktop Infrastructure), also nowadays commonly called DaaS (Desktop as a Service) offering. To be more specific, I am looking forward to a solution covering both Linux and Windows desktops, pre-OS screen access optional, well-integrated (i.e., having automation) with OpenStack Nova, preferably also with Cinder and Glance, optionally with Keystone (apart from self-auth of course!) and Heat, 3D acceleration optional (most of the desktops here don't need it), nice to have read-only sharing of screen. A search brought me to vdi-broker [1] [2] with Guacamole. [3] But the vdi-broker seems dead with no documentation whatsoever. I know Guacamole but normally it offers only static definition of connections to existing machines (VNC or RDP). Looking forward to hearing (reading :D ) your experience with VDI/DaaS on an OpenStack cloud offering. I am also open to collaboration should any of you share the same interest. We can have a relevant SIG created if that is the case. PS: Sadly, I am not able to attend the Berlin summit due to other commitments, so can't ask in person. ;-( [1] https://github.com/cloudbase/vdi-broker [2] https://www.openstack.org/videos/boston-2017/virtual-desktop-infrastructure-vdi-with-openstack [3] https://guacamole.apache.org/ Kind regards, -yoctozepto From haiwu.us at gmail.com Wed Jun 1 16:18:47 2022 From: haiwu.us at gmail.com (hai wu) Date: Wed, 1 Jun 2022 11:18:47 -0500 Subject: [keystone] how to use v3tokenlessauth Message-ID: I could not find any example/usage of v3tokenlessauth, using environment variables. There are no documents to be found. I assume we could need to have this one at least: export OS_AUTH_TYPE="v3tokenlessauth" What other environment variables are required if using this auth type? Would like to use HTTPs x509 auth. This is the only relevant doc found: https://docs.openstack.org/keystone/train/admin/configure_tokenless_x509.html [keystone_authtoken] memcached_servers = localhost:11211 cafile = /etc/keystone/ca.pem project_domain_name = Default project_name = service auth_url = https://192.168.0.10/identity/v3 auth_type = v3tokenlessauth certfile = /etc/glance/certs/glance.pem keyfile = /etc/glance/private/glance_private_key.pem But there are no environment variables mentioned in the above doc. From cboylan at sapwetik.org Wed Jun 1 17:55:08 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Wed, 01 Jun 2022 10:55:08 -0700 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting Message-ID: Hello, Last week the OpenDev team upgraded Zuul which deployed zuul-executors with openstacksdk==0.99.0 installed. After this, test job logs uploaded to the rackspace swift location were no longer viewable in the Zuul dashboard. The reason for this is these objects did not have properly configured CORS headers any longer. The objects uploaded to OVH appear to have been fine (potentially because the headers are not required for OVH). Today we downgraded the version of openstacksdk on the zuul-executors to 0.61.0 and the rackspace uploaded job log objects have functional CORS headers again. For this reason we're fairly confident that something about the 0.99.0 release is impacting the setting of these headers. The code that does these uploads with openstacksdk can be found here [0]. I suspect that the problem is going to be related to this header set on line 227 [1]. In particular that appears to be rackspace specific and this issue is rackspace specific under 0.99.0. However, I wasn't able to find anything in openstacksdk that would have a problem with this. Does anyone else have ideas? I'm somewhat concerned that this represents a class of data loss (metadata is going missing) for swift object uploads. Additionally, we rely on setting the x-delete-after header to timeout, expire, and prune our logs in swift [2]. If there is a general problem with 0.99.0 setting headers on objects it is possible that all of the objects we created using 0.99.0 do not have this metadata set and will leak in their swift containers. [0] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py [1] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L226-L227 [2] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L224 From mtomaska at redhat.com Wed Jun 1 19:21:24 2022 From: mtomaska at redhat.com (Miro Tomaska) Date: Wed, 1 Jun 2022 14:21:24 -0500 Subject: [all][tc] Lets talk about Flake8 E501 In-Reply-To: References: Message-ID: Thank you all for responding and sharing your thoughts. It seems that although the longer limit might be nice it will bring more complications than we want to. For that reason we can consider this discussion as closed. That is, there will be no chances. Thanks again Miro On Tue, May 31, 2022 at 8:22 AM Sylvain Bauza wrote: > > > Le sam. 28 mai 2022 ? 04:36, Miro Tomaska a ?crit : > >> Hello All, >> >> This is probably going to be a hot topic but I was wondering if the >> community ever considered raising the default 79 characters line limit. I >> have seen some places where even a very innocent line of code needs to be >> split into two lines. I have also seen some code where I feel like variable >> names were abbreviated on purpose to squeeze everything into one line. >> >> How does the community feel about raising the E501 limit to 119 >> characters? The 119 character limit is the second most popular limit >> besides the default one. It's long enough to give a developer enough room >> for descriptive variables without being forced to break lines too much. And >> it is short enough for a diff between two files to look OK. >> >> The only downside I can see right now is that it's not easy to convert an >> existing code. So we will end up with files where the new code is 79+ >> characters and the "old" code is <=79. I can also see an argument where >> someone might have trouble reviewing a patch on a laptop screen (assuming >> standard 14" screen) ? >> >> Here is an example of one extreme, a diff of two files maxing out at 119 >> characters >> >> https://review.opendev.org/c/opendev/sandbox/+/843697/1..2 >> >> Thank you for your time and I am looking forward to this conversation :) >> >> > > My personal opinion is while I understand the reasoning behind, my concern > is about the community support. We don't have a lot of contributors at the > moment, and if we would like to modify the default limit, I'm pretty sure > it would take a lot of time for folks at least reviewing some changes that > are not really priorities... > > Also, as said by some folks, changing the default would also create some > problems for some of our contributors and eventually, if some projects > would accept 119 characters while some others not, it would also be a new > community difference between projects... (and yet again some new nit that > new contributors need to understand) > > Sorry, -1 for those reasons. > > > > -- >> Miro Tomaska >> irc: mtomaska >> Red Hat >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Wed Jun 1 20:51:53 2022 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 01 Jun 2022 22:51:53 +0200 Subject: [all][tc] Lets talk about Flake8 E501 In-Reply-To: References: Message-ID: <3430337.j8lITm4QUA@p1> Hi, Dnia wtorek, 31 maja 2022 15:21:50 CEST Sylvain Bauza pisze: > Le sam. 28 mai 2022 ? 04:36, Miro Tomaska a ?crit : > > > Hello All, > > > > This is probably going to be a hot topic but I was wondering if the > > community ever considered raising the default 79 characters line limit. I > > have seen some places where even a very innocent line of code needs to be > > split into two lines. I have also seen some code where I feel like variable > > names were abbreviated on purpose to squeeze everything into one line. > > > > How does the community feel about raising the E501 limit to 119 > > characters? The 119 character limit is the second most popular limit > > besides the default one. It's long enough to give a developer enough room > > for descriptive variables without being forced to break lines too much. And > > it is short enough for a diff between two files to look OK. > > > > The only downside I can see right now is that it's not easy to convert an > > existing code. So we will end up with files where the new code is 79+ > > characters and the "old" code is <=79. I can also see an argument where > > someone might have trouble reviewing a patch on a laptop screen (assuming > > standard 14" screen) ? > > > > Here is an example of one extreme, a diff of two files maxing out at 119 > > characters > > > > https://review.opendev.org/c/opendev/sandbox/+/843697/1..2 > > > > Thank you for your time and I am looking forward to this conversation :) > > > > > > My personal opinion is while I understand the reasoning behind, my concern > is about the community support. We don't have a lot of contributors at the > moment, and if we would like to modify the default limit, I'm pretty sure > it would take a lot of time for folks at least reviewing some changes that > are not really priorities... > > Also, as said by some folks, changing the default would also create some > problems for some of our contributors and eventually, if some projects > would accept 119 characters while some others not, it would also be a new > community difference between projects... (and yet again some new nit that > new contributors need to understand) > > Sorry, -1 for those reasons. I don't have strong opinion about it but I see that e.g. You are saying that this should be changed (or stay like it is now) globally for all projects but earlier fungi said that there wasn't such requirement from e.g. TC to have this configured in same way for all projects. So IIUC every project can actually change it on their own if team wants to do that. Personally I agree with Your point that it should be done for all projects in the same way so maybe TC should have some guidance/rules for that to make sure all projects will follow the same pattern. Wdyt? > > > > -- > > Miro Tomaska > > irc: mtomaska > > Red Hat > > > -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From fungi at yuggoth.org Wed Jun 1 21:41:06 2022 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 1 Jun 2022 21:41:06 +0000 Subject: [all][tc] Lets talk about Flake8 E501 In-Reply-To: <3430337.j8lITm4QUA@p1> References: <3430337.j8lITm4QUA@p1> Message-ID: <20220601214105.ss75dzoidbbfxst3@yuggoth.org> On 2022-06-01 22:51:53 +0200 (+0200), Slawek Kaplonski wrote: [...] > Personally I agree with Your point that it should be done for all > projects in the same way so maybe TC should have some > guidance/rules for that to make sure all projects will follow the > same pattern. Wdyt? Given that the vast majority of the code in OpenStack is consistent on this point even in the absence of any requirement on the part of the TC, I take it as an indication that having the TC make a rule about it would be a redundant time-sink when there's plenty else for them to be working on. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From andy at andybotting.com Wed Jun 1 23:10:22 2022 From: andy at andybotting.com (Andy Botting) Date: Thu, 2 Jun 2022 09:10:22 +1000 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? In-Reply-To: References: Message-ID: Hi Rados?aw, > I am wondering if and how OpenStackers are approaching the subject of > VDI (Virtual Desktop Infrastructure), also nowadays commonly called > DaaS (Desktop as a Service) offering. > To be more specific, I am looking forward to a solution covering both > Linux and Windows desktops, pre-OS screen access optional, > well-integrated (i.e., having automation) with OpenStack Nova, > preferably also with Cinder and Glance, optionally with Keystone > (apart from self-auth of course!) and Heat, 3D acceleration optional > (most of the desktops here don't need it), nice to have read-only > sharing of screen. We just launched a service like this on the Nectar Research Cloud. Like you, I looked around but couldn't find any Open Source projects that would be suitable. In the end, we based ours on a project from the University of Melbourne that did initially support Windows, but we removed some of that support to simplify our codebase as we didn't need it. We called it Bumblebee, and the code is here: https://github.com/NeCTAR-RC/bumblebee It's a Django web app with a Redis-based task scheduler that launches Nova instances, booted from a Cinder volume, cloned from a base OS image. The Django app directly modifies Guacamole's database to create the connections once the OpenStack resources have been provisioned and provides the user with a link that takes them to their desktop. We used Keycloak with OIDC for authentication on the web app and Guacamole and it's fairly seamless transitioning between the two. We did initially look at using Heat, but we wanted to support a workflow of being able to destroy the VM and archive the OS image to Swift (and restore again later if needed) so it was easier to just manage the resources directly. There's no 3D acceleration in our solution yet, but we're planning on looking at things like virtual GPUs later on. Sadly, I'm not going to make it to Berlin either, but I was thinking it might make for a good talk at the next summit. I'm in the process of properly documenting it now, so I don't have much to share at this point, but I'm happy to go into more detail about anything. cheers, Andy From juliaashleykreger at gmail.com Wed Jun 1 23:17:12 2022 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Wed, 1 Jun 2022 16:17:12 -0700 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? In-Reply-To: References: Message-ID: This sounds super interesting. While not a VDI user, I can see a huge appeal. Maybe this might be an interesting topic for OpenInfra Live? https://openinfra.dev/live/ On Wed, Jun 1, 2022 at 4:14 PM Andy Botting wrote: > > Hi Rados?aw, > > > I am wondering if and how OpenStackers are approaching the subject of > > VDI (Virtual Desktop Infrastructure), also nowadays commonly called > > DaaS (Desktop as a Service) offering. > > To be more specific, I am looking forward to a solution covering both > > Linux and Windows desktops, pre-OS screen access optional, > > well-integrated (i.e., having automation) with OpenStack Nova, > > preferably also with Cinder and Glance, optionally with Keystone > > (apart from self-auth of course!) and Heat, 3D acceleration optional > > (most of the desktops here don't need it), nice to have read-only > > sharing of screen. > > We just launched a service like this on the Nectar Research Cloud. > Like you, I looked around but couldn't find any Open Source projects > that would be suitable. > > In the end, we based ours on a project from the University of > Melbourne that did initially support Windows, but we removed some of > that support to simplify our codebase as we didn't need it. > > We called it Bumblebee, and the code is here: > https://github.com/NeCTAR-RC/bumblebee > > It's a Django web app with a Redis-based task scheduler that launches > Nova instances, booted from a Cinder volume, cloned from a base OS > image. > > The Django app directly modifies Guacamole's database to create the > connections once the OpenStack resources have been provisioned and > provides the user with a link that takes them to their desktop. > > We used Keycloak with OIDC for authentication on the web app and > Guacamole and it's fairly seamless transitioning between the two. > > We did initially look at using Heat, but we wanted to support a > workflow of being able to destroy the VM and archive the OS image to > Swift (and restore again later if needed) so it was easier to just > manage the resources directly. > > There's no 3D acceleration in our solution yet, but we're planning on > looking at things like virtual GPUs later on. > > Sadly, I'm not going to make it to Berlin either, but I was thinking > it might make for a good talk at the next summit. > > I'm in the process of properly documenting it now, so I don't have > much to share at this point, but I'm happy to go into more detail > about anything. > > cheers, > Andy > From andy at andybotting.com Thu Jun 2 02:27:41 2022 From: andy at andybotting.com (Andy Botting) Date: Thu, 2 Jun 2022 12:27:41 +1000 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? In-Reply-To: References: Message-ID: > This sounds super interesting. While not a VDI user, I can see a huge appeal. > > Maybe this might be an interesting topic for OpenInfra Live? > https://openinfra.dev/live/ Yeah, I could be down for that. Live demos are the best From ueha.ayumu at fujitsu.com Thu Jun 2 08:38:43 2022 From: ueha.ayumu at fujitsu.com (ueha.ayumu at fujitsu.com) Date: Thu, 2 Jun 2022 08:38:43 +0000 Subject: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team In-Reply-To: References: Message-ID: Hi Rico, This proposal is great helpful for us! There seems to be no particular objection, so could you add it to heat-translator-core if there is no problem? Thanks. Best Regards, Ueha -----Original Message----- From: ueha.ayumu at fujitsu.com Sent: Wednesday, May 25, 2022 1:40 PM To: openstack-discuss at lists.openstack.org Subject: RE: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team +1 Best Regards, Ueha -----Original Message----- From: Yoshito Ito Sent: Wednesday, May 25, 2022 12:31 PM To: openstack-discuss at lists.openstack.org Subject: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team Hi heat-translator team! I would like to propose Hiromu Asahina (h-asahina) as a member of the heat-translator core team. I'm now not able to actively contribute to heat-translator, and he can lead the team instead of me. He is mainly contributing to Tacker by fixing issues, refactoring existing codes, and a lot of helpful reviews, with his experience as a research engineer in NTT. Tacker highly depends on heat-translator in its main functionality, so I believe he can manage it with his knowledge of Tacker. Please vote for his nomination by answering this mail till June 1st. Best Regards, Yoshito Ito From uemit.seren at gmail.com Thu Jun 2 09:13:31 2022 From: uemit.seren at gmail.com (=?iso-8859-2?Q?=DCmit_Seren?=) Date: Thu, 2 Jun 2022 09:13:31 +0000 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? Message-ID: We are using xpra (https://xpra.org/) together with VirtualGL (https://www.virtualgl.org/) on our virtualized OpenStack based SLURM cluster to support OpenGL accelerated applications such as MorphoGraphX (https://morphographx.org/) for our users. This works pretty well. There was a bit of work involved because it requires an X11 server to run on the compute nodes but once this was done it worked fairly well. Best ?mit On 02.06.22, 01:10, "Andy Botting" wrote: Hi Rados?aw, > I am wondering if and how OpenStackers are approaching the subject of > VDI (Virtual Desktop Infrastructure), also nowadays commonly called > DaaS (Desktop as a Service) offering. > To be more specific, I am looking forward to a solution covering both > Linux and Windows desktops, pre-OS screen access optional, > well-integrated (i.e., having automation) with OpenStack Nova, > preferably also with Cinder and Glance, optionally with Keystone > (apart from self-auth of course!) and Heat, 3D acceleration optional > (most of the desktops here don't need it), nice to have read-only > sharing of screen. We just launched a service like this on the Nectar Research Cloud. Like you, I looked around but couldn't find any Open Source projects that would be suitable. In the end, we based ours on a project from the University of Melbourne that did initially support Windows, but we removed some of that support to simplify our codebase as we didn't need it. We called it Bumblebee, and the code is here: https://github.com/NeCTAR-RC/bumblebee It's a Django web app with a Redis-based task scheduler that launches Nova instances, booted from a Cinder volume, cloned from a base OS image. The Django app directly modifies Guacamole's database to create the connections once the OpenStack resources have been provisioned and provides the user with a link that takes them to their desktop. We used Keycloak with OIDC for authentication on the web app and Guacamole and it's fairly seamless transitioning between the two. We did initially look at using Heat, but we wanted to support a workflow of being able to destroy the VM and archive the OS image to Swift (and restore again later if needed) so it was easier to just manage the resources directly. There's no 3D acceleration in our solution yet, but we're planning on looking at things like virtual GPUs later on. Sadly, I'm not going to make it to Berlin either, but I was thinking it might make for a good talk at the next summit. I'm in the process of properly documenting it now, so I don't have much to share at this point, but I'm happy to go into more detail about anything. cheers, Andy -------------- next part -------------- An HTML attachment was scrubbed... URL: From artem.goncharov at gmail.com Thu Jun 2 10:44:36 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Thu, 2 Jun 2022 12:44:36 +0200 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting In-Reply-To: References: Message-ID: Hey, https://review.opendev.org/c/openstack/openstacksdk/+/844414 should address the issue. For R1 of SDK we want to ensure all interfaces of SDK use same code branch and not 3 different like it was before. So with the latest changes we now rely here on the regular mechanism that actually watches how to handle each known parameter. All of the arbitrary parameters in Swift are managed as container/object metadata and should get X-Container-Meta- or X-Object-Meta- prefixes in order to be considered and returned back by manila Swift. This is sadly not the case for Rax as you rightfully mention. So for now I have added header required by Rax into ?known? exceptions and added explicit unit test for that. Would be cool to find way how I could test that really in Rax to ensure particular Zuul use case will work. Generally we would need to think whether SDK should continue ?tell user he is wrong before reaching out to the server? approach or to switch to the ?believe the user and let server reject? approach generally. This here is precisely one of those cases. Alternatively we would need to widen our concept of vendors and move all of the specific Rax logic over there. We surely have lot of OpenStack based clouds, which do not explicitly implement everything in the compatible way. So in order to make SDK/CLI working with all of them we either drop most of the validation logic in SDK or really start implementing cloud specific overrides as part of ?extensions?. Artem > On 1. Jun 2022, at 19:55, Clark Boylan wrote: > > Hello, > > Last week the OpenDev team upgraded Zuul which deployed zuul-executors with openstacksdk==0.99.0 installed. After this, test job logs uploaded to the rackspace swift location were no longer viewable in the Zuul dashboard. The reason for this is these objects did not have properly configured CORS headers any longer. The objects uploaded to OVH appear to have been fine (potentially because the headers are not required for OVH). > > Today we downgraded the version of openstacksdk on the zuul-executors to 0.61.0 and the rackspace uploaded job log objects have functional CORS headers again. For this reason we're fairly confident that something about the 0.99.0 release is impacting the setting of these headers. > > The code that does these uploads with openstacksdk can be found here [0]. I suspect that the problem is going to be related to this header set on line 227 [1]. In particular that appears to be rackspace specific and this issue is rackspace specific under 0.99.0. However, I wasn't able to find anything in openstacksdk that would have a problem with this. Does anyone else have ideas? I'm somewhat concerned that this represents a class of data loss (metadata is going missing) for swift object uploads. > > Additionally, we rely on setting the x-delete-after header to timeout, expire, and prune our logs in swift [2]. If there is a general problem with 0.99.0 setting headers on objects it is possible that all of the objects we created using 0.99.0 do not have this metadata set and will leak in their swift containers. > > [0] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py > [1] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L226-L227 > [2] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L224 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rg at raghavgururajan.name Thu Jun 2 11:22:41 2022 From: rg at raghavgururajan.name (Raghav Gururajan) Date: Thu, 02 Jun 2022 11:22:41 +0000 Subject: Resources to learn IT Infrastructure Message-ID: Hello All, I am stepping into infrastructure side of the IT world and was wondering if any of you could recommend best books, materials or videos for learning? Regards, Raghav "RG" Gururajan. From gmann at ghanshyammann.com Thu Jun 2 14:22:16 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Thu, 02 Jun 2022 09:22:16 -0500 Subject: [all][tc] Technical Committee next weekly meeting on June 2, 2022 at 1500 UTC In-Reply-To: <18116abf909.bcbc5c9a267396.2648997117806913882@ghanshyammann.com> References: <18116abf909.bcbc5c9a267396.2648997117806913882@ghanshyammann.com> Message-ID: <18124cbad36.10270d18c455377.5625384948074577891@ghanshyammann.com> Hello Everyone, Below is the agenda for Today's TC IRC meeting schedule at 1500 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting == Agenda for tomorrow's TC meeting == * Roll call * Follow up on past action items * Gate health check ** Fixing Zuul config error in OpenStack *** https://etherpad.opendev.org/p/zuul-config-error-openstack * New ELK service dashboard: e-r service ** https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global ** https://review.opendev.org/c/openstack/governance-sigs/+/835838 * RBAC feedback in ops meetup ** https://etherpad.opendev.org/p/ops-meetup-berlin-2022-planning#L53 * TC + Board meeting * Open Reviews ** https://review.opendev.org/q/projects:openstack/governance+is:open -gmann ---- On Mon, 30 May 2022 15:32:58 -0500 Ghanshyam Mann wrote ---- > Hello Everyone, > > The technical Committee's next weekly meeting is scheduled for June 2, 2022, at 1500 UTC. > > If you would like to add topics for discussion, please add them to the below wiki page by > Wednesday, June 1, at 2100 UTC. > > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting > > -gmann > > > From fungi at yuggoth.org Thu Jun 2 14:22:54 2022 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 2 Jun 2022 14:22:54 +0000 Subject: [dev][docs][security-sig] Retiring security-analysis process and repo Message-ID: <20220602142254.k2j3uctxkoiumwv7@yuggoth.org> In 2016, what was then the Security Project Team embarked on an effort to centrally collect security analyses of OpenStack components. It accumulated a total of two, one for Barbican as of the Newton release, and another for KeystoneMiddleware as of Pike. The latter was finalized in 2017 and took nearly a year to merge due to already waning enthusiasm and reviewer availability: https://docs.openstack.org/security-analysis Given this effort was effectively abandoned years ago, the Security SIG members agree that the repository should be retired in order to reduce confusion. The vulnerability management oversight requirements were amended in February to remove any reference to this process, and we reached a consensus that this sort of documentation is better off inside the projects which are writing it rather than collected centrally with a disconnected (or in this case absent) group of reviewers and maintainers. This message serves as notice to the community that we will be pushing changes to follow the usual OpenStack repository retirement process for openstack/security-analysis in the coming days. As usual, the final state of the documents will be found in the parent commit of the one which wipes all the old files, but for posterity I'll link it here as well: https://opendev.org/openstack/security-analysis/src/commit/ac43025 Many thanks to those who attempted to provide and review these analyses in years past. The idea of maintaining information on the security risks and considerations for the systems we design is still a good one, and something I hope our contributor community might find more time to focus on in the years to come; but the place to document those things is right alongside the rest of the software's documentation, there's nothing inherently special or different about it. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From elod.illes at est.tech Thu Jun 2 14:58:07 2022 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Thu, 2 Jun 2022 16:58:07 +0200 Subject: [release] Release countdown for week R-17, Jun 06 - 10 Message-ID: <9192a345-1a05-dcf3-7ca1-f1d4fb0e4908@est.tech> Development Focus ----------------- The Zed-2 milestone will happen in next month, on July 14th, 2022. Zed-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/ zed /schedule.html for details. General Information ------------------- Please remember that libraries need to be released at least once per milestone period. At milestone 2, the release team will propose releases for any library that has not been otherwise released since milestone 1. Other non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. At milestone-2 we also freeze the contents of the final release. If you have a new deliverable that should be included in the final release, you should make sure it has a deliverable file in: https://opendev.org/openstack/releases/src/branch/master/deliverables/ zed You should request a beta release (or intermediary release) for those new deliverables by milestone-2. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. See the MembershipFreeze description for more details: https://releases.openstack.org/ zed /schedule.html# z -mf Finally, now may be a good time for teams to check on any stable releases that need to be done for your deliverables. If you have bugfixes that have been backported, but no stable release getting those. If you are unsure what is out there committed but not released, in the openstack/releases repo, running the command "tools/list_stable_unreleased_changes.sh " gives a nice report. Upcoming Deadlines & Dates -------------------------- Zed-2 Milestone: July 14th, 2022 OpenInfra Summit: June 7-9, 2022 (Berlin) El?d Ill?s irc: elodilles -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Thu Jun 2 16:22:25 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Thu, 02 Jun 2022 09:22:25 -0700 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting In-Reply-To: References: Message-ID: <315f833d-2b00-4f01-8ccc-172d7bf7b576@www.fastmail.com> On Thu, Jun 2, 2022, at 3:44 AM, Artem Goncharov wrote: > Hey, > > https://review.opendev.org/c/openstack/openstacksdk/+/844414 should > address the issue. For R1 of SDK we want to ensure all interfaces of > SDK use same code branch and not 3 different like it was before. So > with the latest changes we now rely here on the regular mechanism that > actually watches how to handle each known parameter. All of the > arbitrary parameters in Swift are managed as container/object metadata > and should get X-Container-Meta- or X-Object-Meta- prefixes in order to > be considered and returned back by manila Swift. This is sadly not the > case for Rax as you rightfully mention. Note that the docs [3] still say that headers are passed through, which isn't quite the case in 0.99.0. Filtered headers for what the SDK thinks are valid are passed through. This change should probably update the docs too. > > So for now I have added header required by Rax into ?known? exceptions > and added explicit unit test for that. Would be cool to find way how I > could test that really in Rax to ensure particular Zuul use case will > work. Probably the simplest thing is to have a test that checks the right headers are sent. Then if/when clouds change behavior we can update the tests to follow. > > Generally we would need to think whether SDK should continue ?tell user > he is wrong before reaching out to the server? approach or to switch to > the ?believe the user and let server reject? approach generally. This > here is precisely one of those cases. Alternatively we would need to > widen our concept of vendors and move all of the specific Rax logic > over there. We surely have lot of OpenStack based clouds, which do not > explicitly implement everything in the compatible way. So in order to > make SDK/CLI working with all of them we either drop most of the > validation logic in SDK or really start implementing cloud specific > overrides as part of ?extensions?. The problem with telling the user they are wrong before talking to the server is that it assumes that clouds don't do weird things. History has shown us that cloud definitely do weird things and this isn't related to any single cloud. Off the top of my head I can come up with a few examples and if we look through the git history of nodepool and opendev system-config I'm sure we'll find many many more occurences. This isn't a specific cloud problem either; they all do it one way or another. It is my opinion that tools like openstacksdk should be forgiving in what they accept to enable users to use openstack in its many forms. Shade is a great example of this. Eventually shade was rolled into openstacksdk and doesn't really exist anymore. It feels like the spirit of shade has been lost though and we're regressing back to 2014 when you couldn't talk to two different clouds without 4 different client tools. Similarly support for old versions of APIs have been ripped out of these tools. Unfortunately some clouds are still deploying old versions of clouds and ideally as a user I wouldn't need to think too hard about that. It isn't my fault this is the situation, I just want the software to work. > > Artem > >> On 1. Jun 2022, at 19:55, Clark Boylan wrote: >> >> Hello, >> >> Last week the OpenDev team upgraded Zuul which deployed zuul-executors with openstacksdk==0.99.0 installed. After this, test job logs uploaded to the rackspace swift location were no longer viewable in the Zuul dashboard. The reason for this is these objects did not have properly configured CORS headers any longer. The objects uploaded to OVH appear to have been fine (potentially because the headers are not required for OVH). >> >> Today we downgraded the version of openstacksdk on the zuul-executors to 0.61.0 and the rackspace uploaded job log objects have functional CORS headers again. For this reason we're fairly confident that something about the 0.99.0 release is impacting the setting of these headers. >> >> The code that does these uploads with openstacksdk can be found here [0]. I suspect that the problem is going to be related to this header set on line 227 [1]. In particular that appears to be rackspace specific and this issue is rackspace specific under 0.99.0. However, I wasn't able to find anything in openstacksdk that would have a problem with this. Does anyone else have ideas? I'm somewhat concerned that this represents a class of data loss (metadata is going missing) for swift object uploads. >> >> Additionally, we rely on setting the x-delete-after header to timeout, expire, and prune our logs in swift [2]. If there is a general problem with 0.99.0 setting headers on objects it is possible that all of the objects we created using 0.99.0 do not have this metadata set and will leak in their swift containers. >> >> [0] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py >> [1] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L226-L227 >> [2] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L224 >> [3] https://docs.openstack.org/openstacksdk/latest/user/connection.html#openstack.connection.Connection.create_object From iurygregory at gmail.com Thu Jun 2 16:34:45 2022 From: iurygregory at gmail.com (Iury Gregory) Date: Thu, 2 Jun 2022 13:34:45 -0300 Subject: Fwd: [Forum] Meet the Projects Session! In-Reply-To: <4286047.ni3Bx2OuKx@p1> References: <932f56ec-a9f9-eea0-c3d5-69be9e185397@debian.org> <4286047.ni3Bx2OuKx@p1> Message-ID: Hi Kendall, I will be there, just added to my schedule =) Em qua., 1 de jun. de 2022 ?s 05:40, Slawek Kaplonski escreveu: > Hi, > > Dnia ?roda, 1 czerwca 2022 10:06:34 CEST Thomas Goirand pisze: > > On 5/31/22 17:53, Kendall Nelson wrote: > > > Hello :) > > > > > > I wanted to take a moment to invite project maintainers, core > > > contributors, PTLs, governance officials, etc to a meet the projects > > > (projects being the OIF top level ones- Kata, Zuul, OpenStack, > > > StarlingX, and Airship) session during the Forum at the upcoming > Summit > > > in Berlin. It will take place Tue, June 7, 11:20am - 12:30pm local in > A > > > - A06. > > > > > > The idea is to gather in a single place so that newer members of the > > > community or people unfamiliar with the project can come and meet some > > > of the active participants and ask questions. It's all really informal > > > so there is no need to prepare anything ahead of time. > > > > > > I realize it is pretty late in everyone's planning their schedules for > > > the summit, but if you have a few minutes to spare to come hang out > and > > > meet some people and network, please do! We would love to see you > there. > > > > > > -Kendall > > > > Hi, > > > > Thanks a lot for this proposal, I very much like the idea of meeting > > (new) project members in real. I definitively will join and hope to > > match faces with IRC nicks after the meeting. > > > > Cheers, > > > > Thomas Goirand (zigo) > > > > > > Yeah. Thx for organizing this Kendal. I already added it to my calendar > and will be there too :) > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From iurygregory at gmail.com Thu Jun 2 17:23:51 2022 From: iurygregory at gmail.com (Iury Gregory) Date: Thu, 2 Jun 2022 14:23:51 -0300 Subject: [ironic] No weekly meeting next week (June 6) Message-ID: Hello ironicers, As you probably already know, next week is the Open Infrastructure Summit in Berlin and some of us will be at the event. We will resume our weekly meeting on June 13. Thanks! -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Ironic PTL* *Senior Software Engineer at Red Hat Brazil* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From artem.goncharov at gmail.com Thu Jun 2 19:12:33 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Thu, 2 Jun 2022 21:12:33 +0200 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting In-Reply-To: <315f833d-2b00-4f01-8ccc-172d7bf7b576@www.fastmail.com> References: <315f833d-2b00-4f01-8ccc-172d7bf7b576@www.fastmail.com> Message-ID: <27B7F0D2-E382-49EF-B518-1764D6F83344@gmail.com> Hey Clark, Pls find comments below. I need to justify the product that I maintain, but I perfectly understand you. > On 2. Jun 2022, at 18:22, Clark Boylan wrote: > > On Thu, Jun 2, 2022, at 3:44 AM, Artem Goncharov wrote: >> Hey, >> >> https://review.opendev.org/c/openstack/openstacksdk/+/844414 should >> address the issue. For R1 of SDK we want to ensure all interfaces of >> SDK use same code branch and not 3 different like it was before. So >> with the latest changes we now rely here on the regular mechanism that >> actually watches how to handle each known parameter. All of the >> arbitrary parameters in Swift are managed as container/object metadata >> and should get X-Container-Meta- or X-Object-Meta- prefixes in order to >> be considered and returned back by manila Swift. This is sadly not the >> case for Rax as you rightfully mention. > > Note that the docs [3] still say that headers are passed through, which isn't quite the case in 0.99.0. Filtered headers for what the SDK thinks are valid are passed through. This change should probably update the docs too. Partially yes, partially no. Documentation says it will pass them and the SW does that. It passes, however, only what it know would be accepted. I see the point here and will try to address this, but the issue is that we have generally a mixture of header and metadata props (which are implemented as headers). I will try to reimplement current handling to be able to guess better what user means. > >> >> So for now I have added header required by Rax into ?known? exceptions >> and added explicit unit test for that. Would be cool to find way how I >> could test that really in Rax to ensure particular Zuul use case will >> work. > > Probably the simplest thing is to have a test that checks the right headers are sent. Then if/when clouds change behavior we can update the tests to follow. This is how I implemented that, but it is only a unit test. And here I was rather wondering if there would be possibility to have func test for that to save all of us from another analysis sessions on what the heck is going on right now. Point is that we only know that cloud changed behaviour by noticing Zuul fails now (in similar case to the current issue). > >> >> Generally we would need to think whether SDK should continue ?tell user >> he is wrong before reaching out to the server? approach or to switch to >> the ?believe the user and let server reject? approach generally. This >> here is precisely one of those cases. Alternatively we would need to >> widen our concept of vendors and move all of the specific Rax logic >> over there. We surely have lot of OpenStack based clouds, which do not >> explicitly implement everything in the compatible way. So in order to >> make SDK/CLI working with all of them we either drop most of the >> validation logic in SDK or really start implementing cloud specific >> overrides as part of ?extensions?. > > The problem with telling the user they are wrong before talking to the server is that it assumes that clouds don't do weird things. History has shown us that cloud definitely do weird things and this isn't related to any single cloud. Off the top of my head I can come up with a few examples and if we look through the git history of nodepool and opendev system-config I'm sure we'll find many many more occurences. This isn't a specific cloud problem either; they all do it one way or another. > > It is my opinion that tools like openstacksdk should be forgiving in what they accept to enable users to use openstack in its many forms. Shade is a great example of this. Eventually shade was rolled into openstacksdk and doesn't really exist anymore. It feels like the spirit of shade has been lost though and we're regressing back to 2014 when you couldn't talk to two different clouds without 4 different client tools. > > Similarly support for old versions of APIs have been ripped out of these tools. Unfortunately some clouds are still deploying old versions of clouds and ideally as a user I wouldn't need to think too hard about that. It isn't my fault this is the situation, I just want the software to work. I perfectly see your point. To some extend you are right, but let me describe reasoning for all of those changes we have now: - Zuul is using one interface of SDK - Ansible is using other interface (moreover it mixes 3 different interfaces) - OSC is either not using SDK or using one of 3 mentioned interfaces - Those mentioned interfaces try to do same thing, but eventually differently - Vanila services evolve and require we permanently add new features to all of those interfaces simultaneously - Amount of contributions is incredibly low what makes it impossible to maintain multiple interfaces for doing same job - So what I am trying to do now is to have all of the mentioned tools rely on the single branch in code to be able to maintain it easier and reduce amount of questions like ?hmm, it works if I use OSC, but it doesn?t when I use Ansible. What is wrong??. Target is just to exactly reduce amount of those ?different tools for different tasks and or clouds? as you mention and instead to have a single tool that work for every cloud. We do not take care of shade any more. We try to deprecate all of the individual clients exactly to have a single OSC be able to do everything. And it should just know all of those corner cases. We try to get rid of as much ?reimplementation? from Ansible modules as possible. - Yes, maybe idea of trying to help user to know that he will fail once we reach server was wrong, but you can judge only from experience (when clouds behave differently). Idea as such is not bad, is having its right to live and is generally targeting to be more user friendly than the server response that just returns you 409 with ?good luck figuring out what is wrong?. Here I was spending days trying to find issue in my request with only 409 ?Bad Request? back. And when target clouds are different you as a user are left in even more problems. Generally idea of SDK is to save user from even knowing about differences of clouds he want to talk to. Without that the whole burden is on the user and you need to have all of those cloud hacks directly in your code (is it cool to have all of those hacks inside Zuul and reimplementing them through the different use cases? And repeating the same hacks in OSC and Ansible? Same for all of the user written applications?) - I know this work breaks some corner cases. And I perfectly see the pain my work is causing (even for myself). I have invested huge amount of time in rewriting this and joining code branches and tests. Trust me this was not fun. But unless this is done we wither stuck with old code without possibility to add new features (i.e. micro versions) or we continue living in the mess of tons of different ways to do the same thing with no maintainers. =========== So overall, while I understand all of the issues I am causing to projects right now, I kindly ask, be patient, report broken issues to me and help me fix this insanity. Believe me, I am not doing all of that with bad intents, but rather hoping to resolve issues we have and unify things in the light of low contributions. Otherwise SDK, CLI, Ansible, eventually Zuul are not able to properly keep up with OpenStack services themselves. I listen to complains carefully, and that is the reason for R1.0 not to exist as of now. Artem > >> >> Artem >> >>> On 1. Jun 2022, at 19:55, Clark Boylan wrote: >>> >>> Hello, >>> >>> Last week the OpenDev team upgraded Zuul which deployed zuul-executors with openstacksdk==0.99.0 installed. After this, test job logs uploaded to the rackspace swift location were no longer viewable in the Zuul dashboard. The reason for this is these objects did not have properly configured CORS headers any longer. The objects uploaded to OVH appear to have been fine (potentially because the headers are not required for OVH). >>> >>> Today we downgraded the version of openstacksdk on the zuul-executors to 0.61.0 and the rackspace uploaded job log objects have functional CORS headers again. For this reason we're fairly confident that something about the 0.99.0 release is impacting the setting of these headers. >>> >>> The code that does these uploads with openstacksdk can be found here [0]. I suspect that the problem is going to be related to this header set on line 227 [1]. In particular that appears to be rackspace specific and this issue is rackspace specific under 0.99.0. However, I wasn't able to find anything in openstacksdk that would have a problem with this. Does anyone else have ideas? I'm somewhat concerned that this represents a class of data loss (metadata is going missing) for swift object uploads. >>> >>> Additionally, we rely on setting the x-delete-after header to timeout, expire, and prune our logs in swift [2]. If there is a general problem with 0.99.0 setting headers on objects it is possible that all of the objects we created using 0.99.0 do not have this metadata set and will leak in their swift containers. >>> >>> [0] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py >>> [1] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L226-L227 >>> [2] https://opendev.org/zuul/zuul-jobs/src/commit/e69d879caecb454c529a7d757b80ae49c3caa105/roles/upload-logs-base/library/zuul_swift_upload.py#L224 >>> > > [3] https://docs.openstack.org/openstacksdk/latest/user/connection.html#openstack.connection.Connection.create_object -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Jun 2 19:45:12 2022 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 2 Jun 2022 19:45:12 +0000 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting In-Reply-To: <27B7F0D2-E382-49EF-B518-1764D6F83344@gmail.com> References: <315f833d-2b00-4f01-8ccc-172d7bf7b576@www.fastmail.com> <27B7F0D2-E382-49EF-B518-1764D6F83344@gmail.com> Message-ID: <20220602194511.zkkph5zmasr37jkk@yuggoth.org> On 2022-06-02 21:12:33 +0200 (+0200), Artem Goncharov wrote: [...] > I listen to complains carefully, and that is the reason for R1.0 > not to exist as of now. [...] It's probably also worth pointing out, from a user experience perspective, that we anticipated backward-incompatible changes in 1.0.0 and so merged an openstacksdk<1 pin in nodepool's requirements.txt file. Unfortunately, instead the backward-incompatible changes were released as 0.99.0 so when the container images for our launchers updated we started getting another error back from all of the Rackspace regions we're using (traceback attached). I'm happy to file a bug report on that if nobody has beaten me to it, but further diagnostics will probably have to wait until after summit travel since we've rolled back to 0.61.0 for now. -- Jeremy Stanley -------------- next part -------------- Creating server with hostname ubuntu-bionic-rax-dfw-0029866297 in rax-dfw from image ubuntu-bionic Clearing az, flavor and image caches due to 400 error from nova Launch attempt 1/3 failed: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/nodepool/driver/openstack/handler.py", line 247, in launch self._launchNode() File "/usr/local/lib/python3.9/site-packages/nodepool/driver/openstack/handler.py", line 127, in _launchNode server = self.handler.manager.createServer( File "/usr/local/lib/python3.9/site-packages/nodepool/driver/openstack/provider.py", line 290, in createServer return self._client.create_server(wait=False, **create_args) File "/usr/local/lib/python3.9/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) File "/usr/local/lib/python3.9/site-packages/openstack/cloud/_utils.py", line 365, in func_wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/openstack/cloud/_compute.py", line 862, in create_server server = self.compute.create_server(**kwargs) File "/usr/local/lib/python3.9/site-packages/openstack/compute/v2/_proxy.py", line 629, in create_server return self._create(_server.Server, **attrs) File "/usr/local/lib/python3.9/site-packages/openstack/proxy.py", line 485, in _create return res.create(self, base_path=base_path) File "/usr/local/lib/python3.9/site-packages/openstack/resource.py", line 1388, in create self._translate_response(response, has_body=has_body) File "/usr/local/lib/python3.9/site-packages/openstack/resource.py", line 1200, in _translate_response exceptions.raise_from_response(response, error_message=error_message) File "/usr/local/lib/python3.9/site-packages/openstack/exceptions.py", line 233, in raise_from_response raise cls( openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://dfw.servers.api.rackspacecloud.com/v2/REDACTED/servers, Bad networks format From artem.goncharov at gmail.com Thu Jun 2 19:54:53 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Thu, 2 Jun 2022 21:54:53 +0200 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting In-Reply-To: <20220602194511.zkkph5zmasr37jkk@yuggoth.org> References: <315f833d-2b00-4f01-8ccc-172d7bf7b576@www.fastmail.com> <27B7F0D2-E382-49EF-B518-1764D6F83344@gmail.com> <20220602194511.zkkph5zmasr37jkk@yuggoth.org> Message-ID: Thanks Jeremy, will try to analyze the trace. This is not too extensive, but better then silent pin without possibility to even notice that. I see that there are generally not enough deep tests for Openstack driver in Zuul since mostly issues appear in Prod. I will to try to invest some time there. Artem ---- typed from mobile, auto-correct typos assumed ---- On Thu, Jun 2, 2022, 21:47 Jeremy Stanley wrote: > On 2022-06-02 21:12:33 +0200 (+0200), Artem Goncharov wrote: > [...] > > I listen to complains carefully, and that is the reason for R1.0 > > not to exist as of now. > [...] > > It's probably also worth pointing out, from a user experience > perspective, that we anticipated backward-incompatible changes in > 1.0.0 and so merged an openstacksdk<1 pin in nodepool's > requirements.txt file. Unfortunately, instead the > backward-incompatible changes were released as 0.99.0 so when the > container images for our launchers updated we started getting > another error back from all of the Rackspace regions we're using > (traceback attached). > > I'm happy to file a bug report on that if nobody has beaten me to > it, but further diagnostics will probably have to wait until after > summit travel since we've rolled back to 0.61.0 for now. > -- > Jeremy Stanley > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Thu Jun 2 20:52:27 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Thu, 02 Jun 2022 13:52:27 -0700 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting In-Reply-To: References: <315f833d-2b00-4f01-8ccc-172d7bf7b576@www.fastmail.com> <27B7F0D2-E382-49EF-B518-1764D6F83344@gmail.com> <20220602194511.zkkph5zmasr37jkk@yuggoth.org> Message-ID: <6d960eb3-78c2-4410-8bfd-ec697e329fc9@www.fastmail.com> On Thu, Jun 2, 2022, at 12:54 PM, Artem Goncharov wrote: > Thanks Jeremy, will try to analyze the trace. > > This is not too extensive, but better then silent pin without > possibility to even notice that. > > I see that there are generally not enough deep tests for Openstack > driver in Zuul since mostly issues appear in Prod. I will to try to > invest some time there. We do functional testing against a devstack cloud. Unfortunately, as previously mentioned every cloud is just sufficiently unique that there are always corner cases that devstack can't catch. In general what we are able to test is that we can upload an image to devstack then boot that image and ssh into it. But then different clouds support different image upload methods and networking and even hypervisors. We do our best but once you get to the variety of clouds in the wild there is a lot more opportunity to find bad assumptions. > > Artem > > ---- > typed from mobile, auto-correct typos assumed > ---- > > On Thu, Jun 2, 2022, 21:47 Jeremy Stanley wrote: >> On 2022-06-02 21:12:33 +0200 (+0200), Artem Goncharov wrote: >> [...] >> > I listen to complains carefully, and that is the reason for R1.0 >> > not to exist as of now. >> [...] >> >> It's probably also worth pointing out, from a user experience >> perspective, that we anticipated backward-incompatible changes in >> 1.0.0 and so merged an openstacksdk<1 pin in nodepool's >> requirements.txt file. Unfortunately, instead the >> backward-incompatible changes were released as 0.99.0 so when the >> container images for our launchers updated we started getting >> another error back from all of the Rackspace regions we're using >> (traceback attached). >> >> I'm happy to file a bug report on that if nobody has beaten me to >> it, but further diagnostics will probably have to wait until after >> summit travel since we've rolled back to 0.61.0 for now. >> -- >> Jeremy Stanley From artem.goncharov at gmail.com Fri Jun 3 07:08:26 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Fri, 3 Jun 2022 09:08:26 +0200 Subject: [sdk] openstacksdk 0.99.0 breaks swift object header setting In-Reply-To: <6d960eb3-78c2-4410-8bfd-ec697e329fc9@www.fastmail.com> References: <315f833d-2b00-4f01-8ccc-172d7bf7b576@www.fastmail.com> <27B7F0D2-E382-49EF-B518-1764D6F83344@gmail.com> <20220602194511.zkkph5zmasr37jkk@yuggoth.org> <6d960eb3-78c2-4410-8bfd-ec697e329fc9@www.fastmail.com> Message-ID: <51744E7E-BCAE-4833-ADBC-40496CAEAAB8@gmail.com> > On 2. Jun 2022, at 22:52, Clark Boylan wrote: > > On Thu, Jun 2, 2022, at 12:54 PM, Artem Goncharov wrote: >> Thanks Jeremy, will try to analyze the trace. >> >> This is not too extensive, but better then silent pin without >> possibility to even notice that. >> >> I see that there are generally not enough deep tests for Openstack >> driver in Zuul since mostly issues appear in Prod. I will to try to >> invest some time there. > > We do functional testing against a devstack cloud. Unfortunately, as previously mentioned every cloud is just sufficiently unique that there are always corner cases that devstack can't catch. In general what we are able to test is that we can upload an image to devstack then boot that image and ssh into it. But then different clouds support different image upload methods and networking and even hypervisors. We do our best but once you get to the variety of clouds in the wild there is a lot more opportunity to find bad assumptions. Right. I didn?t want to say there is no testing. I mean in Zuul you have knowledge of the corner cases you face and they need to be included in tests to guarantee we do not break. But actually now I think it would be a real big benefit to everybody to include knowledge of those cases including corresponding tests into the SDK so that others can also benefit from that. Those who rely on SDK should just work no matter which particular cloud they are working with. This brings me slowly to other point. Our dream of real interoperability has not taken off. What is the sense of any certification if this does not save us even in pretty simple scenarios: provision host with custom image or upload object to object_store with CORS on? As a person who is responsible on our cloud for passing this certification I say openly: at the moment this is as good as useless at all. There are tests, and yes, sometimes it is even a challenge to pass those. But if they are not even able to guarantee us interoperability here this is not worth of effort. With this knowledge we can say that there is not a single OpenStack public cloud out there. They are all ?based on OpenStack?. Artem > >> >> Artem >> >> ---- >> typed from mobile, auto-correct typos assumed >> ---- >> >> On Thu, Jun 2, 2022, 21:47 Jeremy Stanley wrote: >>> On 2022-06-02 21:12:33 +0200 (+0200), Artem Goncharov wrote: >>> [...] >>>> I listen to complains carefully, and that is the reason for R1.0 >>>> not to exist as of now. >>> [...] >>> >>> It's probably also worth pointing out, from a user experience >>> perspective, that we anticipated backward-incompatible changes in >>> 1.0.0 and so merged an openstacksdk<1 pin in nodepool's >>> requirements.txt file. Unfortunately, instead the >>> backward-incompatible changes were released as 0.99.0 so when the >>> container images for our launchers updated we started getting >>> another error back from all of the Rackspace regions we're using >>> (traceback attached). >>> >>> I'm happy to file a bug report on that if nobody has beaten me to >>> it, but further diagnostics will probably have to wait until after >>> summit travel since we've rolled back to 0.61.0 for now. >>> -- >>> Jeremy Stanley > From katonalala at gmail.com Fri Jun 3 08:53:31 2022 From: katonalala at gmail.com (Lajos Katona) Date: Fri, 3 Jun 2022 10:53:31 +0200 Subject: [Forum] Meet the Projects Session! In-Reply-To: References: Message-ID: Hi Kendall, Thanks for this event, I add it to my calendar. Lajos Kendall Nelson ezt ?rta (id?pont: 2022. m?j. 31., K, 18:02): > Hello :) > > I wanted to take a moment to invite project maintainers, core > contributors, PTLs, governance officials, etc to a meet the projects > (projects being the OIF top level ones- Kata, Zuul, OpenStack, StarlingX, > and Airship) session during the Forum at the upcoming Summit in Berlin. It > will take place Tue, June 7, 11:20am - 12:30pm local in A - A06. > > The idea is to gather in a single place so that newer members of the > community or people unfamiliar with the project can come and meet some of > the active participants and ask questions. It's all really informal so > there is no need to prepare anything ahead of time. > > I realize it is pretty late in everyone's planning their schedules for the > summit, but if you have a few minutes to spare to come hang out and meet > some people and network, please do! We would love to see you there. > > -Kendall > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.Norrie at cgg.com Fri Jun 3 12:53:13 2022 From: Andrew.Norrie at cgg.com (Norrie, Andrew) Date: Fri, 3 Jun 2022 12:53:13 +0000 Subject: [kolla-ansible][xena] Cell database HA/redundancy? Message-ID: Hi, We are currently planning some large OpenStack deployments utilizing kolla-ansible and I'm curious what folks are doing with the cell database HA/redundancy. With kolla-ansible (xena) it appears that a loadbalancer setup is only allowed for the default database shard (shard 0)... reference: https://docs.openstack.org/kolla-ansible/latest/reference/databases/mariadb-guide.html If we are setting up separate cell database shards with Galera, I'm wondering if there is a convenient work-around or configuration for implementation of HA for these cell databases. In the inventory group_vars directory you can specify the group variables (for each cell database) as: nova_cell_database_address: nova_cell_database_group: but these aren't virtual IP hosts/addresses (non default db shard). This works perfectly fine with a single server cell database but not Galera. If that db server goes down the cell instance information is lost. Many thanks ... Andrew ________________________________ "This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of the email by others is strictly prohibited. If you are not the intended recipient, you must not review, disclose, copy, distribute or use this e-mail; please delete it from your system and notify the sender immediately." -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Fri Jun 3 15:12:04 2022 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 3 Jun 2022 17:12:04 +0200 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? In-Reply-To: References: Message-ID: On Thu, 2 Jun 2022 at 01:10, Andy Botting wrote: > > Hi Rados?aw, Hi Andy, > > I am wondering if and how OpenStackers are approaching the subject of > > VDI (Virtual Desktop Infrastructure), also nowadays commonly called > > DaaS (Desktop as a Service) offering. > > To be more specific, I am looking forward to a solution covering both > > Linux and Windows desktops, pre-OS screen access optional, > > well-integrated (i.e., having automation) with OpenStack Nova, > > preferably also with Cinder and Glance, optionally with Keystone > > (apart from self-auth of course!) and Heat, 3D acceleration optional > > (most of the desktops here don't need it), nice to have read-only > > sharing of screen. > > We just launched a service like this on the Nectar Research Cloud. > Like you, I looked around but couldn't find any Open Source projects > that would be suitable. > > In the end, we based ours on a project from the University of > Melbourne that did initially support Windows, but we removed some of > that support to simplify our codebase as we didn't need it. > > We called it Bumblebee, and the code is here: > https://github.com/NeCTAR-RC/bumblebee > > It's a Django web app with a Redis-based task scheduler that launches > Nova instances, booted from a Cinder volume, cloned from a base OS > image. > > The Django app directly modifies Guacamole's database to create the > connections once the OpenStack resources have been provisioned and > provides the user with a link that takes them to their desktop. First of all, wow, that looks very interesting and in fact very much what I'm looking for. As I mentioned in the original message, the things this solution lacks are not something blocking for me. Regarding the approach to Guacamole, I know that it's preferable to have guacamole extension (that provides the dynamic inventory) developed rather than meddle with the internal database but I guess it is a good start. > We used Keycloak with OIDC for authentication on the web app and > Guacamole and it's fairly seamless transitioning between the two. > > We did initially look at using Heat, but we wanted to support a > workflow of being able to destroy the VM and archive the OS image to > Swift (and restore again later if needed) so it was easier to just > manage the resources directly. Yeah, that's not a problem at all for me, the reasoning is sensible. > There's no 3D acceleration in our solution yet, but we're planning on > looking at things like virtual GPUs later on. > > Sadly, I'm not going to make it to Berlin either, but I was thinking > it might make for a good talk at the next summit. Regarding this, I agree with Julia. I think OpenInfra Live would benefit from a VDI/DaaS-oriented session. > I'm in the process of properly documenting it now, so I don't have > much to share at this point, but I'm happy to go into more detail > about anything. Any "quickstart setting up" would be awesome to have at this stage. As this is a Django app, I think I should be able to figure out the bits and bolts to get it up and running in some shape but obviously it will impede wider adoption. On the note of adoption, if I find it usable, I can provide support for it in Kolla [1] and help grow the project's adoption this way. Also, since this is OpenStack-centric, maybe you could consider migrating to OpenDev at some point to collaborate with interested parties using a common system? Just food for thought at the moment. Kind regards, -yoctozepto [1] https://docs.openstack.org/kolla/latest/ > cheers, > Andy From marios at redhat.com Fri Jun 3 15:22:23 2022 From: marios at redhat.com (Marios Andreou) Date: Fri, 3 Jun 2022 18:22:23 +0300 Subject: [TripleO] move stable/victoria tripleo to End Of Life OK? Message-ID: Hello TripleO the tripleo-ci team proposes that we transition the stable/victoria branch across all tripleo repos [1] to End Of Life [2]. Hopefully this is not a suprise to anyone as it has been discussed for a while (most recent discussion I can point to is at Zed PTG [3]). The stable/victoria branch was moved to Extended Maintenance with [4] so we can no longer make any releases for these repos. Victoria is not an active branch for TripleO (not an FFU branch) so we are just merging patches there on the way back to train. Are there any concerns, objections or any other comment? I'll give this a few days and if there are no further comments I'll post the EOL proposal and update here towards the end of next week, regards, marios [1] https://releases.openstack.org/teams/tripleo.html#victoria [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases [3] https://etherpad.opendev.org/p/tripleo-zed-ci-load [4] https://review.opendev.org/c/openstack/releases/+/837982 From gmann at ghanshyammann.com Fri Jun 3 17:46:59 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 03 Jun 2022 12:46:59 -0500 Subject: [all][tc] Change OpenStack release naming policy proposal In-Reply-To: <180e2904a63.ed97d27316807.1879483015684560107@ghanshyammann.com> References: <2175937.irdbgypaU6@p1> <18076dd3a30.116f2ef97411816.4718977843211132330@ghanshyammann.com> <180c03c9614.10ad4d72e148482.5160564778639516267@ghanshyammann.com> <180e2904a63.ed97d27316807.1879483015684560107@ghanshyammann.com> Message-ID: <1812aad73fb.b9dda98c518480.2929777219543202050@ghanshyammann.com> ---- On Fri, 20 May 2022 12:42:28 -0500 Ghanshyam Mann wrote ---- > ---- On Fri, 13 May 2022 20:43:57 -0500 Ghanshyam Mann wrote ---- > > Hello Everyone, > > > > Writing on the top for quick reading as we have a consensus on this. > > > > In today's TC meeting [3], we discussed this topic with Foundation staff and we all > > agreed to give the release name process handling to Foundation. TC will not be involved > > in the process. The release name will be mainly used for marketing purposes and we will > > use the release number as the primary identifier in our release scheduling, automated > > processes, directory structure etc. > > > > I have proposed the patch to document it in the TC release naming process. > > > > - https://review.opendev.org/c/openstack/governance/+/841800 > > As the next step, we will have a voice call to figure out how to use the number/name in our > development cycle/tooling. Meeting details are below: > > * When Tue May 24, 2022 9am ? 10am Pacific Time (16:00 - 17:00 UTC) > * Joining info Join with Google Meet = meet.google.com/gwi-cphb-rya > * Join by phone: (US) +1 563-447-5079 (PIN: 263292866) To update here: As discussed in the meeting, TC passed the resolution[1] and documented what place to use the release number and name, please reach out to TC if you have any queries on this: - https://governance.openstack.org/tc/reference/release-naming.html [1] https://governance.openstack.org/tc/resolutions/20220524-release-identification-process.html -gmann > > > -gmann > > > > > > [1] https://meetings.opendev.org/meetings/tc/2022/tc.2022-05-12-15.00.log.html#l-245 > > > > -gmann > > > > > > ---- On Fri, 29 Apr 2022 14:47:31 -0500 Ghanshyam Mann wrote ---- > > > > > > ---- On Fri, 29 Apr 2022 14:11:12 -0500 Goutham Pacha Ravi wrote ---- > > > > On Fri, Apr 29, 2022 at 8:36 PM Slawek Kaplonski wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > During the last PTG in April 2022 in the TC meeting we were discussing our release naming policy [1]. > > > > > > > > > > It seems that choosing appropriate name for every releases is very hard and time consuming. There is many factors which needs to be taken into consideration there like legal but also meaning of the chosen name in many different languages. > > > > > > > > > > > > > > > Finally we decided that now, after Zed release, when we will go all round through alphabet it is very good time to change this policy and use only numeric version with "year"."release in the year". It is proposed in [2]. > > > > > > > > > > This is also good timing for such change because in the same release we are going to start our "Tick Tock" release cadence which means that every Tick release will be release with .1 (like 2023.1, 2024.1, etc.) and every Tock release will be one with .2 (2023.2, 2024.2, etc.). > > > > > > > > Beloved TC, > > > > > > > > I'm highly disappointed in this 'decision', and would like for you to > > > > reconsider. I see the reasons you cite, but I feel like we're throwing > > > > the baby out with the bathwater here. Disagreements need not be > > > > feared, why not allow them to be aired publicly? That's a tenet of > > > > this open community. Allow names to be downvoted with reason during > > > > the proposal phase, and they'll organically fall-off from favor. > > > > > > > > Release names have always been a bonding factor. I've been happy to > > > > drum up contributor morale with our release names and the > > > > stories/anecdotes behind them. Release naming will not hurt/help the > > > > tick-tock release IMHO. We can append the release number to the name, > > > > and call it a day if you want. > > > > > > I agree with the disagrement ratio and that should not stop us doing the things. > > > But here we need to understand what type of disagrement we have and on what. > > > Most of the disagrement were cutural or historical where people has shown it > > > emotinally. And I personally as well as a TC or communitiy member does not feel > > > goot to ignore them or give them any reasoning not to listen them (because I do not > > > have any reasoning on these cultural/historical disagrement). > > > > > > Zed cycle was one good example of such thing when it was brought up in TC > > > channel about war thing[1] and asked about change the Zed name. I will be happy > > > to know what is best solution for this. > > > > > > 1. Change Zed name: it involve lot of technical work and communciation too. If yes then > > > let's do this now. > > > > > > 2. Do not listen to these emotional request to change name: We did this at the end and I > > > do not feel ok to do that. At least I do not want to ignore such request in future. > > > > > > Those are main reason we in TC decided to remvoe the name as they are culturally, emotionally > > > tied. That is main reason of droping those not any techncial or work wise issue. > > > > > > [1] https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2022-03-08.log.html#t2022-03-08T14:35:26 > > > > > > -gmann > > > > > > > > > > > I do believe our current release naming process is a step out of the > > > > TC's perceived charter. There are many technical challenges that the > > > > TC is tackling, and coordinating a vote/slugfest about names isn't as > > > > important as those. > > > > As Allison suggests, we could seek help from the foundation to run the > > > > community voting and vetting for the release naming process - and > > > > expect the same level of transparency as the 4 opens that the > > > > OpenStack community espouses. > > > > > > Yes we will offcourse open to that but at the same time we will be waiting > > > for the foudnation proposal to sovle such issue irespective of who is doing name > > > selection. So let's wait for that. > > > > > > -gmann > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] https://etherpad.opendev.org/p/tc-zed-ptg#L265 > > > > > > > > > > [2] https://review.opendev.org/c/openstack/governance/+/839897 > > > > > > > > > > > > > > > -- > > > > > > > > > > Slawek Kaplonski > > > > > > > > > > Principal Software Engineer > > > > > > > > > > Red Hat > > > > > > > > > > > > > > > > > > > > From gmann at ghanshyammann.com Fri Jun 3 19:21:53 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 03 Jun 2022 14:21:53 -0500 Subject: [all][tc] What's happening in Technical Committee: summary 03 June 2022: Reading: 5 min Message-ID: <1812b04558e.f62a6fa3520078.8768492805426232760@ghanshyammann.com> Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We had this week's meeting on 02 June. Most of the topics I am summarizing in this email. This was a video call and recordings are available @ https://www.youtube.com/watch?v=nHG2WJNceMs And logs summary are available @https://meetings.opendev.org/meetings/tc/2022/tc.2022-06-02-15.02.log.html * We are cancelling next week's meeting. The next TC weekly meeting will be on 16 June Thursday at 15:00 UTC, feel free to add the topic to the agenda[1] by 15 June. 2. What we completed this week: ========================= * TC resolution[2] and documentation[3] for OpenStack release identification schema. This clarify what place we need to use the release number and name. * Security SIG chair rotation from Gage to Jeremy[4] * Removed security-analysis repo from Security SIG[5] * Added a repository for the Large Scale SIG[6] * Added project stats check tool[7] 3. Activities In progress: ================== TC Tracker for Zed cycle ------------------------------ * Zed tracker etherpad includes the TC working items[8], many items are in-progress. Open Reviews ----------------- * Four open reviews for ongoing activities[9]. New release cadence "SLURP" open items -------------------------------------------------- 1. release notes strategy: Brian proposal for ""SLURP release notes approach is up for review[10]. Improve project governance --------------------------------- Slawek has the proposal the framework up and it is under review[11]. New ELK service dashboard: e-r service ----------------------------------------------- dpawlik working on ansible playbooks to deploy e-r and functional tests. TripleO team and dpawlik are working on the e-r repo merge. Consistent and Secure Default RBAC ------------------------------------------- We had a call yesterday and discussed about presenting this to the ops meetup in berlin and getting feedback, especially on the 'scope' concept. knikolla plan to moderate this topic in the ops meetup but unfortunately he will not be to make it to the summit. I have added a few notes and what questions we need to ask the operator in etherpad[12]. Please note that operator feedback is very important for us to proceed on 'scope' in SRBAC. If you are operator or know anyone around you, we request you to provide feedback. 2021 User Survey TC Question Analysis ----------------------------------------------- No update on this. The survey summary is up for review[13]. Feel free to check and provide feedback. Zed cycle Leaderless projects ---------------------------------- No updates on this. Only Adjutant project is leaderless/maintainer-less. We will check Adjutant's the situation again on ML and hope Braden will be ready with their company side permission[14]. Fixing Zuul config error ---------------------------- Requesting projects with zuul config error to look into those and fix them which should not take much time[15][16]. Project updates ------------------- * Add a repository for the Large Scale SIG[17] 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/843214 [3] https://review.opendev.org/c/openstack/governance/+/841800 [4] https://review.opendev.org/c/openstack/governance-sigs/+/844446 [5] https://review.opendev.org/c/openstack/governance/+/844463 [6] https://review.opendev.org/c/openstack/governance/+/843535 [7] https://review.opendev.org/c/openstack/governance/+/810037 [8] https://etherpad.opendev.org/p/tc-zed-tracker [9] https://review.opendev.org/q/projects:openstack/governance+status:open [10] https://review.opendev.org/c/openstack/project-team-guide/+/843457 [11] https://review.opendev.org/c/openstack/governance/+/839880 [12] https://etherpad.opendev.org/p/ops-meetup-berlin-2022-planning#L53 [13] https://review.opendev.org/c/openstack/governance/+/836888 [14] http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027626.html [15] https://etherpad.opendev.org/p/zuul-config-error-openstack [16] http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html [17] https://review.opendev.org/c/openstack/project-config/+/843534 [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting -gmann From haiwu.us at gmail.com Fri Jun 3 20:37:52 2022 From: haiwu.us at gmail.com (hai wu) Date: Fri, 3 Jun 2022 15:37:52 -0500 Subject: [sdk][openstacksdk][keystone] Does openstacksdk support v3tokenlessauth? Message-ID: Does openstacksdk support keystone auth type "v3tokenlessauth"? I got this error message when trying to enable openstacksdk with auth_type="v3tokenlessauth" openstack.exceptions.NotSupported: The identity service for TEST: exists but does not have any supported versions. The following are the possible relevant documents I could find, and there is no place mentioning the support for tokenless auth using openstacksdk: https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html https://docs.openstack.org/keystone/train/admin/configure_tokenless_x509.html From artem.goncharov at gmail.com Sat Jun 4 06:25:28 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Sat, 4 Jun 2022 08:25:28 +0200 Subject: [sdk][openstacksdk][keystone] Does openstacksdk support v3tokenlessauth? In-Reply-To: References: Message-ID: The error you posted suggests you reached keystone, got some answer, but it is not sufficient for SDK to continue. It can be caused by misconfiguration on keystone. But in any way you should provide logs (add `openstack.enable_logging(debug=True)`) and provide it here. Without this it is not possible to give you any other information. Artem ---- typed from mobile, auto-correct typos assumed ---- On Fri, Jun 3, 2022, 22:40 hai wu wrote: > Does openstacksdk support keystone auth type "v3tokenlessauth"? > > I got this error message when trying to enable openstacksdk with > auth_type="v3tokenlessauth" > > openstack.exceptions.NotSupported: The identity service for TEST: > exists but does not have any supported versions. > > The following are the possible relevant documents I could find, and > there is no place mentioning the support for tokenless auth using > openstacksdk: > > https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html > > https://docs.openstack.org/keystone/train/admin/configure_tokenless_x509.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haiwu.us at gmail.com Sat Jun 4 13:37:24 2022 From: haiwu.us at gmail.com (hai wu) Date: Sat, 4 Jun 2022 08:37:24 -0500 Subject: [sdk][openstacksdk][keystone] Does openstacksdk support v3tokenlessauth? In-Reply-To: References: Message-ID: I did add the above debug line in the openstacksdk script before, but I did not get any more information.. With the current keystone setting, the regular 'openstack' command line is working with x509 tokenless (v3tokenlessauth) authentication for this simple test case (with test command: openstack project list), but I could not get openstacksdk script to work the same way.. ~ ? cat /tmp/test.py import openstack from openstack import ( # type: ignore # pylint: disable=import-self connection, exceptions, ) openstack.enable_logging(debug=True) conn = connection.from_config(cloud_name='TEST') for project in sorted(conn.identity.projects(), key=lambda p: p.name): print(project) ~ ? python /tmp/test.py Traceback (most recent call last): File "/tmp/test.py", line 10, in for project in sorted(conn.identity.projects(), key=lambda p: p.name): File "/home/hai/venv/openstack/lib/python3.10/site-packages/openstack/service_description.py", line 87, in __get__ proxy = self._make_proxy(instance) File "/home/hai/venv/openstack/lib/python3.10/site-packages/openstack/service_description.py", line 266, in _make_proxy raise exceptions.NotSupported( openstack.exceptions.NotSupported: The identity service for TEST: exists but does not have any supported versions. So the 'found_version' ends up as 'None' from this line in 'service_description.py' file: found_version = temp_adapter.get_api_major_version() Here is my clouds.yaml for 'TEST': TEST: auth_type: "v3tokenlessauth" auth: project_name: testme auth_url: https://keystonetest:5050/v3 project_domain_name: Default key: /home/hai/hai.key cert: /home/hai/hai.pem cacert: /home/hai/myca.crt region: RegionOne identity_api_version: 3 On Sat, Jun 4, 2022 at 1:25 AM Artem Goncharov wrote: > > The error you posted suggests you reached keystone, got some answer, but it is not sufficient for SDK to continue. It can be caused by misconfiguration on keystone. > > But in any way you should provide logs (add `openstack.enable_logging(debug=True)`) and provide it here. Without this it is not possible to give you any other information. > > Artem > > ---- > typed from mobile, auto-correct typos assumed > ---- > > On Fri, Jun 3, 2022, 22:40 hai wu wrote: >> >> Does openstacksdk support keystone auth type "v3tokenlessauth"? >> >> I got this error message when trying to enable openstacksdk with >> auth_type="v3tokenlessauth" >> >> openstack.exceptions.NotSupported: The identity service for TEST: >> exists but does not have any supported versions. >> >> The following are the possible relevant documents I could find, and >> there is no place mentioning the support for tokenless auth using >> openstacksdk: >> https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html >> https://docs.openstack.org/keystone/train/admin/configure_tokenless_x509.html >> From radoslaw.piliszek at gmail.com Sat Jun 4 18:41:28 2022 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Sat, 4 Jun 2022 20:41:28 +0200 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? In-Reply-To: References: Message-ID: Writing to let you know I have also found the following related paper: [1] and reached out to its authors in the hope to enable further collaboration to happen. The paper is not open access so I have only obtained it for myself and am unsure if licensing permits me to share, thus I also asked the authors to share their copy (that they have copyrights to). I have obviously let them know of the existence of this thread. ;-) Let's stay tuned. [1] https://link.springer.com/chapter/10.1007/978-3-030-99191-3_12 Kind regards, -yoctozepto From amonster369 at gmail.com Sun Jun 5 13:15:11 2022 From: amonster369 at gmail.com (A Monster) Date: Sun, 5 Jun 2022 14:15:11 +0100 Subject: Using infiniband for openstack network communications Message-ID: Is it possible to use the infiniband port for openstack networks without having to use the infiniband port as an ethernet port ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Sun Jun 5 13:22:59 2022 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Sun, 5 Jun 2022 15:22:59 +0200 Subject: Using infiniband for openstack network communications In-Reply-To: References: Message-ID: Yes, why not? As long as your infiniband card support sr-iov there should be no problems with that. You can also check one blog which describes the experience https://satishdotpatel.github.io/HPC-on-openstack/ ??, 5 ???. 2022 ?., 15:20 A Monster : > Is it possible to use the infiniband port for openstack networks without > having to use the infiniband port as an ethernet port ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yasufum.o at gmail.com Sun Jun 5 18:01:43 2022 From: yasufum.o at gmail.com (Yasufumi Ogawa) Date: Mon, 6 Jun 2022 03:01:43 +0900 Subject: [tacker] Weekly IRC meeting on Jun 7th cancelled Message-ID: <19dcb1e5-4fb7-0361-8cf7-14712e5efc94@gmail.com> Hi team, The next meeting on Jun 7th is cancelled due to the Berlin summit. Cheers, Yasufumi From noonedeadpunk at gmail.com Sun Jun 5 18:38:15 2022 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Sun, 5 Jun 2022 20:38:15 +0200 Subject: [openstack-ansible] Meeting on Jun 7th 2022 is cancelled Message-ID: Hi everybody! The next meeting on Jun 7th is cancelled due to the Berlin summit. Hope seeing you in person instead! We will have next meeting Jun 14th. -- Kind regards, Dmitriy Rabotyagov From p.aminian.server at gmail.com Sun Jun 5 19:06:50 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Sun, 5 Jun 2022 23:36:50 +0430 Subject: masakari-hostmonitor error Message-ID: hello on kolla ansible masakari-hostmonitor container on wallaby version I have this error : 2022-06-05 23:30:51.880 7 WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication using 'eth0' is failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2022-06-05 23:30:51.881 7 ERROR masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication is failed. 2022-06-05 23:31:22.042 7 WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] Exception caught: Unexpected error while running command. Command: crmadmin -S myhostname Exit code: 102 Stdout: '' Stderr: 'error: Could not connect to controller: Connection refused\nerror: Command failed: Connection refused\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Could you please help me ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.aminian.server at gmail.com Sun Jun 5 19:39:17 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Mon, 6 Jun 2022 00:09:17 +0430 Subject: masakari bug Message-ID: on kolla ansible masakari-hostmonitor container on wallaby version I have this error : 2022-06-05 23:30:51.880 7 WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication using 'eth0' is failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2022-06-05 23:30:51.881 7 ERROR masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication is failed. 2022-06-05 23:31:22.042 7 WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] Exception caught: Unexpected error while running command. Command: crmadmin -S myhostname Exit code: 102 Stdout: '' Stderr: 'error: Could not connect to controller: Connection refused\nerror: Command failed: Connection refused\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkajinam at redhat.com Mon Jun 6 04:15:50 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Mon, 6 Jun 2022 13:15:50 +0900 Subject: [all][doc] the openstack-tox-docs job is broken with Sphinx >= 5.0.0 Message-ID: Hi, It seems the openstack-tox-docs job is broken in some projects because of the following change made in Sphinx >= 5.0.0 . https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 We currently have Sphinx 4.5.0 in upper-constraints.txt but some projects (eg. Heat, Nova, Neutron) are using only their own docs-constraints.txt and pull the latest version which is 5.0.1 . IMO we need to fix the problem by the following two steps, but I'd appreciate if anybody can share a better idea. 1. Update tox.ini to honor upper-constraints.txt. This is needed in master and stable/branches as well. 2. Update docs/source/conf.py to hard-code the language parameter to 'en'. This is required in master so that we can bump the upper version later. Thank you, Takashi Kajinami -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkajinam at redhat.com Mon Jun 6 04:27:51 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Mon, 6 Jun 2022 13:27:51 +0900 Subject: [all][doc] the openstack-tox-docs job is broken with Sphinx >= 5.0.0 In-Reply-To: References: Message-ID: Sorry it seems I was looking at something wrong and it seems the projects I mentioned as examples(Heat, Nova and Neutron) are already using upper constraints for doc build. However I'd recommend reach project reviews their current tox.ini. Updating conf.py would be still required so it's better to work on it earlier before the upper constraint is updated. On Mon, Jun 6, 2022 at 1:15 PM Takashi Kajinami wrote: > Hi, > > > It seems the openstack-tox-docs job is broken in some projects because of > the following > change made in Sphinx >= 5.0.0 . > > https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 > > We currently have Sphinx 4.5.0 in upper-constraints.txt but some projects > (eg. Heat, Nova, > Neutron) are using only their own docs-constraints.txt and pull the latest > version which is 5.0.1 . > > IMO we need to fix the problem by the following two steps, but I'd > appreciate > if anybody can share a better idea. > > 1. Update tox.ini to honor upper-constraints.txt. This is needed in > master and > stable/branches as well. > > 2. Update docs/source/conf.py to hard-code the language parameter to 'en'. > This is required in master so that we can bump the upper version > later. > > Thank you, > Takashi Kajinami > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Mon Jun 6 08:22:22 2022 From: mark at stackhpc.com (Mark Goddard) Date: Mon, 6 Jun 2022 09:22:22 +0100 Subject: [kolla-ansible][xena] Cell database HA/redundancy? In-Reply-To: References: Message-ID: On Fri, 3 Jun 2022 at 14:10, Norrie, Andrew wrote: > > > Hi, > > > > We are currently planning some large OpenStack deployments utilizing > kolla-ansible and > > I?m curious what folks are doing with the cell database HA/redundancy. > > > > With kolla-ansible (xena) it appears that a loadbalancer setup is only > allowed for the > > default database shard (shard 0)... reference: > https://docs.openstack.org/kolla-ansible/latest/reference/databases/mariadb-guide.html > > > > If we are setting up separate cell database shards with Galera, I?m > wondering if there is a convenient work-around or configuration for > implementation of HA for these cell databases. > > > > In the inventory group_vars directory you can specify the group variables > (for each cell database) as: > > > > nova_cell_database_address: > > nova_cell_database_group: > > > > but these aren?t virtual IP hosts/addresses (non default db shard). This > works perfectly fine with > > a single server cell database but not Galera. If that db server goes down > the cell instance information is lost. > > > Hi Andrew, You are correct - only the first shard is load balanced. The sharding feature was actually just the first patch in a series intended to support proxysql. I believe this would add the functionality you are looking for. In fact, the proposed test case for proxysql in our CI is a multi-cell deployment. Here is the patch series: https://review.opendev.org/q/topic:proxysql Unfortunately it has been stuck for a while, largely due to core reviewer bandwidth. The author is Michael Arbet, who is often on IRC as kevko. I'd suggest registering your interest in these patches via gerrit, and at one of the weekly kolla IRC meetings (this week is cancelled due to the summit). If you have time available, then testing and providing reviews could help to get the patches moving. In the meantime, you can configure HAProxy to load balance additional services, by placing an HAProxy config snippet in /etc/kolla/config/haproxy/services.d/*.cfg. Regards, Mark Many thanks ... Andrew > ------------------------------ > ?This e-mail and any accompanying attachments are confidential. The > information is intended solely for the use of the individual to whom it is > addressed. Any review, disclosure, copying, distribution, or use of the > email by others is strictly prohibited. If you are not the intended > recipient, you must not review, disclose, copy, distribute or use this > e-mail; please delete it from your system and notify the sender > immediately.? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Mon Jun 6 10:55:42 2022 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Mon, 6 Jun 2022 12:55:42 +0200 Subject: masakari-hostmonitor error In-Reply-To: References: Message-ID: set enable_hacluster to true and redeploy -yoctozepto On Sun, Jun 5, 2022, 21:08 Parsa Aminian wrote: > hello > on kolla ansible masakari-hostmonitor container on wallaby version I have > this error : > > 2022-06-05 23:30:51.880 7 WARNING > masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync > communication using 'eth0' is failed.: > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while > running command. > 2022-06-05 23:30:51.881 7 ERROR > masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync > communication is failed. > 2022-06-05 23:31:22.042 7 WARNING > masakarimonitors.hostmonitor.host_handler.handle_host [-] Exception caught: > Unexpected error while running command. > Command: crmadmin -S myhostname > Exit code: 102 > Stdout: '' > Stderr: 'error: Could not connect to controller: Connection > refused\nerror: Command failed: Connection refused\n': > oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while > running command. > > Could you please help me ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnasiadka at gmail.com Mon Jun 6 11:02:24 2022 From: mnasiadka at gmail.com (=?UTF-8?Q?Micha=C5=82_Nasiadka?=) Date: Mon, 6 Jun 2022 13:02:24 +0200 Subject: [kolla] weekly meeting 8th June Message-ID: Hello koalas, This Wednesday (8th June) meeting is cancelled due to OpenInfra Summit in Berlin. See you on a meeting next week. Michal -- Micha? Nasiadka mnasiadka at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Mon Jun 6 13:14:03 2022 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Mon, 6 Jun 2022 09:14:03 -0400 Subject: [all][doc] the openstack-tox-docs job is broken with Sphinx >= 5.0.0 In-Reply-To: References: Message-ID: <4bff2d02-3bd7-48bc-5acc-33233dd63a0b@gmail.com> On 6/6/22 12:27 AM, Takashi Kajinami wrote: [snip] > Updating conf.py would be still required so it's better to work on it > earlier > before the upper constraint is updated. I was just looking into this, and as long as your conf.py does not explicitly set language = None the default language is set to 'en' internally by Sphinx in 5.0.0+ [0]. So as long as you have the language setting either missing or commented out in your conf.py, you should not see this warning (which, when treated as an error, breaks the doc build). [0] https://github.com/sphinx-doc/sphinx/commit/e4e58a4f2791e528cdaa861b96636a1e37a558ba > > On Mon, Jun 6, 2022 at 1:15 PM Takashi Kajinami > wrote: > > Hi, > > > It seems the openstack-tox-docs job is broken in some projects > because of the following > change made in Sphinx >= 5.0.0 . > https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 > > > We currently have Sphinx 4.5.0 in upper-constraints.txt but some > projects (eg. Heat, Nova, > Neutron) are using only their own docs-constraints.txt and pull the > latest version which is 5.0.1 . > > IMO we need to fix the problem by the following two steps, but I'd > appreciate > if anybody can share a better idea. > > ?1. Update tox.ini to honor upper-constraints.txt. This is needed > in master and > ???? stable/branches as well. > > ?2. Update docs/source/conf.py to hard-code the language parameter > to 'en'. > ???? This is required in master so that we can bump the upper > version later. > > Thank you, > Takashi Kajinami > From amonster369 at gmail.com Mon Jun 6 13:15:56 2022 From: amonster369 at gmail.com (A Monster) Date: Mon, 6 Jun 2022 14:15:56 +0100 Subject: Using infiniband for openstack network communications Message-ID: Thank you for your reply, I'll check the link you've sent to me immediately. -------------- next part -------------- An HTML attachment was scrubbed... URL: From amonster369 at gmail.com Mon Jun 6 13:36:38 2022 From: amonster369 at gmail.com (A Monster) Date: Mon, 6 Jun 2022 14:36:38 +0100 Subject: Using infiniband for openstack network communications In-Reply-To: References: Message-ID: What I wanted to actually do is not what is done in this link https://satishdotpatel.github.io/HPC-on-openstack/ , because here vm's are given the possibility to use the infiniband port through Virtual functions, but in my case, I want to use infiniband for openstack management, compute and storage networks, to get better performance than when using ethernet and rg45 cables. so I'm wondering if it's feasible and whether it's a good thing or not. Thank you. On Mon, 6 Jun 2022 at 14:15, A Monster wrote: > Thank you for your reply, I'll check the link you've sent to me > immediately. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Mon Jun 6 13:51:43 2022 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Mon, 6 Jun 2022 15:51:43 +0200 Subject: Using infiniband for openstack network communications In-Reply-To: References: Message-ID: Ah, ok. Well, you can use infiniband for control plane only with IPoIB. There's no RDMA support if that's what you're asking. ??, 6 ???. 2022 ?., 15:36 A Monster : > What I wanted to actually do is not what is done in this link > https://satishdotpatel.github.io/HPC-on-openstack/ , because here vm's > are given the possibility to use the infiniband port through Virtual > functions, but in my case, I want to use infiniband for openstack > management, compute and storage networks, to get better performance than > when using ethernet and rg45 cables. so I'm wondering if it's feasible and > whether it's a good thing or not. > Thank you. > > On Mon, 6 Jun 2022 at 14:15, A Monster wrote: > >> Thank you for your reply, I'll check the link you've sent to me >> immediately. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkajinam at redhat.com Mon Jun 6 14:15:54 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Mon, 6 Jun 2022 23:15:54 +0900 Subject: [all][doc] the openstack-tox-docs job is broken with Sphinx >= 5.0.0 In-Reply-To: <4bff2d02-3bd7-48bc-5acc-33233dd63a0b@gmail.com> References: <4bff2d02-3bd7-48bc-5acc-33233dd63a0b@gmail.com> Message-ID: Thanks Brian, Actually I hit this problem initially with heat-cfntools which is not using upper-constraints in doc build but has language=None explicitly set, so what you mentioned makes quite clear sense. Quick grep shows a few instances of the hard-coded language=None . These should be commented out before we get the constraint bumped. openstack/python-swiftclient/releasenotes/source/conf.py:language = None openstack/blazar/api-ref/source/conf.py:language = None openstack/heat-agents/doc/source/conf.py:language = None openstack/heat-agents/releasenotes/source/conf.py:language = None openstack/swift/releasenotes/source/conf.py:language = None openstack/storlets/releasenotes/source/conf.py:language = None openstack/placement/releasenotes/source/conf.py:language = None openstack/osc-lib/releasenotes/source/conf.py:language = None openstack/python-heatclient/releasenotes/source/conf.py:language = None openstack/trove-dashboard/releasenotes/source/conf.py:language = None openstack/virtualbmc/releasenotes/source/conf.py:language = None openstack/heat-templates/doc/source/conf.py:language = None Disclaimer: The above list might be missing some projects. On Mon, Jun 6, 2022 at 10:19 PM Brian Rosmaita wrote: > On 6/6/22 12:27 AM, Takashi Kajinami wrote: > [snip] > > Updating conf.py would be still required so it's better to work on it > > earlier > > before the upper constraint is updated. > > I was just looking into this, and as long as your conf.py does not > explicitly set > language = None > the default language is set to 'en' internally by Sphinx in 5.0.0+ [0]. > So as long as you have the language setting either missing or > commented out in your conf.py, you should not see this warning (which, > when treated as an error, breaks the doc build). > > [0] > > https://github.com/sphinx-doc/sphinx/commit/e4e58a4f2791e528cdaa861b96636a1e37a558ba > > > > > On Mon, Jun 6, 2022 at 1:15 PM Takashi Kajinami > > wrote: > > > > Hi, > > > > > > It seems the openstack-tox-docs job is broken in some projects > > because of the following > > change made in Sphinx >= 5.0.0 . > > > https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 > > < > https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 > > > > > > We currently have Sphinx 4.5.0 in upper-constraints.txt but some > > projects (eg. Heat, Nova, > > Neutron) are using only their own docs-constraints.txt and pull the > > latest version which is 5.0.1 . > > > > IMO we need to fix the problem by the following two steps, but I'd > > appreciate > > if anybody can share a better idea. > > > > 1. Update tox.ini to honor upper-constraints.txt. This is needed > > in master and > > stable/branches as well. > > > > 2. Update docs/source/conf.py to hard-code the language parameter > > to 'en'. > > This is required in master so that we can bump the upper > > version later. > > > > Thank you, > > Takashi Kajinami > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amonster369 at gmail.com Mon Jun 6 14:30:37 2022 From: amonster369 at gmail.com (A Monster) Date: Mon, 6 Jun 2022 15:30:37 +0100 Subject: Using infiniband for openstack network communications Message-ID: Alright, thank you again Sir. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at stackhpc.com Mon Jun 6 15:01:51 2022 From: pierre at stackhpc.com (Pierre Riteau) Date: Mon, 6 Jun 2022 17:01:51 +0200 Subject: [all][doc] the openstack-tox-docs job is broken with Sphinx >= 5.0.0 In-Reply-To: References: <4bff2d02-3bd7-48bc-5acc-33233dd63a0b@gmail.com> Message-ID: Thank you Takashi for raising this. I also found one in the release notes configuration for python-blazarclient. I will fix the blazar repositories. On Mon, 6 Jun 2022 at 16:22, Takashi Kajinami wrote: > Thanks Brian, > > > Actually I hit this problem initially with heat-cfntools which is not > using upper-constraints in doc build > but has language=None explicitly set, so what you mentioned makes quite > clear sense. > > Quick grep shows a few instances of the hard-coded language=None . These > should be commented out > before we get the constraint bumped. > > openstack/python-swiftclient/releasenotes/source/conf.py:language = None > openstack/blazar/api-ref/source/conf.py:language = None > openstack/heat-agents/doc/source/conf.py:language = None > openstack/heat-agents/releasenotes/source/conf.py:language = None > openstack/swift/releasenotes/source/conf.py:language = None > openstack/storlets/releasenotes/source/conf.py:language = None > openstack/placement/releasenotes/source/conf.py:language = None > openstack/osc-lib/releasenotes/source/conf.py:language = None > openstack/python-heatclient/releasenotes/source/conf.py:language = None > openstack/trove-dashboard/releasenotes/source/conf.py:language = None > openstack/virtualbmc/releasenotes/source/conf.py:language = None > openstack/heat-templates/doc/source/conf.py:language = None > > Disclaimer: The above list might be missing some projects. > > On Mon, Jun 6, 2022 at 10:19 PM Brian Rosmaita > wrote: > >> On 6/6/22 12:27 AM, Takashi Kajinami wrote: >> [snip] >> > Updating conf.py would be still required so it's better to work on it >> > earlier >> > before the upper constraint is updated. >> >> I was just looking into this, and as long as your conf.py does not >> explicitly set >> language = None >> the default language is set to 'en' internally by Sphinx in 5.0.0+ [0]. >> So as long as you have the language setting either missing or >> commented out in your conf.py, you should not see this warning (which, >> when treated as an error, breaks the doc build). >> >> [0] >> >> https://github.com/sphinx-doc/sphinx/commit/e4e58a4f2791e528cdaa861b96636a1e37a558ba >> >> > >> > On Mon, Jun 6, 2022 at 1:15 PM Takashi Kajinami > > > wrote: >> > >> > Hi, >> > >> > >> > It seems the openstack-tox-docs job is broken in some projects >> > because of the following >> > change made in Sphinx >= 5.0.0 . >> > >> https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 >> > < >> https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 >> > >> > >> > We currently have Sphinx 4.5.0 in upper-constraints.txt but some >> > projects (eg. Heat, Nova, >> > Neutron) are using only their own docs-constraints.txt and pull the >> > latest version which is 5.0.1 . >> > >> > IMO we need to fix the problem by the following two steps, but I'd >> > appreciate >> > if anybody can share a better idea. >> > >> > 1. Update tox.ini to honor upper-constraints.txt. This is needed >> > in master and >> > stable/branches as well. >> > >> > 2. Update docs/source/conf.py to hard-code the language parameter >> > to 'en'. >> > This is required in master so that we can bump the upper >> > version later. >> > >> > Thank you, >> > Takashi Kajinami >> > >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpodivin at redhat.com Mon Jun 6 15:05:13 2022 From: jpodivin at redhat.com (Jiri Podivin) Date: Mon, 6 Jun 2022 17:05:13 +0200 Subject: [all][doc] the openstack-tox-docs job is broken with Sphinx >= 5.0.0 In-Reply-To: References: <4bff2d02-3bd7-48bc-5acc-33233dd63a0b@gmail.com> Message-ID: Cursory search appears to indicate that these are part of some boilerplate configuration code. So I believe we should be fine if we remove it. Still, I wonder why was the language set to `None` in the first place. On Mon, Jun 6, 2022 at 4:21 PM Takashi Kajinami wrote: > Thanks Brian, > > > Actually I hit this problem initially with heat-cfntools which is not > using upper-constraints in doc build > but has language=None explicitly set, so what you mentioned makes quite > clear sense. > > Quick grep shows a few instances of the hard-coded language=None . These > should be commented out > before we get the constraint bumped. > > openstack/python-swiftclient/releasenotes/source/conf.py:language = None > openstack/blazar/api-ref/source/conf.py:language = None > openstack/heat-agents/doc/source/conf.py:language = None > openstack/heat-agents/releasenotes/source/conf.py:language = None > openstack/swift/releasenotes/source/conf.py:language = None > openstack/storlets/releasenotes/source/conf.py:language = None > openstack/placement/releasenotes/source/conf.py:language = None > openstack/osc-lib/releasenotes/source/conf.py:language = None > openstack/python-heatclient/releasenotes/source/conf.py:language = None > openstack/trove-dashboard/releasenotes/source/conf.py:language = None > openstack/virtualbmc/releasenotes/source/conf.py:language = None > openstack/heat-templates/doc/source/conf.py:language = None > > Disclaimer: The above list might be missing some projects. > > On Mon, Jun 6, 2022 at 10:19 PM Brian Rosmaita > wrote: > >> On 6/6/22 12:27 AM, Takashi Kajinami wrote: >> [snip] >> > Updating conf.py would be still required so it's better to work on it >> > earlier >> > before the upper constraint is updated. >> >> I was just looking into this, and as long as your conf.py does not >> explicitly set >> language = None >> the default language is set to 'en' internally by Sphinx in 5.0.0+ [0]. >> So as long as you have the language setting either missing or >> commented out in your conf.py, you should not see this warning (which, >> when treated as an error, breaks the doc build). >> >> [0] >> >> https://github.com/sphinx-doc/sphinx/commit/e4e58a4f2791e528cdaa861b96636a1e37a558ba >> >> > >> > On Mon, Jun 6, 2022 at 1:15 PM Takashi Kajinami > > > wrote: >> > >> > Hi, >> > >> > >> > It seems the openstack-tox-docs job is broken in some projects >> > because of the following >> > change made in Sphinx >= 5.0.0 . >> > >> https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 >> > < >> https://github.com/sphinx-doc/sphinx/commit/a3d09835522e6487cf93dcc871179db8d69e4180 >> > >> > >> > We currently have Sphinx 4.5.0 in upper-constraints.txt but some >> > projects (eg. Heat, Nova, >> > Neutron) are using only their own docs-constraints.txt and pull the >> > latest version which is 5.0.1 . >> > >> > IMO we need to fix the problem by the following two steps, but I'd >> > appreciate >> > if anybody can share a better idea. >> > >> > 1. Update tox.ini to honor upper-constraints.txt. This is needed >> > in master and >> > stable/branches as well. >> > >> > 2. Update docs/source/conf.py to hard-code the language parameter >> > to 'en'. >> > This is required in master so that we can bump the upper >> > version later. >> > >> > Thank you, >> > Takashi Kajinami >> > >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.harrison at manchester.ac.uk Mon Jun 6 15:11:29 2022 From: paul.harrison at manchester.ac.uk (Paul Harrison) Date: Mon, 6 Jun 2022 15:11:29 +0000 Subject: [nova] local LVM volume on compute hosts Message-ID: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> I have installed a small test cluster using openstack-ansible (xena) and would like to use local LVM volumes for storing instance volumes - and following https://docs.openstack.org/nova/latest/configuration/config.html and (e.g. https://cloudnull.io/2017/12/nova-lvm-an-iop-love-story/) I have the libvirt section of my /etc/nova/nova.conf as below. -8<--- [libvirt] inject_partition = -2 inject_password = False inject_key = False virt_type = kvm live_migration_with_native_tls = true live_migration_scheme = tls live_migration_inbound_addr = 172.29.236.5 hw_disk_discard = ignore disk_cachemodes = none images_type = lvm images_volume_group = nova volume_clear = zero -8 From kdhall at binghamton.edu Mon Jun 6 16:50:24 2022 From: kdhall at binghamton.edu (Dave Hall) Date: Mon, 6 Jun 2022 12:50:24 -0400 Subject: Openstack-Ansible vs Kolla-Ansible -Xena or later Message-ID: Hello, My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am sensitive to the effort that has been put into both of these projects, what I really need right now is the most fool-proof way to deploy and manage a small production cluster for academic instructional use. I have been working for a couple months with OpenStack-Ansible to deploy a small (3-node) test cluster on VirtualBox VMs in preparation for a larger deployment on real hardware - 6 to 10 nodes. (Details: Debian 11 deployment, Debian 10 infra, compute, and storage nodes) It has been slow going, at least partially due to some issues and limitations with VirtualBox. However, deploying a test cluster on VMs still seems preferable to just diving deployment on real hardware and going through multiple scrubs/reinstalls. Today I saw that Mark asked a question about Kolla-Ansible. So, as a 'beginner', should I shift and look at Kolla-Ansible, or should I stay the course and continue with Openstack-Ansible? What are the pros and cons of each? Thanks. -Dave -- Dave Hall Binghamton University kdhall at binghamton.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian.rohmann at inovex.de Mon Jun 6 16:59:41 2022 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Mon, 6 Jun 2022 18:59:41 +0200 Subject: [designate] How to avoid NXDOMAIN or stale data during cold start of a (new) machine In-Reply-To: References: <69ab8e54-f419-4cd1-f289-a0b5efb7f723@inovex.de> Message-ID: Hey Michael, all, sorry for the late reply, just got around to playing with Designate and BIND9 again ... On 10/05/2022 19:04, Michael Johnson wrote: > On startup, BIND9 will start sending SOA serial number queries for all > of the zones it knows about. In the case of Designate, that means > BIND9 will send out requests to the miniDNS instances to check if the > serial number in Designate is newer than the one in BIND9. If the > serial number in Designate is newer, BIND9 will initiate a zone > transfer from the miniDNS in Designate. > > BIND9, by default, will do 20 SOA serial number queries at a time > (less on older versions of BIND). See the serial-query-rate setting in > the rate limiter knowledge base article[1]. > > The tuning knowledge base article[2] also discusses settings that can > be adjusted for secondary servers that may also help speed up a cold > startup. > > Off my head, I don't know of a way to tell BIND9 to not answer queries > via rdnc or such. I usually block network access to a new BIND9 > instance until the "rdnc status" shows the "soa queries in progress" > and "xfers running" drop to 0 or a low number. Thanks for your thoughts and for distinguishing the sync of existing domains via AXFR/IXFR zone-transfers, because I was actually talking about an empty BIND9 in need to actually learn about zones. Maybe I was not clear about that on my initial post. But in the case of BIND9 not having a zone configured it requires input from designate before asking for the latest zone data from mDNS. Think about a freshly restored or a newly added server to the pool that requires to have ALL zones added and until then is not doing any IXFR / AXFR, but wrongfully sending NXDOMAIN. For new zones there is https://github.com/openstack/designate/blob/5d5d83e511acbf5d6f34e9a998ff16d96c5bb162/designate/backend/impl_bind9.py#L76 which creates the zone (via rndc addzone - https://docs.openstack.org/designate/latest/admin/backends/bind9.html#bind9-configuration). and triggers mDNS. But I don't quite get how Designate would recognize that a zone is "missing" on one of the BIND9 servers in the pool. What would be the task that checks or determines that a server of the pool does not have some / all of the zones (anymore)? Is there no form of health check for backends to see if they have all zones? Like a simple count or comparison of the zone list? I'd like to form a readiness check to e.g. block all external queries until all zones exist. There is an open issues about this even: https://gitlab.isc.org/isc-projects/bind9/-/issues/3081 Or is there any way / designate-manage command to trigger this or a re-creating of all zone for a backend or a way to remove and re-add a BIND9 backend to have designate treat it as new and not configured to then do all those "rndc addzone" commands? Regards Christian From gmann at ghanshyammann.com Mon Jun 6 19:56:12 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 06 Jun 2022 14:56:12 -0500 Subject: [all][tc] Canceling this week TC meetings Message-ID: <1813a96d713.b1ad2f74618627.5691800160918441653@ghanshyammann.com> Hello Everyone, We are cancelling this week's TC meeting and will be meeting next on 16 June. -gmann From haleyb.dev at gmail.com Mon Jun 6 21:52:25 2022 From: haleyb.dev at gmail.com (Brian Haley) Date: Mon, 6 Jun 2022 17:52:25 -0400 Subject: [neutron] Bug deputy report for week of May 30th Message-ID: Hi, I was Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * None High bugs --------- * https://bugs.launchpad.net/neutron/+bug/1976357 - [sqlalchemy-20] Missing DB context in subnetpool methods - https://review.opendev.org/c/openstack/neutron/+/844024 * https://bugs.launchpad.net/neutron/+bug/1976360 - [sqlalchemy-20] Missing DB context in networking-sfc flow classifier - Rodolfo assigned to himself as he's working on a number of these Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/1976292 - [OVN] AgentCache.agents can change during an iteration operation - https://review.opendev.org/c/openstack/neutron/+/843933 * https://bugs.launchpad.net/neutron/+bug/1976323 - [functional/fullstack][master] move fips periodic job to CentOS 9-Stream - https://review.opendev.org/c/openstack/neutron/+/843252 * https://bugs.launchpad.net/neutron/+bug/1976345 - The sync_routers interface takes too long - Needs owner and further investigation * https://bugs.launchpad.net/neutron/+bug/1976366 - OVN: Metadata is not provisioned for a network with DHCP turned off - Related to https://bugs.launchpad.net/neutron/+bug/1918914 ? * https://bugs.launchpad.net/neutron/+bug/1976439 - The database ml2_port_binding_levels and ml2_distributed_port_bindings tables have a lot of redundant data - Needs assignee * https://bugs.launchpad.net/neutron/+bug/1976355 - remove eager subquery load for PortBindingLevel - Recent fix https://review.opendev.org/c/openstack/neutron/+/841823 - https://review.opendev.org/c/openstack/neutron/+/844699 (potential) - Other suggestions from Oleg as well Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/1977629 - [ovn]agent's heartbeat time is error - Zhou Heng took ownership Misc bugs --------- * https://bugs.launchpad.net/neutron/+bug/1977485 - Neutron deletes port in use and nova errors out when cleaning the VM XML - Possible Nova issue, posted comment * https://bugs.launchpad.net/neutron/+bug/1977470 - [networking-bgpvpn] zed unit test failures - Closed as invalid Wishlist bugs ------------- * https://bugs.launchpad.net/neutron/+bug/1976270 - [RFE] Improve performance of bulk port create/delete with networking-generic-switch From johnsomor at gmail.com Tue Jun 7 00:04:12 2022 From: johnsomor at gmail.com (Michael Johnson) Date: Mon, 6 Jun 2022 17:04:12 -0700 Subject: [designate] How to avoid NXDOMAIN or stale data during cold start of a (new) machine In-Reply-To: References: <69ab8e54-f419-4cd1-f289-a0b5efb7f723@inovex.de> Message-ID: Hi Christian, Sorry I misunderstood the scenario. There are two ways zones can be resynced: 1. Using the "designate-manage pool update" command. This will force an update/recreate of all of the zones. 2. When a zone is in ERROR or PENDING for too long, the WorkerPeriodicRecovery task in producer will attempt to repair the zone. I don't think there is a periodic task that checks the BIND instances for missing content at the moment. Adding one would have to be done very carefully as it would be easy to get into race conditions when new zones are being created and deployed. Michael On Mon, Jun 6, 2022 at 9:59 AM Christian Rohmann wrote: > > Hey Michael, all, > > sorry for the late reply, just got around to playing with Designate and > BIND9 again ... > > > On 10/05/2022 19:04, Michael Johnson wrote: > > On startup, BIND9 will start sending SOA serial number queries for all > > of the zones it knows about. In the case of Designate, that means > > BIND9 will send out requests to the miniDNS instances to check if the > > serial number in Designate is newer than the one in BIND9. If the > > serial number in Designate is newer, BIND9 will initiate a zone > > transfer from the miniDNS in Designate. > > > > BIND9, by default, will do 20 SOA serial number queries at a time > > (less on older versions of BIND). See the serial-query-rate setting in > > the rate limiter knowledge base article[1]. > > > > The tuning knowledge base article[2] also discusses settings that can > > be adjusted for secondary servers that may also help speed up a cold > > startup. > > > > Off my head, I don't know of a way to tell BIND9 to not answer queries > > via rdnc or such. I usually block network access to a new BIND9 > > instance until the "rdnc status" shows the "soa queries in progress" > > and "xfers running" drop to 0 or a low number. > > Thanks for your thoughts and for distinguishing the sync of existing > domains via AXFR/IXFR zone-transfers, because I was actually talking > about an empty BIND9 in need to actually learn about zones. > Maybe I was not clear about that on my initial post. But in the case of > BIND9 not having a zone configured it requires input from designate > before asking for the latest zone data from mDNS. > Think about a freshly restored or a newly added server to the pool that > requires to have ALL zones added and until then is not doing any IXFR / > AXFR, but wrongfully sending NXDOMAIN. > > For new zones there is > https://github.com/openstack/designate/blob/5d5d83e511acbf5d6f34e9a998ff16d96c5bb162/designate/backend/impl_bind9.py#L76 > which creates the zone (via rndc addzone - > https://docs.openstack.org/designate/latest/admin/backends/bind9.html#bind9-configuration). > and triggers mDNS. > > But I don't quite get how Designate would recognize that a zone is > "missing" on one of the BIND9 servers in the pool. > What would be the task that checks or determines that a server of the > pool does not have some / all of the zones (anymore)? > Is there no form of health check for backends to see if they have all > zones? Like a simple count or comparison of the zone list? > I'd like to form a readiness check to e.g. block all external queries > until all zones exist. There is an open issues about this even: > https://gitlab.isc.org/isc-projects/bind9/-/issues/3081 > > Or is there any way / designate-manage command to trigger this or a > re-creating of all zone for a backend or a way to remove and re-add a > BIND9 backend to have designate > treat it as new and not configured to then do all those "rndc addzone" > commands? > > > Regards > > Christian > From sharath.madhava at gmail.com Tue Jun 7 05:09:27 2022 From: sharath.madhava at gmail.com (Sharath Ck) Date: Tue, 7 Jun 2022 10:39:27 +0530 Subject: [kolla-ansible][kolla] merge_configs module replacing colon to equal Message-ID: Hi, In kolla-ansible, I am using the merge_configs module to merge service configs files. I want to merge api-paste.ini config with my custom changes, however while merging if a line consists of one colon, it is getting replaced with equal sign. merge_configs splits the line by considering colon as delimiter and treats two parts as key and value. Later which will be substituted as key = value. Original */v2.1: oscomputeversion_v2* Merged */v2.1= oscomputeversion_v2 * Below, colon will be 1 and equal will be -1, *def _split_key_value(self, line): colon = line.find(':') equal = line.find('=') if colon < 0 and equal < 0: return self.error_invalid_assignment(line) if colon < 0 or (equal >= 0 and equal < colon): key, value = line[:equal], line[equal + 1:] else: key, value = line[:colon], line[colon + 1:] value = value.strip() if value and value[0] == value[-1] and value.startswith(("\"", "'")): value = value[1:-1] return key.strip(), [value]* I want to keep the config as it is and merge only my custom changes. Please let me know how to achieve this or colon can be escaped somehow. Regards, Sharath Regards, Sharath -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian.rohmann at inovex.de Tue Jun 7 06:54:22 2022 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Tue, 7 Jun 2022 08:54:22 +0200 Subject: [designate] How to avoid NXDOMAIN or stale data during cold start of a (new) machine In-Reply-To: References: <69ab8e54-f419-4cd1-f289-a0b5efb7f723@inovex.de> Message-ID: <21cce5d5-5d21-dc08-4b08-89fdabeb6625@inovex.de> On 07/06/2022 02:04, Michael Johnson wrote: > There are two ways zones can be resynced: > 1. Using the "designate-manage pool update" command. This will force > an update/recreate of all of the zones. > 2. When a zone is in ERROR or PENDING for too long, the > WorkerPeriodicRecovery task in producer will attempt to repair the > zone. Thanks again for these details! I suppose there is no option for a targeted run of this "pool update" command to just hit one backend server? If you don't mind me asking yet another question ... what is the difference between running a regular backend or an agent backend? So https://github.com/openstack/designate/blob/master/designate/backend/impl_bind9.py vs. https://github.com/openstack/designate/blob/master/designate/backend/agent_backend/impl_bind9.py Regards and thanks again, Christian From michal.arbet at ultimum.io Tue Jun 7 07:25:33 2022 From: michal.arbet at ultimum.io (Michal Arbet) Date: Tue, 7 Jun 2022 09:25:33 +0200 Subject: [cinder] Failed to delete volume Message-ID: Hi, I would like to ask if someone saw issue when volume failed to delete, log below : This is happening from time to time. LOG : https://paste.opendev.org/show/b1jRCxuBFnnVzf64i9yV/ Thanks, Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Po???? 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet at ultimum.io *https://ultimum.io * LinkedIn | Twitter | Facebook -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbauza at redhat.com Tue Jun 7 09:13:35 2022 From: sbauza at redhat.com (Sylvain Bauza) Date: Tue, 7 Jun 2022 11:13:35 +0200 Subject: [summit][ops][nova] Ops feedback session Message-ID: Hi all, You probably haven't noticed but we'll run a Forum session at Berlin about Nova pain points and features you'd like to have : https://etherpad.opendev.org/p/nova-berlin-meet-and-greet This will be held tomorrow at 14:50 CEST but your feedback is absolutely welcome even if you can't attend our session. Please take a few minutes to look at the etherpad above and provide your thoughts, that'd be more than welcome. The more we have feedback, the better we could have a productive discussion :-) See you hopefully in-person for whose who are lucky to be present at the Summit. -Sylvain and gibi -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Tue Jun 7 12:13:51 2022 From: smooney at redhat.com (Sean Mooney) Date: Tue, 07 Jun 2022 13:13:51 +0100 Subject: Using infiniband for openstack network communications In-Reply-To: References: Message-ID: On Sun, 2022-06-05 at 15:22 +0200, Dmitriy Rabotyagov wrote: > Yes, why not? As long as your infiniband card support sr-iov there should > be no problems with that. > > You can also check one blog which describes the experience > https://satishdotpatel.github.io/HPC-on-openstack/ well it could work but its not really a tested usecase. security groups for example (which yes i know dont work with normal sriov) more or less assume ethernet. ovs and other backends do assuem ethernet so you cant use an infinaband interface for ovs, ovn or linux bridge. neutrop ports also kind of implictly assume ethernet via the mac adress filed so you cant really use infinaband without ethernet in openstack other then via a direct passthough to a guest in which case you are not using infinaband with neturon netowrks you are using infinaband without integration with neutron netowrking api via flaovr based pci passthoug. that is what https://satishdotpatel.github.io/HPC-on-openstack/ descripbes you will notice that the passthrough_whitelist = { "vendor_id": "10de", "product_id": "1df6" } does not contain a physical_network tag which means this VF is not usable via neutron sriov ports (vnic_type=direct in the case of a vf) they are instead creating a pci alias alias = { "vendor_id":"15b3", "product_id":"101c", "device_type":"type-VF", "name":"mlx5-sriov-ib" } then requesting that in the flavor openstack flavor create --vcpus 8 --ram 16384 --disk 100 --property hw:mem_page_size='large' --property hw:cpu_policy='dedicated' --property "pci_passthrough:alias"="mlx5-sriov-ib" ib.small then when the vm is create i has a neutron network for managment openstack server create --flavor ib.small --image ubuntu_20_04 --nic net-id=private1 ib-vm1 and a passed through infinaband VF. so i i would not cinsider this to be "use the infiniband port for openstack networks" since the infinaband port is not assocated with any neutron network or modeled as a neutron port. as far as i am aware there is no support for using infinaband with a neutron port and vnic_type=direct. so can you use infinaband with openstack yes. can you use infinaband with onpenstack networking no. > > ??, 5 ???. 2022 ?., 15:20 A Monster : > > > Is it possible to use the infiniband port for openstack networks without > > having to use the infiniband port as an ethernet port ? > > From smooney at redhat.com Tue Jun 7 12:25:35 2022 From: smooney at redhat.com (Sean Mooney) Date: Tue, 07 Jun 2022 13:25:35 +0100 Subject: [nova] local LVM volume on compute hosts In-Reply-To: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> References: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> Message-ID: <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> On Mon, 2022-06-06 at 15:11 +0000, Paul Harrison wrote: > I have installed a small test cluster using openstack-ansible (xena) and would like to use local LVM volumes for storing instance volumes - and following https://docs.openstack.org/nova/latest/configuration/config.html and (e.g. https://cloudnull.io/2017/12/nova-lvm-an-iop-love-story/) I have the libvirt section of my /etc/nova/nova.conf as below. > > -8<--- > [libvirt] > inject_partition = -2 > inject_password = False > inject_key = False > virt_type = kvm > live_migration_with_native_tls = true > live_migration_scheme = tls > live_migration_inbound_addr = 172.29.236.5 > hw_disk_discard = ignore > disk_cachemodes = none > images_type = lvm > images_volume_group = nova > volume_clear = zero > -8 > however, the nova service continues to use cinder volumes from my storage host to store instances. Is there something else that needs to be configured to make this work, or is it obsolete? no there is noting else you need to configure but this option is not what you think it is. the images_type option contols what storage will be used for all non cinder storage. i.e. vms that are booted with out usign a boot volume. by default i belive we woudl use qcow or raw files on disk with images_type=lvm we will instead create a lvm volume for the root disk but only if you do not use the boot form volume workflow. there is no way curretnly to prevent usign boot form volume on a specific host and force only local storage directly via config. you can do this indirectly but with sideffect. effectivly if you have a set of hosts that cant or should not have acess to cinder you can create a spereate avaiablity zone. you would then set [cinder]/cross_az_attch=false. e.g. create local-only az and if the cinder backend is configured to be in a differnt az then the cross_az_attach config will prevent the vm form booting in the local-only az. https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that is what they will recive but if they use teh default storage provided by the flaovr then it will be local. > > > From hjensas at redhat.com Tue Jun 7 12:32:54 2022 From: hjensas at redhat.com (Harald Jensas) Date: Tue, 7 Jun 2022 14:32:54 +0200 Subject: [Tripleo] - Provisioning Baremetal Instances in Wallaby Openstack In-Reply-To: References: Message-ID: <86dca991-09df-afd9-7e84-7fc60bb9efbd@redhat.com> On 6/7/22 11:49, Anirudh Gupta wrote: > Hi Harald/Julia/Openstack Team > > I had successfully?installed Ironic Service in Tripleo Train, Ussuri > Release with the help of openstack discuss team members. > > https://lists.openstack.org/pipermail/openstack-discuss/2022-February/027047.html > > > I had custom network configuration for which I had passed Service net > map as below in network-environment file. > > parameter_defaults: > ? ServiceNetMap: > ? ? IronicApiNetwork: oc_provisioning > ? ? IronicNetwork: oc_provisioning > > But with the Wallaby Release, since the deployment format has changed > There is no network-environment file. > > So I tried passing it in a separate?environment.yaml file along with the > templates. > Command to deploy? in wallaby: > > openstack overcloud deploy --templates \ > ? ? --networks-file /home/stack/templates/custom_network_data.yaml \ > ? ? --vip-file ?/home/stack/templates/custom_vip_data.yaml \ > ? ? --baremetal-deployment > ?/home/stack/templates/overcloud-baremetal-deploy.yaml --deployed-server \ > ? ? --network-config \ > ? ? -e /home/stack/templates/environment.yaml \ > ? ? -e > /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml > \ > ? ? -e > /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml > \ > ? ? -e > /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml > \ > ? ? -e /home/stack/templates/ironic-config.yaml \ > ? ? -e > /usr/share/openstack-tripleo-heat-templates/environments/services/ptp.yaml \ > ? ? -e > /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ > ? ? -e > /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ > ? ? -e /home/stack/containers-prepare-parameter.yaml > > Without Service Net Map, deployment was successful but TFTP of image was > an issue since we are going with custom network approach. > > Passing Service Net map details in a seperate file environment.yaml, is > giving the below error: > > 24d-f43f-6f76-000000006ce9 | TASK | Create containers managed by Podman > for /var/lib/tripleo-config/container-startup-config/step_3 > 2022-06-07 11:16:46.806982 | | WARNING | ERROR: Can't run container > ironic_db_sync > stderr: Error: statfs /var/lib/config-data/puppet-generated/ironic_api: > no such file or directory > 2022-06-07 11:16:46.809252 | 525400fd-f24d-f43f-6f76-000000006c79 | > FATAL | Create containers managed by Podman for > /var/lib/tripleo-config/container-startup-config/step_3 | > overcloud-controller-1 | error={"changed": false, "msg": "Failed > containers: ironic_db_sync"} > 2022-06-07 11:16:46.810197 | 525400fd-f24d-f43f-6f76-000000006c79 | > TIMING | tripleo_container_manage : Create containers managed by Podman > for /var/lib/tripleo-config/container-startup-config/step_3 | > overcloud-controller-1 | 0:11:40.574944 | 10.62s > 2022-06-07 11:16:47.683400 | | WARNING | ERROR: Can't run container > ironic_db_sync > stderr: Error: statfs /var/lib/config-data/puppet-generated/ironic_api: > no such file or directory > 2022-06-07 11:16:47.684455 | 525400fd-f24d-f43f-6f76-000000006ce9 | > FATAL | Create containers managed by Podman for > /var/lib/tripleo-config/container-startup-config/step_3 | > overcloud-controller-2 | error={"changed": false, "msg": "Failed > containers: ironic_db_sync"} > > Can you please suggest the place we can pass the service net map details > in Wallaby Release to resolve this issue? > You should be able to pass it in any of the environment files, like you did. Can you try adding the following to the environment file where you override the ServiceNetMap? parameter_merge_strategies: ServiceNetMap: merge -- Harald From hjensas at redhat.com Tue Jun 7 13:31:47 2022 From: hjensas at redhat.com (Harald Jensas) Date: Tue, 7 Jun 2022 15:31:47 +0200 Subject: [Tripleo] - Provisioning Baremetal Instances in Wallaby Openstack In-Reply-To: <86dca991-09df-afd9-7e84-7fc60bb9efbd@redhat.com> References: <86dca991-09df-afd9-7e84-7fc60bb9efbd@redhat.com> Message-ID: <5ee4d9f4-772d-18c5-bc30-7d5f849f63b3@redhat.com> On 6/7/22 14:32, Harald Jensas wrote: > On 6/7/22 11:49, Anirudh Gupta wrote: >> Hi Harald/Julia/Openstack Team >> >> I had successfully?installed Ironic Service in Tripleo Train, Ussuri >> Release with the help of openstack discuss team members. >> >> https://lists.openstack.org/pipermail/openstack-discuss/2022-February/027047.html >> >> >> >> I had custom network configuration for which I had passed Service net >> map as below in network-environment file. >> >> parameter_defaults: >> ?? ServiceNetMap: >> ?? ? IronicApiNetwork: oc_provisioning >> ?? ? IronicNetwork: oc_provisioning >> >> But with the Wallaby Release, since the deployment format has changed >> There is no network-environment file. >> >> So I tried passing it in a separate?environment.yaml file along with >> the templates. >> Command to deploy? in wallaby: >> >> openstack overcloud deploy --templates \ >> ?? ? --networks-file /home/stack/templates/custom_network_data.yaml \ >> ?? ? --vip-file ?/home/stack/templates/custom_vip_data.yaml \ >> ?? ? --baremetal-deployment >> ??/home/stack/templates/overcloud-baremetal-deploy.yaml >> --deployed-server \ >> ?? ? --network-config \ >> ?? ? -e /home/stack/templates/environment.yaml \ >> ?? ? -e >> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml >> \ >> ?? ? -e >> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml >> \ >> ?? ? -e >> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml >> \ >> ?? ? -e /home/stack/templates/ironic-config.yaml \ >> ?? ? -e >> /usr/share/openstack-tripleo-heat-templates/environments/services/ptp.yaml >> \ >> ?? ? -e >> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ >> ?? ? -e >> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ >> ?? ? -e /home/stack/containers-prepare-parameter.yaml >> >> Without Service Net Map, deployment was successful but TFTP of image >> was an issue since we are going with custom network approach. >> >> Passing Service Net map details in a seperate file environment.yaml, >> is giving the below error: >> >> 24d-f43f-6f76-000000006ce9 | TASK | Create containers managed by >> Podman for /var/lib/tripleo-config/container-startup-config/step_3 >> 2022-06-07 11:16:46.806982 | | WARNING | ERROR: Can't run container >> ironic_db_sync >> stderr: Error: statfs >> /var/lib/config-data/puppet-generated/ironic_api: no such file or >> directory >> 2022-06-07 11:16:46.809252 | 525400fd-f24d-f43f-6f76-000000006c79 | >> FATAL | Create containers managed by Podman for >> /var/lib/tripleo-config/container-startup-config/step_3 | >> overcloud-controller-1 | error={"changed": false, "msg": "Failed >> containers: ironic_db_sync"} >> 2022-06-07 11:16:46.810197 | 525400fd-f24d-f43f-6f76-000000006c79 | >> TIMING | tripleo_container_manage : Create containers managed by >> Podman for /var/lib/tripleo-config/container-startup-config/step_3 | >> overcloud-controller-1 | 0:11:40.574944 | 10.62s >> 2022-06-07 11:16:47.683400 | | WARNING | ERROR: Can't run container >> ironic_db_sync >> stderr: Error: statfs >> /var/lib/config-data/puppet-generated/ironic_api: no such file or >> directory >> 2022-06-07 11:16:47.684455 | 525400fd-f24d-f43f-6f76-000000006ce9 | >> FATAL | Create containers managed by Podman for >> /var/lib/tripleo-config/container-startup-config/step_3 | >> overcloud-controller-2 | error={"changed": false, "msg": "Failed >> containers: ironic_db_sync"} >> >> Can you please suggest the place we can pass the service net map >> details in Wallaby Release to resolve this issue? >> > > You should be able to pass it in any of the environment files, like you > did. Can you try adding the following to the environment file where you > override the ServiceNetMap? > > parameter_merge_strategies: > ? ServiceNetMap: merge > Actually, my bad. (Thanks Rabi for reminding me on IRC.) The merge strategies is after a change in master. With Wallaby we still have ServiceNetMapDefaults which is merged with overrides in ServiceNetMap. AFICT from this error ... > 2022-06-07 11:16:47.683400 | | WARNING | ERROR: Can't run container > ironic_db_sync > stderr: Error: statfs > /var/lib/config-data/puppet-generated/ironic_api: no such file or > directory ... it did not execute $THT/common/container-puppet.sh for the ironic_api service correctly. Could it be because you don't use a custom roles file? The default[1] role file does not have the `oc_provisioning` provisioning network. I wonder if some task is skipped because the node is not on the network the service is mapped to. Can you try using a custom roles file with the `oc_provisioning` network defined on the controller role? I think we probably need more information to understand the problem. Would it be possible to share more data? i.e full deploy log, all custom environment files? Thanks! [1]?https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/wallaby/roles_data.yaml#L18 From jdratlif at globalnoc.iu.edu Tue Jun 7 13:51:05 2022 From: jdratlif at globalnoc.iu.edu (jdratlif at globalnoc.iu.edu) Date: Tue, 7 Jun 2022 09:51:05 -0400 Subject: [sdk] specify mac address when creating ports with nova Message-ID: <007701d87a75$a24d8020$e6e88060$@globalnoc.iu.edu> When using connection.compute.create_server(), what is the structure of the networks parameter? I see in the example is a list of objects with a single key in the list being the uuid of the network. I know there are other parameters that could be in this list though. The os-migrate ansible playbooks add fixed_ip to this. fixed_ip is not a Network or a Port class parameter. Port has fixed_ips, but that would be a list. I want to add mac address to this port, but I'm not sure if that's possible or what key I would need to add. Is the structure of the networks object documented somewhere? Thanks --- John Ratliff Systems Automation Engineer GlobalNOC @ Indiana University -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6687 bytes Desc: not available URL: From smooney at redhat.com Tue Jun 7 14:26:22 2022 From: smooney at redhat.com (Sean Mooney) Date: Tue, 07 Jun 2022 15:26:22 +0100 Subject: [sdk] specify mac address when creating ports with nova In-Reply-To: <007701d87a75$a24d8020$e6e88060$@globalnoc.iu.edu> References: <007701d87a75$a24d8020$e6e88060$@globalnoc.iu.edu> Message-ID: <833f128b350476c0ef66339bd9f33f1e7b99524c.camel@redhat.com> On Tue, 2022-06-07 at 09:51 -0400, jdratlif at globalnoc.iu.edu wrote: > When using connection.compute.create_server(), what is the structure of the > networks parameter? > > > > I see in the example is a list of objects with a single key in the list > being the uuid of the network. I know there are other parameters that could > be in this list though. The os-migrate ansible playbooks add fixed_ip to > this. fixed_ip is not a Network or a Port class parameter. Port has > fixed_ips, but that would be a list. I want to add mac address to this port, > but I'm not sure if that's possible or what key I would need to add. this is not something that shoudl be added to the sdk or nova. id you want to create a vm with a set of ports with specific mac or fixed ips you shoudl first precreate the port in neutron with the relevent setting applied and then create teh server with a list of neutron port uuids. nova has legacy proxy apis for create vms with interfaces on spefic neturon networks or subnets but we do not plan to extend them to allow any finer grained contol then is there today https://docs.openstack.org/nova/latest/contributor/project-scope.html#no-more-api-proxies """ No more API Proxies? Nova API current has some APIs that are now (in kilo) mostly just a proxy to other OpenStack services. If it were possible to remove a public API, these are some we might start with. As such, we don?t want to add any more. The first example is the API that is a proxy to the Glance v1 API. As Glance moves to deprecate its v1 API, we need to translate calls from the old v1 API we expose, to Glance?s v2 API. The next API to mention is the networking APIs, in particular the security groups API. Most of these APIs exist from when nova-network existed and the proxies were added during the transition. However, security groups has a much richer Neutron API, and if you use both Nova API and Neutron API, the mismatch can lead to some very unexpected results, in certain cases. Our intention is to avoid adding to the problems we already have in this area. """ the sdk shoudl likel avoid introduceding simiarl proxy apis. os-migrate shoudl simple use the neutron port apis in the sdk to create the ports on the migrate neutron netowrks with the fixed ips and macs that are required and then boot the vms with those ports instead of encodeign this in the sdk. this is they type of logic that shoudl live in the calling application (os-migrate) not in the sdk its self. > > > > Is the structure of the networks object documented somewhere? > > > > Thanks > > > > --- > > John Ratliff > > Systems Automation Engineer > > GlobalNOC @ Indiana University > > > From jdratlif at globalnoc.iu.edu Tue Jun 7 14:50:34 2022 From: jdratlif at globalnoc.iu.edu (jdratlif at globalnoc.iu.edu) Date: Tue, 7 Jun 2022 10:50:34 -0400 Subject: [ansible] [openstack-cloud] how to find port by IP address in ansible Message-ID: <01de01d87a7d$f1491970$d3db4c50$@globalnoc.iu.edu> How can I get a port by it's IP address using the openstack cloud ansible modules? This is what I'm trying, but it never yields any results. I do get results if I use mac_address or device_id until filters, but I need to find the port by it's IP. - name: get port info openstack.cloud.port_info: filters: fixed_ips: - 'ip_address=123.124.125.126' auth: "{{ auth_dict }}" region_name: "{{ region }}" Thanks --- John Ratliff Systems Automation Engineer GlobalNOC @ Indiana University -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6687 bytes Desc: not available URL: From jdratlif at globalnoc.iu.edu Tue Jun 7 15:12:52 2022 From: jdratlif at globalnoc.iu.edu (jdratlif at globalnoc.iu.edu) Date: Tue, 7 Jun 2022 11:12:52 -0400 Subject: [sdk] specify mac address when creating ports with nova In-Reply-To: <833f128b350476c0ef66339bd9f33f1e7b99524c.camel@redhat.com> References: <007701d87a75$a24d8020$e6e88060$@globalnoc.iu.edu> <833f128b350476c0ef66339bd9f33f1e7b99524c.camel@redhat.com> Message-ID: <021f01d87a81$0eb8edc0$2c2ac940$@globalnoc.iu.edu> I'm looking at these docs here. https://docs.openstack.org/openstacksdk/xena/user/proxies/compute.html There is a create_server() function for the compute object on the connection. It says to pass attrs, which is comprised of properties of the Server class. When I go to the server class, all I see for ports is the networks property, which I can't find any description of. Assuming I already used neutron to create the ports, how would I tell the create_server function to use those ports? Is this the proxy function you said I shouldn't use anymore? I'm not clear on that. Thanks --- John Ratliff Systems Automation Engineer GlobalNOC @ Indiana University -----Original Message----- From: Sean Mooney Sent: Tuesday, June 7, 2022 10:26 AM To: jdratlif at globalnoc.iu.edu; openstack-discuss at lists.openstack.org Cc: Ratliff, John Subject: Re: [sdk] specify mac address when creating ports with nova On Tue, 2022-06-07 at 09:51 -0400, jdratlif at globalnoc.iu.edu wrote: > When using connection.compute.create_server(), what is the structure > of the networks parameter? > > > > I see in the example is a list of objects with a single key in the > list being the uuid of the network. I know there are other parameters > that could be in this list though. The os-migrate ansible playbooks > add fixed_ip to this. fixed_ip is not a Network or a Port class > parameter. Port has fixed_ips, but that would be a list. I want to add > mac address to this port, but I'm not sure if that's possible or what key I would need to add. this is not something that shoudl be added to the sdk or nova. id you want to create a vm with a set of ports with specific mac or fixed ips you shoudl first precreate the port in neutron with the relevent setting applied and then create teh server with a list of neutron port uuids. nova has legacy proxy apis for create vms with interfaces on spefic neturon networks or subnets but we do not plan to extend them to allow any finer grained contol then is there today https://docs.openstack.org/nova/latest/contributor/project-scope.html#no-more-api-proxies """ No more API Proxies? Nova API current has some APIs that are now (in kilo) mostly just a proxy to other OpenStack services. If it were possible to remove a public API, these are some we might start with. As such, we don?t want to add any more. The first example is the API that is a proxy to the Glance v1 API. As Glance moves to deprecate its v1 API, we need to translate calls from the old v1 API we expose, to Glance?s v2 API. The next API to mention is the networking APIs, in particular the security groups API. Most of these APIs exist from when nova-network existed and the proxies were added during the transition. However, security groups has a much richer Neutron API, and if you use both Nova API and Neutron API, the mismatch can lead to some very unexpected results, in certain cases. Our intention is to avoid adding to the problems we already have in this area. """ the sdk shoudl likel avoid introduceding simiarl proxy apis. os-migrate shoudl simple use the neutron port apis in the sdk to create the ports on the migrate neutron netowrks with the fixed ips and macs that are required and then boot the vms with those ports instead of encodeign this in the sdk. this is they type of logic that shoudl live in the calling application (os-migrate) not in the sdk its self. > > > > Is the structure of the networks object documented somewhere? > > > > Thanks > > > > --- > > John Ratliff > > Systems Automation Engineer > > GlobalNOC @ Indiana University > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6687 bytes Desc: not available URL: From smooney at redhat.com Tue Jun 7 15:32:00 2022 From: smooney at redhat.com (Sean Mooney) Date: Tue, 07 Jun 2022 16:32:00 +0100 Subject: [sdk] specify mac address when creating ports with nova In-Reply-To: <021f01d87a81$0eb8edc0$2c2ac940$@globalnoc.iu.edu> References: <007701d87a75$a24d8020$e6e88060$@globalnoc.iu.edu> <833f128b350476c0ef66339bd9f33f1e7b99524c.camel@redhat.com> <021f01d87a81$0eb8edc0$2c2ac940$@globalnoc.iu.edu> Message-ID: On Tue, 2022-06-07 at 11:12 -0400, jdratlif at globalnoc.iu.edu wrote: > I'm looking at these docs here. https://docs.openstack.org/openstacksdk/xena/user/proxies/compute.html > > There is a create_server() function for the compute object on the connection. It says to pass attrs, which is comprised of properties of the Server class. When I go to the server class, all I see for ports is the networks property, which I can't find any description of. > > Assuming I already used neutron to create the ports, how would I tell the create_server function to use those ports? Is this the proxy function you said I shouldn't use anymore? I'm not clear on that. > i think you would use the networks object https://docs.openstack.org/openstacksdk/xena/user/resources/compute/v2/server.html#openstack.compute.v2.server.Server.networks based on https://docs.openstack.org/openstacksdk/xena/user/guides/compute.html#create-server and https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server= you woudl change there default example form server = conn.compute.create_server( name=SERVER_NAME, image_id=image.id, flavor_id=flavor.id, networks=[{"uuid": network.id}], key_name=keypair.name) to server = conn.compute.create_server( name=SERVER_NAME, image_id=image.id, flavor_id=flavor.id, networks=[{"port": port.id}], key_name=keypair.name) where port is a https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/network/v2/port.py object that you previously created via https://docs.openstack.org/openstacksdk/xena/user/proxies/network.html#port-operations or https://docs.openstack.org/openstacksdk/xena/user/resources/network/v2/port.html#openstack.network.v2.port.Port depending on if you want to use the service proxies https://docs.openstack.org/openstacksdk/xena/user/index.html#service-proxies or resource interface https://docs.openstack.org/openstacksdk/xena/user/index.html#resource-interface im kind of surprised there isnt an example fo this already. > Thanks > > --- > John Ratliff > Systems Automation Engineer > GlobalNOC @ Indiana University > > -----Original Message----- > From: Sean Mooney > Sent: Tuesday, June 7, 2022 10:26 AM > To: jdratlif at globalnoc.iu.edu; openstack-discuss at lists.openstack.org > Cc: Ratliff, John > Subject: Re: [sdk] specify mac address when creating ports with nova > > On Tue, 2022-06-07 at 09:51 -0400, jdratlif at globalnoc.iu.edu wrote: > > When using connection.compute.create_server(), what is the structure > > of the networks parameter? > > > > > > > > I see in the example is a list of objects with a single key in the > > list being the uuid of the network. I know there are other parameters > > that could be in this list though. The os-migrate ansible playbooks > > add fixed_ip to this. fixed_ip is not a Network or a Port class > > parameter. Port has fixed_ips, but that would be a list. I want to add > > mac address to this port, but I'm not sure if that's possible or what key I would need to add. > > this is not something that shoudl be added to the sdk or nova. > > id you want to create a vm with a set of ports with specific mac or fixed ips you shoudl first precreate the port in neutron with the relevent setting applied and then create teh server with a list of neutron port uuids. > > nova has legacy proxy apis for create vms with interfaces on spefic neturon networks or subnets but we do not plan to extend them to allow any finer grained contol then is there today https://docs.openstack.org/nova/latest/contributor/project-scope.html#no-more-api-proxies > > """ > No more API Proxies? > > Nova API current has some APIs that are now (in kilo) mostly just a proxy to other OpenStack services. If it were possible to remove a public API, these are some we might start with. As such, we don?t want to add any more. > > The first example is the API that is a proxy to the Glance v1 API. As Glance moves to deprecate its v1 API, we need to translate calls from the old v1 API we expose, to Glance?s v2 API. > > The next API to mention is the networking APIs, in particular the security groups API. Most of these APIs exist from when nova-network existed and the proxies were added during the transition. However, security groups has a much richer Neutron API, and if you use both Nova API and Neutron API, the mismatch can lead to some very unexpected results, in certain cases. > > Our intention is to avoid adding to the problems we already have in this area. > """ > > the sdk shoudl likel avoid introduceding simiarl proxy apis. > > os-migrate shoudl simple use the neutron port apis in the sdk to create the ports on the migrate neutron netowrks with the fixed ips and macs that are required and then boot the vms with those ports instead of encodeign this in the sdk. this is they type of logic that shoudl live in the calling application (os-migrate) not in the sdk its self. > > > > > > > > Is the structure of the networks object documented somewhere? > > > > > > > > Thanks > > > > > > > > --- > > > > John Ratliff > > > > Systems Automation Engineer > > > > GlobalNOC @ Indiana University > > > > > > > From johnsomor at gmail.com Tue Jun 7 16:24:19 2022 From: johnsomor at gmail.com (Michael Johnson) Date: Tue, 7 Jun 2022 09:24:19 -0700 Subject: [designate] How to avoid NXDOMAIN or stale data during cold start of a (new) machine In-Reply-To: <21cce5d5-5d21-dc08-4b08-89fdabeb6625@inovex.de> References: <69ab8e54-f419-4cd1-f289-a0b5efb7f723@inovex.de> <21cce5d5-5d21-dc08-4b08-89fdabeb6625@inovex.de> Message-ID: The agent based backends use an agent process to interact with the backend DNS server as opposed to directly talking to them over a management protocol. I wasn't part of the project when the agent based BIND9 backend was added, so I don't have the full context of why there is an agent for BIND9. I will note that all of the agent based backends are in the "untested" or "experimental" category: https://docs.openstack.org/designate/latest/admin/support-matrix.html Since the agent based drivers don't have many contributors now, I would expect to see more of them marked deprecated and removed in a future release. I would highly recommend you use the native backend if possible. Michael On Mon, Jun 6, 2022 at 11:54 PM Christian Rohmann wrote: > > On 07/06/2022 02:04, Michael Johnson wrote: > > There are two ways zones can be resynced: > > 1. Using the "designate-manage pool update" command. This will force > > an update/recreate of all of the zones. > > 2. When a zone is in ERROR or PENDING for too long, the > > WorkerPeriodicRecovery task in producer will attempt to repair the > > zone. > > Thanks again for these details! I suppose there is no option for a > targeted run of this "pool update" command to just hit one backend server? > > If you don't mind me asking yet another question ... what is the > difference between running a regular backend or an agent backend? > So > https://github.com/openstack/designate/blob/master/designate/backend/impl_bind9.py > vs. > https://github.com/openstack/designate/blob/master/designate/backend/agent_backend/impl_bind9.py > > > > Regards and thanks again, > > > Christian > From rlandy at redhat.com Tue Jun 7 17:08:31 2022 From: rlandy at redhat.com (Ronelle Landy) Date: Tue, 7 Jun 2022 13:08:31 -0400 Subject: [tripleo] gate blocker - master/wallaby - "SELinux boolean os_enable_vtpm does not exist." Message-ID: Hello All, We have a consistent failure on multiple jobs running on check/gate/integration lines on master and wallaby changes - which shows up as a deployment error: FATAL | Enable os_enable_vtpm SELinux boolean for vTPM | standalone | error={"changed": false, "msg": "SELinux boolean os_enable_vtpm does not exist."} Details of the failure are in: https://bugs.launchpad.net/tripleo/+bug/1977873 We are investigating a fix/workaround. Please hold rechecks - we will post out again with updates. Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Tue Jun 7 17:27:29 2022 From: senrique at redhat.com (Sofia Enriquez) Date: Tue, 7 Jun 2022 14:27:29 -0300 Subject: [cinder] Failed to delete volume In-Reply-To: References: Message-ID: Hi Michal, I do not think we have such a bug on the Launchpad platform.[1] Could you create a new bug report indicating which backend you are using, which Cinder version you have, and if the error occurs after following some steps? Thanks in advance, Sofia [1] https://bugs.launchpad.net/cinder On Tue, Jun 7, 2022 at 4:28 AM Michal Arbet wrote: > Hi, > > I would like to ask if someone saw issue when volume failed to delete, log > below : > This is happening from time to time. > > LOG : https://paste.opendev.org/show/b1jRCxuBFnnVzf64i9yV/ > > Thanks, > > Michal Arbet > Openstack Engineer > > Ultimum Technologies a.s. > Na Po???? 1047/26, 11000 Praha 1 > Czech Republic > > +420 604 228 897 > michal.arbet at ultimum.io > *https://ultimum.io * > > LinkedIn | Twitter > | Facebook > > -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From gthiemonge at redhat.com Tue Jun 7 17:36:29 2022 From: gthiemonge at redhat.com (Gregory Thiemonge) Date: Tue, 7 Jun 2022 19:36:29 +0200 Subject: [Octavia] Weekly meeting cancelled Message-ID: Hi, This week's upstream meeting is cancelled, because it conflicts with the OpenInfra Summit (and I'm also on PTO), See you next week, Thanks, Gregory -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Tue Jun 7 18:05:40 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 07 Jun 2022 13:05:40 -0500 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) Message-ID: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Hello Everyone, As you might know, we are redesigning the OpenStack default RBAC. The new design target two things: 1. 'new defaults (reader role)' 2. "Scope" concept It is hard to explain the details in email but the below doc is a good place to start understanding this: - https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html We as a community think 1st target (reader role) is a good thing to do and it will definitely be useful in many cases. But we need feedback on the "Scope" concept. To understand what it is and how it can impact your existing use case/deployment, please ref the documentation mentioned in the etherpad[1] (if there is any question about its design/usage we are planning, feel free to reply here or contact us in #openstack-tc IRC channel). * If you are an operator, we really need your feedback if the 'Scope' concept is a useful thing for your deployment/use-case or not. * If you are attending events have operators also attending (for example, project operator feedback (like nova[2]), forum sessions in berlin summit, ops meetup or any local operator event), please communicate about the required feedback. * Due to various reasons, many of us involved in RBAC work are not travelling to Berlin and we have this topic to be discussed in Berlin ops meetup[3] but we require someone knowing RBAC new design moderate this topic. Please reach out to us if you would like to help. Central Etherpad to collect feedback (this can be used to collect from various forums/places): * https://etherpad.opendev.org/p/rbac-operator-feedback [1] https://etherpad.opendev.org/p/rbac-operator-feedback [2] https://etherpad.opendev.org/p/nova-berlin-meet-and-greet [3]https://etherpad.opendev.org/p/ops-meetup-berlin-2022-planning#L74 -gmann From rajatdhasmana at gmail.com Tue Jun 7 19:14:56 2022 From: rajatdhasmana at gmail.com (Rajat Dhasmana) Date: Wed, 8 Jun 2022 00:44:56 +0530 Subject: [cinder][drivers] Register blueprints for your drivers Message-ID: Hello Driver Maintainers, If you are planning to submit your driver in the Zed development cycle, make sure you have a blueprint registered against your driver. There is a doc available[1] describing how to register a blueprint and also information related to its attributes. Also take a look at the previously filed blueprints here[2] for reference. We had a discussion in the last mid cycle (1st June) and the following is a list of drivers that are currently proposed. If your driver is in the list or not, please file a blueprint and mention the driver patches in the work items section of the blueprint that will help us prioritize the review of that driver for the Zed development cycle. 1) DataCore (reintroduced), FC and iSCSI: https://review.opendev.org/c/openstack/cinder/+/836996 2) YADRO Tatlin.UNIFIED, FC and iSCSI: https://review.opendev.org/c/openstack/cinder/+/825492 3) Dell EMC PowerStore, NFS: https://review.opendev.org/c/openstack/cinder/+/797608 4) Pure Storage, NVMe-RoCE: https://review.opendev.org/c/openstack/cinder/+/799871 5) HPE XP Storage, FC and iSCSI: https://review.opendev.org/c/openstack/cinder/+/815582 [1] https://wiki.openstack.org/wiki/Blueprints [2] https://blueprints.launchpad.net/cinder Thanks and regards Rajat Dhasmana -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlandy at redhat.com Tue Jun 7 19:53:33 2022 From: rlandy at redhat.com (Ronelle Landy) Date: Tue, 7 Jun 2022 15:53:33 -0400 Subject: [tripleo] gate blocker - master/wallaby - "SELinux boolean os_enable_vtpm does not exist." In-Reply-To: References: Message-ID: On Tue, Jun 7, 2022 at 1:08 PM Ronelle Landy wrote: > Hello All, > > We have a consistent failure on multiple jobs running on > check/gate/integration lines on master and wallaby changes - which shows up > as a deployment error: > > FATAL | Enable os_enable_vtpm SELinux boolean for vTPM | standalone > | error={"changed": false, "msg": "SELinux boolean os_enable_vtpm does not > exist."} > > Details of the failure are in: > https://bugs.launchpad.net/tripleo/+bug/1977873 > > We are investigating a fix/workaround. > > Please hold rechecks - we will post out again with updates. > > Thank you! > > A fix is in progress - thank you Rafael and Lon: https://bugs.launchpad.net/tripleo/+bug/1977873/comments/7 https://bugs.launchpad.net/tripleo/+bug/1977873/comments/8 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andy at andybotting.com Tue Jun 7 23:19:38 2022 From: andy at andybotting.com (Andy Botting) Date: Wed, 8 Jun 2022 09:19:38 +1000 Subject: [vdi][daas][ops] What are your solutions to VDI/DaaS on OpenStack? In-Reply-To: References: Message-ID: Hi Rados?aw, > First of all, wow, that looks very interesting and in fact very much > what I'm looking for. As I mentioned in the original message, the > things this solution lacks are not something blocking for me. > Regarding the approach to Guacamole, I know that it's preferable to > have guacamole extension (that provides the dynamic inventory) > developed rather than meddle with the internal database but I guess it > is a good start. An even better approach would be something like the Guacozy project (https://guacozy.readthedocs.io) They were able to use the Guacmole JavaScript libraries directly to embed the HTML5 desktop within a React? app. I think this is a much better approach, and I'd love to be able to do something similar in the future. Would make the integration that much nicer. > > Any "quickstart setting up" would be awesome to have at this stage. As > this is a Django app, I think I should be able to figure out the bits > and bolts to get it up and running in some shape but obviously it will > impede wider adoption. Yeah I agree. I'm in the process of documenting it, so I'll aim to get a quickstart guide together. I have a private repo with code to set up a development environment which uses Heat and Ansible - this might be the quickest way to get started. I'm happy to share this with you privately if you like. > On the note of adoption, if I find it usable, I can provide support > for it in Kolla [1] and help grow the project's adoption this way. Kolla could be useful. We're already using containers for this project now, and I have a helm chart for deploying to k8s. https://github.com/NeCTAR-RC/bumblebee-helm Also, an important part is making sure the images are set up correctly with XRDP, etc. Our images are built using Packer, and the config for them can be found at https://github.com/NeCTAR-RC/bumblebee-images > Also, since this is OpenStack-centric, maybe you could consider > migrating to OpenDev at some point to collaborate with interested > parties using a common system? > Just food for thought at the moment. I think it would be more appropriate to start a new project. I think our codebase has too many assumptions about the underlying cloud. We inherited the code from another project too, so it's got twice the cruft. > Writing to let you know I have also found the following related paper: [1] > and reached out to its authors in the hope to enable further > collaboration to happen. > The paper is not open access so I have only obtained it for myself and > am unsure if licensing permits me to share, thus I also asked the > authors to share their copy (that they have copyrights to). > I have obviously let them know of the existence of this thread. ;-) > Let's stay tuned. > > [1] https://link.springer.com/chapter/10.1007/978-3-030-99191-3_12 This looks interesting. A collaboration would be good if there is enough interest in the community. cheers, Andy From dsneddon at redhat.com Wed Jun 8 02:43:21 2022 From: dsneddon at redhat.com (Dan Sneddon) Date: Tue, 7 Jun 2022 19:43:21 -0700 Subject: Using infiniband for openstack network communications In-Reply-To: References: Message-ID: <005599d1-fcb2-bed5-b63b-0512cd1eb012@redhat.com> On 6/5/22 06:15, A Monster wrote: > Is it possible to use the infiniband port for openstack networks without > having to use the infiniband port as an ethernet port ? In theory you could use Infiniband with IPoIB for API/RPC control plane, and possibly for tenant virtual networks using VXLAN. I think you must use Ethernet mode for provider networks or external provider networks, but I could be mistaken. As of Stein it was not possible to use Infiniband for bare metal nodes using Ironic [0] with DHCP and PXE boot. There may have been additional work done since, however I never saw completion of the dependencies to make this work, but it might be possible using config-drive rather than PXE boot and DHCP. I have worked with Mellanox to use some Infiniband-capable NICs (MLX-ConnectX-5) with ML2/OVS, but only in Ethernet mode. These cards can also be used with DPDK using the "mlx5_core" driver. -- Dan Sneddon | Senior Principal Software Engineer dsneddon at redhat.com | redhat.com/cloud dsneddon:irc | @dxs:twitter From juliaashleykreger at gmail.com Wed Jun 8 05:49:18 2022 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Wed, 8 Jun 2022 07:49:18 +0200 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Message-ID: On Tue, Jun 7, 2022 at 8:10 PM Ghanshyam Mann wrote: > Hello Everyone, > > As you might know, we are redesigning the OpenStack default RBAC. The new > design target two things: > > 1. 'new defaults (reader role)' > 2. "Scope" concept > > It is hard to explain the details in email but the below doc is a good > place to start understanding this: > - > https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html > > We as a community think 1st target (reader role) is a good thing to do and > it will definitely be useful > in many cases. > > But we need feedback on the "Scope" concept. To understand what it is and > how it can impact your existing > use case/deployment, please ref the documentation mentioned in the > etherpad[1] (if there is any question > about its design/usage we are planning, feel free to reply here or contact > us in #openstack-tc IRC channel). > > * If you are an operator, we really need your feedback if the 'Scope' > concept is a useful thing for your deployment/use-case > or not. > > * If you are attending events have operators also attending (for example, > project operator feedback (like nova[2]), forum sessions > in berlin summit, ops meetup or any local operator event), please > communicate about the required feedback. > > * Due to various reasons, many of us involved in RBAC work are not > travelling to Berlin and > we have this topic to be discussed in Berlin ops meetup[3] but we > require someone knowing RBAC new design moderate > this topic. Please reach out to us if you would like to help. I previously volunteered to facilitate this at the operators meet up and given others have had to drop out, I discussed it with the ops meetup leaders and will be facilitating a session with the interested operators on Friday. I know from previous discussions I?ve had, there was quite an interest in the system level of scope access to be able to see everything across a system, so I suspect there is tons of value there, but our developer perception is obvious different if we?re questioning it at this point. > > Central Etherpad to collect feedback (this can be used to collect from > various forums/places): > > * https://etherpad.opendev.org/p/rbac-operator-feedback > > > [1] https://etherpad.opendev.org/p/rbac-operator-feedback > [2] https://etherpad.opendev.org/p/nova-berlin-meet-and-greet > [3]https://etherpad.opendev.org/p/ops-meetup-berlin-2022-planning#L74 > > > -gmann > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.harrison at manchester.ac.uk Wed Jun 8 07:26:34 2022 From: paul.harrison at manchester.ac.uk (Paul Harrison) Date: Wed, 8 Jun 2022 07:26:34 +0000 Subject: [nova] local LVM volume on compute hosts In-Reply-To: <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> References: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> Message-ID: <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> > On 2022-06 -07, at 13:25, Sean Mooney wrote: > > no there is noting else you need to configure but this option is not what you think it is. > the images_type option contols what storage will be used for all non cinder storage. > i.e. vms that are booted with out usign a boot volume. > > by default i belive we woudl use qcow or raw files on disk > with images_type=lvm we will instead create a lvm volume for the root disk but only if you do not use the boot form volume workflow. > > there is no way curretnly to prevent usign boot form volume on a specific host and force only local storage directly via config. > > you can do this indirectly but with sideffect. > > effectivly if you have a set of hosts that cant or should not have acess to cinder you can create a spereate avaiablity zone. > you would then set [cinder]/cross_az_attch=false. e.g. create local-only az and if the cinder backend is configured to be in a differnt az then > the cross_az_attach config will prevent the vm form booting in the local-only az. > https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach > > > so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks > but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that > is what they will recive but if they use teh default storage provided by the flaovr then it will be local. thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools has never got enough votes for implementation. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2893 bytes Desc: not available URL: From smooney at redhat.com Wed Jun 8 09:35:01 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 08 Jun 2022 10:35:01 +0100 Subject: [nova] local LVM volume on compute hosts In-Reply-To: <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> References: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> Message-ID: On Wed, 2022-06-08 at 07:26 +0000, Paul Harrison wrote: > > > On 2022-06 -07, at 13:25, Sean Mooney wrote: > > > > no there is noting else you need to configure but this option is not what you think it is. > > the images_type option contols what storage will be used for all non cinder storage. > > i.e. vms that are booted with out usign a boot volume. > > > > by default i belive we woudl use qcow or raw files on disk > > with images_type=lvm we will instead create a lvm volume for the root disk but only if you do not use the boot form volume workflow. > > > > there is no way curretnly to prevent usign boot form volume on a specific host and force only local storage directly via config. > > > > you can do this indirectly but with sideffect. > > > > effectivly if you have a set of hosts that cant or should not have acess to cinder you can create a spereate avaiablity zone. > > you would then set [cinder]/cross_az_attch=false. e.g. create local-only az and if the cinder backend is configured to be in a differnt az then > > the cross_az_attach config will prevent the vm form booting in the local-only az. > > https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach > > > > > > so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks > > but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that > > is what they will recive but if they use teh default storage provided by the flaovr then it will be local. > > thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools has never got enough votes for implementation. > thats not what your looking for really, libvirt has a way to create pools of storage locally but nova has its own way to do it itslef. so we never used libvirts way of managing storage pools becasuse in the end it did not really provide any advantage over what we alreay had. the libvirt driver can use local lvm storage or qcow/raw files but you cannot force people to use local storage without breaking interoperablity. if they request to use cinder storage that request will be fulfilled if posible or the vm will fail to boot. by default nova always uses local stroage unless you config the image backedn to rbd/ceph or the end user request boot form volume its extra work for user to boot form volume so normally they do not do this. if your users are explcitly going out of there way to not use local storage and instead are using the boot form volume flow there might be a reason for that worflow wise. nova will never use or create a cinder volume if the user does not ask for one. if you really want to prevent them more directly and interoperablity is not a concern you can alter novas rbac policy. https://github.com/openstack/nova/blob/master/nova/policies/servers.py#L254-L264= policy.DocumentedRuleDefault( name=SERVERS % 'create:attach_volume', check_str=base.PROJECT_MEMBER, description="Create a server with the requested volume attached to it", operations=[ { 'method': 'POST', 'path': '/servers' } ], scope_types=['project']), unfortunetly i belive that policy applies to both boot form cinder root volume and boot form local storage with a secondary data volume. your other option to acive your goal is to write custom midelware like this https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f635d/nova/api/openstack/requestlog.py and then enable it via api-paste.ini so you declare the middelware with the fully qualifed python calsse name https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f635d/etc/nova/api-paste.ini#L45-L46= https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f635d/etc/nova/api-paste.ini#L62-L63= then you add it to the pipeline https://github.com/openstack/nova/blob/b5029890c1c5b1b5153c9ca2fc9a8ea2437f635d/etc/nova/api-paste.ini#L77-L79= so if you want to block boot form volume but not block boot form local disk with an attached data volume you can do that with middelware. but in general i think it would be good to talk to your uses and understand why they are going through the extra effort to create boot form volume guests instead of just use the local storage which as i said above is the default for server create. > From smooney at redhat.com Wed Jun 8 09:39:46 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 08 Jun 2022 10:39:46 +0100 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Message-ID: On Wed, 2022-06-08 at 07:49 +0200, Julia Kreger wrote: > On Tue, Jun 7, 2022 at 8:10 PM Ghanshyam Mann > wrote: > > > Hello Everyone, > > > > As you might know, we are redesigning the OpenStack default RBAC. The new > > design target two things: > > > > 1. 'new defaults (reader role)' > > 2. "Scope" concept > > > > It is hard to explain the details in email but the below doc is a good > > place to start understanding this: > > - > > https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html > > > > We as a community think 1st target (reader role) is a good thing to do and > > it will definitely be useful > > in many cases. > > > > But we need feedback on the "Scope" concept. To understand what it is and > > how it can impact your existing > > use case/deployment, please ref the documentation mentioned in the > > etherpad[1] (if there is any question > > about its design/usage we are planning, feel free to reply here or contact > > us in #openstack-tc IRC channel). > > > > * If you are an operator, we really need your feedback if the 'Scope' > > concept is a useful thing for your deployment/use-case > > or not. > > > > * If you are attending events have operators also attending (for example, > > project operator feedback (like nova[2]), forum sessions > > in berlin summit, ops meetup or any local operator event), please > > communicate about the required feedback. > > > > * Due to various reasons, many of us involved in RBAC work are not > > travelling to Berlin and > > we have this topic to be discussed in Berlin ops meetup[3] but we > > require someone knowing RBAC new design moderate > > this topic. Please reach out to us if you would like to help. > > > I previously volunteered to facilitate this at the operators meet up and > given others have had to drop out, I discussed it with the ops meetup > leaders and will be facilitating a session with the interested operators on > Friday. > > I know from previous discussions I?ve had, there was quite an interest in > the system level of scope access to be able to see everything across a > system, so I suspect there is tons of value there, but our developer > perception is obvious different if we?re questioning it at this point. the system level of scope does not allow you to see everything across the system it only allows you to see the non project related resouces so you can see the flavors and host aggreates but not the instances as instances are project scoped. and project scoped resouces like ports, instances, images and volumes cannot be accessed with a system scope token if you enabel scope enforcement. that is one of the things we want to get clarity on form operators. is the disticntion between system level resouces and project level resouces useful. > > > > > > Central Etherpad to collect feedback (this can be used to collect from > > various forums/places): > > > > * https://etherpad.opendev.org/p/rbac-operator-feedback > > > > > > [1] https://etherpad.opendev.org/p/rbac-operator-feedback > > [2] https://etherpad.opendev.org/p/nova-berlin-meet-and-greet > > [3]https://etherpad.opendev.org/p/ops-meetup-berlin-2022-planning#L74 > > > > > > -gmann > > > > From senrique at redhat.com Wed Jun 8 13:22:54 2022 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 8 Jun 2022 10:22:54 -0300 Subject: [cinder] Bug deputy report for week of 06-08-2022 Message-ID: This is a bug report from 06-01-2022 to 06-08-2022. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Low - https://bugs.launchpad.net/cinder/+bug/1976499 "[IBM Storwize] lsfcportsetmember is being called in the wrong SVC code level." Fix proposed to master. Wishlist - https://bugs.launchpad.net/cinder/+bug/1977868 "When create volume from image, image is fetched twice." Unassigned. Cheers, -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From dms at danplanet.com Wed Jun 8 13:53:24 2022 From: dms at danplanet.com (Dan Smith) Date: Wed, 08 Jun 2022 06:53:24 -0700 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: (Sean Mooney's message of "Wed, 08 Jun 2022 10:39:46 +0100") References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Message-ID: > the system level of scope does not allow you to see everything across the system > it only allows you to see the non project related resouces > > so you can see the flavors and host aggreates but not the instances as instances are project scoped. > and project scoped resouces like ports, instances, images and volumes cannot be accessed with a system scope > token if you enabel scope enforcement. > > that is one of the things we want to get clarity on form operators. > is the disticntion between system level resouces and project level resouces useful. Yep, exactly this. Given the amount of breakage it brings for things like Heat and Tacker, as well as the potential workflow annoyance for human admins, I really want to measure whether any operators see a benefit here. The persona roles, things like a standardized service role, and getting out of this current situation of having two sets of defaults are priorities for me. --Dan From juliaashleykreger at gmail.com Wed Jun 8 14:19:43 2022 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Wed, 8 Jun 2022 07:19:43 -0700 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Message-ID: Is that Nova's interpretation, specifically the delineation that non-project owned should only be viewable by system, or was system scope changed at some point? I interpreted it differently, but haven't circled back recently. I guess interpretation and evolution in specific pockets after initial implementation work started ultimately resulted in different perceptions. I'll make sure operators are aware there may be significant nuances and perception differences since they are using their own etherpads and their own communications flow. i.e. we're unlikely to see many find/use our own etherpads as they have their own. And yes, this is a problem, but it is a higher level community/communications feedback issue under active discussion. Granted, I get that the system scope ideas were breaking for some projects in specific use patterns since not everything would be the same nor possible (which is actually a good thing, context of use and all), but it was in theory perfect for a lot of the external audit tooling use cases which arise in so many different ways. Anyway, off to the next $thing with my scattered brain. On Wed, Jun 8, 2022 at 6:53 AM Dan Smith wrote: > > > the system level of scope does not allow you to see everything across the system > > it only allows you to see the non project related resouces > > > > so you can see the flavors and host aggreates but not the instances as instances are project scoped. > > and project scoped resouces like ports, instances, images and volumes cannot be accessed with a system scope > > token if you enabel scope enforcement. > > > > that is one of the things we want to get clarity on form operators. > > is the disticntion between system level resouces and project level resouces useful. > > Yep, exactly this. Given the amount of breakage it brings for things > like Heat and Tacker, as well as the potential workflow annoyance for > human admins, I really want to measure whether any operators see a > benefit here. The persona roles, things like a standardized service > role, and getting out of this current situation of having two sets of > defaults are priorities for me. > > --Dan From michal.arbet at ultimum.io Tue Jun 7 07:23:56 2022 From: michal.arbet at ultimum.io (Michal Arbet) Date: Tue, 7 Jun 2022 09:23:56 +0200 Subject: [cinder] Failed to delete volume Message-ID: Hi, I would like to ask if someone saw issue when volume failed to delete, log below : This is happening from time to time. 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server [req-80b56352-81b5-47b6-841e-94fface5974e fce19e20993647db8ae39d67094a4294 c8d8a385912c4731832bf1f2cf4727fb - - -] Exception during message handling: EOFError 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 359, in _recv_loop 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server self._read_trampoline() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 331, in _read_trampoline 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server timeout_exc=socket_timeout('timed out')) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 210, in _trampoline 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server mark_as_closed=self._mark_as_closed) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/hubs/__init__.py", line 159, in trampoline 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return hub.switch() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 298, in switch 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return self.greenlet.switch() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server eventlet.hubs.IOClosed: [Errno 107] Operation on closed file 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 160, in wrapper 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 160, in wrapper 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "", line 2, in delete_volume 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/coordination.py", line 144, in _synchronized 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server with lock(blocking): 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/tooz/locking.py", line 29, in __enter__ 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return self.lock.__enter__(*self.args, **self.kwargs) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/tooz/locking.py", line 49, in __enter__ 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server acquired = self.acquire(*args, **kwargs) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/tooz/drivers/redis.py", line 58, in wrapper 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return func(*args, **kwargs) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/tooz/drivers/redis.py", line 110, in acquire 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server blocking=blocking, blocking_timeout=timeout) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/lock.py", line 182, in acquire 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server if self.do_acquire(token): 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/lock.py", line 197, in do_acquire 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server if self.redis.set(self.name, token, nx=True, px=timeout): 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/client.py", line 1519, in set 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return self.execute_command('SET', *pieces) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/client.py", line 839, in execute_command 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return self.parse_response(conn, command_name, **options) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/client.py", line 853, in parse_response 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server response = connection.read_response() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/sentinel.py", line 55, in read_response 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return super(SentinelManagedConnection, self).read_response() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/connection.py", line 700, in read_response 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server response = self._parser.read_response() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/connection.py", line 310, in read_response 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server response = self._buffer.readline() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/connection.py", line 242, in readline 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server self._read_from_socket() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/connection.py", line 184, in _read_from_socket 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server data = recv(self._sock, socket_read_size) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/redis/_compat.py", line 71, in recv 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return sock.recv(*args, **kwargs) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 365, in recv 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server return self._recv_loop(self.fd.recv, b'', bufsize, flags) 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 362, in _recv_loop 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server raise EOFError() 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server EOFError 2022-05-23 10:15:24.414 833 ERROR oslo_messaging.rpc.server Thanks, Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Po???? 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet at ultimum.io *https://ultimum.io * LinkedIn | Twitter | Facebook -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andrew.Norrie at cgg.com Mon Jun 6 14:13:22 2022 From: Andrew.Norrie at cgg.com (Norrie, Andrew) Date: Mon, 6 Jun 2022 14:13:22 +0000 Subject: [kolla-ansible][xena] Cell database HA/redundancy? In-Reply-To: References: Message-ID: Hi Mark, Thanks for the reply. I'll look into the HAProxy "custom config" as suggested and using different ports for the cell databases. Maybe that can be worked in. We are expanding our OpenStack/kolla-ansible personnel/knowledge so will look into the possibility for us to contribute as reviewers/testers as a start. Best regards .... Andrew From: Mark Goddard Sent: Monday, June 6, 2022 2:22 AM To: Norrie, Andrew Cc: openstack-discuss at lists.openstack.org Subject: Re: [kolla-ansible][xena] Cell database HA/redundancy? On Fri, 3 Jun 2022 at 14:10, Norrie, Andrew > wrote: Hi, We are currently planning some large OpenStack deployments utilizing kolla-ansible and I'm curious what folks are doing with the cell database HA/redundancy. With kolla-ansible (xena) it appears that a loadbalancer setup is only allowed for the default database shard (shard 0)... reference: https://docs.openstack.org/kolla-ansible/latest/reference/databases/mariadb-guide.html If we are setting up separate cell database shards with Galera, I'm wondering if there is a convenient work-around or configuration for implementation of HA for these cell databases. In the inventory group_vars directory you can specify the group variables (for each cell database) as: nova_cell_database_address: nova_cell_database_group: but these aren't virtual IP hosts/addresses (non default db shard). This works perfectly fine with a single server cell database but not Galera. If that db server goes down the cell instance information is lost. Hi Andrew, You are correct - only the first shard is load balanced. The sharding feature was actually just the first patch in a series intended to support proxysql. I believe this would add the functionality you are looking for. In fact, the proposed test case for proxysql in our CI is a multi-cell deployment. Here is the patch series: https://review.opendev.org/q/topic:proxysql Unfortunately it has been stuck for a while, largely due to core reviewer bandwidth. The author is Michael Arbet, who is often on IRC as kevko. I'd suggest registering your interest in these patches via gerrit, and at one of the weekly kolla IRC meetings (this week is cancelled due to the summit). If you have time available, then testing and providing reviews could help to get the patches moving. In the meantime, you can configure HAProxy to load balance additional services, by placing an HAProxy config snippet in /etc/kolla/config/haproxy/services.d/*.cfg. Regards, Mark Many thanks ... Andrew ________________________________ "This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of the email by others is strictly prohibited. If you are not the intended recipient, you must not review, disclose, copy, distribute or use this e-mail; please delete it from your system and notify the sender immediately." ________________________________ "This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of the email by others is strictly prohibited. If you are not the intended recipient, you must not review, disclose, copy, distribute or use this e-mail; please delete it from your system and notify the sender immediately." -------------- next part -------------- An HTML attachment was scrubbed... URL: From moshele at nvidia.com Mon Jun 6 20:48:56 2022 From: moshele at nvidia.com (Moshe Levi) Date: Mon, 6 Jun 2022 20:48:56 +0000 Subject: Using infiniband for openstack network communications In-Reply-To: References: Message-ID: If you are using Mellanox (Nvidia) NIC with inbox driver the default mode is ipoib Enhanced mode [1] Which support for acceleration for the IPoIB [1] - https://www.spinics.net/lists/linux-rdma/msg46802.html From: Dmitriy Rabotyagov Sent: Monday, June 6, 2022 4:52 PM Cc: openstack-discuss Subject: Re: Using infiniband for openstack network communications You don't often get email from noonedeadpunk at gmail.com. Learn why this is important External email: Use caution opening links or attachments Ah, ok. Well, you can use infiniband for control plane only with IPoIB. There's no RDMA support if that's what you're asking. ??, 6 ???. 2022 ?., 15:36 A Monster >: What I wanted to actually do is not what is done in this link https://satishdotpatel.github.io/HPC-on-openstack/ , because here vm's are given the possibility to use the infiniband port through Virtual functions, but in my case, I want to use infiniband for openstack management, compute and storage networks, to get better performance than when using ethernet and rg45 cables. so I'm wondering if it's feasible and whether it's a good thing or not. Thank you. On Mon, 6 Jun 2022 at 14:15, A Monster > wrote: Thank you for your reply, I'll check the link you've sent to me immediately. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lokendrarathour at gmail.com Tue Jun 7 07:24:20 2022 From: lokendrarathour at gmail.com (Lokendra Rathour) Date: Tue, 7 Jun 2022 12:54:20 +0530 Subject: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version In-Reply-To: References: Message-ID: Hi Swogat, Yes after correcting the j2 template, as suggested by you, it started working. Thanks a lot once again. Now the Overcloud is deployed. Further, as a next requirement, we were also trying to add an *additional Node as a bare-metal Instance* on the overcloud. As it is the composable network approach, we add an entry in parameters default for ServiceNetMap, as like below in environments/network-environments.yaml parameter_defaults: # This section is where deployment-specific configuration is done ServiceNetMap: IronicApiNetwork: oc_provisioning IronicNetwork: oc_provisioning this is now passed as in the environment.yaml as parameter_defaults, but somehow we are not able to deploy the setup. it gives the error for containers. *2022-06-06 19:08:37.560800 | 525400fd-f24d-9703-b0e7-0000000084b2 | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | overcloud-controller-0 | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"}* and if we remove this setting as stated above, the deployment is successful, but at the time of node provisioning it is not able to reach the TFTP server to pull files, mainly because of the composable network (oc_provisioning), we faced this issue in Train, so we did the above config changes and it working. But as it looks something more is needed in Wallaby for Baremetal provisioning as an instance. Please share your learnings with the reference to it. Thanks once again for your all-time support. reference link: Bare Metal Instances in Overcloud ? TripleO 3.0.0 documentation (openstack.org) -Lokendra On Wed, Jun 1, 2022 at 12:19 AM Lokendra Rathour wrote: > Ok Swogat, > Thanks once again. > i will try this approach once and will let you know. > > > On Wed, 1 Jun 2022, 00:04 Swogat Pradhan, > wrote: > >> Hi Lokendra, >> Like i said, >> > NOTE: when passing --network-config parameter in node provision step, >> it creates a directory in /etc/os-net-config and in it creates a file >> > config.yaml, do check the indentation of that file. (in my case the >> indentation was wrong when i was using bonding everytime, so i had to >> > manually change the script and run a while loop and then my node >> provision step was successful) >> this parameter --network-config creates a config.yaml(network config) >> file in /etc/os-net-config directory in the overcloud nodes and then >> ansible tries to apply the network config from the generated config file. >> And in wallaby version if you are using bonding then the syntax in the >> generated conf is wrong (atleast was in my case) and then the ansible tries >> to apply the network config (with wrong syntax) so your overcloud nodes >> become unavailable. >> >> Please run node provision separately, and keep on monitoring "metalsmith >> list" command. Once IP is assigned to the overcloud nodes, ssh to the nodes >> and assign a password to heat-admin user so that even if the network >> becomes unavailable you still will be able to access the nodes via console >> access, then you can visit /etc/os-net-config directory and verify the >> config.yaml file. >> >> With regards, >> Swogat pradhan. >> >> On Tue, May 31, 2022 at 11:56 PM Lokendra Rathour < >> lokendrarathour at gmail.com> wrote: >> >>> Hi Swogat, >>> Thanks once again for your input it really helped much. >>> >>> instead of running mentioned those three provisioning steps i used >>> alternate method and passed directly in deploy command. now my current >>> deploy command is: >>> >>> openstack overcloud deploy --templates \ >>> --networks-file /home/stack/templates/custom_network_data.yaml \ >>> --vip-file /home/stack/templates/custom_vip_data.yaml \ >>> --baremetal-deployment >>> /home/stack/templates/overcloud-baremetal-deploy.yaml \ >>> --network-config \ >>> -e /home/stack/templates/environment.yaml \ >>> -e >>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml >>> \ >>> -e >>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml >>> \ >>> -e >>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml >>> \ >>> -e /home/stack/templates/ironic-config.yaml \ >>> -e >>> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ >>> -e >>> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ >>> -e /home/stack/containers-prepare-parameter.yaml >>> >>> The files as suggested by you are well created. >>> >>> But once we run the deployment I get this error for all nodes: >>> >>> 0:00:16.064781 | 1.16s >>> 2022-05-31 19:13:00.276954 | 525400ef-b928-9ded-fecc-000000000094 | >>> TASK | Run tripleo_os_net_config_module with network_config >>> 2022-05-31 19:40:30.061582 | 525400ef-b928-9ded-fecc-000000000094 | >>> FATAL | Run tripleo_os_net_config_module with network_config | >>> overcloud-controller-1 | error={"msg": "Data could not be sent to remote >>> host \"30.30.30.117\". Make sure this host can be reached over ssh: ssh: >>> connect to host 30.30.30.117 port 22: No route to host\r\n"} >>> 2022- >>> >>> >>> Baremetal node list are showing in as active. >>> >>> (undercloud) [stack at undercloud ~]$ openstack baremetal node list >>> /usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: >>> DeprecationWarning: The _yaml extension module is now located at yaml._yaml >>> and its location is subject to change. To use the LibYAML-based parser and >>> emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper >>> as Dumper`. >>> DeprecationWarning >>> >>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>> | UUID | Name | Instance UUID >>> | Power State | Provisioning State | Maintenance | >>> >>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>> | 1a4d873c-f9f7-4504-a3af-92c11f954171 | node-a | >>> 901453a1-183f-4de8-aaab-0f38be2be455 | power on | active | >>> False | >>> | d18610fc-9532-410c-918e-8efc326c89f8 | node-b | >>> d059b94a-8357-4f8e-a0d8-15a24b0c1afe | power on | active | >>> False | >>> | b69f2d5a-5b18-4453-8843-15c6af79aca0 | node-c | >>> f196ef3a-7950-47b9-a5ae-751f06b18f75 | power on | active | >>> False | >>> | 8a38c584-f812-4ebc-a0b1-4299f0917637 | node-d | >>> 1636517c-2ab2-43d7-8205-9f02c5290207 | power on | active | >>> False | >>> >>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>> >>> Some config is missing it seems, please check once and advise. >>> >>> >>> On Tue, May 31, 2022 at 11:40 AM Swogat Pradhan < >>> swogatpradhan22 at gmail.com> wrote: >>> >>>> Hi Lokendra, >>>> You need to generate another file also in the following step: openstack >>>> overcloud node provision --stack overcloud --overcloud-ssh-key >>>> /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml also you >>>> need to pass another parameter --network-config. >>>> example: >>>> openstack overcloud node provision --stack overcloud >>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa *--network-config* *--output >>>> overcloud-baremetal-deployed.yaml* overcloud-baremetal-deploy.yaml >>>> >>>> And then all these output files will be passed on to the openstack >>>> overcloud deploy command. >>>> NOTE: when passing --network-config parameter in node provision step, >>>> it creates a directory in /etc/os-net-config and in it creates a file >>>> config.yaml, do check the indentation of that file. (in my case the >>>> indentation was wrong when i was using bondind everytime, so i had to >>>> manually change the script and run a while loop and then my node provision >>>> step was successful) >>>> >>>> On Tue, May 31, 2022 at 8:59 AM Lokendra Rathour < >>>> lokendrarathour at gmail.com> wrote: >>>> >>>>> Hi Swogat, >>>>> I tried generating the scripts as used by you in your deployments >>>>> using the >>>>> >>>>> >>>>> #openstack overcloud network provision --stack overcloud --output >>>>> networks-deployed-environment.yaml custom_network_data.yaml >>>>> # openstack overcloud network vip provision --stack overcloud >>>>> --output vip-deployed-environment.yaml custom_vip_data.yaml >>>>> # openstack overcloud node provision --stack overcloud >>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa >>>>> overcloud-baremetal-deploy.yaml >>>>> >>>>> and used the first two in the final deployment script, but it gives >>>>> the error: >>>>> >>>>> heatclient.exc.HTTPInternalServerError: ERROR: Internal Error >>>>> 2022-05-30 14:14:39.772 479668 ERROR >>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent >>>>> call last):\n', ' File "/usr/lib/python3.6/ted_stack\n >>>>> nested_stack.validate()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in >>>>> wrapper\n result = f(*args, ine 969, in validate\n result = >>>>> res.validate()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/openstack/neutron/port.py", >>>>> line 454site-packages/heat/engine/resources/openstack/neutron/neutron.py", >>>>> line 43, in validate\n res = super(NeutronResource, self).validate()\n', >>>>> ' File "/un return self.validate_template()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 1882, in >>>>> validate_template\n self.t.rpy", line 200, in >>>>> _validate_service_availability\n raise ex\n', >>>>> 'heat.common.exception.ResourceTypeUnavailable: HEAT-E99001 Service neutron >>>>> is not avaieutron network endpoint is not in service catalog.\n', '\nDuring >>>>> handling of the above exception, another exception occurred:\n\n', >>>>> 'Traceback (most recens/stack_resource.py", line 75, in >>>>> validate_nested_stack\n nested_stack.validate()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py"thon3.6/site-packages/heat/engine/stack.py", >>>>> line 969, in validate\n result = res.validate()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/ateResource, >>>>> self).validate()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", >>>>> line 65, in validate\n self.validources/stack_resource.py", line 81, in >>>>> validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', >>>>> 'heat.common.exception.StackValidationFaileeploy/overcloud/tripleo-heat-templates/deployed-server/deployed-server.yaml>: >>>>> HEAT-E99001 Service neutron is not available for resource type >>>>> OS::TripleO::vice catalog.\n', '\nDuring handling of the above exception, >>>>> another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' >>>>> File "/usr/lib/pline 320, in validate_nested_stack\n >>>>> nested_stack.validate()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in >>>>> wrappe/heat/engine/stack.py", line 969, in validate\n result = >>>>> res.validate()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/template_relidate()\n', >>>>> ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", >>>>> line 65, in validate\n self.validate_nested_stack()\n'.py", line 81, in >>>>> validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', >>>>> 'heat.common.exception.StackValidationFailed: >>>>> ResourceTypeUnavaimplates/puppet/compute-role.yaml>.resources.NovaCompute>>>> reason: neutron network endpoint is not in service catalog.\n', '\nDuring >>>>> handling of the above exception, >>>>> another/lib/python3.6/site-packages/heat/common/context.py", line 416, in >>>>> wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File >>>>> "/usr/lib/python3.6/sirce_name, template_id)\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 756, in >>>>> _parse_template_and_validate_stack\n stack.v line 160, in wrapper\n >>>>> result = f(*args, **kwargs)\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 969, in >>>>> validate\n resesources/stack_resource.py", line 65, in validate\n >>>>> self.validate_nested_stack()\n', ' File >>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/oph=[self.stack.t.RESOURCES, >>>>> path])\n', 'heat.common.exception.StackValidationFailed: >>>>> ResourceTypeUnavailable: >>>>> resources.Compute.resources.0.resources.NovaCompute: >>>>> HEAT-E9900::ControlPlanePort, reason: neutron network endpoint is not in >>>>> service catalog.\n']. >>>>> >>>>> Request you to check once, please. >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, May 30, 2022 at 11:06 AM Lokendra Rathour < >>>>> lokendrarathour at gmail.com> wrote: >>>>> >>>>>> Hi Swogat, >>>>>> Thanks once again. >>>>>> >>>>>> with the files as shown below I am running the overcloud deploy for >>>>>> wallaby using this command: >>>>>> >>>>>> (undercloud) [stack at undercloud ~]$ cat deploy_overcloud_working_1.sh >>>>>> openstack overcloud deploy --templates \ >>>>>> -n /home/stack/templates/network_data.yaml \ >>>>>> -r /home/stack/templates/roles_data.yaml \ >>>>>> -e /home/stack/templates/environment.yaml \ >>>>>> -e /home/stack/templates/environments/network-isolation.yaml \ >>>>>> -e /home/stack/templates/environments/network-environment.yaml \ >>>>>> -e >>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml >>>>>> \ >>>>>> -e >>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml >>>>>> \ >>>>>> -e >>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml >>>>>> \ >>>>>> -e /home/stack/templates/ironic-config.yaml \ >>>>>> -e >>>>>> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ >>>>>> -e >>>>>> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ >>>>>> -e /home/stack/containers-prepare-parameter.yaml >>>>>> (undercloud) [stack at undercloud ~]$ >>>>>> >>>>>> >>>>>> This deployment is on ipv6 using triple0 wallaby, templates, as >>>>>> mentioned below, are generated using rendering steps and the >>>>>> network_data.yaml the roles_data.yaml >>>>>> Steps used to render the templates: >>>>>> cd /usr/share/openstack-tripleo-heat-templates/ >>>>>> ./tools/process-templates.py -o >>>>>> ~/openstack-tripleo-heat-templates-rendered_at_wallaby -n >>>>>> /home/stack/templates/network_data.yaml -r >>>>>> /home/stack/templates/roles_data.yaml >>>>>> >>>>>> *Now if i try adding the related to VIP port I do get the error as:* >>>>>> >>>>>> 2022-05-30 10:37:12.792 979387 WARNING >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template >>>>>> to file: >>>>>> /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/controller-role.yaml >>>>>> 2022-05-30 10:37:12.792 979387 WARNING >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template >>>>>> to file: >>>>>> /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/compute-role.yaml >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured >>>>>> while running the command: ValueError: The environment is not a valid YAML >>>>>> mapping data type. >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent >>>>>> call last): >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, >>>>>> self).run(parsed_args) >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in >>>>>> run >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, >>>>>> self).run(parsed_args) >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = >>>>>> self.take_action(parsed_args) or 0 >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>> line 1189, in take_action >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud stack, parsed_args, >>>>>> new_tht_root, user_tht_root) >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>> line 227, in create_env_files >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud created_env_files, >>>>>> parsed_args, new_tht_root, user_tht_root) >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>> line 204, in build_image_params >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud cleanup=(not >>>>>> parsed_args.no_cleanup)) >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1929, in >>>>>> process_multiple_environments >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud env_path=env_path, >>>>>> include_env_in_files=include_env_in_files) >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", >>>>>> line 326, in process_environment_and_files >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud env = >>>>>> environment_format.parse(raw_env) >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>> "/usr/lib/python3.6/site-packages/heatclient/common/environment_format.py", >>>>>> line 50, in parse >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud raise >>>>>> ValueError(_('The environment is not a valid ' >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: The >>>>>> environment is not a valid YAML mapping data type. >>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud >>>>>> 2022-05-30 10:37:14.457 979387 ERROR openstack [-] The environment is >>>>>> not a valid YAML mapping data type. >>>>>> 2022-05-30 10:37:14.457 979387 INFO osc_lib.shell [-] END return >>>>>> value: 1 >>>>>> (undercloud) [stack at undercloud ~]$ >>>>>> >>>>>> This is more of a syntax error where it is not able to understand the >>>>>> passed VIP data file: >>>>>> >>>>>> undercloud) [stack at undercloud ~]$ cat >>>>>> /home/stack/templates/vip-data-default-network-isolation.yaml >>>>>> - >>>>>> dns_name: overcloud >>>>>> network: internal_api >>>>>> - >>>>>> dns_name: overcloud >>>>>> network: external >>>>>> - >>>>>> dns_name: overcloud >>>>>> network: ctlplane >>>>>> - >>>>>> dns_name: overcloud >>>>>> network: oc_provisioning >>>>>> - >>>>>> dns_name: overcloud >>>>>> network: j3mgmt >>>>>> >>>>>> >>>>>> Please advise, also please note that similar templates generated in >>>>>> prior releases such as train/ussuri works perfectly. >>>>>> >>>>>> >>>>>> >>>>>> Please check the list of *templates *files: >>>>>> >>>>>> drwxr-xr-x. 2 stack stack 68 May 30 09:22 environments >>>>>> -rw-r--r--. 1 stack stack 265 May 27 13:47 environment.yaml >>>>>> -rw-rw-r--. 1 stack stack 297 May 27 13:47 init-repo.yaml >>>>>> -rw-r--r--. 1 stack stack 570 May 27 13:47 ironic-config.yaml >>>>>> drwxrwxr-x. 4 stack stack 4096 May 27 13:53 network >>>>>> -rw-r--r--. 1 stack stack 6370 May 27 14:26 network_data.yaml >>>>>> -rw-r--r--. 1 stack stack 11137 May 27 13:53 roles_data.yaml >>>>>> -rw-r--r--. 1 stack stack 234 May 30 09:23 >>>>>> vip-data-default-network-isolation.yaml >>>>>> >>>>>> >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ cat environment.yaml >>>>>> >>>>>> parameter_defaults: >>>>>> OvercloudControllerFlavor: control >>>>>> OvercloudComputeFlavor: compute >>>>>> ControllerCount: 3 >>>>>> ComputeCount: 1 >>>>>> TimeZone: 'Asia/Kolkata' >>>>>> NtpServer: ['30.30.30.3'] >>>>>> NeutronBridgeMappings: datacentre:br-tenant >>>>>> NeutronFlatNetworks: datacentre >>>>>> (undercloud) [stack at undercloud templates]$ >>>>>> >>>>>> >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ cat ironic-config.yaml >>>>>> >>>>>> parameter_defaults: >>>>>> NovaSchedulerDefaultFilters: >>>>>> - AggregateInstanceExtraSpecsFilter >>>>>> - AvailabilityZoneFilter >>>>>> - ComputeFilter >>>>>> - ComputeCapabilitiesFilter >>>>>> - ImagePropertiesFilter >>>>>> IronicEnabledHardwareTypes: >>>>>> - ipmi >>>>>> - redfish >>>>>> IronicEnabledPowerInterfaces: >>>>>> - ipmitool >>>>>> - redfish >>>>>> IronicEnabledManagementInterfaces: >>>>>> - ipmitool >>>>>> - redfish >>>>>> IronicCleaningDiskErase: metadata >>>>>> IronicIPXEEnabled: true >>>>>> IronicInspectorSubnets: >>>>>> - ip_range: 172.23.3.100,172.23.3.150 >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ cat network_data.yaml >>>>>> >>>>>> - name: J3Mgmt >>>>>> name_lower: j3mgmt >>>>>> vip: true >>>>>> vlan: 400 >>>>>> ipv6: true >>>>>> ipv6_subnet: 'fd80:fd00:fd00:4000::/64' >>>>>> ipv6_allocation_pools: [{'start': 'fd80:fd00:fd00:4000::10', 'end': >>>>>> 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] >>>>>> mtu: 9000 >>>>>> >>>>>> >>>>>> >>>>>> - name: InternalApi >>>>>> name_lower: internal_api >>>>>> vip: true >>>>>> vlan: 418 >>>>>> ipv6: true >>>>>> ipv6_subnet: 'fd00:fd00:fd00:2000::/64' >>>>>> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': >>>>>> 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] >>>>>> mtu: 9000 >>>>>> >>>>>> >>>>>> - name: External >>>>>> vip: true >>>>>> name_lower: external >>>>>> vlan: 408 >>>>>> ipv6: true >>>>>> gateway_ipv6: 'fd00:fd00:fd00:9900::1' >>>>>> ipv6_subnet: 'fd00:fd00:fd00:9900::/64' >>>>>> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:9900::10', 'end': >>>>>> 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}] >>>>>> mtu: 9000 >>>>>> >>>>>> >>>>>> - name: OCProvisioning >>>>>> vip: true >>>>>> name_lower: oc_provisioning >>>>>> vlan: 412 >>>>>> ip_subnet: '172.23.3.0/24' >>>>>> allocation_pools: [{'start': '172.23.3.10', 'end': '172.23.3.50'}] >>>>>> mtu: 9000 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ cat roles_data.yaml >>>>>> >>>>>> >>>>>> ############################################################################### >>>>>> # File generated by TripleO >>>>>> >>>>>> ############################################################################### >>>>>> >>>>>> ############################################################################### >>>>>> # Role: Controller >>>>>> # >>>>>> >>>>>> ############################################################################### >>>>>> - name: Controller >>>>>> description: | >>>>>> Controller role that has all the controller services loaded and >>>>>> handles >>>>>> Database, Messaging, and Network functions. >>>>>> CountDefault: 1 >>>>>> tags: >>>>>> - primary >>>>>> - controller >>>>>> # Create external Neutron bridge for SNAT (and floating IPs when >>>>>> using >>>>>> # ML2/OVS without DVR) >>>>>> - external_bridge >>>>>> networks: >>>>>> External: >>>>>> subnet: external_subnet >>>>>> InternalApi: >>>>>> subnet: internal_api_subnet >>>>>> OCProvisioning: >>>>>> subnet: oc_provisioning_subnet >>>>>> J3Mgmt: >>>>>> subnet: j3mgmt_subnet >>>>>> >>>>>> >>>>>> # For systems with both IPv4 and IPv6, you may specify a gateway >>>>>> network for >>>>>> # each, such as ['ControlPlane', 'External'] >>>>>> default_route_networks: ['External'] >>>>>> HostnameFormatDefault: '%stackname%-controller-%index%' >>>>>> RoleParametersDefault: >>>>>> OVNCMSOptions: "enable-chassis-as-gw" >>>>>> # Deprecated & backward-compatible values (FIXME: Make parameters >>>>>> consistent) >>>>>> # Set uses_deprecated_params to True if any deprecated params are >>>>>> used. >>>>>> uses_deprecated_params: True >>>>>> deprecated_param_extraconfig: 'controllerExtraConfig' >>>>>> deprecated_param_flavor: 'OvercloudControlFlavor' >>>>>> deprecated_param_image: 'controllerImage' >>>>>> deprecated_nic_config_name: 'controller.yaml' >>>>>> update_serial: 1 >>>>>> ServicesDefault: >>>>>> - OS::TripleO::Services::Aide >>>>>> - OS::TripleO::Services::AodhApi >>>>>> - OS::TripleO::Services::AodhEvaluator >>>>>> >>>>>> .. >>>>>> . >>>>>> >>>>>> >>>>>> ..############################################################################### >>>>>> # Role: Compute >>>>>> # >>>>>> >>>>>> ############################################################################### >>>>>> - name: Compute >>>>>> description: | >>>>>> Basic Compute Node role >>>>>> CountDefault: 1 >>>>>> # Create external Neutron bridge (unset if using ML2/OVS without >>>>>> DVR) >>>>>> tags: >>>>>> - compute >>>>>> - external_bridge >>>>>> networks: >>>>>> InternalApi: >>>>>> subnet: internal_api_subnet >>>>>> J3Mgmt: >>>>>> subnet: j3mgmt_subnet >>>>>> HostnameFormatDefault: '%stackname%-novacompute-%index%' >>>>>> RoleParametersDefault: >>>>>> FsAioMaxNumber: 1048576 >>>>>> TunedProfileName: "virtual-host" >>>>>> # Deprecated & backward-compatible values (FIXME: Make parameters >>>>>> consistent) >>>>>> # Set uses_deprecated_params to True if any deprecated params are >>>>>> used. >>>>>> # These deprecated_params only need to be used for existing roles >>>>>> and not for >>>>>> # composable roles. >>>>>> uses_deprecated_params: True >>>>>> deprecated_param_image: 'NovaImage' >>>>>> deprecated_param_extraconfig: 'NovaComputeExtraConfig' >>>>>> deprecated_param_metadata: 'NovaComputeServerMetadata' >>>>>> deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints' >>>>>> deprecated_param_ips: 'NovaComputeIPs' >>>>>> deprecated_server_resource_name: 'NovaCompute' >>>>>> >>>>>> deprecated_nic_config_name: 'compute.yaml' >>>>>> update_serial: 25 >>>>>> ServicesDefault: >>>>>> - OS::TripleO::Services::Aide >>>>>> - OS::TripleO::Services::AuditD >>>>>> - OS::TripleO::Services::BootParams >>>>>> >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ cat >>>>>> environments/network-environment.yaml >>>>>> >>>>>> #This file is an example of an environment file for defining the >>>>>> isolated >>>>>> #networks and related parameters. >>>>>> resource_registry: >>>>>> # Network Interface templates to use (these files must exist). You >>>>>> can >>>>>> # override these by including one of the net-*.yaml environment >>>>>> files, >>>>>> # such as net-bond-with-vlans.yaml, or modifying the list here. >>>>>> # Port assignments for the Controller >>>>>> OS::TripleO::Controller::Net::SoftwareConfig: OS::Heat::None >>>>>> # Port assignments for the Compute >>>>>> OS::TripleO::Compute::Net::SoftwareConfig: OS::Heat::None >>>>>> >>>>>> >>>>>> parameter_defaults: >>>>>> # This section is where deployment-specific configuration is done >>>>>> # >>>>>> ServiceNetMap: >>>>>> IronicApiNetwork: oc_provisioning >>>>>> IronicNetwork: oc_provisioning >>>>>> >>>>>> >>>>>> >>>>>> # This section is where deployment-specific configuration is done >>>>>> ControllerNetworkConfigTemplate: >>>>>> 'templates/bonds_vlans/bonds_vlans.j2' >>>>>> ComputeNetworkConfigTemplate: 'templates/bonds_vlans/bonds_vlans.j2' >>>>>> >>>>>> >>>>>> >>>>>> # Customize the IP subnet to match the local environment >>>>>> J3MgmtNetCidr: 'fd80:fd00:fd00:4000::/64' >>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>> J3MgmtAllocationPools: [{'start': 'fd80:fd00:fd00:4000::10', 'end': >>>>>> 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] >>>>>> # Customize the VLAN ID to match the local environment >>>>>> J3MgmtNetworkVlanID: 400 >>>>>> >>>>>> >>>>>> >>>>>> # Customize the IP subnet to match the local environment >>>>>> InternalApiNetCidr: 'fd00:fd00:fd00:2000::/64' >>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>> InternalApiAllocationPools: [{'start': 'fd00:fd00:fd00:2000::10', >>>>>> 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] >>>>>> # Customize the VLAN ID to match the local environment >>>>>> InternalApiNetworkVlanID: 418 >>>>>> >>>>>> >>>>>> >>>>>> # Customize the IP subnet to match the local environment >>>>>> ExternalNetCidr: 'fd00:fd00:fd00:9900::/64' >>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>> # Leave room if the external network is also used for floating IPs >>>>>> ExternalAllocationPools: [{'start': 'fd00:fd00:fd00:9900::10', >>>>>> 'end': 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}] >>>>>> # Gateway router for routable networks >>>>>> ExternalInterfaceDefaultRoute: 'fd00:fd00:fd00:9900::1' >>>>>> # Customize the VLAN ID to match the local environment >>>>>> ExternalNetworkVlanID: 408 >>>>>> >>>>>> >>>>>> >>>>>> # Customize the IP subnet to match the local environment >>>>>> OCProvisioningNetCidr: '172.23.3.0/24' >>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>> OCProvisioningAllocationPools: [{'start': '172.23.3.10', 'end': >>>>>> '172.23.3.50'}] >>>>>> # Customize the VLAN ID to match the local environment >>>>>> OCProvisioningNetworkVlanID: 412 >>>>>> >>>>>> >>>>>> >>>>>> # List of Neutron network types for tenant networks (will be used >>>>>> in order) >>>>>> NeutronNetworkType: 'geneve,vlan' >>>>>> # Neutron VLAN ranges per network, for example >>>>>> 'datacentre:1:499,tenant:500:1000': >>>>>> NeutronNetworkVLANRanges: 'datacentre:1:1000' >>>>>> # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 >>>>>> miimon=100" >>>>>> # for Linux bonds w/LACP, or "bond_mode=active-backup" for OVS >>>>>> active/backup. >>>>>> BondInterfaceOvsOptions: "bond_mode=active-backup" >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ >>>>>> >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ cat >>>>>> environments/network-isolation.yaml >>>>>> >>>>>> # NOTE: This template is now deprecated, and is only included for >>>>>> compatibility >>>>>> # when upgrading a deployment where this template was originally >>>>>> used. For new >>>>>> # deployments, set "ipv6: true" on desired networks in >>>>>> network_data.yaml, and >>>>>> # include network-isolation.yaml. >>>>>> # >>>>>> # Enable the creation of Neutron networks for isolated Overcloud >>>>>> # traffic and configure each role to assign ports (related >>>>>> # to that role) on these networks. >>>>>> resource_registry: >>>>>> # networks as defined in network_data.yaml >>>>>> OS::TripleO::Network::J3Mgmt: ../network/j3mgmt_v6.yaml >>>>>> OS::TripleO::Network::InternalApi: ../network/internal_api_v6.yaml >>>>>> OS::TripleO::Network::External: ../network/external_v6.yaml >>>>>> OS::TripleO::Network::OCProvisioning: >>>>>> ../network/oc_provisioning.yaml >>>>>> >>>>>> >>>>>> # Port assignments for the VIPs >>>>>> OS::TripleO::Network::Ports::J3MgmtVipPort: >>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>> OS::TripleO::Network::Ports::InternalApiVipPort: >>>>>> ../network/ports/internal_api_v6.yaml >>>>>> OS::TripleO::Network::Ports::ExternalVipPort: >>>>>> ../network/ports/external_v6.yaml >>>>>> OS::TripleO::Network::Ports::OCProvisioningVipPort: >>>>>> ../network/ports/oc_provisioning.yaml >>>>>> >>>>>> >>>>>> >>>>>> # Port assignments by role, edit role definition to assign networks >>>>>> to roles. >>>>>> # Port assignments for the Controller >>>>>> OS::TripleO::Controller::Ports::J3MgmtPort: >>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>> OS::TripleO::Controller::Ports::InternalApiPort: >>>>>> ../network/ports/internal_api_v6.yaml >>>>>> OS::TripleO::Controller::Ports::ExternalPort: >>>>>> ../network/ports/external_v6.yaml >>>>>> OS::TripleO::Controller::Ports::OCProvisioningPort: >>>>>> ../network/ports/oc_provisioning.yaml >>>>>> # Port assignments for the Compute >>>>>> OS::TripleO::Compute::Ports::J3MgmtPort: >>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>> OS::TripleO::Compute::Ports::InternalApiPort: >>>>>> ../network/ports/internal_api_v6.yaml >>>>>> >>>>>> >>>>>> >>>>>> parameter_defaults: >>>>>> # Enable IPv6 environment for Manila >>>>>> ManilaIPv6: True >>>>>> >>>>>> (undercloud) [stack at undercloud templates]$ >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, May 24, 2022 at 5:04 PM Lokendra Rathour < >>>>>> lokendrarathour at gmail.com> wrote: >>>>>> >>>>>>> Thanks, I'll check them out. >>>>>>> will let you know in case it works out. >>>>>>> >>>>>>> On Tue, May 24, 2022 at 2:37 PM Swogat Pradhan < >>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> Please find the below templates: >>>>>>>> These are for openstack wallaby release: >>>>>>>> >>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>> custom_network_data.yaml >>>>>>>> - name: Storage >>>>>>>> name_lower: storage >>>>>>>> vip: true >>>>>>>> mtu: 1500 >>>>>>>> subnets: >>>>>>>> storage_subnet: >>>>>>>> ip_subnet: 172.25.202.0/26 >>>>>>>> allocation_pools: >>>>>>>> - start: 172.25.202.6 >>>>>>>> end: 172.25.202.20 >>>>>>>> vlan: 1105 >>>>>>>> - name: StorageMgmt >>>>>>>> name_lower: storage_mgmt >>>>>>>> vip: true >>>>>>>> mtu: 1500 >>>>>>>> subnets: >>>>>>>> storage_mgmt_subnet: >>>>>>>> ip_subnet: 172.25.202.64/26 >>>>>>>> allocation_pools: >>>>>>>> - start: 172.25.202.72 >>>>>>>> end: 172.25.202.87 >>>>>>>> vlan: 1106 >>>>>>>> - name: InternalApi >>>>>>>> name_lower: internal_api >>>>>>>> vip: true >>>>>>>> mtu: 1500 >>>>>>>> subnets: >>>>>>>> internal_api_subnet: >>>>>>>> ip_subnet: 172.25.201.192/26 >>>>>>>> allocation_pools: >>>>>>>> - start: 172.25.201.198 >>>>>>>> end: 172.25.201.212 >>>>>>>> vlan: 1104 >>>>>>>> - name: Tenant >>>>>>>> vip: false # Tenant network does not use VIPs >>>>>>>> mtu: 1500 >>>>>>>> name_lower: tenant >>>>>>>> subnets: >>>>>>>> tenant_subnet: >>>>>>>> ip_subnet: 172.25.202.128/26 >>>>>>>> allocation_pools: >>>>>>>> - start: 172.25.202.135 >>>>>>>> end: 172.25.202.150 >>>>>>>> vlan: 1108 >>>>>>>> - name: External >>>>>>>> name_lower: external >>>>>>>> vip: true >>>>>>>> mtu: 1500 >>>>>>>> subnets: >>>>>>>> external_subnet: >>>>>>>> ip_subnet: 172.25.201.128/26 >>>>>>>> allocation_pools: >>>>>>>> - start: 172.25.201.135 >>>>>>>> end: 172.25.201.150 >>>>>>>> gateway_ip: 172.25.201.129 >>>>>>>> vlan: 1103 >>>>>>>> >>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>> custom_vip_data.yaml >>>>>>>> - network: ctlplane >>>>>>>> #dns_name: overcloud >>>>>>>> ip_address: 172.25.201.91 >>>>>>>> subnet: ctlplane-subnet >>>>>>>> - network: external >>>>>>>> #dns_name: overcloud >>>>>>>> ip_address: 172.25.201.150 >>>>>>>> subnet: external_subnet >>>>>>>> - network: internal_api >>>>>>>> #dns_name: overcloud >>>>>>>> ip_address: 172.25.201.250 >>>>>>>> subnet: internal_api_subnet >>>>>>>> - network: storage >>>>>>>> #dns_name: overcloud >>>>>>>> ip_address: 172.25.202.50 >>>>>>>> subnet: storage_subnet >>>>>>>> - network: storage_mgmt >>>>>>>> #dns_name: overcloud >>>>>>>> ip_address: 172.25.202.90 >>>>>>>> subnet: storage_mgmt_subnet >>>>>>>> >>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>> overcloud-baremetal-deploy.yaml >>>>>>>> - name: Controller >>>>>>>> count: 4 >>>>>>>> defaults: >>>>>>>> networks: >>>>>>>> - network: ctlplane >>>>>>>> vif: true >>>>>>>> - network: external >>>>>>>> subnet: external_subnet >>>>>>>> - network: internal_api >>>>>>>> subnet: internal_api_subnet >>>>>>>> - network: storage >>>>>>>> subnet: storage_subnet >>>>>>>> - network: storage_mgmt >>>>>>>> subnet: storage_mgmt_subnet >>>>>>>> - network: tenant >>>>>>>> subnet: tenant_subnet >>>>>>>> network_config: >>>>>>>> template: /home/stack/templates/controller.j2 >>>>>>>> default_route_network: >>>>>>>> - external >>>>>>>> instances: >>>>>>>> - hostname: overcloud-controller-0 >>>>>>>> name: dc1-controller2 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-controller-1 >>>>>>>> name: dc2-controller2 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-controller-2 >>>>>>>> name: dc1-controller1 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-controller-no-ceph-3 >>>>>>>> name: dc2-ceph2 >>>>>>>> #provisioned: false >>>>>>>> #- hostname: overcloud-controller-3 >>>>>>>> #name: dc2-compute3 >>>>>>>> #provisioned: false >>>>>>>> >>>>>>>> - name: Compute >>>>>>>> count: 5 >>>>>>>> defaults: >>>>>>>> networks: >>>>>>>> - network: ctlplane >>>>>>>> vif: true >>>>>>>> - network: internal_api >>>>>>>> subnet: internal_api_subnet >>>>>>>> - network: tenant >>>>>>>> subnet: tenant_subnet >>>>>>>> - network: storage >>>>>>>> subnet: storage_subnet >>>>>>>> network_config: >>>>>>>> template: /home/stack/templates/compute.j2 >>>>>>>> instances: >>>>>>>> - hostname: overcloud-novacompute-0 >>>>>>>> name: dc2-compute1 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-novacompute-1 >>>>>>>> name: dc2-ceph1 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-novacompute-2 >>>>>>>> name: dc1-compute1 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-novacompute-3 >>>>>>>> name: dc1-compute2 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-novacompute-4 >>>>>>>> name: dc2-compute3 >>>>>>>> #provisioned: false >>>>>>>> >>>>>>>> - name: CephStorage >>>>>>>> count: 4 >>>>>>>> defaults: >>>>>>>> networks: >>>>>>>> - network: ctlplane >>>>>>>> vif: true >>>>>>>> - network: internal_api >>>>>>>> subnet: internal_api_subnet >>>>>>>> - network: storage >>>>>>>> subnet: storage_subnet >>>>>>>> - network: storage_mgmt >>>>>>>> subnet: storage_mgmt_subnet >>>>>>>> network_config: >>>>>>>> template: /home/stack/templates/ceph-storage.j2 >>>>>>>> instances: >>>>>>>> - hostname: overcloud-cephstorage-0 >>>>>>>> name: dc2-controller1 >>>>>>>> #provisioned: false >>>>>>>> # - hostname: overcloud-cephstorage-1 >>>>>>>> # name: dc2-ceph2 >>>>>>>> - hostname: overcloud-cephstorage-1 >>>>>>>> name: dc1-ceph1 >>>>>>>> # provisioned: false >>>>>>>> - hostname: overcloud-cephstorage-2 >>>>>>>> name: dc1-ceph2 >>>>>>>> #provisioned: false >>>>>>>> - hostname: overcloud-cephstorage-3 >>>>>>>> name: dc2-compute2 >>>>>>>> #provisioned: false >>>>>>>> >>>>>>>> >>>>>>>> You must use these templates to provision network, vip and nodes. >>>>>>>> You must use the output files generated during the provisioning >>>>>>>> step in openstack overcloud deploy command using -e parameter. >>>>>>>> >>>>>>>> With regards, >>>>>>>> Swogat Pradhan >>>>>>>> >>>>>>>> >>>>>>>> On Mon, May 23, 2022 at 8:33 PM Lokendra Rathour < >>>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Swogat, >>>>>>>>> I tried checking your solution and my templates but could not >>>>>>>>> relate much. >>>>>>>>> But issue seems the same >>>>>>>>> >>>>>>>>> http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028401.html >>>>>>>>> >>>>>>>>> I tried somemore ways but looks like some issue with templates. >>>>>>>>> Can you please share the templates used to deploy the overcloud. >>>>>>>>> >>>>>>>>> Mysetup have 3 controller and 1 compute. >>>>>>>>> >>>>>>>>> Thanks once again for reading my mail. >>>>>>>>> >>>>>>>>> Waiting for your reply. >>>>>>>>> >>>>>>>>> -Lokendra >>>>>>>>> >>>>>>>>> On Fri, 20 May 2022, 08:25 Swogat Pradhan, < >>>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> Yes I was able to find the issue and fix it. >>>>>>>>>> The issue was with the overcloud-baremetal-deployed.yaml file i >>>>>>>>>> was trying to provision controller-0, controller-1 and controller-3 and >>>>>>>>>> kept controller-2 aside for later, but the tripleo scripts are built in >>>>>>>>>> such a way that they were taking controller- 0, 1 and 2 inplace of >>>>>>>>>> controller-3, so the network ports and vip were created for controller 0,1 >>>>>>>>>> and 2 but not for 3 , so this error was popping off. Also i would request >>>>>>>>>> you to check the jinja nic templates and once the node provisioning is done >>>>>>>>>> check the /etc/os-net-config/config.json/yaml file for syntax if using >>>>>>>>>> bonded nic template. >>>>>>>>>> If you need any more infor please let me know. >>>>>>>>>> >>>>>>>>>> With regards, >>>>>>>>>> Swogat Pradhan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, May 20, 2022 at 8:01 AM Lokendra Rathour < >>>>>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Swogat, >>>>>>>>>>> Thanks for raising this issue. >>>>>>>>>>> Did you find any solution? to this problem ? >>>>>>>>>>> >>>>>>>>>>> Please let me know it might be helpful >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 19, 2022 at 12:43 PM Swogat Pradhan < >>>>>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> I am currently trying to deploy openstack wallaby using tripleo >>>>>>>>>>>> arch. >>>>>>>>>>>> I created the network jinja templates, ran the following >>>>>>>>>>>> commands also: >>>>>>>>>>>> >>>>>>>>>>>> #openstack overcloud network provision --stack overcloud >>>>>>>>>>>> --output networks-deployed-environment.yaml custom_network_data.yaml >>>>>>>>>>>> # openstack overcloud network vip provision --stack overcloud >>>>>>>>>>>> --output vip-deployed-environment.yaml custom_vip_data.yaml >>>>>>>>>>>> # openstack overcloud node provision --stack overcloud >>>>>>>>>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa >>>>>>>>>>>> overcloud-baremetal-deploy.yaml >>>>>>>>>>>> >>>>>>>>>>>> and used the environment files in the openstack overcloud >>>>>>>>>>>> deploy command: >>>>>>>>>>>> >>>>>>>>>>>> (undercloud) [stack at hkg2director ~]$ cat deploy.sh >>>>>>>>>>>> #!/bin/bash >>>>>>>>>>>> THT=/usr/share/openstack-tripleo-heat-templates/ >>>>>>>>>>>> CNF=/home/stack/ >>>>>>>>>>>> openstack overcloud deploy --templates $THT \ >>>>>>>>>>>> -r $CNF/templates/roles_data.yaml \ >>>>>>>>>>>> -n $CNF/workplace/custom_network_data.yaml \ >>>>>>>>>>>> -e ~/containers-prepare-parameter.yaml \ >>>>>>>>>>>> -e $CNF/templates/node-info.yaml \ >>>>>>>>>>>> -e $CNF/templates/scheduler-hints.yaml \ >>>>>>>>>>>> -e $CNF/workplace/networks-deployed-environment.yaml \ >>>>>>>>>>>> -e $CNF/workplace/vip-deployed-environment.yaml \ >>>>>>>>>>>> -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ >>>>>>>>>>>> -e $CNF/workplace/custom-net-bond-with-vlans.yaml >>>>>>>>>>>> >>>>>>>>>>>> Now when i run the ./deploy.sh script i encounter an error >>>>>>>>>>>> stating: >>>>>>>>>>>> >>>>>>>>>>>> ERROR openstack [-] Resource >>>>>>>>>>>> OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type >>>>>>>>>>>> OS::Neutron::Port and the Neutron service is not available when using >>>>>>>>>>>> ephemeral Heat. The generated environments from 'openstack overcloud >>>>>>>>>>>> baremetal provision' and 'openstack overcloud network provision' must be >>>>>>>>>>>> included with the deployment command.: >>>>>>>>>>>> tripleoclient.exceptions.InvalidConfiguration: Resource >>>>>>>>>>>> OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type >>>>>>>>>>>> OS::Neutron::Port and the Neutron service is not available when using >>>>>>>>>>>> ephemeral Heat. The generated environments from 'openstack overcloud >>>>>>>>>>>> baremetal provision' and 'openstack overcloud network provision' must be >>>>>>>>>>>> included with the deployment command. >>>>>>>>>>>> 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END >>>>>>>>>>>> return value: 1 >>>>>>>>>>>> >>>>>>>>>>>> Can someone tell me where the mistake is? >>>>>>>>>>>> >>>>>>>>>>>> With regards, >>>>>>>>>>>> Swogat Pradhan >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>> >>>>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bragin75 at gmail.com Tue Jun 7 09:40:00 2022 From: bragin75 at gmail.com (Vasiliy Bragin) Date: Tue, 7 Jun 2022 14:40:00 +0500 Subject: trove, error - Exception during message handling: AttributeError: 'TroveContext' object has no attribute 'notification', please help to fix Message-ID: Hello, team trove! trove, error - Exception during message handling: AttributeError: 'TroveContext' object has no attribute 'notification', please help to fix release openstack - yoga, CentOS, CentOS Stream release 8 instance trove - ubuntu, Linux version 5.4.0-113-generic (buildd at lcy02-amd64-067) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 On Yoga Trove guest-agent inside instance not work at all: log 2022-06-01 15:31:13.263 908 DEBUG trove.common.rpc.service [-] Creating RPC server for service guestagent.15e4e2e9-13a4-42f6-bafb-f63fdd074d76 start /opt/guest-agent-venv/lib/python3.8/site-packages/trove/common/rpc/service.py:56 *2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server [-] Exception during message handling: AttributeError: 'TroveContext' object has no attribute 'notification'* 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.8/site-packages/osprofiler/profiler.py", line 160, in wrapper 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.8/site-packages/trove/guestagent/datastore/manager.py", line 213, in prepare 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server with EndNotification(context, instance_id=CONF.guest_id): 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.8/site-packages/trove/common/notification.py", line 48, in __init__ 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server self.context.notification.payload.update(kwargs) 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server AttributeError: 'TroveContext' object has no attribute 'notification' 2022-06-01 15:31:13.610 908 ERROR oslo_messaging.rpc.server Trove guest-agent on instance execute. But DB don't work and not started. After restart guest-agent on instance next log: 2022-06-06 12:51:09.443 4503 DEBUG trove.common.rpc.service [-] Creating RPC server for service guestagent.15e4e2e9-13a4-42f6-bafb-f63fdd074d76 start /opt/guest-agent-venv/lib/python3.8/site-packages/trove/common/rpc/service.py:56 2022-06-06 12:52:09.541 4503 DEBUG oslo_service.periodic_task [-] Running periodic task Manager.update_status run_periodic_tasks /opt/guest-agent-venv/lib/python3.8/site-packages/oslo_service/periodic_task.py:210 *2022-06-06 12:52:09.546 4503 INFO trove.guestagent.datastore.manager [-] Database service is not installed, skip status check* Please tell me how can I fix this errors? -- Vasily Bragin -------------- next part -------------- An HTML attachment was scrubbed... URL: From anyrude10 at gmail.com Tue Jun 7 09:49:05 2022 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Tue, 7 Jun 2022 15:19:05 +0530 Subject: [Tripleo] - Provisioning Baremetal Instances in Wallaby Openstack Message-ID: Hi Harald/Julia/Openstack Team I had successfully installed Ironic Service in Tripleo Train, Ussuri Release with the help of openstack discuss team members. https://lists.openstack.org/pipermail/openstack-discuss/2022-February/027047.html I had custom network configuration for which I had passed Service net map as below in network-environment file. parameter_defaults: ServiceNetMap: IronicApiNetwork: oc_provisioning IronicNetwork: oc_provisioning But with the Wallaby Release, since the deployment format has changed There is no network-environment file. So I tried passing it in a separate environment.yaml file along with the templates. Command to deploy in wallaby: openstack overcloud deploy --templates \ --networks-file /home/stack/templates/custom_network_data.yaml \ --vip-file /home/stack/templates/custom_vip_data.yaml \ --baremetal-deployment /home/stack/templates/overcloud-baremetal-deploy.yaml --deployed-server \ --network-config \ -e /home/stack/templates/environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml \ -e /home/stack/templates/ironic-config.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ptp.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ -e /home/stack/containers-prepare-parameter.yaml Without Service Net Map, deployment was successful but TFTP of image was an issue since we are going with custom network approach. Passing Service Net map details in a seperate file environment.yaml, is giving the below error: 24d-f43f-6f76-000000006ce9 | TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 2022-06-07 11:16:46.806982 | | WARNING | ERROR: Can't run container ironic_db_sync stderr: Error: statfs /var/lib/config-data/puppet-generated/ironic_api: no such file or directory 2022-06-07 11:16:46.809252 | 525400fd-f24d-f43f-6f76-000000006c79 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | overcloud-controller-1 | error={"changed": false, "msg": "Failed containers: ironic_db_sync"} 2022-06-07 11:16:46.810197 | 525400fd-f24d-f43f-6f76-000000006c79 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | overcloud-controller-1 | 0:11:40.574944 | 10.62s 2022-06-07 11:16:47.683400 | | WARNING | ERROR: Can't run container ironic_db_sync stderr: Error: statfs /var/lib/config-data/puppet-generated/ironic_api: no such file or directory 2022-06-07 11:16:47.684455 | 525400fd-f24d-f43f-6f76-000000006ce9 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | overcloud-controller-2 | error={"changed": false, "msg": "Failed containers: ironic_db_sync"} Can you please suggest the place we can pass the service net map details in Wallaby Release to resolve this issue? Regards Anirudh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From swogatpradhan22 at gmail.com Wed Jun 8 05:34:22 2022 From: swogatpradhan22 at gmail.com (Swogat Pradhan) Date: Wed, 8 Jun 2022 11:04:22 +0530 Subject: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version In-Reply-To: References: Message-ID: Hi Lokendra, I am not sure why this issue is caused, can you please check the tripleo_dnsmasq service and the tripleo_ironic_pxe service in your director node? IF the services are running fine then please report this here for fast resolvement https://bugzilla.redhat.com/ With regards, Swogat Pradhan On Tue, Jun 7, 2022 at 12:54 PM Lokendra Rathour wrote: > Hi Swogat, > Yes after correcting the j2 template, as suggested by you, it started > working. Thanks a lot once again. > Now the Overcloud is deployed. > > Further, as a next requirement, we were also trying to add an *additional > Node as a bare-metal Instance* on the overcloud. As it is the composable > network approach, we add an entry in parameters default for ServiceNetMap, > as like below in environments/network-environments.yaml > > parameter_defaults: > # This section is where deployment-specific configuration is done > ServiceNetMap: > IronicApiNetwork: oc_provisioning > IronicNetwork: oc_provisioning > > this is now passed as in the environment.yaml as parameter_defaults, but > somehow we are not able to deploy the setup. it gives the error for > containers. > > > *2022-06-06 19:08:37.560800 | 525400fd-f24d-9703-b0e7-0000000084b2 | FATAL > | Manage container systemd services and cleanup old systemd healthchecks > for /var/lib/tripleo-config/container-startup-config/step_4 | > overcloud-controller-0 | error={"changed": false, "msg": "Service > ironic_pxe_tftp has not started yet"}* > > > and if we remove this setting as stated above, the deployment is > successful, but at the time of node provisioning it is not able to reach > the TFTP server to pull files, mainly because of the composable network > (oc_provisioning), we faced this issue in Train, so we did the above config > changes and it working. But as it looks something more is needed in > Wallaby for Baremetal provisioning as an instance. > > Please share your learnings with the reference to it. Thanks once again > for your all-time support. > reference link: Bare Metal Instances in Overcloud ? TripleO 3.0.0 > documentation (openstack.org) > > > -Lokendra > > > On Wed, Jun 1, 2022 at 12:19 AM Lokendra Rathour < > lokendrarathour at gmail.com> wrote: > >> Ok Swogat, >> Thanks once again. >> i will try this approach once and will let you know. >> >> >> On Wed, 1 Jun 2022, 00:04 Swogat Pradhan, >> wrote: >> >>> Hi Lokendra, >>> Like i said, >>> > NOTE: when passing --network-config parameter in node provision step, >>> it creates a directory in /etc/os-net-config and in it creates a file >>> > config.yaml, do check the indentation of that file. (in my case the >>> indentation was wrong when i was using bonding everytime, so i had to >>> > manually change the script and run a while loop and then my node >>> provision step was successful) >>> this parameter --network-config creates a config.yaml(network config) >>> file in /etc/os-net-config directory in the overcloud nodes and then >>> ansible tries to apply the network config from the generated config file. >>> And in wallaby version if you are using bonding then the syntax in the >>> generated conf is wrong (atleast was in my case) and then the ansible tries >>> to apply the network config (with wrong syntax) so your overcloud nodes >>> become unavailable. >>> >>> Please run node provision separately, and keep on monitoring "metalsmith >>> list" command. Once IP is assigned to the overcloud nodes, ssh to the nodes >>> and assign a password to heat-admin user so that even if the network >>> becomes unavailable you still will be able to access the nodes via console >>> access, then you can visit /etc/os-net-config directory and verify the >>> config.yaml file. >>> >>> With regards, >>> Swogat pradhan. >>> >>> On Tue, May 31, 2022 at 11:56 PM Lokendra Rathour < >>> lokendrarathour at gmail.com> wrote: >>> >>>> Hi Swogat, >>>> Thanks once again for your input it really helped much. >>>> >>>> instead of running mentioned those three provisioning steps i used >>>> alternate method and passed directly in deploy command. now my current >>>> deploy command is: >>>> >>>> openstack overcloud deploy --templates \ >>>> --networks-file /home/stack/templates/custom_network_data.yaml \ >>>> --vip-file /home/stack/templates/custom_vip_data.yaml \ >>>> --baremetal-deployment >>>> /home/stack/templates/overcloud-baremetal-deploy.yaml \ >>>> --network-config \ >>>> -e /home/stack/templates/environment.yaml \ >>>> -e >>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml >>>> \ >>>> -e >>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml >>>> \ >>>> -e >>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml >>>> \ >>>> -e /home/stack/templates/ironic-config.yaml \ >>>> -e >>>> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ >>>> -e >>>> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ >>>> -e /home/stack/containers-prepare-parameter.yaml >>>> >>>> The files as suggested by you are well created. >>>> >>>> But once we run the deployment I get this error for all nodes: >>>> >>>> 0:00:16.064781 | 1.16s >>>> 2022-05-31 19:13:00.276954 | 525400ef-b928-9ded-fecc-000000000094 | >>>> TASK | Run tripleo_os_net_config_module with network_config >>>> 2022-05-31 19:40:30.061582 | 525400ef-b928-9ded-fecc-000000000094 | >>>> FATAL | Run tripleo_os_net_config_module with network_config | >>>> overcloud-controller-1 | error={"msg": "Data could not be sent to remote >>>> host \"30.30.30.117\". Make sure this host can be reached over ssh: ssh: >>>> connect to host 30.30.30.117 port 22: No route to host\r\n"} >>>> 2022- >>>> >>>> >>>> Baremetal node list are showing in as active. >>>> >>>> (undercloud) [stack at undercloud ~]$ openstack baremetal node list >>>> /usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: >>>> DeprecationWarning: The _yaml extension module is now located at yaml._yaml >>>> and its location is subject to change. To use the LibYAML-based parser and >>>> emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper >>>> as Dumper`. >>>> DeprecationWarning >>>> >>>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>>> | UUID | Name | Instance UUID >>>> | Power State | Provisioning State | Maintenance | >>>> >>>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>>> | 1a4d873c-f9f7-4504-a3af-92c11f954171 | node-a | >>>> 901453a1-183f-4de8-aaab-0f38be2be455 | power on | active | >>>> False | >>>> | d18610fc-9532-410c-918e-8efc326c89f8 | node-b | >>>> d059b94a-8357-4f8e-a0d8-15a24b0c1afe | power on | active | >>>> False | >>>> | b69f2d5a-5b18-4453-8843-15c6af79aca0 | node-c | >>>> f196ef3a-7950-47b9-a5ae-751f06b18f75 | power on | active | >>>> False | >>>> | 8a38c584-f812-4ebc-a0b1-4299f0917637 | node-d | >>>> 1636517c-2ab2-43d7-8205-9f02c5290207 | power on | active | >>>> False | >>>> >>>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>>> >>>> Some config is missing it seems, please check once and advise. >>>> >>>> >>>> On Tue, May 31, 2022 at 11:40 AM Swogat Pradhan < >>>> swogatpradhan22 at gmail.com> wrote: >>>> >>>>> Hi Lokendra, >>>>> You need to generate another file also in the following step: openstack >>>>> overcloud node provision --stack overcloud --overcloud-ssh-key >>>>> /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml also you >>>>> need to pass another parameter --network-config. >>>>> example: >>>>> openstack overcloud node provision --stack overcloud >>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa *--network-config* *--output >>>>> overcloud-baremetal-deployed.yaml* overcloud-baremetal-deploy.yaml >>>>> >>>>> And then all these output files will be passed on to the openstack >>>>> overcloud deploy command. >>>>> NOTE: when passing --network-config parameter in node provision step, >>>>> it creates a directory in /etc/os-net-config and in it creates a file >>>>> config.yaml, do check the indentation of that file. (in my case the >>>>> indentation was wrong when i was using bondind everytime, so i had to >>>>> manually change the script and run a while loop and then my node provision >>>>> step was successful) >>>>> >>>>> On Tue, May 31, 2022 at 8:59 AM Lokendra Rathour < >>>>> lokendrarathour at gmail.com> wrote: >>>>> >>>>>> Hi Swogat, >>>>>> I tried generating the scripts as used by you in your deployments >>>>>> using the >>>>>> >>>>>> >>>>>> #openstack overcloud network provision --stack overcloud --output >>>>>> networks-deployed-environment.yaml custom_network_data.yaml >>>>>> # openstack overcloud network vip provision --stack overcloud >>>>>> --output vip-deployed-environment.yaml custom_vip_data.yaml >>>>>> # openstack overcloud node provision --stack overcloud >>>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa >>>>>> overcloud-baremetal-deploy.yaml >>>>>> >>>>>> and used the first two in the final deployment script, but it gives >>>>>> the error: >>>>>> >>>>>> heatclient.exc.HTTPInternalServerError: ERROR: Internal Error >>>>>> 2022-05-30 14:14:39.772 479668 ERROR >>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent >>>>>> call last):\n', ' File "/usr/lib/python3.6/ted_stack\n >>>>>> nested_stack.validate()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in >>>>>> wrapper\n result = f(*args, ine 969, in validate\n result = >>>>>> res.validate()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/openstack/neutron/port.py", >>>>>> line 454site-packages/heat/engine/resources/openstack/neutron/neutron.py", >>>>>> line 43, in validate\n res = super(NeutronResource, self).validate()\n', >>>>>> ' File "/un return self.validate_template()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 1882, in >>>>>> validate_template\n self.t.rpy", line 200, in >>>>>> _validate_service_availability\n raise ex\n', >>>>>> 'heat.common.exception.ResourceTypeUnavailable: HEAT-E99001 Service neutron >>>>>> is not avaieutron network endpoint is not in service catalog.\n', '\nDuring >>>>>> handling of the above exception, another exception occurred:\n\n', >>>>>> 'Traceback (most recens/stack_resource.py", line 75, in >>>>>> validate_nested_stack\n nested_stack.validate()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py"thon3.6/site-packages/heat/engine/stack.py", >>>>>> line 969, in validate\n result = res.validate()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/ateResource, >>>>>> self).validate()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", >>>>>> line 65, in validate\n self.validources/stack_resource.py", line 81, in >>>>>> validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', >>>>>> 'heat.common.exception.StackValidationFaileeploy/overcloud/tripleo-heat-templates/deployed-server/deployed-server.yaml>: >>>>>> HEAT-E99001 Service neutron is not available for resource type >>>>>> OS::TripleO::vice catalog.\n', '\nDuring handling of the above exception, >>>>>> another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' >>>>>> File "/usr/lib/pline 320, in validate_nested_stack\n >>>>>> nested_stack.validate()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in >>>>>> wrappe/heat/engine/stack.py", line 969, in validate\n result = >>>>>> res.validate()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/template_relidate()\n', >>>>>> ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", >>>>>> line 65, in validate\n self.validate_nested_stack()\n'.py", line 81, in >>>>>> validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', >>>>>> 'heat.common.exception.StackValidationFailed: >>>>>> ResourceTypeUnavaimplates/puppet/compute-role.yaml>.resources.NovaCompute>>>>> reason: neutron network endpoint is not in service catalog.\n', '\nDuring >>>>>> handling of the above exception, >>>>>> another/lib/python3.6/site-packages/heat/common/context.py", line 416, in >>>>>> wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File >>>>>> "/usr/lib/python3.6/sirce_name, template_id)\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 756, in >>>>>> _parse_template_and_validate_stack\n stack.v line 160, in wrapper\n >>>>>> result = f(*args, **kwargs)\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 969, in >>>>>> validate\n resesources/stack_resource.py", line 65, in validate\n >>>>>> self.validate_nested_stack()\n', ' File >>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/oph=[self.stack.t.RESOURCES, >>>>>> path])\n', 'heat.common.exception.StackValidationFailed: >>>>>> ResourceTypeUnavailable: >>>>>> resources.Compute.resources.0.resources.NovaCompute: >>>>>> HEAT-E9900::ControlPlanePort, reason: neutron network endpoint is not in >>>>>> service catalog.\n']. >>>>>> >>>>>> Request you to check once, please. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, May 30, 2022 at 11:06 AM Lokendra Rathour < >>>>>> lokendrarathour at gmail.com> wrote: >>>>>> >>>>>>> Hi Swogat, >>>>>>> Thanks once again. >>>>>>> >>>>>>> with the files as shown below I am running the overcloud deploy for >>>>>>> wallaby using this command: >>>>>>> >>>>>>> (undercloud) [stack at undercloud ~]$ cat deploy_overcloud_working_1.sh >>>>>>> openstack overcloud deploy --templates \ >>>>>>> -n /home/stack/templates/network_data.yaml \ >>>>>>> -r /home/stack/templates/roles_data.yaml \ >>>>>>> -e /home/stack/templates/environment.yaml \ >>>>>>> -e /home/stack/templates/environments/network-isolation.yaml \ >>>>>>> -e /home/stack/templates/environments/network-environment.yaml \ >>>>>>> -e >>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml >>>>>>> \ >>>>>>> -e >>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml >>>>>>> \ >>>>>>> -e >>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml >>>>>>> \ >>>>>>> -e /home/stack/templates/ironic-config.yaml \ >>>>>>> -e >>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ >>>>>>> -e >>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ >>>>>>> -e /home/stack/containers-prepare-parameter.yaml >>>>>>> (undercloud) [stack at undercloud ~]$ >>>>>>> >>>>>>> >>>>>>> This deployment is on ipv6 using triple0 wallaby, templates, as >>>>>>> mentioned below, are generated using rendering steps and the >>>>>>> network_data.yaml the roles_data.yaml >>>>>>> Steps used to render the templates: >>>>>>> cd /usr/share/openstack-tripleo-heat-templates/ >>>>>>> ./tools/process-templates.py -o >>>>>>> ~/openstack-tripleo-heat-templates-rendered_at_wallaby -n >>>>>>> /home/stack/templates/network_data.yaml -r >>>>>>> /home/stack/templates/roles_data.yaml >>>>>>> >>>>>>> *Now if i try adding the related to VIP port I do get the error as:* >>>>>>> >>>>>>> 2022-05-30 10:37:12.792 979387 WARNING >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template >>>>>>> to file: >>>>>>> /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/controller-role.yaml >>>>>>> 2022-05-30 10:37:12.792 979387 WARNING >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template >>>>>>> to file: >>>>>>> /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/compute-role.yaml >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured >>>>>>> while running the command: ValueError: The environment is not a valid YAML >>>>>>> mapping data type. >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent >>>>>>> call last): >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, >>>>>>> self).run(parsed_args) >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in >>>>>>> run >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, >>>>>>> self).run(parsed_args) >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = >>>>>>> self.take_action(parsed_args) or 0 >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>>> line 1189, in take_action >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud stack, parsed_args, >>>>>>> new_tht_root, user_tht_root) >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>>> line 227, in create_env_files >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud created_env_files, >>>>>>> parsed_args, new_tht_root, user_tht_root) >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>>> line 204, in build_image_params >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud cleanup=(not >>>>>>> parsed_args.no_cleanup)) >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1929, in >>>>>>> process_multiple_environments >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud env_path=env_path, >>>>>>> include_env_in_files=include_env_in_files) >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", >>>>>>> line 326, in process_environment_and_files >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud env = >>>>>>> environment_format.parse(raw_env) >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>> "/usr/lib/python3.6/site-packages/heatclient/common/environment_format.py", >>>>>>> line 50, in parse >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud raise >>>>>>> ValueError(_('The environment is not a valid ' >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: The >>>>>>> environment is not a valid YAML mapping data type. >>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud >>>>>>> 2022-05-30 10:37:14.457 979387 ERROR openstack [-] The environment >>>>>>> is not a valid YAML mapping data type. >>>>>>> 2022-05-30 10:37:14.457 979387 INFO osc_lib.shell [-] END return >>>>>>> value: 1 >>>>>>> (undercloud) [stack at undercloud ~]$ >>>>>>> >>>>>>> This is more of a syntax error where it is not able to understand >>>>>>> the passed VIP data file: >>>>>>> >>>>>>> undercloud) [stack at undercloud ~]$ cat >>>>>>> /home/stack/templates/vip-data-default-network-isolation.yaml >>>>>>> - >>>>>>> dns_name: overcloud >>>>>>> network: internal_api >>>>>>> - >>>>>>> dns_name: overcloud >>>>>>> network: external >>>>>>> - >>>>>>> dns_name: overcloud >>>>>>> network: ctlplane >>>>>>> - >>>>>>> dns_name: overcloud >>>>>>> network: oc_provisioning >>>>>>> - >>>>>>> dns_name: overcloud >>>>>>> network: j3mgmt >>>>>>> >>>>>>> >>>>>>> Please advise, also please note that similar templates generated in >>>>>>> prior releases such as train/ussuri works perfectly. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Please check the list of *templates *files: >>>>>>> >>>>>>> drwxr-xr-x. 2 stack stack 68 May 30 09:22 environments >>>>>>> -rw-r--r--. 1 stack stack 265 May 27 13:47 environment.yaml >>>>>>> -rw-rw-r--. 1 stack stack 297 May 27 13:47 init-repo.yaml >>>>>>> -rw-r--r--. 1 stack stack 570 May 27 13:47 ironic-config.yaml >>>>>>> drwxrwxr-x. 4 stack stack 4096 May 27 13:53 network >>>>>>> -rw-r--r--. 1 stack stack 6370 May 27 14:26 network_data.yaml >>>>>>> -rw-r--r--. 1 stack stack 11137 May 27 13:53 roles_data.yaml >>>>>>> -rw-r--r--. 1 stack stack 234 May 30 09:23 >>>>>>> vip-data-default-network-isolation.yaml >>>>>>> >>>>>>> >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ cat environment.yaml >>>>>>> >>>>>>> parameter_defaults: >>>>>>> OvercloudControllerFlavor: control >>>>>>> OvercloudComputeFlavor: compute >>>>>>> ControllerCount: 3 >>>>>>> ComputeCount: 1 >>>>>>> TimeZone: 'Asia/Kolkata' >>>>>>> NtpServer: ['30.30.30.3'] >>>>>>> NeutronBridgeMappings: datacentre:br-tenant >>>>>>> NeutronFlatNetworks: datacentre >>>>>>> (undercloud) [stack at undercloud templates]$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ cat ironic-config.yaml >>>>>>> >>>>>>> parameter_defaults: >>>>>>> NovaSchedulerDefaultFilters: >>>>>>> - AggregateInstanceExtraSpecsFilter >>>>>>> - AvailabilityZoneFilter >>>>>>> - ComputeFilter >>>>>>> - ComputeCapabilitiesFilter >>>>>>> - ImagePropertiesFilter >>>>>>> IronicEnabledHardwareTypes: >>>>>>> - ipmi >>>>>>> - redfish >>>>>>> IronicEnabledPowerInterfaces: >>>>>>> - ipmitool >>>>>>> - redfish >>>>>>> IronicEnabledManagementInterfaces: >>>>>>> - ipmitool >>>>>>> - redfish >>>>>>> IronicCleaningDiskErase: metadata >>>>>>> IronicIPXEEnabled: true >>>>>>> IronicInspectorSubnets: >>>>>>> - ip_range: 172.23.3.100,172.23.3.150 >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ cat network_data.yaml >>>>>>> >>>>>>> - name: J3Mgmt >>>>>>> name_lower: j3mgmt >>>>>>> vip: true >>>>>>> vlan: 400 >>>>>>> ipv6: true >>>>>>> ipv6_subnet: 'fd80:fd00:fd00:4000::/64' >>>>>>> ipv6_allocation_pools: [{'start': 'fd80:fd00:fd00:4000::10', >>>>>>> 'end': 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] >>>>>>> mtu: 9000 >>>>>>> >>>>>>> >>>>>>> >>>>>>> - name: InternalApi >>>>>>> name_lower: internal_api >>>>>>> vip: true >>>>>>> vlan: 418 >>>>>>> ipv6: true >>>>>>> ipv6_subnet: 'fd00:fd00:fd00:2000::/64' >>>>>>> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', >>>>>>> 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] >>>>>>> mtu: 9000 >>>>>>> >>>>>>> >>>>>>> - name: External >>>>>>> vip: true >>>>>>> name_lower: external >>>>>>> vlan: 408 >>>>>>> ipv6: true >>>>>>> gateway_ipv6: 'fd00:fd00:fd00:9900::1' >>>>>>> ipv6_subnet: 'fd00:fd00:fd00:9900::/64' >>>>>>> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:9900::10', >>>>>>> 'end': 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}] >>>>>>> mtu: 9000 >>>>>>> >>>>>>> >>>>>>> - name: OCProvisioning >>>>>>> vip: true >>>>>>> name_lower: oc_provisioning >>>>>>> vlan: 412 >>>>>>> ip_subnet: '172.23.3.0/24' >>>>>>> allocation_pools: [{'start': '172.23.3.10', 'end': '172.23.3.50'}] >>>>>>> mtu: 9000 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ cat roles_data.yaml >>>>>>> >>>>>>> >>>>>>> ############################################################################### >>>>>>> # File generated by TripleO >>>>>>> >>>>>>> ############################################################################### >>>>>>> >>>>>>> ############################################################################### >>>>>>> # Role: Controller >>>>>>> # >>>>>>> >>>>>>> ############################################################################### >>>>>>> - name: Controller >>>>>>> description: | >>>>>>> Controller role that has all the controller services loaded and >>>>>>> handles >>>>>>> Database, Messaging, and Network functions. >>>>>>> CountDefault: 1 >>>>>>> tags: >>>>>>> - primary >>>>>>> - controller >>>>>>> # Create external Neutron bridge for SNAT (and floating IPs when >>>>>>> using >>>>>>> # ML2/OVS without DVR) >>>>>>> - external_bridge >>>>>>> networks: >>>>>>> External: >>>>>>> subnet: external_subnet >>>>>>> InternalApi: >>>>>>> subnet: internal_api_subnet >>>>>>> OCProvisioning: >>>>>>> subnet: oc_provisioning_subnet >>>>>>> J3Mgmt: >>>>>>> subnet: j3mgmt_subnet >>>>>>> >>>>>>> >>>>>>> # For systems with both IPv4 and IPv6, you may specify a gateway >>>>>>> network for >>>>>>> # each, such as ['ControlPlane', 'External'] >>>>>>> default_route_networks: ['External'] >>>>>>> HostnameFormatDefault: '%stackname%-controller-%index%' >>>>>>> RoleParametersDefault: >>>>>>> OVNCMSOptions: "enable-chassis-as-gw" >>>>>>> # Deprecated & backward-compatible values (FIXME: Make parameters >>>>>>> consistent) >>>>>>> # Set uses_deprecated_params to True if any deprecated params are >>>>>>> used. >>>>>>> uses_deprecated_params: True >>>>>>> deprecated_param_extraconfig: 'controllerExtraConfig' >>>>>>> deprecated_param_flavor: 'OvercloudControlFlavor' >>>>>>> deprecated_param_image: 'controllerImage' >>>>>>> deprecated_nic_config_name: 'controller.yaml' >>>>>>> update_serial: 1 >>>>>>> ServicesDefault: >>>>>>> - OS::TripleO::Services::Aide >>>>>>> - OS::TripleO::Services::AodhApi >>>>>>> - OS::TripleO::Services::AodhEvaluator >>>>>>> >>>>>>> .. >>>>>>> . >>>>>>> >>>>>>> >>>>>>> ..############################################################################### >>>>>>> # Role: Compute >>>>>>> # >>>>>>> >>>>>>> ############################################################################### >>>>>>> - name: Compute >>>>>>> description: | >>>>>>> Basic Compute Node role >>>>>>> CountDefault: 1 >>>>>>> # Create external Neutron bridge (unset if using ML2/OVS without >>>>>>> DVR) >>>>>>> tags: >>>>>>> - compute >>>>>>> - external_bridge >>>>>>> networks: >>>>>>> InternalApi: >>>>>>> subnet: internal_api_subnet >>>>>>> J3Mgmt: >>>>>>> subnet: j3mgmt_subnet >>>>>>> HostnameFormatDefault: '%stackname%-novacompute-%index%' >>>>>>> RoleParametersDefault: >>>>>>> FsAioMaxNumber: 1048576 >>>>>>> TunedProfileName: "virtual-host" >>>>>>> # Deprecated & backward-compatible values (FIXME: Make parameters >>>>>>> consistent) >>>>>>> # Set uses_deprecated_params to True if any deprecated params are >>>>>>> used. >>>>>>> # These deprecated_params only need to be used for existing roles >>>>>>> and not for >>>>>>> # composable roles. >>>>>>> uses_deprecated_params: True >>>>>>> deprecated_param_image: 'NovaImage' >>>>>>> deprecated_param_extraconfig: 'NovaComputeExtraConfig' >>>>>>> deprecated_param_metadata: 'NovaComputeServerMetadata' >>>>>>> deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints' >>>>>>> deprecated_param_ips: 'NovaComputeIPs' >>>>>>> deprecated_server_resource_name: 'NovaCompute' >>>>>>> >>>>>>> deprecated_nic_config_name: 'compute.yaml' >>>>>>> update_serial: 25 >>>>>>> ServicesDefault: >>>>>>> - OS::TripleO::Services::Aide >>>>>>> - OS::TripleO::Services::AuditD >>>>>>> - OS::TripleO::Services::BootParams >>>>>>> >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ cat >>>>>>> environments/network-environment.yaml >>>>>>> >>>>>>> #This file is an example of an environment file for defining the >>>>>>> isolated >>>>>>> #networks and related parameters. >>>>>>> resource_registry: >>>>>>> # Network Interface templates to use (these files must exist). You >>>>>>> can >>>>>>> # override these by including one of the net-*.yaml environment >>>>>>> files, >>>>>>> # such as net-bond-with-vlans.yaml, or modifying the list here. >>>>>>> # Port assignments for the Controller >>>>>>> OS::TripleO::Controller::Net::SoftwareConfig: OS::Heat::None >>>>>>> # Port assignments for the Compute >>>>>>> OS::TripleO::Compute::Net::SoftwareConfig: OS::Heat::None >>>>>>> >>>>>>> >>>>>>> parameter_defaults: >>>>>>> # This section is where deployment-specific configuration is done >>>>>>> # >>>>>>> ServiceNetMap: >>>>>>> IronicApiNetwork: oc_provisioning >>>>>>> IronicNetwork: oc_provisioning >>>>>>> >>>>>>> >>>>>>> >>>>>>> # This section is where deployment-specific configuration is done >>>>>>> ControllerNetworkConfigTemplate: >>>>>>> 'templates/bonds_vlans/bonds_vlans.j2' >>>>>>> ComputeNetworkConfigTemplate: >>>>>>> 'templates/bonds_vlans/bonds_vlans.j2' >>>>>>> >>>>>>> >>>>>>> >>>>>>> # Customize the IP subnet to match the local environment >>>>>>> J3MgmtNetCidr: 'fd80:fd00:fd00:4000::/64' >>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>> J3MgmtAllocationPools: [{'start': 'fd80:fd00:fd00:4000::10', >>>>>>> 'end': 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] >>>>>>> # Customize the VLAN ID to match the local environment >>>>>>> J3MgmtNetworkVlanID: 400 >>>>>>> >>>>>>> >>>>>>> >>>>>>> # Customize the IP subnet to match the local environment >>>>>>> InternalApiNetCidr: 'fd00:fd00:fd00:2000::/64' >>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>> InternalApiAllocationPools: [{'start': 'fd00:fd00:fd00:2000::10', >>>>>>> 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] >>>>>>> # Customize the VLAN ID to match the local environment >>>>>>> InternalApiNetworkVlanID: 418 >>>>>>> >>>>>>> >>>>>>> >>>>>>> # Customize the IP subnet to match the local environment >>>>>>> ExternalNetCidr: 'fd00:fd00:fd00:9900::/64' >>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>> # Leave room if the external network is also used for floating IPs >>>>>>> ExternalAllocationPools: [{'start': 'fd00:fd00:fd00:9900::10', >>>>>>> 'end': 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}] >>>>>>> # Gateway router for routable networks >>>>>>> ExternalInterfaceDefaultRoute: 'fd00:fd00:fd00:9900::1' >>>>>>> # Customize the VLAN ID to match the local environment >>>>>>> ExternalNetworkVlanID: 408 >>>>>>> >>>>>>> >>>>>>> >>>>>>> # Customize the IP subnet to match the local environment >>>>>>> OCProvisioningNetCidr: '172.23.3.0/24' >>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>> OCProvisioningAllocationPools: [{'start': '172.23.3.10', 'end': >>>>>>> '172.23.3.50'}] >>>>>>> # Customize the VLAN ID to match the local environment >>>>>>> OCProvisioningNetworkVlanID: 412 >>>>>>> >>>>>>> >>>>>>> >>>>>>> # List of Neutron network types for tenant networks (will be used >>>>>>> in order) >>>>>>> NeutronNetworkType: 'geneve,vlan' >>>>>>> # Neutron VLAN ranges per network, for example >>>>>>> 'datacentre:1:499,tenant:500:1000': >>>>>>> NeutronNetworkVLANRanges: 'datacentre:1:1000' >>>>>>> # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 >>>>>>> miimon=100" >>>>>>> # for Linux bonds w/LACP, or "bond_mode=active-backup" for OVS >>>>>>> active/backup. >>>>>>> BondInterfaceOvsOptions: "bond_mode=active-backup" >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ >>>>>>> >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ cat >>>>>>> environments/network-isolation.yaml >>>>>>> >>>>>>> # NOTE: This template is now deprecated, and is only included for >>>>>>> compatibility >>>>>>> # when upgrading a deployment where this template was originally >>>>>>> used. For new >>>>>>> # deployments, set "ipv6: true" on desired networks in >>>>>>> network_data.yaml, and >>>>>>> # include network-isolation.yaml. >>>>>>> # >>>>>>> # Enable the creation of Neutron networks for isolated Overcloud >>>>>>> # traffic and configure each role to assign ports (related >>>>>>> # to that role) on these networks. >>>>>>> resource_registry: >>>>>>> # networks as defined in network_data.yaml >>>>>>> OS::TripleO::Network::J3Mgmt: ../network/j3mgmt_v6.yaml >>>>>>> OS::TripleO::Network::InternalApi: ../network/internal_api_v6.yaml >>>>>>> OS::TripleO::Network::External: ../network/external_v6.yaml >>>>>>> OS::TripleO::Network::OCProvisioning: >>>>>>> ../network/oc_provisioning.yaml >>>>>>> >>>>>>> >>>>>>> # Port assignments for the VIPs >>>>>>> OS::TripleO::Network::Ports::J3MgmtVipPort: >>>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>>> OS::TripleO::Network::Ports::InternalApiVipPort: >>>>>>> ../network/ports/internal_api_v6.yaml >>>>>>> OS::TripleO::Network::Ports::ExternalVipPort: >>>>>>> ../network/ports/external_v6.yaml >>>>>>> OS::TripleO::Network::Ports::OCProvisioningVipPort: >>>>>>> ../network/ports/oc_provisioning.yaml >>>>>>> >>>>>>> >>>>>>> >>>>>>> # Port assignments by role, edit role definition to assign >>>>>>> networks to roles. >>>>>>> # Port assignments for the Controller >>>>>>> OS::TripleO::Controller::Ports::J3MgmtPort: >>>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>>> OS::TripleO::Controller::Ports::InternalApiPort: >>>>>>> ../network/ports/internal_api_v6.yaml >>>>>>> OS::TripleO::Controller::Ports::ExternalPort: >>>>>>> ../network/ports/external_v6.yaml >>>>>>> OS::TripleO::Controller::Ports::OCProvisioningPort: >>>>>>> ../network/ports/oc_provisioning.yaml >>>>>>> # Port assignments for the Compute >>>>>>> OS::TripleO::Compute::Ports::J3MgmtPort: >>>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>>> OS::TripleO::Compute::Ports::InternalApiPort: >>>>>>> ../network/ports/internal_api_v6.yaml >>>>>>> >>>>>>> >>>>>>> >>>>>>> parameter_defaults: >>>>>>> # Enable IPv6 environment for Manila >>>>>>> ManilaIPv6: True >>>>>>> >>>>>>> (undercloud) [stack at undercloud templates]$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, May 24, 2022 at 5:04 PM Lokendra Rathour < >>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>> >>>>>>>> Thanks, I'll check them out. >>>>>>>> will let you know in case it works out. >>>>>>>> >>>>>>>> On Tue, May 24, 2022 at 2:37 PM Swogat Pradhan < >>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Please find the below templates: >>>>>>>>> These are for openstack wallaby release: >>>>>>>>> >>>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>>> custom_network_data.yaml >>>>>>>>> - name: Storage >>>>>>>>> name_lower: storage >>>>>>>>> vip: true >>>>>>>>> mtu: 1500 >>>>>>>>> subnets: >>>>>>>>> storage_subnet: >>>>>>>>> ip_subnet: 172.25.202.0/26 >>>>>>>>> allocation_pools: >>>>>>>>> - start: 172.25.202.6 >>>>>>>>> end: 172.25.202.20 >>>>>>>>> vlan: 1105 >>>>>>>>> - name: StorageMgmt >>>>>>>>> name_lower: storage_mgmt >>>>>>>>> vip: true >>>>>>>>> mtu: 1500 >>>>>>>>> subnets: >>>>>>>>> storage_mgmt_subnet: >>>>>>>>> ip_subnet: 172.25.202.64/26 >>>>>>>>> allocation_pools: >>>>>>>>> - start: 172.25.202.72 >>>>>>>>> end: 172.25.202.87 >>>>>>>>> vlan: 1106 >>>>>>>>> - name: InternalApi >>>>>>>>> name_lower: internal_api >>>>>>>>> vip: true >>>>>>>>> mtu: 1500 >>>>>>>>> subnets: >>>>>>>>> internal_api_subnet: >>>>>>>>> ip_subnet: 172.25.201.192/26 >>>>>>>>> allocation_pools: >>>>>>>>> - start: 172.25.201.198 >>>>>>>>> end: 172.25.201.212 >>>>>>>>> vlan: 1104 >>>>>>>>> - name: Tenant >>>>>>>>> vip: false # Tenant network does not use VIPs >>>>>>>>> mtu: 1500 >>>>>>>>> name_lower: tenant >>>>>>>>> subnets: >>>>>>>>> tenant_subnet: >>>>>>>>> ip_subnet: 172.25.202.128/26 >>>>>>>>> allocation_pools: >>>>>>>>> - start: 172.25.202.135 >>>>>>>>> end: 172.25.202.150 >>>>>>>>> vlan: 1108 >>>>>>>>> - name: External >>>>>>>>> name_lower: external >>>>>>>>> vip: true >>>>>>>>> mtu: 1500 >>>>>>>>> subnets: >>>>>>>>> external_subnet: >>>>>>>>> ip_subnet: 172.25.201.128/26 >>>>>>>>> allocation_pools: >>>>>>>>> - start: 172.25.201.135 >>>>>>>>> end: 172.25.201.150 >>>>>>>>> gateway_ip: 172.25.201.129 >>>>>>>>> vlan: 1103 >>>>>>>>> >>>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>>> custom_vip_data.yaml >>>>>>>>> - network: ctlplane >>>>>>>>> #dns_name: overcloud >>>>>>>>> ip_address: 172.25.201.91 >>>>>>>>> subnet: ctlplane-subnet >>>>>>>>> - network: external >>>>>>>>> #dns_name: overcloud >>>>>>>>> ip_address: 172.25.201.150 >>>>>>>>> subnet: external_subnet >>>>>>>>> - network: internal_api >>>>>>>>> #dns_name: overcloud >>>>>>>>> ip_address: 172.25.201.250 >>>>>>>>> subnet: internal_api_subnet >>>>>>>>> - network: storage >>>>>>>>> #dns_name: overcloud >>>>>>>>> ip_address: 172.25.202.50 >>>>>>>>> subnet: storage_subnet >>>>>>>>> - network: storage_mgmt >>>>>>>>> #dns_name: overcloud >>>>>>>>> ip_address: 172.25.202.90 >>>>>>>>> subnet: storage_mgmt_subnet >>>>>>>>> >>>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>>> overcloud-baremetal-deploy.yaml >>>>>>>>> - name: Controller >>>>>>>>> count: 4 >>>>>>>>> defaults: >>>>>>>>> networks: >>>>>>>>> - network: ctlplane >>>>>>>>> vif: true >>>>>>>>> - network: external >>>>>>>>> subnet: external_subnet >>>>>>>>> - network: internal_api >>>>>>>>> subnet: internal_api_subnet >>>>>>>>> - network: storage >>>>>>>>> subnet: storage_subnet >>>>>>>>> - network: storage_mgmt >>>>>>>>> subnet: storage_mgmt_subnet >>>>>>>>> - network: tenant >>>>>>>>> subnet: tenant_subnet >>>>>>>>> network_config: >>>>>>>>> template: /home/stack/templates/controller.j2 >>>>>>>>> default_route_network: >>>>>>>>> - external >>>>>>>>> instances: >>>>>>>>> - hostname: overcloud-controller-0 >>>>>>>>> name: dc1-controller2 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-controller-1 >>>>>>>>> name: dc2-controller2 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-controller-2 >>>>>>>>> name: dc1-controller1 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-controller-no-ceph-3 >>>>>>>>> name: dc2-ceph2 >>>>>>>>> #provisioned: false >>>>>>>>> #- hostname: overcloud-controller-3 >>>>>>>>> #name: dc2-compute3 >>>>>>>>> #provisioned: false >>>>>>>>> >>>>>>>>> - name: Compute >>>>>>>>> count: 5 >>>>>>>>> defaults: >>>>>>>>> networks: >>>>>>>>> - network: ctlplane >>>>>>>>> vif: true >>>>>>>>> - network: internal_api >>>>>>>>> subnet: internal_api_subnet >>>>>>>>> - network: tenant >>>>>>>>> subnet: tenant_subnet >>>>>>>>> - network: storage >>>>>>>>> subnet: storage_subnet >>>>>>>>> network_config: >>>>>>>>> template: /home/stack/templates/compute.j2 >>>>>>>>> instances: >>>>>>>>> - hostname: overcloud-novacompute-0 >>>>>>>>> name: dc2-compute1 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-novacompute-1 >>>>>>>>> name: dc2-ceph1 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-novacompute-2 >>>>>>>>> name: dc1-compute1 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-novacompute-3 >>>>>>>>> name: dc1-compute2 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-novacompute-4 >>>>>>>>> name: dc2-compute3 >>>>>>>>> #provisioned: false >>>>>>>>> >>>>>>>>> - name: CephStorage >>>>>>>>> count: 4 >>>>>>>>> defaults: >>>>>>>>> networks: >>>>>>>>> - network: ctlplane >>>>>>>>> vif: true >>>>>>>>> - network: internal_api >>>>>>>>> subnet: internal_api_subnet >>>>>>>>> - network: storage >>>>>>>>> subnet: storage_subnet >>>>>>>>> - network: storage_mgmt >>>>>>>>> subnet: storage_mgmt_subnet >>>>>>>>> network_config: >>>>>>>>> template: /home/stack/templates/ceph-storage.j2 >>>>>>>>> instances: >>>>>>>>> - hostname: overcloud-cephstorage-0 >>>>>>>>> name: dc2-controller1 >>>>>>>>> #provisioned: false >>>>>>>>> # - hostname: overcloud-cephstorage-1 >>>>>>>>> # name: dc2-ceph2 >>>>>>>>> - hostname: overcloud-cephstorage-1 >>>>>>>>> name: dc1-ceph1 >>>>>>>>> # provisioned: false >>>>>>>>> - hostname: overcloud-cephstorage-2 >>>>>>>>> name: dc1-ceph2 >>>>>>>>> #provisioned: false >>>>>>>>> - hostname: overcloud-cephstorage-3 >>>>>>>>> name: dc2-compute2 >>>>>>>>> #provisioned: false >>>>>>>>> >>>>>>>>> >>>>>>>>> You must use these templates to provision network, vip and nodes. >>>>>>>>> You must use the output files generated during the provisioning >>>>>>>>> step in openstack overcloud deploy command using -e parameter. >>>>>>>>> >>>>>>>>> With regards, >>>>>>>>> Swogat Pradhan >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, May 23, 2022 at 8:33 PM Lokendra Rathour < >>>>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Swogat, >>>>>>>>>> I tried checking your solution and my templates but could not >>>>>>>>>> relate much. >>>>>>>>>> But issue seems the same >>>>>>>>>> >>>>>>>>>> http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028401.html >>>>>>>>>> >>>>>>>>>> I tried somemore ways but looks like some issue with templates. >>>>>>>>>> Can you please share the templates used to deploy the overcloud. >>>>>>>>>> >>>>>>>>>> Mysetup have 3 controller and 1 compute. >>>>>>>>>> >>>>>>>>>> Thanks once again for reading my mail. >>>>>>>>>> >>>>>>>>>> Waiting for your reply. >>>>>>>>>> >>>>>>>>>> -Lokendra >>>>>>>>>> >>>>>>>>>> On Fri, 20 May 2022, 08:25 Swogat Pradhan, < >>>>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> Yes I was able to find the issue and fix it. >>>>>>>>>>> The issue was with the overcloud-baremetal-deployed.yaml file i >>>>>>>>>>> was trying to provision controller-0, controller-1 and controller-3 and >>>>>>>>>>> kept controller-2 aside for later, but the tripleo scripts are built in >>>>>>>>>>> such a way that they were taking controller- 0, 1 and 2 inplace of >>>>>>>>>>> controller-3, so the network ports and vip were created for controller 0,1 >>>>>>>>>>> and 2 but not for 3 , so this error was popping off. Also i would request >>>>>>>>>>> you to check the jinja nic templates and once the node provisioning is done >>>>>>>>>>> check the /etc/os-net-config/config.json/yaml file for syntax if using >>>>>>>>>>> bonded nic template. >>>>>>>>>>> If you need any more infor please let me know. >>>>>>>>>>> >>>>>>>>>>> With regards, >>>>>>>>>>> Swogat Pradhan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, May 20, 2022 at 8:01 AM Lokendra Rathour < >>>>>>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Swogat, >>>>>>>>>>>> Thanks for raising this issue. >>>>>>>>>>>> Did you find any solution? to this problem ? >>>>>>>>>>>> >>>>>>>>>>>> Please let me know it might be helpful >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Apr 19, 2022 at 12:43 PM Swogat Pradhan < >>>>>>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> I am currently trying to deploy openstack wallaby using >>>>>>>>>>>>> tripleo arch. >>>>>>>>>>>>> I created the network jinja templates, ran the following >>>>>>>>>>>>> commands also: >>>>>>>>>>>>> >>>>>>>>>>>>> #openstack overcloud network provision --stack overcloud >>>>>>>>>>>>> --output networks-deployed-environment.yaml custom_network_data.yaml >>>>>>>>>>>>> # openstack overcloud network vip provision --stack overcloud >>>>>>>>>>>>> --output vip-deployed-environment.yaml custom_vip_data.yaml >>>>>>>>>>>>> # openstack overcloud node provision --stack overcloud >>>>>>>>>>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa >>>>>>>>>>>>> overcloud-baremetal-deploy.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> and used the environment files in the openstack overcloud >>>>>>>>>>>>> deploy command: >>>>>>>>>>>>> >>>>>>>>>>>>> (undercloud) [stack at hkg2director ~]$ cat deploy.sh >>>>>>>>>>>>> #!/bin/bash >>>>>>>>>>>>> THT=/usr/share/openstack-tripleo-heat-templates/ >>>>>>>>>>>>> CNF=/home/stack/ >>>>>>>>>>>>> openstack overcloud deploy --templates $THT \ >>>>>>>>>>>>> -r $CNF/templates/roles_data.yaml \ >>>>>>>>>>>>> -n $CNF/workplace/custom_network_data.yaml \ >>>>>>>>>>>>> -e ~/containers-prepare-parameter.yaml \ >>>>>>>>>>>>> -e $CNF/templates/node-info.yaml \ >>>>>>>>>>>>> -e $CNF/templates/scheduler-hints.yaml \ >>>>>>>>>>>>> -e $CNF/workplace/networks-deployed-environment.yaml \ >>>>>>>>>>>>> -e $CNF/workplace/vip-deployed-environment.yaml \ >>>>>>>>>>>>> -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ >>>>>>>>>>>>> -e $CNF/workplace/custom-net-bond-with-vlans.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> Now when i run the ./deploy.sh script i encounter an error >>>>>>>>>>>>> stating: >>>>>>>>>>>>> >>>>>>>>>>>>> ERROR openstack [-] Resource >>>>>>>>>>>>> OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type >>>>>>>>>>>>> OS::Neutron::Port and the Neutron service is not available when using >>>>>>>>>>>>> ephemeral Heat. The generated environments from 'openstack overcloud >>>>>>>>>>>>> baremetal provision' and 'openstack overcloud network provision' must be >>>>>>>>>>>>> included with the deployment command.: >>>>>>>>>>>>> tripleoclient.exceptions.InvalidConfiguration: Resource >>>>>>>>>>>>> OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type >>>>>>>>>>>>> OS::Neutron::Port and the Neutron service is not available when using >>>>>>>>>>>>> ephemeral Heat. The generated environments from 'openstack overcloud >>>>>>>>>>>>> baremetal provision' and 'openstack overcloud network provision' must be >>>>>>>>>>>>> included with the deployment command. >>>>>>>>>>>>> 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END >>>>>>>>>>>>> return value: 1 >>>>>>>>>>>>> >>>>>>>>>>>>> Can someone tell me where the mistake is? >>>>>>>>>>>>> >>>>>>>>>>>>> With regards, >>>>>>>>>>>>> Swogat Pradhan >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>> >>>>>> >>>> >>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anyrude10 at gmail.com Wed Jun 8 13:08:33 2022 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Wed, 8 Jun 2022 18:38:33 +0530 Subject: [Tripleo] - Provisioning Baremetal Instances in Wallaby Openstack In-Reply-To: <5ee4d9f4-772d-18c5-bc30-7d5f849f63b3@redhat.com> References: <86dca991-09df-afd9-7e84-7fc60bb9efbd@redhat.com> <5ee4d9f4-772d-18c5-bc30-7d5f849f63b3@redhat.com> Message-ID: Hi Harald Thanks for your response. After passing the role data with relevant details we were able to successfully deploy Overcloud and pass Service NetMap as well. Thanks a lot for your help Regards Anirudh Gupta On Tue, Jun 7, 2022 at 7:01 PM Harald Jensas wrote: > On 6/7/22 14:32, Harald Jensas wrote: > > On 6/7/22 11:49, Anirudh Gupta wrote: > >> Hi Harald/Julia/Openstack Team > >> > >> I had successfully installed Ironic Service in Tripleo Train, Ussuri > >> Release with the help of openstack discuss team members. > >> > >> > https://lists.openstack.org/pipermail/openstack-discuss/2022-February/027047.html > >> < > https://lists.openstack.org/pipermail/openstack-discuss/2022-February/027047.html> > > >> > >> > >> I had custom network configuration for which I had passed Service net > >> map as below in network-environment file. > >> > >> parameter_defaults: > >> ServiceNetMap: > >> IronicApiNetwork: oc_provisioning > >> IronicNetwork: oc_provisioning > >> > >> But with the Wallaby Release, since the deployment format has changed > >> There is no network-environment file. > >> > >> So I tried passing it in a separate environment.yaml file along with > >> the templates. > >> Command to deploy in wallaby: > >> > >> openstack overcloud deploy --templates \ > >> --networks-file /home/stack/templates/custom_network_data.yaml \ > >> --vip-file /home/stack/templates/custom_vip_data.yaml \ > >> --baremetal-deployment > >> /home/stack/templates/overcloud-baremetal-deploy.yaml > >> --deployed-server \ > >> --network-config \ > >> -e /home/stack/templates/environment.yaml \ > >> -e > >> > /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml > > >> \ > >> -e > >> > /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml > > >> \ > >> -e > >> > /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml > > >> \ > >> -e /home/stack/templates/ironic-config.yaml \ > >> -e > >> > /usr/share/openstack-tripleo-heat-templates/environments/services/ptp.yaml > >> \ > >> -e > >> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml > \ > >> -e > >> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ > >> -e /home/stack/containers-prepare-parameter.yaml > >> > >> Without Service Net Map, deployment was successful but TFTP of image > >> was an issue since we are going with custom network approach. > >> > >> Passing Service Net map details in a seperate file environment.yaml, > >> is giving the below error: > >> > >> 24d-f43f-6f76-000000006ce9 | TASK | Create containers managed by > >> Podman for /var/lib/tripleo-config/container-startup-config/step_3 > >> 2022-06-07 11:16:46.806982 | | WARNING | ERROR: Can't run container > >> ironic_db_sync > >> stderr: Error: statfs > >> /var/lib/config-data/puppet-generated/ironic_api: no such file or > >> directory > >> 2022-06-07 11:16:46.809252 | 525400fd-f24d-f43f-6f76-000000006c79 | > >> FATAL | Create containers managed by Podman for > >> /var/lib/tripleo-config/container-startup-config/step_3 | > >> overcloud-controller-1 | error={"changed": false, "msg": "Failed > >> containers: ironic_db_sync"} > >> 2022-06-07 11:16:46.810197 | 525400fd-f24d-f43f-6f76-000000006c79 | > >> TIMING | tripleo_container_manage : Create containers managed by > >> Podman for /var/lib/tripleo-config/container-startup-config/step_3 | > >> overcloud-controller-1 | 0:11:40.574944 | 10.62s > >> 2022-06-07 11:16:47.683400 | | WARNING | ERROR: Can't run container > >> ironic_db_sync > >> stderr: Error: statfs > >> /var/lib/config-data/puppet-generated/ironic_api: no such file or > >> directory > >> 2022-06-07 11:16:47.684455 | 525400fd-f24d-f43f-6f76-000000006ce9 | > >> FATAL | Create containers managed by Podman for > >> /var/lib/tripleo-config/container-startup-config/step_3 | > >> overcloud-controller-2 | error={"changed": false, "msg": "Failed > >> containers: ironic_db_sync"} > >> > >> Can you please suggest the place we can pass the service net map > >> details in Wallaby Release to resolve this issue? > >> > > > > You should be able to pass it in any of the environment files, like you > > did. Can you try adding the following to the environment file where you > > override the ServiceNetMap? > > > > parameter_merge_strategies: > > ServiceNetMap: merge > > > > Actually, my bad. (Thanks Rabi for reminding me on IRC.) The merge > strategies is after a change in master. With Wallaby we still have > ServiceNetMapDefaults which is merged with overrides in ServiceNetMap. > > AFICT from this error ... > > > 2022-06-07 11:16:47.683400 | | WARNING | ERROR: Can't run container > > ironic_db_sync > > stderr: Error: statfs > > /var/lib/config-data/puppet-generated/ironic_api: no such file or > > directory > > ... it did not execute $THT/common/container-puppet.sh for the > ironic_api service correctly. > > Could it be because you don't use a custom roles file? > The default[1] role file does not have the `oc_provisioning` > provisioning network. I wonder if some task is skipped because the node > is not on the network the service is mapped to. Can you try using a > custom roles file with the `oc_provisioning` network defined on the > controller role? > > I think we probably need more information to understand the problem. > Would it be possible to share more data? i.e full deploy log, all custom > environment files? > > Thanks! > > [1] > https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/wallaby/roles_data.yaml#L18 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafaelweingartner at gmail.com Wed Jun 8 13:59:11 2022 From: rafaelweingartner at gmail.com (=?UTF-8?Q?Rafael_Weing=C3=A4rtner?=) Date: Wed, 8 Jun 2022 10:59:11 -0300 Subject: [CLOUDKITTY] presentation used in the forum panels Message-ID: Hello CloudKitters! It was a pleasure to meet you guys during the OpenStack summit. Also, it was a good surprise to see that so many people already use CloudKitty. I really hope that we can all collaborate together to keep improving CloudKitty, and making it more and more solid. Attached is the presentation we used during the Forum, so you all can consult this material. Also, as always, if you need some guidance/help/or anything else, let us know. Link for the presentation online: https://docs.google.com/presentation/d/15e37xGKCJoTMwbarrGcSlLpECUkcJ2exazW_TztCehk/edit?usp=sharing -- Rafael Weing?rtner -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: CloudKitty onboarding and operators feedback forum panels.pdf Type: application/pdf Size: 1091998 bytes Desc: not available URL: From tkajinam at redhat.com Wed Jun 8 14:29:40 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Wed, 8 Jun 2022 23:29:40 +0900 Subject: [neutron][networking-l2gw][puppet] Status of networking-l2gw project Message-ID: Hello team, # I understand networking-l2gw is not under neutron stadium # but adding [neutron] tag hoping it would attract more attention. We found a recent change in neutron[1] broke networking-l2gw[2], and some jobs in puppet repos and rdo, with l2gw enabled, are failing. I proposed a potential fix for the bug[3], but I noticed the project looks unmaintained according to the following points. - No change has been merged for more than 1 year - No release was created since Wallaby - The zuul setting still points the victoria job template (openstack-python3-victoria-jobs-neutron) I'm wondering if anybody is actually interested in maintaining the project. If not, I'll propose retiring the plugin support from puppet (and TripleO because it requires puppet implementation). I appreciate any feedback about this. [1] https://review.opendev.org/c/openstack/neutron/+/837392 [2] https://bugs.launchpad.net/networking-l2gw/+bug/1977980 [3] https://review.opendev.org/c/x/networking-l2gw/+/845141 Thank you, Takashi -------------- next part -------------- An HTML attachment was scrubbed... URL: From dms at danplanet.com Wed Jun 8 14:43:59 2022 From: dms at danplanet.com (Dan Smith) Date: Wed, 08 Jun 2022 07:43:59 -0700 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: (Julia Kreger's message of "Wed, 8 Jun 2022 07:19:43 -0700") References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Message-ID: Julia Kreger writes: > Is that Nova's interpretation, specifically the delineation that > non-project owned should only be viewable by system, or was system > scope changed at some point? I interpreted it differently, but haven't > circled back recently. I guess interpretation and evolution in > specific pockets after initial implementation work started ultimately > resulted in different perceptions. Nope, not a Nova thing. Here's the relevant course correction from two PTGs ago: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#direction-change Mohammed is going to be there and primed to discuss this as well. I think he's pretty well caught up on the current state of things. Having your experience with what it means in Ironic, as well as his context from the sticky implementation issues in the other projects should mean we have pretty good coverage. Thanks! --Dan From smooney at redhat.com Wed Jun 8 14:48:04 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 08 Jun 2022 15:48:04 +0100 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Message-ID: <913612422bb6844e05dfee31dc3caab60f71200a.camel@redhat.com> On Wed, 2022-06-08 at 07:19 -0700, Julia Kreger wrote: > Is that Nova's interpretation, specifically the delineation that > non-project owned should only be viewable by system, or was system > scope changed at some point? > no its not just the nova interpreation. there was some differnt view in differnt porjects but we did an openstack wide reset on this in yoga and on the defintions of the personas when it became apprent we coudl not proceed with the previous approch. > I interpreted it differently, but haven't > circled back recently. I guess interpretation and evolution in > specific pockets after initial implementation work started ultimately > resulted in different perceptions. > > I'll make sure operators are aware there may be significant nuances > and perception differences since they are using their own etherpads > and their own communications flow. i.e. we're unlikely to see many > find/use our own etherpads as they have their own. And yes, this is a > problem, but it is a higher level community/communications feedback > issue under active discussion. well it was under active discussion for a few cycle both at the ptgs and mailing lists, and common understaind was codeified as part of a comunity wide goal https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html with pahses agree to be implemnted across all project based on the updated common understanding of the personas and definitons to ensure that seperate interpreation and perspective nolonger differed. the direction change and rational are documend in the goal document https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#direction-change with the high level roadmap cpature in teh pahse 1,2 and 3 defintions. but in general we woudl like to get feedback from operatros if what is being propsoed in that document is useful. As dan noted the scope enforcment aspect is proving a challange for higher level porject like heat and tacker to support. the is more agreement on the usfullnes of a standard reader role, service role and perhaps manager role as thos standarised role dont really break exising usage but scope enforcement is a big change. so the dicusion of RBAC is not finalised but a direction has been chosen and there is a certin amount of moment behind it now. parts of the design can still change an evlonve but other parts are more concrate like the reader role. > > Granted, I get that the system scope ideas were breaking for some > projects in specific use patterns since not everything would be the > same nor possible (which is actually a good thing, context of use and > all), but it was in theory perfect for a lot of the external audit > tooling use cases which arise in so many different ways. > > Anyway, off to the next $thing with my scattered brain. > > On Wed, Jun 8, 2022 at 6:53 AM Dan Smith wrote: > > > > > the system level of scope does not allow you to see everything across the system > > > it only allows you to see the non project related resouces > > > > > > so you can see the flavors and host aggreates but not the instances as instances are project scoped. > > > and project scoped resouces like ports, instances, images and volumes cannot be accessed with a system scope > > > token if you enabel scope enforcement. > > > > > > that is one of the things we want to get clarity on form operators. > > > is the disticntion between system level resouces and project level resouces useful. > > > > Yep, exactly this. Given the amount of breakage it brings for things > > like Heat and Tacker, as well as the potential workflow annoyance for > > human admins, I really want to measure whether any operators see a > > benefit here. The persona roles, things like a standardized service > > role, and getting out of this current situation of having two sets of > > defaults are priorities for me. > > > > --Dan > From rlandy at redhat.com Wed Jun 8 15:13:33 2022 From: rlandy at redhat.com (Ronelle Landy) Date: Wed, 8 Jun 2022 11:13:33 -0400 Subject: [tripleo] gate blocker - master/wallaby - "SELinux boolean os_enable_vtpm does not exist." In-Reply-To: References: Message-ID: On Tue, Jun 7, 2022 at 3:53 PM Ronelle Landy wrote: > > > On Tue, Jun 7, 2022 at 1:08 PM Ronelle Landy wrote: > >> Hello All, >> >> We have a consistent failure on multiple jobs running on >> check/gate/integration lines on master and wallaby changes - which shows up >> as a deployment error: >> >> FATAL | Enable os_enable_vtpm SELinux boolean for vTPM | standalone >> | error={"changed": false, "msg": "SELinux boolean os_enable_vtpm does not >> exist."} >> >> Details of the failure are in: >> https://bugs.launchpad.net/tripleo/+bug/1977873 >> >> We are investigating a fix/workaround. >> >> Please hold rechecks - we will post out again with updates. >> >> Thank you! >> >> > A fix is in progress - thank you Rafael and Lon: > > https://bugs.launchpad.net/tripleo/+bug/1977873/comments/7 > https://bugs.launchpad.net/tripleo/+bug/1977873/comments/8 > > Updated linselinux versions are available in centos-9 production composes: libselinux-3.4-2.el9.i686.rpm 2022-06-07 12:12 93K libselinux-3.4-2.el9.x86_64.rpm 2022-06-07 12:12 86K https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/?C=M;O=A Waiting on mirrors to get this update before closing this out. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Wed Jun 8 16:49:57 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 08 Jun 2022 11:49:57 -0500 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> Message-ID: <18144390aeb.f7aa2f0856684.396006809241741590@ghanshyammann.com> ---- On Wed, 08 Jun 2022 09:43:59 -0500 Dan Smith wrote ---- > Julia Kreger writes: > > > Is that Nova's interpretation, specifically the delineation that > > non-project owned should only be viewable by system, or was system > > scope changed at some point? I interpreted it differently, but haven't > > circled back recently. I guess interpretation and evolution in > > specific pockets after initial implementation work started ultimately > > resulted in different perceptions. > > Nope, not a Nova thing. Here's the relevant course correction from two > PTGs ago: > > https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#direction-change > > Mohammed is going to be there and primed to discuss this as well. I > think he's pretty well caught up on the current state of things. Having > your experience with what it means in Ironic, as well as his context > from the sticky implementation issues in the other projects should mean > we have pretty good coverage. Yes. and it is more than just a single service use case especially when heat discussion[1] came up and the scope complexity for heat/NVF users is brought up. We want to make sure by introducing scope at the service level which is all good for us does not break others users/tooling like heat, tacker, and deployment projects. We discussed one solution for heat[2] which is sent on ML for feedback not still now response and that is why operators' feedback is critical before we try to implement something that can break them. [1] https://etherpad.opendev.org/p/rbac-zed-ptg#L104 [2] http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028490.html -gmann > > Thanks! > > --Dan > > From gmann at ghanshyammann.com Wed Jun 8 16:53:03 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 08 Jun 2022 11:53:03 -0500 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: <913612422bb6844e05dfee31dc3caab60f71200a.camel@redhat.com> References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> <913612422bb6844e05dfee31dc3caab60f71200a.camel@redhat.com> Message-ID: <181443bdf1f.12767362256831.7510285921554510482@ghanshyammann.com> ---- On Wed, 08 Jun 2022 09:48:04 -0500 Sean Mooney wrote ---- > On Wed, 2022-06-08 at 07:19 -0700, Julia Kreger wrote: > > Is that Nova's interpretation, specifically the delineation that > > non-project owned should only be viewable by system, or was system > > scope changed at some point? > > > no its not just the nova interpreation. there was some differnt view in differnt porjects but > we did an openstack wide reset on this in yoga and on the defintions of the personas > when it became apprent we coudl not proceed with the previous approch. > > > I interpreted it differently, but haven't > > circled back recently. I guess interpretation and evolution in > > specific pockets after initial implementation work started ultimately > > resulted in different perceptions. > > > > I'll make sure operators are aware there may be significant nuances > > and perception differences since they are using their own etherpads > > and their own communications flow. i.e. we're unlikely to see many > > find/use our own etherpads as they have their own. And yes, this is a > > problem, but it is a higher level community/communications feedback > > issue under active discussion. > well it was under active discussion for a few cycle both at the ptgs and mailing lists, and common understaind was codeified as > part of a comunity wide goal https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html > with pahses agree to be implemnted across all project based on the updated common understanding of the personas and definitons > to ensure that seperate interpreation and perspective nolonger differed. > > the direction change and rational are documend in the goal document > https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#direction-change > > with the high level roadmap cpature in teh pahse 1,2 and 3 defintions. > > but in general we woudl like to get feedback from operatros if what is being propsoed in that document is useful. > As dan noted the scope enforcment aspect is proving a challange for higher level porject like heat and tacker to support. > the is more agreement on the usfullnes of a standard reader role, service role and perhaps manager role as thos standarised > role dont really break exising usage but scope enforcement is a big change. > > so the dicusion of RBAC is not finalised but a direction has been chosen and there is a certin amount of moment behind it now. > parts of the design can still change an evlonve but other parts are more concrate like the reader role. True, our overall goal is to ship improvement without breaking any use case. For example, 'scope' can be good for just nova admin/users but from heat, tacker users these might be breaking change more than improvement so we want to make sure what operators think by considering all use cases. -gmann > > > > Granted, I get that the system scope ideas were breaking for some > > projects in specific use patterns since not everything would be the > > same nor possible (which is actually a good thing, context of use and > > all), but it was in theory perfect for a lot of the external audit > > tooling use cases which arise in so many different ways. > > > > Anyway, off to the next $thing with my scattered brain. > > > > On Wed, Jun 8, 2022 at 6:53 AM Dan Smith wrote: > > > > > > > the system level of scope does not allow you to see everything across the system > > > > it only allows you to see the non project related resouces > > > > > > > > so you can see the flavors and host aggreates but not the instances as instances are project scoped. > > > > and project scoped resouces like ports, instances, images and volumes cannot be accessed with a system scope > > > > token if you enabel scope enforcement. > > > > > > > > that is one of the things we want to get clarity on form operators. > > > > is the disticntion between system level resouces and project level resouces useful. > > > > > > Yep, exactly this. Given the amount of breakage it brings for things > > > like Heat and Tacker, as well as the potential workflow annoyance for > > > human admins, I really want to measure whether any operators see a > > > benefit here. The persona roles, things like a standardized service > > > role, and getting out of this current situation of having two sets of > > > defaults are priorities for me. > > > > > > --Dan > > > > > From james.slagle at gmail.com Wed Jun 8 19:35:55 2022 From: james.slagle at gmail.com (James Slagle) Date: Wed, 8 Jun 2022 15:35:55 -0400 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami Message-ID: I'm formally proposing that we add Brendan Shephard and Takashi Kajinami to the core reviewers team for TripleO. Takashi is already considered core on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and likewise for Brendan. They both have been contributing and reviewing for a little while now, and I feel it's time to add them to the team. Please +1 or let me know any concerns before Friday June 15th. Thanks! -- -- James Slagle -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From hjensas at redhat.com Wed Jun 8 19:47:20 2022 From: hjensas at redhat.com (Harald Jensas) Date: Wed, 8 Jun 2022 21:47:20 +0200 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: On Wed, 8 Jun 2022, 21:36 James Slagle, wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered core > on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and > likewise for Brendan. > > They both have been contributing and reviewing for a little while now, and > I feel it's time to add them to the team. Please +1 or let me know any > concerns before Friday June 15th. Thanks! > > -- > -- James Slagle > -- > +1 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abishop at redhat.com Wed Jun 8 19:56:19 2022 From: abishop at redhat.com (Alan Bishop) Date: Wed, 8 Jun 2022 12:56:19 -0700 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: On Wed, Jun 8, 2022 at 12:38 PM James Slagle wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered core > on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and > likewise for Brendan. > +1 for both of them! > They both have been contributing and reviewing for a little while now, and > I feel it's time to add them to the team. Please +1 or let me know any > concerns before Friday June 15th. Thanks! > > -- > -- James Slagle > -- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbaker at redhat.com Wed Jun 8 21:09:37 2022 From: sbaker at redhat.com (Steve Baker) Date: Thu, 9 Jun 2022 09:09:37 +1200 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: +1! On 9/06/22 07:35, James Slagle wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi > Kajinami to the core reviewers team for TripleO. Takashi is already > considered core on puppet-tripleo, and I'm proposing we expand that to > all of TripleO, and likewise for Brendan. > > They both have been contributing and reviewing for a little while now, > and I feel it's time to add them to the team. Please +1 or let me know > any concerns? before Friday June 15th. Thanks! > > -- > -- James Slagle > -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From johfulto at redhat.com Wed Jun 8 22:29:49 2022 From: johfulto at redhat.com (John Fulton) Date: Wed, 8 Jun 2022 18:29:49 -0400 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: +1 On Wed, Jun 8, 2022, 5:11 PM Steve Baker wrote: > +1! > On 9/06/22 07:35, James Slagle wrote: > > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered core > on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and > likewise for Brendan. > > They both have been contributing and reviewing for a little while now, and > I feel it's time to add them to the team. Please +1 or let me know any > concerns before Friday June 15th. Thanks! > > -- > -- James Slagle > -- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramishra at redhat.com Thu Jun 9 02:27:49 2022 From: ramishra at redhat.com (Rabi Mishra) Date: Thu, 9 Jun 2022 07:57:49 +0530 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: On Thu, Jun 9, 2022 at 1:08 AM James Slagle wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered core > on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and > likewise for Brendan. > > They both have been contributing and reviewing for a little while now, and > I feel it's time to add them to the team. Please +1 or let me know any > concerns before Friday June 15th. Thanks! > +1! > -- > -- James Slagle > -- > -- Regards, Rabi Mishra -------------- next part -------------- An HTML attachment was scrubbed... URL: From ueha.ayumu at fujitsu.com Thu Jun 9 05:14:15 2022 From: ueha.ayumu at fujitsu.com (ueha.ayumu at fujitsu.com) Date: Thu, 9 Jun 2022 05:14:15 +0000 Subject: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team In-Reply-To: References: Message-ID: Hi Rico, This is just a friendly reminder that we are waiting for your action. Or please let us know if there are any other necessary processes. Thanks. Best Regards, Ueha -----Original Message----- From: ueha.ayumu at fujitsu.com Sent: Thursday, June 2, 2022 5:39 PM To: openstack-discuss at lists.openstack.org; 'Rico Lin' Subject: RE: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team Hi Rico, This proposal is great helpful for us! There seems to be no particular objection, so could you add it to heat-translator-core if there is no problem? Thanks. Best Regards, Ueha -----Original Message----- From: ueha.ayumu at fujitsu.com Sent: Wednesday, May 25, 2022 1:40 PM To: openstack-discuss at lists.openstack.org Subject: RE: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team +1 Best Regards, Ueha -----Original Message----- From: Yoshito Ito Sent: Wednesday, May 25, 2022 12:31 PM To: openstack-discuss at lists.openstack.org Subject: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team Hi heat-translator team! I would like to propose Hiromu Asahina (h-asahina) as a member of the heat-translator core team. I'm now not able to actively contribute to heat-translator, and he can lead the team instead of me. He is mainly contributing to Tacker by fixing issues, refactoring existing codes, and a lot of helpful reviews, with his experience as a research engineer in NTT. Tacker highly depends on heat-translator in its main functionality, so I believe he can manage it with his knowledge of Tacker. Please vote for his nomination by answering this mail till June 1st. Best Regards, Yoshito Ito From marios at redhat.com Thu Jun 9 05:15:26 2022 From: marios at redhat.com (Marios Andreou) Date: Thu, 9 Jun 2022 08:15:26 +0300 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: On Wednesday, June 8, 2022, James Slagle wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered core > on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and > likewise for Brendan. > > +1 and +1 easily ;) > They both have been contributing and reviewing for a little while now, and > I feel it's time to add them to the team. Please +1 or let me know any > concerns before Friday June 15th. Thanks! > > -- > -- James Slagle > -- > -- _sent from my mobile - sorry for spacing spelling etc_ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.orange at pawsey.org.au Thu Jun 9 08:10:49 2022 From: gregory.orange at pawsey.org.au (Gregory Orange) Date: Thu, 9 Jun 2022 16:10:49 +0800 Subject: [kolla] restart mariadb container on host 1 Message-ID: Hi all, I wanted to temporarily change a mariadb config in /etc/kolla/mariadb/galera.cnf so I intended to stop mariadb on host 1, make the change, start it again, then repeat in turn for hosts 2 and 3. Upon starting it on host 1, it doesn't rejoin, but creates its own separate cluster. Dutiful really, since logs show: WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster How do I avoid that? General advice on handling this kind of activity would also be welcome. Thank you, Greg. From juliaashleykreger at gmail.com Thu Jun 9 08:41:09 2022 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Thu, 9 Jun 2022 01:41:09 -0700 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: <18144390aeb.f7aa2f0856684.396006809241741590@ghanshyammann.com> References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> <18144390aeb.f7aa2f0856684.396006809241741590@ghanshyammann.com> Message-ID: On Wed, Jun 8, 2022 at 9:50 AM Ghanshyam Mann wrote: > > ---- On Wed, 08 Jun 2022 09:43:59 -0500 Dan Smith wrote ---- > > Julia Kreger writes: > > > > > Is that Nova's interpretation, specifically the delineation that > > > non-project owned should only be viewable by system, or was system > > > scope changed at some point? I interpreted it differently, but haven't > > > circled back recently. I guess interpretation and evolution in > > > specific pockets after initial implementation work started ultimately > > > resulted in different perceptions. > > > > Nope, not a Nova thing. Here's the relevant course correction from two > > PTGs ago: > > > > https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#direction-change Thanks Sean! Now I'm remembering! This is what happens when we implement things and then change directions and already have some variety of models. I guess I mentally dropped some of the context due to this after reading it since it was already "done" and committed from my perspective. Thanks for the refresher! > > > > Mohammed is going to be there and primed to discuss this as well. I > > think he's pretty well caught up on the current state of things. Having > > your experience with what it means in Ironic, as well as his context > > from the sticky implementation issues in the other projects should mean > > we have pretty good coverage. > > Yes. and it is more than just a single service use case especially when heat discussion[1] > came up and the scope complexity for heat/NVF users is brought up. We want to make > sure by introducing scope at the service level which is all good for us does not break > others users/tooling like heat, tacker, and deployment projects. > > We discussed one solution for heat[2] which is sent on ML for feedback not still now response and that > is why operators' feedback is critical before we try to implement something that can break them. > Thanks! I think we will try to remember to bring this up, but we also need to let the operators drive based upon their wants and needs. Operators tend to want to focus on the nuts and bolts of older software, so hopefully we will be able to move the discussion forward. There is entirely the possibility they just won't want to discuss it because it is well over the horizon for them. Semi-on topic, but also on topic to the base different topic: How many people would be interested in actually trying to meet up with operators on some semi-regular basis? Having a past operations background would be a major bonus point because the same words would be able to be coalesced far faster. Additionally, travel? How far? etc? I'm feeling like we need to build a "bridging" group of some sort. There are a ton of threads I'm on on trying to help bridge some of these communication gaps right now. > [1] https://etherpad.opendev.org/p/rbac-zed-ptg#L104 > [2] http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028490.html > > -gmann > > > > > Thanks! > > > > --Dan > > > > From mark at stackhpc.com Thu Jun 9 08:50:22 2022 From: mark at stackhpc.com (Mark Goddard) Date: Thu, 9 Jun 2022 09:50:22 +0100 Subject: [kolla-ansible][kolla] merge_configs module replacing colon to equal In-Reply-To: References: Message-ID: Hi Sharath, I had not realised this. Wikipedia suggests colons are sometimes used instead of equals. Perhaps the function should return which delimiter was found, and use it to generate the merged file? Mark On Tue, 7 Jun 2022 at 06:19, Sharath Ck wrote: > Hi, > > In kolla-ansible, I am using the merge_configs module to merge service > configs files. I want to merge api-paste.ini config with my custom changes, > however while merging if a line consists of one colon, it is getting > replaced with equal sign. merge_configs splits the line by considering > colon as delimiter and treats two parts as key and value. Later which will > be substituted as key = value. > > Original > */v2.1: oscomputeversion_v2* > Merged > */v2.1= oscomputeversion_v2 * > > Below, colon will be 1 and equal will be -1, > > > > > > > > > > > > > > > > *def _split_key_value(self, line): colon = line.find(':') > equal = line.find('=') if colon < 0 and equal < 0: return > self.error_invalid_assignment(line) if colon < 0 or (equal >= 0 and > equal < colon): key, value = line[:equal], line[equal + 1:] > else: key, value = line[:colon], line[colon + 1:] value > = value.strip() if value and value[0] == value[-1] and > value.startswith(("\"", "'")): value = value[1:-1] return > key.strip(), [value]* > > I want to keep the config as it is and merge only my custom changes. > Please let me know how to achieve this or colon can be escaped somehow. > > Regards, > Sharath > Regards, > Sharath > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Thu Jun 9 08:53:52 2022 From: mark at stackhpc.com (Mark Goddard) Date: Thu, 9 Jun 2022 09:53:52 +0100 Subject: [kolla] restart mariadb container on host 1 In-Reply-To: References: Message-ID: Hi Greg, It should only have that option set when bootstrapping the cluster. How are you stopping and starting the container? Could you provide logs? Mark On Thu, 9 Jun 2022 at 09:17, Gregory Orange wrote: > Hi all, > > I wanted to temporarily change a mariadb config in > /etc/kolla/mariadb/galera.cnf so I intended to stop mariadb on host 1, > make the change, start it again, then repeat in turn for hosts 2 and 3. > > Upon starting it on host 1, it doesn't rejoin, but creates its own > separate cluster. Dutiful really, since logs show: > > WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster > > How do I avoid that? General advice on handling this kind of activity > would also be welcome. > > Thank you, > Greg. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Thu Jun 9 08:55:27 2022 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Thu, 9 Jun 2022 01:55:27 -0700 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: <181443bdf1f.12767362256831.7510285921554510482@ghanshyammann.com> References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> <913612422bb6844e05dfee31dc3caab60f71200a.camel@redhat.com> <181443bdf1f.12767362256831.7510285921554510482@ghanshyammann.com> Message-ID: On Wed, Jun 8, 2022 at 9:53 AM Ghanshyam Mann wrote: > > ---- On Wed, 08 Jun 2022 09:48:04 -0500 Sean Mooney wrote ---- [trim] > > so the dicusion of RBAC is not finalised but a direction has been chosen and there is a certin amount of moment behind it now. > > parts of the design can still change an evlonve but other parts are more concrate like the reader role. > > True, our overall goal is to ship improvement without breaking any use case. For example, 'scope' can be good > for just nova admin/users but from heat, tacker users these might be breaking change more than improvement > so we want to make sure what operators think by considering all use cases. > > -gmann So, one thought. Ironic views system scope as *critical* for our usage based upon the consensus we built before the direction change, because the system fundamentally is the owner/manager of $things. We can and likely should extend that out to project admin (granted, I suspect at least one ironic admin will reply with a strong -1 to such a change... :\. ) given the direction change. We also have had some operators jump on it, but... again, entirely different models of usage/interaction given the base state. If system scope were to suddenly disappear or be completely redefined, it would be a hard break for us at this point. > > > > > > > Granted, I get that the system scope ideas were breaking for some > > > projects in specific use patterns since not everything would be the > > > same nor possible (which is actually a good thing, context of use and > > > all), but it was in theory perfect for a lot of the external audit > > > tooling use cases which arise in so many different ways. > > > > > > Anyway, off to the next $thing with my scattered brain. > > > > > > On Wed, Jun 8, 2022 at 6:53 AM Dan Smith wrote: > > > > > > > > > the system level of scope does not allow you to see everything across the system > > > > > it only allows you to see the non project related resouces > > > > > > > > > > so you can see the flavors and host aggreates but not the instances as instances are project scoped. > > > > > and project scoped resouces like ports, instances, images and volumes cannot be accessed with a system scope > > > > > token if you enabel scope enforcement. > > > > > > > > > > that is one of the things we want to get clarity on form operators. > > > > > is the disticntion between system level resouces and project level resouces useful. > > > > > > > > Yep, exactly this. Given the amount of breakage it brings for things > > > > like Heat and Tacker, as well as the potential workflow annoyance for > > > > human admins, I really want to measure whether any operators see a > > > > benefit here. The persona roles, things like a standardized service > > > > role, and getting out of this current situation of having two sets of > > > > defaults are priorities for me. > > > > > > > > --Dan > > > > > > > > > From sharath.madhava at gmail.com Thu Jun 9 08:57:03 2022 From: sharath.madhava at gmail.com (Sharath Ck) Date: Thu, 9 Jun 2022 14:27:03 +0530 Subject: [kolla-ansible][kolla] merge_configs module replacing colon to equal In-Reply-To: References: Message-ID: Hi Mark, Yes, that would be correct. If the function returns the delimiter and uses the same while mapping key value, it would keep the config content intact. If colon can be replaced by equal and vice versa in openstack configs, especially api-paste.ini. I can continue with the merge now and update the merge_configs module when a fix is available. Regards, Sharath On Thu, Jun 9, 2022 at 2:20 PM Mark Goddard wrote: > Hi Sharath, > > I had not realised this. Wikipedia suggests colons are sometimes used > instead of equals. Perhaps the function should return which delimiter was > found, and use it to generate the merged file? > > Mark > > On Tue, 7 Jun 2022 at 06:19, Sharath Ck wrote: > >> Hi, >> >> In kolla-ansible, I am using the merge_configs module to merge service >> configs files. I want to merge api-paste.ini config with my custom changes, >> however while merging if a line consists of one colon, it is getting >> replaced with equal sign. merge_configs splits the line by considering >> colon as delimiter and treats two parts as key and value. Later which will >> be substituted as key = value. >> >> Original >> */v2.1: oscomputeversion_v2* >> Merged >> */v2.1= oscomputeversion_v2 * >> >> Below, colon will be 1 and equal will be -1, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> *def _split_key_value(self, line): colon = line.find(':') >> equal = line.find('=') if colon < 0 and equal < 0: return >> self.error_invalid_assignment(line) if colon < 0 or (equal >= 0 and >> equal < colon): key, value = line[:equal], line[equal + 1:] >> else: key, value = line[:colon], line[colon + 1:] value >> = value.strip() if value and value[0] == value[-1] and >> value.startswith(("\"", "'")): value = value[1:-1] return >> key.strip(), [value]* >> >> I want to keep the config as it is and merge only my custom changes. >> Please let me know how to achieve this or colon can be escaped somehow. >> >> Regards, >> Sharath >> Regards, >> Sharath >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjeanner at redhat.com Thu Jun 9 09:02:37 2022 From: cjeanner at redhat.com (=?UTF-8?Q?C=c3=a9dric_Jeanneret?=) Date: Thu, 9 Jun 2022 11:02:37 +0200 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: <6c26f05b-1814-fb7d-7a85-8070dd9b7b24@redhat.com> Of course +42 !! On 6/8/22 21:35, James Slagle wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered > core on puppet-tripleo, and I'm proposing we expand that to all of > TripleO, and likewise for Brendan. > > They both have been contributing and reviewing for a little while now, > and I feel it's time to add them to the team. Please +1 or let me know > any concerns? before Friday June 15th. Thanks! > > -- > -- James Slagle > -- -- C?dric Jeanneret (He/Him/His) Sr. Software Engineer - OpenStack Platform Deployment Framework TC Red Hat EMEA https://www.redhat.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From christian.rohmann at inovex.de Thu Jun 9 09:11:46 2022 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Thu, 9 Jun 2022 11:11:46 +0200 Subject: [rbac][nova][cinder] Canned roles for service users / inter-service communication (e.g. event submission) Message-ID: Hey openstack-discuss, I posted to the ML quite a while ago about an issue of resized (Cinder) volumes not being propagated to the (Nova) instance. See http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020476.html. The issue there was Cinder being not allowed to send the "volume-extended" event (or any event for that matter via the Nova API just using the user token. For this a configurable additional "privileged user" was added to the config quite a while back with https://opendev.org/openstack/cinder/commit/04003d7c513ed4dd5129cbd5ad1af14a5b200677. While the functionality then works I suppose there should be canned and maintained RBAC roles for such kind of inter service to service communications, e.g. to emit events to others. Otherwise deployments likely will use admin privileged users ignoring the least privilege principle and creating an large attack surface. And there are quite few of those relations even with the most commonly used services. Cinder -> nova, nova-> cincer, nova->ironic, .... nova-> neutron, .... Are such canned RBAC rules for "special" inter service users on the backlog somewhere? Or am I totally misconceiving the issue here? I know there is https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#isolate-system-specific-api-policies and also the question for feedback at https://etherpad.opendev.org/p/rbac-operator-feedback, but that all seems to focus on the impact of roles used by humans / users and not about service roles at all. Regards Christian From christian.rohmann at inovex.de Thu Jun 9 10:39:44 2022 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Thu, 9 Jun 2022 12:39:44 +0200 Subject: [rbac][nova][cinder] Canned roles for service users / inter-service communication (e.g. event submission) In-Reply-To: References: Message-ID: <26833ce2-e486-d953-b7ea-999d94862631@inovex.de> On 09/06/2022 11:11, Christian Rohmann wrote: > And there are quite few of those relations even with the most commonly > used services. > Cinder -> nova, nova-> cincer, nova->ironic, .... nova-> neutron, .... > > Are such canned RBAC rules for "special" inter service users on the > backlog somewhere? Or am I totally misconceiving the issue here? > > I know there is > https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#isolate-system-specific-api-policies > and also the question for feedback at > https://etherpad.opendev.org/p/rbac-operator-feedback, but that all > seems to focus on the impact of roles used by humans / users and not > about service roles at all. I just noticed that Christian Berendt does a forum talk on "Deprivilization of the internal service accounts" TODAY at 2:40pm - 3:10pm at A05 on apparently that exact question :-) Regards Christian From Russell.Stather at ignitiongroup.co.za Thu Jun 9 10:41:10 2022 From: Russell.Stather at ignitiongroup.co.za (Russell Stather) Date: Thu, 9 Jun 2022 10:41:10 +0000 Subject: Error creating octavia amphora Message-ID: Hi I am seeing the below error when creating a load balancer. The security group does exist, and it is the correct group (tagged with charm-octavia) What am I missing here to resolve this? 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2022-06-09T10:09:59Z', 'message': 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 654, in build_instances\n scheduler_utils.populate_retry(\n File "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 989, in populate_retry\n raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.\n'} 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.page at canonical.com Thu Jun 9 10:53:30 2022 From: james.page at canonical.com (James Page) Date: Thu, 9 Jun 2022 12:53:30 +0200 Subject: [charms][octavia] Re: Error creating octavia amphora In-Reply-To: References: Message-ID: Hi Russell On Thu, Jun 9, 2022 at 12:44 PM Russell Stather < Russell.Stather at ignitiongroup.co.za> wrote: > Hi > > I am seeing the below error when creating a load balancer. The security > group does exist, and it is the correct group (tagged with charm-octavia) > > What am I missing here to resolve this? > I'm wondering whether the configured security group is not owned by the right project - how did you create the required network and security configuration for Octavia? > > 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server > octavia.common.exceptions.ComputeBuildException: Failed to build compute > instance due to: {'code': 500, 'created': '2022-06-09T10:09:59Z', > 'message': 'Exceeded maximum number of retries. Exceeded max scheduling > attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last > exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.', > 'details': 'Traceback (most recent call last):\n File > "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 654, in > build_instances\n scheduler_utils.populate_retry(\n File > "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 989, in > populate_retry\n raise > exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: > Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for > instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security > group 0b683c75-d900-4e45-acb2-8bc321580666 not found.\n'} > 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bshephar at redhat.com Thu Jun 9 11:16:30 2022 From: bshephar at redhat.com (Brendan Shephard) Date: Thu, 9 Jun 2022 21:16:30 +1000 Subject: Error creating octavia amphora In-Reply-To: References: Message-ID: Hey Russell, Are you able to share the outputs from: $ openstack server show 524ae27b-1542-4c2d-9118-138d9e7f3770 -c id -c project_id -c security_groups -c status -f yaml And: $ openstack security group show0b683c75-d900-4e45-acb2-8bc321580666 -c id -c project_id -f yaml I agree with James, my assumption would be that we will find they aren't in the same project, so Nova can't use that security group for the Amphorae. I'm not familiar with charmed openstack though, but it does look like James is from Canonical and might be able to advise on the specifics. All the best, Brendan Shephard Software Engineer Red Hat APAC 193 N Quay Brisbane City QLD 4000 @RedHat Red Hat Red Hat On Thu, Jun 9, 2022 at 8:48 PM Russell Stather < Russell.Stather at ignitiongroup.co.za> wrote: > Hi > > I am seeing the below error when creating a load balancer. The security > group does exist, and it is the correct group (tagged with charm-octavia) > > What am I missing here to resolve this? > > 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server > octavia.common.exceptions.ComputeBuildException: Failed to build compute > instance due to: {'code': 500, 'created': '2022-06-09T10:09:59Z', > 'message': 'Exceeded maximum number of retries. Exceeded max scheduling > attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last > exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.', > 'details': 'Traceback (most recent call last):\n File > "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 654, in > build_instances\n scheduler_utils.populate_retry(\n File > "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 989, in > populate_retry\n raise > exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: > Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for > instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security > group 0b683c75-d900-4e45-acb2-8bc321580666 not found.\n'} > 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Jun 9 11:24:10 2022 From: smooney at redhat.com (Sean Mooney) Date: Thu, 09 Jun 2022 12:24:10 +0100 Subject: [kolla-ansible][kolla] merge_configs module replacing colon to equal In-Reply-To: References: Message-ID: <042a070e63901559e99fddf3a9a45724f88d82d0.camel@redhat.com> On Thu, 2022-06-09 at 14:27 +0530, Sharath Ck wrote: > Hi Mark, > > Yes, that would be correct. If the function returns the delimiter and uses > the same while mapping key value, it would keep the config content intact. > > If colon can be replaced by equal and vice versa in openstack configs, > especially api-paste.ini. I can continue with the merge now and update the > merge_configs module when a fix is available. just not that openstack for most file does not use ini format. it juse conf format or strictly speaking the oslo config format but the openstack service dont actully use ini unless we are using lib that have config that we need to generate. so not evenything that is vaild in conf format is vlaid in ini format and visversa for example ini formant defient that if repeate a key the last value is used with oslo config if the key is defiend as a multiopt type repeated values are formed into a list. so? key=val1 key=val2 in ini format result in key=val2 when parsed but in conforma that can be intpertated as k=[val1,val2] the merge_configs module is based on pythons config parser which is inteded for ini files but we use them for conf files. api-paste.ini is not form openstack it s a config for a lib we use so in principal it might support : instead of = but that replacemnt is not intended to be support in nova.conf for example which is not an ini file its a .conf or oslo conf file. kolla can extend merge_config to alow : for ini file if they want too but its not a bug if you try to use : in nova.conf and it does not work. we dont support that for openstack's config files intentinally. it may work because of the the code oslo.config is built on but that woudl not be intentinal as far as im aware and it might break. in this case you are modifying nova's api-paste.ini https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini specificaly https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini#L21-L22= im not entirely sure why we use : [composite:osapi_compute] use = call:nova.api.openstack.urlmap:urlmap_factory /: oscomputeversions /v2: oscomputeversion_legacy_v2 /v2.1: oscomputeversion_v2 # v21 is an exactly feature match for v2, except it has more stringent # input validation on the wsgi surface (prevents fuzzing early on the # API). It also provides new features via API microversions which are # opt into for clients. Unaware clients will receive the same frozen # v2 API feature set, but with some relaxed validation /v2/+: openstack_compute_api_v21_legacy_v2_compatible /v2.1/+: openstack_compute_api_v21 looking a the pastedeploy docs breifly https://docs.pylonsproject.org/projects/pastedeploy/en/latest/index.html#paste-composite-factory it seams to be using = and none of the test code used : so i dont think they inted to suport that either. it might be better for nova and other poject to not use : in the api-paste.ini as we seem to be in undefiend behavior land on this one. https://github.com/openstack/nova/blob/master/nova/api/openstack/urlmap.py implementes the factory in nova which delegets the path parsign to paste https://github.com/cdent/paste/blob/225c7ebf34a9c90435932b690967538493a943ce/paste/urlmap.py#L35-L74= [composite:osapi_compute] use = call:nova.api.openstack.urlmap:urlmap_factory / = oscomputeversions /v2 = oscomputeversion_legacy_v2 /v2.1 = oscomputeversion_v2 # v21 is an exactly feature match for v2, except it has more stringent # input validation on the wsgi surface (prevents fuzzing early on the # API). It also provides new features via API microversions which are # opt into for clients. Unaware clients will receive the same frozen # v2 API feature set, but with some relaxed validation /v2/+ = openstack_compute_api_v21_legacy_v2_compatible /v2.1/+ = openstack_compute_api_v21 i assume ^ does not work today? i have not tried it. by all means add supprot to kolla for this use case if you would like too but longter we proably shoudl remove the use of paste since it undeploy or at least consider a syntax that is tested by them. if this syntax is somethign we added we do not appear to have any testing for it lookign at nova quickly. > > Regards, > Sharath > > > On Thu, Jun 9, 2022 at 2:20 PM Mark Goddard wrote: > > > Hi Sharath, > > > > I had not realised this. Wikipedia suggests colons are sometimes used > > instead of equals. Perhaps the function should return which delimiter was > > found, and use it to generate the merged file? > > > > Mark > > > > On Tue, 7 Jun 2022 at 06:19, Sharath Ck wrote: > > > > > Hi, > > > > > > In kolla-ansible, I am using the merge_configs module to merge service > > > configs files. I want to merge api-paste.ini config with my custom changes, > > > however while merging if a line consists of one colon, it is getting > > > replaced with equal sign. merge_configs splits the line by considering > > > colon as delimiter and treats two parts as key and value. Later which will > > > be substituted as key = value. > > > > > > Original > > > */v2.1: oscomputeversion_v2* > > > Merged > > > */v2.1= oscomputeversion_v2 * > > > > > > Below, colon will be 1 and equal will be -1, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *def _split_key_value(self, line): colon = line.find(':') > > > equal = line.find('=') if colon < 0 and equal < 0: return > > > self.error_invalid_assignment(line) if colon < 0 or (equal >= 0 and > > > equal < colon): key, value = line[:equal], line[equal + 1:] > > > else: key, value = line[:colon], line[colon + 1:] value > > > = value.strip() if value and value[0] == value[-1] and > > > value.startswith(("\"", "'")): value = value[1:-1] return > > > key.strip(), [value]* > > > > > > I want to keep the config as it is and merge only my custom changes. > > > Please let me know how to achieve this or colon can be escaped somehow. > > > > > > Regards, > > > Sharath > > > Regards, > > > Sharath > > > > > From smooney at redhat.com Thu Jun 9 11:28:54 2022 From: smooney at redhat.com (Sean Mooney) Date: Thu, 09 Jun 2022 12:28:54 +0100 Subject: [rbac][nova][cinder] Canned roles for service users / inter-service communication (e.g. event submission) In-Reply-To: References: Message-ID: On Thu, 2022-06-09 at 11:11 +0200, Christian Rohmann wrote: > Hey openstack-discuss, > > I posted to the ML quite a while ago about an issue of resized (Cinder) > volumes not being propagated to the (Nova) instance. > See > http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020476.html. > > The issue there was Cinder being not allowed to send the > "volume-extended" event (or any event for that matter via the Nova API > just using the user token. > For this a configurable additional "privileged user" was added to the > config quite a while back with > https://opendev.org/openstack/cinder/commit/04003d7c513ed4dd5129cbd5ad1af14a5b200677. > > While the functionality then works I suppose there should be canned and > maintained RBAC roles for such kind of inter service to service > communications, e.g. to emit events to others. Otherwise deployments > likely will use admin privileged users ignoring the least privilege > principle and creating an large attack surface. > > And there are quite few of those relations even with the most commonly > used services. > Cinder -> nova, nova-> cincer, nova->ironic, .... nova-> neutron, .... > > Are such canned RBAC rules for "special" inter service users on the > backlog somewhere? Or am I totally misconceiving the issue here? yes its called the service role its been discussed many times over the laste 3-4 virutal ptgs. its part of phase 2 https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#phase-2 """Isolate service-to-service APIs to the service role""" so the target is to start implementing the service role in AA and compelte it in all service by BB https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#aa-release-timeline > > I know there is > https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#isolate-system-specific-api-policies > and also the question for feedback at > https://etherpad.opendev.org/p/rbac-operator-feedback, but that all > seems to focus on the impact of roles used by humans / users and not > about service roles at all. > > > Regards > > Christian > > From tim+openstack.org at coote.org Thu Jun 9 11:43:39 2022 From: tim+openstack.org at coote.org (tim+openstack.org at coote.org) Date: Thu, 9 Jun 2022 12:43:39 +0100 Subject: simple build fails to allocate network interface In-Reply-To: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> References: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> Message-ID: <0044A33F-0E87-433C-8130-77C66D79926A@coote.org> On 9 Jun 2022, at 09:58, tim+openstack.org at coote.org wrote: Hullo I?m trying to spin up a simple OS environment in a single vm and think that I?m making some silly mistake, but cannot spot it. Any hints would be greatly expected. I?ve tried this on a couple of laptops, with the same result, so I suspect that it?s something to do with how I?ve set up my vm, but I cannot identify it. (I think that there may be a few other devstack issues, but I?ve got to get it working to log any bugs). I?m using a vagrant box and VirtualBox on x86 Macs (Monterey), and the devstack install/build process. Everything seems to install, but when I try: `openstack server create --flavor 42 --image cirros-0.5.2-x86_64-disk --nic net-id=2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 --security-group default wibble` The network id is identified from this: ??" [vagrant at localhost devstack]$ openstack network list +--------------------------------------+---------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+---------+----------------------------------------------------------------------------+ | 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f | public | 1bbbfa3c-8e0a-483f-a084-4e38241b3315, eecfc603-c057-4b28-90bd-950a47345410 | | 2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 | private | e7377bed-3e22-4e8d-9c2d-ea7ba740fcfd, f915b88c-9988-4e1f-9060-a6295465699a | | fab3fb15-cbd2-45fe-8451-3f0063d59454 | shared | 97e6d8e6-0f57-48bc-8f1d-e30587086153 | +--------------------------------------+---------+??????????????????????????????????????+ ??? Using a public network fails with an error that this isn?t allowed. With the private network above, the failure throws out the following errors. ??" [vagrant at localhost devstack]$ sudo journalctl -f |grep ERROR Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): [snip] ??" The Vagrantfile (commented out lines relate to trying this with Ubunutu, which failed the same way, and attempts to use multiple vms): ??" # -*- mode: ruby -*- # vi: set ft=ruby : [snip] Vagrant.configure(2) do |config| config.vm.box = "eurolinux-vagrant/centos-stream-9" #config.vm.box = "hashicorp/bionic64" config.hostmanager.enabled = true config.hostmanager.manage_host = true config.hostmanager.manage_guest = true config.hostmanager.ignore_private_ip = false config.hostmanager.include_offline = true config.ssh.pty = true config.vm.provision "shell", inline: $script config.vm.network "forwarded_port", guest: 80, host: 8080 config.vm.provider "virtualbox" do | v | v.memory = "9216" v.cpus = "2" end end ??? I think that the vm has enough RAM, although there is minimal swap being used, but I think that this is not significant as there is much more RAM used for caching files. Any obvious hints as to why the spin up fails to create the NIC - or somewhere to look for further and better details? Tim From gregory.orange at pawsey.org.au Thu Jun 9 11:45:20 2022 From: gregory.orange at pawsey.org.au (Gregory Orange) Date: Thu, 9 Jun 2022 19:45:20 +0800 Subject: [kolla] restart mariadb container on host 1 In-Reply-To: References: Message-ID: <55d9fd44-aaca-2bd7-adce-e3cda10140e0@pawsey.org.au> Right, thank you - I thought and hoped that was the intent. I'm just doing: docker stop mariadb docker start mariadb I've just done the same with host 3, and while stopped grastate.dat changed to `safe_to_bootstrap: 1` on host 2, and seqno: appeared in host 3's grastate.dat. Starting it again worked fine. docker inspect mariadb differs on the hosts, 1 has: "Env": [ ... "BOOTSTRAP_ARGS=--wsrep-new-cluster", while the other two don't. /var/log/kolla/mariadb/mariadb.log https://termbin.com/px15 with safe_to_bootstrap: 0, container stops again https://termbin.com/meli after removing /var/lib/docker/volumes/mariadb/_data/, separate cluster results I've just tested a dev cluster with the same steps, and it doesn't have the same problem - no Env entry, no problems restarting. I'm now wondering if it relates to the database migration and restore we did a while back, coming from another cluster with rsync replication and switching over to mariabackup. Next, I deleted the container in the dev cluster, and ran mariadb_recovery play - worked fine. I was hesitant to do this in the test cluster I'm working on because of other settings we have handcrafted for now, but after a few trial runs, I'm more confident. ... Yep, that fixed it - docker stop/start mariadb now works. BOOTSTRAP_ARGS is gone from the Env in the container too. Thank you for giving me the confidence to prod it further. Greg. On 9/6/22 16:53, Mark Goddard wrote: > Hi Greg, > > It should only have that option set when bootstrapping the cluster. How > are you stopping and starting the container? Could you provide logs? > > Mark > > On Thu, 9 Jun 2022 at 09:17, Gregory Orange > > wrote: > > Hi all, > > I wanted to temporarily change a mariadb config in > /etc/kolla/mariadb/galera.cnf so I intended to stop mariadb on host 1, > make the change, start it again, then repeat in turn for hosts 2 and 3. > > Upon starting it on host 1, it doesn't rejoin, but creates its own > separate cluster. Dutiful really, since logs show: > > WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster > > How do I avoid that? General advice on handling this kind of activity > would also be welcome. > > Thank you, > Greg. > -- Gregory Orange Cloud System Administrator Scientific Platforms Team building representative Pawsey Supercomputing Centre, CSIRO From ralonsoh at redhat.com Thu Jun 9 12:20:22 2022 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Thu, 9 Jun 2022 14:20:22 +0200 Subject: simple build fails to allocate network interface In-Reply-To: <0044A33F-0E87-433C-8130-77C66D79926A@coote.org> References: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> <0044A33F-0E87-433C-8130-77C66D79926A@coote.org> Message-ID: Hello: This seems to be a problem in Neutron. Please provide what backend you are using, and the Neutron server logs. If the ML2 driver has agents (DHCP, ML2 agent), provide the agents logs too. Regards. On Thu, Jun 9, 2022 at 1:53 PM wrote: > > > On 9 Jun 2022, at 09:58, tim+openstack.org at coote.org wrote: > > Hullo > > I?m trying to spin up a simple OS environment in a single vm and think > that I?m making some silly mistake, but cannot spot it. Any hints would be > greatly expected. I?ve tried this on a couple of laptops, with the same > result, so I suspect that it?s something to do with how I?ve set up my vm, > but I cannot identify it. (I think that there may be a few other devstack > issues, but I?ve got to get it working to log any bugs). > > I?m using a vagrant box and VirtualBox on x86 Macs (Monterey), and the > devstack install/build process. Everything seems to install, but when I try: > > `openstack server create --flavor 42 --image cirros-0.5.2-x86_64-disk > --nic net-id=2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 --security-group default > wibble` > > The network id is identified from this: > ??" > [vagrant at localhost devstack]$ openstack network list > > +--------------------------------------+---------+----------------------------------------------------------------------------+ > | ID | Name | Subnets > | > > +--------------------------------------+---------+----------------------------------------------------------------------------+ > | 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f | public | > 1bbbfa3c-8e0a-483f-a084-4e38241b3315, eecfc603-c057-4b28-90bd-950a47345410 | > | 2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 | private | > e7377bed-3e22-4e8d-9c2d-ea7ba740fcfd, f915b88c-9988-4e1f-9060-a6295465699a | > | fab3fb15-cbd2-45fe-8451-3f0063d59454 | shared | > 97e6d8e6-0f57-48bc-8f1d-e30587086153 | > > +--------------------------------------+---------+??????????????????????????????????????+ > ??? > > Using a public network fails with an error that this isn?t allowed. > > With the private network above, the failure throws out the following > errors. > ??" > [vagrant at localhost devstack]$ sudo journalctl -f |grep ERROR > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR > nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo > demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Instance failed to > spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface > creation failed > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR > nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] > Traceback (most recent call last): > > [snip] > ??" > > The Vagrantfile (commented out lines relate to trying this with Ubunutu, > which failed the same way, and attempts to use multiple vms): > ??" > # -*- mode: ruby -*- > # vi: set ft=ruby : > > [snip] > Vagrant.configure(2) do |config| > > config.vm.box = "eurolinux-vagrant/centos-stream-9" > #config.vm.box = "hashicorp/bionic64" > > > config.hostmanager.enabled = true > config.hostmanager.manage_host = true > config.hostmanager.manage_guest = true > config.hostmanager.ignore_private_ip = false > config.hostmanager.include_offline = true > > config.ssh.pty = true > > config.vm.provision "shell", inline: $script > > config.vm.network "forwarded_port", guest: 80, host: 8080 > > config.vm.provider "virtualbox" do | v | > v.memory = "9216" > v.cpus = "2" > end > > end > ??? > > I think that the vm has enough RAM, although there is minimal swap being > used, but I think that this is not significant as there is much more RAM > used for caching files. > > Any obvious hints as to why the spin up fails to create the NIC - or > somewhere to look for further and better details? > > Tim > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlandy at redhat.com Thu Jun 9 12:51:12 2022 From: rlandy at redhat.com (Ronelle Landy) Date: Thu, 9 Jun 2022 08:51:12 -0400 Subject: [tripleo] gate blocker - master/wallaby - "SELinux boolean os_enable_vtpm does not exist." In-Reply-To: References: Message-ID: On Wed, Jun 8, 2022 at 11:13 AM Ronelle Landy wrote: > > On Tue, Jun 7, 2022 at 3:53 PM Ronelle Landy wrote: > >> >> >> On Tue, Jun 7, 2022 at 1:08 PM Ronelle Landy wrote: >> >>> Hello All, >>> >>> We have a consistent failure on multiple jobs running on >>> check/gate/integration lines on master and wallaby changes - which shows up >>> as a deployment error: >>> >>> FATAL | Enable os_enable_vtpm SELinux boolean for vTPM | >>> standalone | error={"changed": false, "msg": "SELinux boolean >>> os_enable_vtpm does not exist."} >>> >>> Details of the failure are in: >>> https://bugs.launchpad.net/tripleo/+bug/1977873 >>> >>> We are investigating a fix/workaround. >>> >>> Please hold rechecks - we will post out again with updates. >>> >>> Thank you! >>> >>> >> A fix is in progress - thank you Rafael and Lon: >> >> https://bugs.launchpad.net/tripleo/+bug/1977873/comments/7 >> https://bugs.launchpad.net/tripleo/+bug/1977873/comments/8 >> >> > Updated linselinux versions are available in centos-9 production composes: > > libselinux-3.4-2.el9.i686.rpm 2022-06-07 12:12 93K libselinux-3.4-2.el9.x86_64.rpm 2022-06-07 12:12 86K > > > > https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/?C=M;O=A > > Waiting on mirrors to get this update before closing this out. > https://review.opendev.org/c/openstack/tripleo-quickstart/+/845229 merged as a workaround to unblcok most check/gate jobs on Master and Wallaby c9. Overcloud build image jobs are still failing at the moment - patches to address this are in progress -------------- next part -------------- An HTML attachment was scrubbed... URL: From dms at danplanet.com Thu Jun 9 13:30:08 2022 From: dms at danplanet.com (Dan Smith) Date: Thu, 09 Jun 2022 06:30:08 -0700 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: (Julia Kreger's message of "Thu, 9 Jun 2022 01:55:27 -0700") References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> <913612422bb6844e05dfee31dc3caab60f71200a.camel@redhat.com> <181443bdf1f.12767362256831.7510285921554510482@ghanshyammann.com> Message-ID: > So, one thought. Ironic views system scope as *critical* for our usage > based upon the consensus we built before the direction change, because > the system fundamentally is the owner/manager of $things. We can and > likely should extend that out to project admin (granted, I suspect at > least one ironic admin will reply with a strong -1 to such a change... > :\. ) given the direction change. We also have had some operators jump > on it, but... again, entirely different models of usage/interaction > given the base state. If system scope were to suddenly disappear or be > completely redefined, it would be a hard break for us at this point. I don't think system scope would (or could) disappear at this point, so I don't think there's much to worry about. I think it's totally reasonable to say that there are certain things that a user would never interact with directly, which are entirely system-scoped. This might be things like ironic and maybe even placement. You could also make the argument that a good chunk of keystone's API is system-only. If people are already using ironic with scopes turned on, it proves the point that it's isolated enough that it doesn't suffer from all the other problems that caused the direction change. --Dan From bdobreli at redhat.com Thu Jun 9 13:53:54 2022 From: bdobreli at redhat.com (Bogdan Dobrelya) Date: Thu, 9 Jun 2022 15:53:54 +0200 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: <29949013-7b00-daa3-f10a-be25a596c5b2@redhat.com> +1 On 6/8/22 21:35, James Slagle wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered > core on puppet-tripleo, and I'm proposing we expand that to all of > TripleO, and likewise for Brendan. > > They both have been contributing and reviewing for a little while now, > and I feel it's time to add them to the team. Please +1 or let me know > any concerns? before Friday June 15th. Thanks! > > -- > -- James Slagle > -- -- Best regards, Bogdan Dobrelya, Irc #bogdando From Russell.Stather at ignitiongroup.co.za Thu Jun 9 14:24:01 2022 From: Russell.Stather at ignitiongroup.co.za (Russell Stather) Date: Thu, 9 Jun 2022 14:24:01 +0000 Subject: Error creating octavia amphora In-Reply-To: References: Message-ID: Hi I've built new security groups to make sure they are in the correct project. I can start a new server manually using these security groups. The error is coming from the nova compute node. Same error can't find security group. I am creating the load balancer and specifying the project on the command line even. openstack loadbalancer create --name lb25 --vip-subnet-id admin_subnet1 --project 5e168b652a374a02aff855a5e250b7f8 Kind of running out of ideas. ________________________________ From: Brendan Shephard Sent: 09 June 2022 13:16 To: Russell Stather Cc: openstack-discuss at lists.openstack.org Subject: Re: Error creating octavia amphora Hey Russell, Are you able to share the outputs from: $ openstack server show 524ae27b-1542-4c2d-9118-138d9e7f3770 -c id -c project_id -c security_groups -c status -f yaml And: $ openstack security group show0b683c75-d900-4e45-acb2-8bc321580666 -c id -c project_id -f yaml I agree with James, my assumption would be that we will find they aren't in the same project, so Nova can't use that security group for the Amphorae. I'm not familiar with charmed openstack though, but it does look like James is from Canonical and might be able to advise on the specifics. All the best, Brendan Shephard Software Engineer Red Hat APAC 193 N Quay Brisbane City QLD 4000 @RedHat Red Hat Red Hat [https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png] [https://static.redhat.com/libs/redhat/brand-assets/latest/events/red-hat-summit.png] On Thu, Jun 9, 2022 at 8:48 PM Russell Stather > wrote: Hi I am seeing the below error when creating a load balancer. The security group does exist, and it is the correct group (tagged with charm-octavia) What am I missing here to resolve this? 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2022-06-09T10:09:59Z', 'message': 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 654, in build_instances\n scheduler_utils.populate_retry(\n File "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 989, in populate_retry\n raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.\n'} 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server -------------- next part -------------- An HTML attachment was scrubbed... URL: From kdhall at binghamton.edu Thu Jun 9 14:58:41 2022 From: kdhall at binghamton.edu (Dave Hall) Date: Thu, 9 Jun 2022 10:58:41 -0400 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: Message-ID: Hello, My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am sensitive to the effort that has been put into both of these projects, what I really need right now is the most fool-proof way to deploy and manage a small production cluster for academic instructional use. (BTW, I do understand that there are other differences between Kolla and regular openstack.) I have been working for a couple months with OpenStack-Ansible to deploy a small (3-node) Xena test cluster on VirtualBox VMs in preparation for a larger deployment on real hardware - 6 to 10 nodes. My VirtualBox deployment has been: Debian 11 deployment, Debian 10 infra, compute, and storage nodes It has been slow going, at least partially due to some issues and limitations with VirtualBox (and Vagrant early on). However, deploying a test cluster on VMs still seems preferable to just diving into deployment on real hardware and going through multiple scrubs/reinstalls. I've recently seen more posts in the list about Kolla-Ansible. So, as a 'beginner', should I shift and look at Kolla-Ansible, or should I stay the course and continue with Openstack-Ansible? What are the pros and cons of each? For that matter, is there some other deployment mechanism that would be more fool-proof for a first-timer? Although I'm more familiar with Ansible than the other tools (Chef, Puppet, etc.) I'm most interested in how to get a cluster up and running regardless of the underlying tools. Thanks. -Dave -- Dave Hall Binghamton University kdhall at binghamton.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Thu Jun 9 15:00:57 2022 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Thu, 9 Jun 2022 08:00:57 -0700 Subject: [all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC) In-Reply-To: References: <1813f5800a2.dea83620696760.7937991544769180537@ghanshyammann.com> <913612422bb6844e05dfee31dc3caab60f71200a.camel@redhat.com> <181443bdf1f.12767362256831.7510285921554510482@ghanshyammann.com> Message-ID: On Thu, Jun 9, 2022 at 6:30 AM Dan Smith wrote: > > > So, one thought. Ironic views system scope as *critical* for our usage > > based upon the consensus we built before the direction change, because > > the system fundamentally is the owner/manager of $things. We can and > > likely should extend that out to project admin (granted, I suspect at > > least one ironic admin will reply with a strong -1 to such a change... > > :\. ) given the direction change. We also have had some operators jump > > on it, but... again, entirely different models of usage/interaction > > given the base state. If system scope were to suddenly disappear or be > > completely redefined, it would be a hard break for us at this point. > > I don't think system scope would (or could) disappear at this point, so > I don't think there's much to worry about. Whew! That is a weight off my shoulders! > I think it's totally > reasonable to say that there are certain things that a user would never > interact with directly, which are entirely system-scoped. This might be > things like ironic and maybe even placement. You could also make the > argument that a good chunk of keystone's API is system-only. Definitely. > If people > are already using ironic with scopes turned on, it proves the point that > it's isolated enough that it doesn't suffer from all the other problems > that caused the direction change. > > --Dan Really, starting out as an admin-only service and then later adding multi-tenancy support which only got turned on by default with SRBAC meant we never had to deal with the can of worms that were initially encountered with system scope. From jonathan.rosser at rd.bbc.co.uk Thu Jun 9 15:34:40 2022 From: jonathan.rosser at rd.bbc.co.uk (Jonathan Rosser) Date: Thu, 9 Jun 2022 16:34:40 +0100 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: Message-ID: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> Hi Dave, I have been hesitating to reply to your mailing list post because it doesn't feel right to pitch two community tools against each other here on the mailing list - so i won't do that here. I would say that the deployment tool is a means to an end. So you should look at the technology choices, upgrade paths, support for "day 2 operations", how bugs get addressed, documentation, operator experience etc etc. Only you can decide which is appropriate for the constraints and requirements of your deployment. My reply will obviously be biassed, as I am a big contributor to openstack-ansible. My observation is that the operators that gain the most out of any of these tools are the ones who engage with the community around those tools, primarily in the case of openstack-ansible that would be through our IRC channel, weekly meetings and bug reports on Launchpad. You will gain insight and be able to leverage the knowledge of other operators who in some cases have literally written the book on various aspects of OpenStack. Trying to fight though every decision or problem on your own is the worst way to approach using any of these community driven tools. If you instead want a more "shrink wrap" approach to an installer, or more formal support, then it would be wise to look at the product oriented offerings from the large vendors. Both openstack-ansible and kolla-ansible will expect you to make a good number of decisions about the specifics of your deployment, for example storage, networking and security concerns. Both would also expect you to gain sufficient knowledge about how OpenStack itself works to be able to make good use of the customization opportunities that both provide. This is really the unique selling point of the community tooling, you get a very high degree of customization potential but that can come at the cost of some complexity. As you are already using openstack-ansible I would suggest that you continue, but as I've already said I have an existing interest here and I really don't want to start a tooling debate. Join us in IRC in #openstack-ansible and discuss any pain points you have. This way we can hopefully help you out, or address specific issues in the code - you may have discovered legitimate bugs or a use case that is not straightforward to fulfill. This is how all of the community tools get improved and evolved over time. On one specific point I would recommend that you move entirely to Debian 11 as Xena will be the last release that openstack-ansible supports Buster. I'm not sure there is a fool-proof installer really. Installing the code is one thing, being effective though upgrades and applying bugfixes to a production environment is a different and a more important concern in the long term. Both openstack-ansible and kolla-ansible offer "all-in-one" deployments which are intended as "should-just-work" demonstrators of how things fit together and for lightweight testing. Scaling those out to larger deployments is where the real work is, and neither tool sets out to be particularly prescriptive about some parts of how you build your environment. Hopefully this is helpful, Jonathan. On 09/06/2022 15:58, Dave Hall wrote: > Hello, > > My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am > sensitive to the effort that has been put into both of these projects, > what I really need right now is the most fool-proof way to deploy and > manage a small production cluster for academic instructional use. > > (BTW, I do understand that there are other differences between Kolla > and regular openstack.) > > I have been working for a couple months with OpenStack-Ansible?to > deploy a small (3-node) Xena test cluster on VirtualBox VMs in > preparation for a larger deployment on real hardware - 6 to 10 nodes.? > My VirtualBox deployment has been: > > Debian 11 deployment, Debian 10 infra, compute, and storage nodes > > > It has been slow going, at least partially due to some issues and > limitations with VirtualBox (and Vagrant early on).? However, > deploying a test cluster on VMs still seems preferable to just > diving?into deployment on real hardware and going through multiple > scrubs/reinstalls. > > I've?recently seen?more posts in the list about Kolla-Ansible. So, as > a 'beginner', should I shift and look at Kolla-Ansible, or should I > stay the course and continue with Openstack-Ansible? What are the pros > and cons of each? > > For that matter, is there some other deployment mechanism that would > be more fool-proof for a first-timer?? Although I'm more familiar with > Ansible than the other tools (Chef, Puppet, etc.) I'm most interested > in how to get a cluster up and running regardless of the underlying tools. > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian.rohmann at inovex.de Thu Jun 9 16:18:00 2022 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Thu, 9 Jun 2022 18:18:00 +0200 Subject: [rbac][nova][cinder] Canned roles for service users / inter-service communication (e.g. event submission) In-Reply-To: <26833ce2-e486-d953-b7ea-999d94862631@inovex.de> References: <26833ce2-e486-d953-b7ea-999d94862631@inovex.de> Message-ID: <6c7210b0-fad6-e31e-ecf5-3a977079e33a@inovex.de> On 09/06/2022 12:39, Christian Rohmann wrote: > I just noticed that Christian Berendt does a forum talk on > "Deprivilization of the internal service accounts" TODAY at 2:40pm - > 3:10pm at A05 on apparently that exact question :-) The notes on the discussion can be found here: https://etherpad.opendev.org/p/deprivilization-of-service-accounts Regards Christian From kdhall at binghamton.edu Thu Jun 9 18:03:47 2022 From: kdhall at binghamton.edu (Dave Hall) Date: Thu, 9 Jun 2022 14:03:47 -0400 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> References: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> Message-ID: Jonathan, Thank you for replying. I want to stress that I also understand that both Openstack-Ansible and Kolla-Ansible are serious projects and that I would not try to play one against the other. I do perceive that although both use Ansible, both also take different approaches in their organization and configuration. My query was whether one might be easier for beginners to grok whereas the other might be the tool of choice for the enlightened ones. Or perhaps it's just that Kolla uses Docker instead of LXC. It's just a lot to learn either way - this is big software. Regarding Openstack-Ansible, I will check out the IRC, although last I knew the University was blocking all IRC. I will also ask more questions here. Thanks. -Dave -- Dave Hall Binghamton University kdhall at binghamton.edu On Thu, Jun 9, 2022 at 11:35 AM Jonathan Rosser < jonathan.rosser at rd.bbc.co.uk> wrote: > Hi Dave, > > I have been hesitating to reply to your mailing list post because it > doesn't feel right to pitch two community tools against each other here on > the mailing list - so i won't do that here. > > I would say that the deployment tool is a means to an end. So you should > look at the technology choices, upgrade paths, support for "day 2 > operations", how bugs get addressed, documentation, operator experience etc > etc. Only you can decide which is appropriate for the constraints and > requirements of your deployment. > My reply will obviously be biassed, as I am a big contributor to > openstack-ansible. My observation is that the operators that gain the most > out of any of these tools are the ones who engage with the community around > those tools, primarily in the case of openstack-ansible that would be > through our IRC channel, weekly meetings and bug reports on Launchpad. You > will gain insight and be able to leverage the knowledge of other operators > who in some cases have literally written the book on various aspects of > OpenStack. Trying to fight though every decision or problem on your own is > the worst way to approach using any of these community driven tools. > > If you instead want a more "shrink wrap" approach to an installer, or more > formal support, then it would be wise to look at the product oriented > offerings from the large vendors. > > Both openstack-ansible and kolla-ansible will expect you to make a good > number of decisions about the specifics of your deployment, for example > storage, networking and security concerns. Both would also expect you to > gain sufficient knowledge about how OpenStack itself works to be able to > make good use of the customization opportunities that both provide. This is > really the unique selling point of the community tooling, you get a very > high degree of customization potential but that can come at the cost of > some complexity. > > As you are already using openstack-ansible I would suggest that you > continue, but as I've already said I have an existing interest here and I > really don't want to start a tooling debate. Join us in IRC in > #openstack-ansible and discuss any pain points you have. This way we can > hopefully help you out, or address specific issues in the code - you may > have discovered legitimate bugs or a use case that is not straightforward > to fulfill. This is how all of the community tools get improved and evolved > over time. > > On one specific point I would recommend that you move entirely to Debian > 11 as Xena will be the last release that openstack-ansible supports Buster. > > I'm not sure there is a fool-proof installer really. Installing the code > is one thing, being effective though upgrades and applying bugfixes to a > production environment is a different and a more important concern in the > long term. Both openstack-ansible and kolla-ansible offer "all-in-one" > deployments which are intended as "should-just-work" demonstrators of how > things fit together and for lightweight testing. Scaling those out to > larger deployments is where the real work is, and neither tool sets out to > be particularly prescriptive about some parts of how you build your > environment. > > Hopefully this is helpful, > Jonathan. > > On 09/06/2022 15:58, Dave Hall wrote: > > Hello, > > My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am > sensitive to the effort that has been put into both of these projects, what > I really need right now is the most fool-proof way to deploy and manage a > small production cluster for academic instructional use. > > (BTW, I do understand that there are other differences between Kolla and > regular openstack.) > > I have been working for a couple months with OpenStack-Ansible to deploy > a small (3-node) Xena test cluster on VirtualBox VMs in preparation for a > larger deployment on real hardware - 6 to 10 nodes. My VirtualBox > deployment has been: > > Debian 11 deployment, Debian 10 infra, compute, and storage nodes > > > It has been slow going, at least partially due to some issues and > limitations with VirtualBox (and Vagrant early on). However, deploying a > test cluster on VMs still seems preferable to just diving into deployment > on real hardware and going through multiple scrubs/reinstalls. > > I've recently seen more posts in the list about Kolla-Ansible. So, as a > 'beginner', should I shift and look at Kolla-Ansible, or should I stay > the course and continue with Openstack-Ansible? What are the pros and > cons of each? > > For that matter, is there some other deployment mechanism that would be > more fool-proof for a first-timer? Although I'm more familiar with Ansible > than the other tools (Chef, Puppet, etc.) I'm most interested in how to get > a cluster up and running regardless of the underlying tools. > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kdhall at binghamton.edu Thu Jun 9 18:19:11 2022 From: kdhall at binghamton.edu (Dave Hall) Date: Thu, 9 Jun 2022 14:19:11 -0400 Subject: [nova][openstack-ansible] "No Valid Host Found" when running on VirtualBox Message-ID: Hello, I'm trying to run a test/learning cluster on VirtualBox VMs deployed with Openstack-Ansible prior to deploying on real hardware. The playbooks succeed, and the basic functional checks all look good. However, I am unable to deploy an instance due to 'No Valid Host Found'. I have seen that this might be because VirtualBox does not support KVM. But I am also perplexed - I am able to deploy instances on an AIO running in VirtualBox on the same physical host. If there is an easy fix for this please point me in the right direction. If I need to destroy and re-clone my VMs I can try that as well. In an effort to learn, though, I'd really like to learn how to adjust the current deployment. Thanks. -Dave -- Dave Hall Binghamton University kdhall at binghamton.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Thu Jun 9 19:18:01 2022 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Thu, 9 Jun 2022 21:18:01 +0200 Subject: [nova][openstack-ansible] "No Valid Host Found" when running on VirtualBox In-Reply-To: References: Message-ID: Hey, One question - have you set nova_virt_type to qemu in overrides? As otherwise you indeed need to have nested virtualization enabled. ??, 9 ???. 2022 ?., 20:21 Dave Hall : > Hello, > > I'm trying to run a test/learning cluster on VirtualBox VMs deployed with > Openstack-Ansible prior to deploying on real hardware. > > The playbooks succeed, and the basic functional checks all look good. > However, I am unable to deploy an instance due to 'No Valid Host Found'. I > have seen that this might be because VirtualBox does not support KVM. But > I am also perplexed - I am able to deploy instances on an AIO running in > VirtualBox on the same physical host. > > If there is an easy fix for this please point me in the right direction. > If I need to destroy and re-clone my VMs I can try that as well. In an > effort to learn, though, I'd really like to learn how to adjust the current > deployment. > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Thu Jun 9 20:49:43 2022 From: johnsomor at gmail.com (Michael Johnson) Date: Thu, 9 Jun 2022 13:49:43 -0700 Subject: Error creating octavia amphora In-Reply-To: References: Message-ID: This isn't an issue with how you are creating the load balancer. It's a nova error booting a VM, and most likely a configuration issue in the deployment. It sounds like it is a charms bug. At boot time, we only plug the lb-mgmt-network in the amphora instance. All of the required settings for this are in the octavia.conf. First thing to check is the security group Octavia is configured to used: [controller_worker] amp_secgroup_list For each security group ID in that list, do an "openstack security group show " and note the project ID that owns the group. Then, also in the octavia.conf, check the project ID used to create the amphora instances in nova: [service_auth] project_id If project_id is not specified, then look up the project_name with "openstack project show ". All of these project IDs must match. You can also lookup the project IDs of the amphora VM "openstack server show" while it's attempting to boot and compare that to the security group project ID, just to make sure the current running configuration is also correct (as mentioned in a previous email). I haven't seen this error before, but I also don't use charms to deploy OpenStack. Michael On Thu, Jun 9, 2022 at 7:37 AM Russell Stather < Russell.Stather at ignitiongroup.co.za> wrote: > Hi > > I've built new security groups to make sure they are in the correct > project. I can start a new server manually using these security groups. > > The error is coming from the nova compute node. Same error can't find > security group. > > I am creating the load balancer and specifying the project on the command > line even. > > openstack loadbalancer create --name lb25 --vip-subnet-id admin_subnet1 > --project 5e168b652a374a02aff855a5e250b7f8 > > Kind of running out of ideas. > > > ------------------------------ > *From:* Brendan Shephard > *Sent:* 09 June 2022 13:16 > *To:* Russell Stather > *Cc:* openstack-discuss at lists.openstack.org < > openstack-discuss at lists.openstack.org> > *Subject:* Re: Error creating octavia amphora > > Hey Russell, > > Are you able to share the outputs from: > $ openstack server show 524ae27b-1542-4c2d-9118-138d9e7f3770 -c id -c > project_id -c security_groups -c status -f yaml > > And: > $ openstack security group show0b683c75-d900-4e45-acb2-8bc321580666 -c id > -c project_id -f yaml > > I agree with James, my assumption would be that we will find they aren't > in the same project, so Nova can't use that security group for the > Amphorae. I'm not familiar with charmed openstack though, but it does look > like James is from Canonical and might be able to advise on the specifics. > > All the best, > > Brendan Shephard > > Software Engineer > > Red Hat APAC > > 193 N Quay > > Brisbane City QLD 4000 > @RedHat Red Hat > Red Hat > > > > > > On Thu, Jun 9, 2022 at 8:48 PM Russell Stather < > Russell.Stather at ignitiongroup.co.za> wrote: > > Hi > > I am seeing the below error when creating a load balancer. The security > group does exist, and it is the correct group (tagged with charm-octavia) > > What am I missing here to resolve this? > > 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server > octavia.common.exceptions.ComputeBuildException: Failed to build compute > instance due to: {'code': 500, 'created': '2022-06-09T10:09:59Z', > 'message': 'Exceeded maximum number of retries. Exceeded max scheduling > attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last > exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.', > 'details': 'Traceback (most recent call last):\n File > "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 654, in > build_instances\n scheduler_utils.populate_retry(\n File > "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 989, in > populate_retry\n raise > exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: > Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for > instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security > group 0b683c75-d900-4e45-acb2-8bc321580666 not found.\n'} > 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mihalis68 at gmail.com Thu Jun 9 21:32:24 2022 From: mihalis68 at gmail.com (Chris Morgan) Date: Thu, 9 Jun 2022 23:32:24 +0200 Subject: [ops] details for Ops Meetup, Berlin, Tomorrow, Friday 2022-6-10 Message-ID: A brief twitter thread here https://twitter.com/osopsmeetup/status/1535004027032940544?s=20&t=AgLzV6QvgMUFiOm0lG6Agg has some reminders about tomorrow's event. Looking forward to seeing everyone. Chris -- Chris Morgan -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurentfdumont at gmail.com Fri Jun 10 00:02:26 2022 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Thu, 9 Jun 2022 20:02:26 -0400 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> Message-ID: For the sake of figuring out your options, I would also look at what features do you actually require at the Openstack level. I would start with the following questions to identify gaps or advantages between the two tools. - What Openstack projects do you require beyond the basic services? - Manilla, Trove, Skydive, Octavia, Ceilometer? - What network topologies are you looking to support? - OVS, OVN, SRIOV? - What support model are you looking for? - Using community deployers means you wear many different hats. Don't get me wrong, support through IRC and mailing lists is great but it's a community effort. If this breaks at 2AM, you will be mostly on your own. - Canonical can offer a Juju based deployment for Openstack with commercial support. - RedHat offers RHOSP that is based around tripleo with commercial support. - Are you expecting to configure CEPH through your deployer? - Are there any specific tech you are already familiar with in either tool? I am actually working on some deep dive into kolla-ansible and ansible-openstack for $work but I am a bit early in that process. In the end, like Jonathan mentioned, they both are a means to an end. That said, it is fair to say that having two different tools that essentially serve the same purpose is a strange position to be in. But I guess that comes with the flexibility of opensource! On Thu, Jun 9, 2022 at 2:07 PM Dave Hall wrote: > Jonathan, > > Thank you for replying. I want to stress that I also understand that both > Openstack-Ansible and Kolla-Ansible are serious projects and that I > would not try to play one against the other. I do perceive that although > both use Ansible, both also take different approaches in their organization > and configuration. > > My query was whether one might be easier for beginners to grok whereas the > other might be the tool of choice for the enlightened ones. Or perhaps > it's just that Kolla uses Docker instead of LXC. > > It's just a lot to learn either way - this is big software. > > Regarding Openstack-Ansible, I will check out the IRC, although last I > knew the University was blocking all IRC. > > I will also ask more questions here. > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > > On Thu, Jun 9, 2022 at 11:35 AM Jonathan Rosser < > jonathan.rosser at rd.bbc.co.uk> wrote: > >> Hi Dave, >> >> I have been hesitating to reply to your mailing list post because it >> doesn't feel right to pitch two community tools against each other here on >> the mailing list - so i won't do that here. >> >> I would say that the deployment tool is a means to an end. So you should >> look at the technology choices, upgrade paths, support for "day 2 >> operations", how bugs get addressed, documentation, operator experience etc >> etc. Only you can decide which is appropriate for the constraints and >> requirements of your deployment. >> My reply will obviously be biassed, as I am a big contributor to >> openstack-ansible. My observation is that the operators that gain the most >> out of any of these tools are the ones who engage with the community around >> those tools, primarily in the case of openstack-ansible that would be >> through our IRC channel, weekly meetings and bug reports on Launchpad. You >> will gain insight and be able to leverage the knowledge of other operators >> who in some cases have literally written the book on various aspects of >> OpenStack. Trying to fight though every decision or problem on your own is >> the worst way to approach using any of these community driven tools. >> >> If you instead want a more "shrink wrap" approach to an installer, or >> more formal support, then it would be wise to look at the product oriented >> offerings from the large vendors. >> >> Both openstack-ansible and kolla-ansible will expect you to make a good >> number of decisions about the specifics of your deployment, for example >> storage, networking and security concerns. Both would also expect you to >> gain sufficient knowledge about how OpenStack itself works to be able to >> make good use of the customization opportunities that both provide. This is >> really the unique selling point of the community tooling, you get a very >> high degree of customization potential but that can come at the cost of >> some complexity. >> >> As you are already using openstack-ansible I would suggest that you >> continue, but as I've already said I have an existing interest here and I >> really don't want to start a tooling debate. Join us in IRC in >> #openstack-ansible and discuss any pain points you have. This way we can >> hopefully help you out, or address specific issues in the code - you may >> have discovered legitimate bugs or a use case that is not straightforward >> to fulfill. This is how all of the community tools get improved and evolved >> over time. >> >> On one specific point I would recommend that you move entirely to Debian >> 11 as Xena will be the last release that openstack-ansible supports Buster. >> >> I'm not sure there is a fool-proof installer really. Installing the code >> is one thing, being effective though upgrades and applying bugfixes to a >> production environment is a different and a more important concern in the >> long term. Both openstack-ansible and kolla-ansible offer "all-in-one" >> deployments which are intended as "should-just-work" demonstrators of how >> things fit together and for lightweight testing. Scaling those out to >> larger deployments is where the real work is, and neither tool sets out to >> be particularly prescriptive about some parts of how you build your >> environment. >> >> Hopefully this is helpful, >> Jonathan. >> >> On 09/06/2022 15:58, Dave Hall wrote: >> >> Hello, >> >> My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am >> sensitive to the effort that has been put into both of these projects, what >> I really need right now is the most fool-proof way to deploy and manage a >> small production cluster for academic instructional use. >> >> (BTW, I do understand that there are other differences between Kolla and >> regular openstack.) >> >> I have been working for a couple months with OpenStack-Ansible to deploy >> a small (3-node) Xena test cluster on VirtualBox VMs in preparation for a >> larger deployment on real hardware - 6 to 10 nodes. My VirtualBox >> deployment has been: >> >> Debian 11 deployment, Debian 10 infra, compute, and storage nodes >> >> >> It has been slow going, at least partially due to some issues and >> limitations with VirtualBox (and Vagrant early on). However, deploying a >> test cluster on VMs still seems preferable to just diving into deployment >> on real hardware and going through multiple scrubs/reinstalls. >> >> I've recently seen more posts in the list about Kolla-Ansible. So, as a >> 'beginner', should I shift and look at Kolla-Ansible, or should I stay >> the course and continue with Openstack-Ansible? What are the pros and >> cons of each? >> >> For that matter, is there some other deployment mechanism that would be >> more fool-proof for a first-timer? Although I'm more familiar with Ansible >> than the other tools (Chef, Puppet, etc.) I'm most interested in how to get >> a cluster up and running regardless of the underlying tools. >> >> Thanks. >> >> -Dave >> >> -- >> Dave Hall >> Binghamton University >> kdhall at binghamton.edu >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From Russell.Stather at ignitiongroup.co.za Fri Jun 10 06:01:26 2022 From: Russell.Stather at ignitiongroup.co.za (Russell Stather) Date: Fri, 10 Jun 2022 06:01:26 +0000 Subject: Error creating octavia amphora In-Reply-To: References: Message-ID: Hi all Thank you for the assistance. checking octavia.conf I found that this: [service_auth] auth_section = keystone_authtoken [keystone_authtoken] auth_uri = https://10.0.143.28:5000 auth_url = https://10.0.143.28:35357 auth_type = password project_domain_name = service_domain user_domain_name = service_domain project_name = services username = octavia password = FN9BztN8tYcWrtxbHYPR2myrpz5HgSJ4T7479ynkCNFyWPhbcfgdbfc5xSCxkPH6 memcached_servers = inet6:[::1]:11211 So I rebult the security groups and put them in the services project. All working fine now. Thanks so much! Russell ________________________________ From: Michael Johnson Sent: 09 June 2022 22:49 To: Russell Stather Cc: Brendan Shephard ; openstack-discuss at lists.openstack.org Subject: Re: Error creating octavia amphora This isn't an issue with how you are creating the load balancer. It's a nova error booting a VM, and most likely a configuration issue in the deployment. It sounds like it is a charms bug. At boot time, we only plug the lb-mgmt-network in the amphora instance. All of the required settings for this are in the octavia.conf. First thing to check is the security group Octavia is configured to used: [controller_worker] amp_secgroup_list For each security group ID in that list, do an "openstack security group show " and note the project ID that owns the group. Then, also in the octavia.conf, check the project ID used to create the amphora instances in nova: [service_auth] project_id If project_id is not specified, then look up the project_name with "openstack project show ". All of these project IDs must match. You can also lookup the project IDs of the amphora VM "openstack server show" while it's attempting to boot and compare that to the security group project ID, just to make sure the current running configuration is also correct (as mentioned in a previous email). I haven't seen this error before, but I also don't use charms to deploy OpenStack. Michael On Thu, Jun 9, 2022 at 7:37 AM Russell Stather > wrote: Hi I've built new security groups to make sure they are in the correct project. I can start a new server manually using these security groups. The error is coming from the nova compute node. Same error can't find security group. I am creating the load balancer and specifying the project on the command line even. openstack loadbalancer create --name lb25 --vip-subnet-id admin_subnet1 --project 5e168b652a374a02aff855a5e250b7f8 Kind of running out of ideas. ________________________________ From: Brendan Shephard > Sent: 09 June 2022 13:16 To: Russell Stather > Cc: openstack-discuss at lists.openstack.org > Subject: Re: Error creating octavia amphora Hey Russell, Are you able to share the outputs from: $ openstack server show 524ae27b-1542-4c2d-9118-138d9e7f3770 -c id -c project_id -c security_groups -c status -f yaml And: $ openstack security group show0b683c75-d900-4e45-acb2-8bc321580666 -c id -c project_id -f yaml I agree with James, my assumption would be that we will find they aren't in the same project, so Nova can't use that security group for the Amphorae. I'm not familiar with charmed openstack though, but it does look like James is from Canonical and might be able to advise on the specifics. All the best, Brendan Shephard Software Engineer Red Hat APAC 193 N Quay Brisbane City QLD 4000 @RedHat Red Hat Red Hat [X] [X] On Thu, Jun 9, 2022 at 8:48 PM Russell Stather > wrote: Hi I am seeing the below error when creating a load balancer. The security group does exist, and it is the correct group (tagged with charm-octavia) What am I missing here to resolve this? 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2022-06-09T10:09:59Z', 'message': 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 654, in build_instances\n scheduler_utils.populate_retry(\n File "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 989, in populate_retry\n raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.\n'} 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server -------------- next part -------------- An HTML attachment was scrubbed... URL: From fpantano at redhat.com Fri Jun 10 06:57:28 2022 From: fpantano at redhat.com (Francesco Pantano) Date: Fri, 10 Jun 2022 08:57:28 +0200 Subject: [TripleO] Proposing Brendan Shephard and Takashi Kajinami In-Reply-To: References: Message-ID: +1 On Wed, Jun 8, 2022 at 9:45 PM James Slagle wrote: > > I'm formally proposing that we add Brendan Shephard and Takashi Kajinami > to the core reviewers team for TripleO. Takashi is already considered core > on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and > likewise for Brendan. > > They both have been contributing and reviewing for a little while now, and > I feel it's time to add them to the team. Please +1 or let me know any > concerns before Friday June 15th. Thanks! > > -- > -- James Slagle > -- > -- Francesco Pantano GPG KEY: F41BD75C -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri Jun 10 07:02:29 2022 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 10 Jun 2022 07:02:29 +0000 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> Message-ID: <20220610070009.3syvxwzeuhos2odn@yuggoth.org> On 2022-06-09 14:03:47 -0400 (-0400), Dave Hall wrote: [...] > I will check out the IRC, although last I knew the University was > blocking all IRC. [...] If you're able to reach Matrix, you can participate quite effectively in OFTC's IRC channels through their Matrix bridge. A concise writeup can be found here: https://blog.christophersmart.com/2022/03/21/joining-a-bridged-irc-network-on-element-matrix/ -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Fri Jun 10 07:06:35 2022 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 10 Jun 2022 07:06:35 +0000 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> Message-ID: <20220610070635.cahozvgv6klxuoib@yuggoth.org> On 2022-06-09 20:02:26 -0400 (-0400), Laurent Dumont wrote: [...] > That said, it is fair to say that having two different tools that > essentially serve the same purpose is a strange position to be in. > But I guess that comes with the flexibility of opensource! [...] As I'm sure you can imagine, that's not how things started out, it's just where they ended up. In the long-ago, one was a container-focused deployment project while the other was an Ansible-focused deployment project. Over time, they each made convergent tooling choices, leading to the present situation. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From jonathan.rosser at rd.bbc.co.uk Fri Jun 10 07:09:06 2022 From: jonathan.rosser at rd.bbc.co.uk (Jonathan Rosser) Date: Fri, 10 Jun 2022 08:09:06 +0100 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> Message-ID: On 09/06/2022 19:03, Dave Hall wrote: > > > Regarding Openstack-Ansible, I will check out the IRC, although last I > knew the?University was blocking all IRC. > I see you managed to join the IRC channel yesterday. A lot of people will have been at the OpenInfra summit so things are quiet this week. Your best bet it to also try to overlap with the EU timezone. The weekly meeting is held at Tuesday at 15:00 UTC. Web clients such as IRCCloud can help overcome direct blocking of IRC and also can provide a persistent connection, plus there is a bridge to Matrix chat documented here https://docs.openstack.org/contributors/common/irc.html#irc-chatting-with-matrix, although unfortunately that doc does not give a specific example for joining an openstack related channel. Jonathan. From mark at stackhpc.com Fri Jun 10 08:53:30 2022 From: mark at stackhpc.com (Mark Goddard) Date: Fri, 10 Jun 2022 09:53:30 +0100 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: Message-ID: Hi Dave, I would suggest getting a minimal deployment up and running with each environment, before making your choice. For Kolla Ansible, you can follow the quickstart documentation to get a single node deployment up and running: https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html If you also need bare metal provisioning capabilities for your control & compute nodes, as well as more advanced host configuration, you might consider Kayobe, which builds on top of Kolla Ansible. Here is a guide to building up the configuration for a single node deployment: https://docs.openstack.org/kayobe/latest/configuration/scenarios/all-in-one/index.html Regards, Mark On Thu, 9 Jun 2022 at 16:05, Dave Hall wrote: > Hello, > > My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am > sensitive to the effort that has been put into both of these projects, what > I really need right now is the most fool-proof way to deploy and manage a > small production cluster for academic instructional use. > > (BTW, I do understand that there are other differences between Kolla and > regular openstack.) > > I have been working for a couple months with OpenStack-Ansible to deploy > a small (3-node) Xena test cluster on VirtualBox VMs in preparation for a > larger deployment on real hardware - 6 to 10 nodes. My VirtualBox > deployment has been: > > Debian 11 deployment, Debian 10 infra, compute, and storage nodes > > > It has been slow going, at least partially due to some issues and > limitations with VirtualBox (and Vagrant early on). However, deploying a > test cluster on VMs still seems preferable to just diving into deployment > on real hardware and going through multiple scrubs/reinstalls. > > I've recently seen more posts in the list about Kolla-Ansible. So, as a > 'beginner', should I shift and look at Kolla-Ansible, or should I stay > the course and continue with Openstack-Ansible? What are the pros and > cons of each? > > For that matter, is there some other deployment mechanism that would be > more fool-proof for a first-timer? Although I'm more familiar with Ansible > than the other tools (Chef, Puppet, etc.) I'm most interested in how to get > a cluster up and running regardless of the underlying tools. > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Fri Jun 10 08:54:30 2022 From: mark at stackhpc.com (Mark Goddard) Date: Fri, 10 Jun 2022 09:54:30 +0100 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> References: <8b0c70a6-53d4-f314-c6d9-8de65a11826d@rd.bbc.co.uk> Message-ID: This has been discussed before, but it would be nice if we as a community had an easier way to answer this question. I know many of us feel conflicted about aggressively promoting our own projects over alternatives, but also want to give them a fair chance. Maybe each project needs a "why pick me" page that could be linked to in a standard response. Mark On Thu, 9 Jun 2022 at 16:40, Jonathan Rosser wrote: > Hi Dave, > > I have been hesitating to reply to your mailing list post because it > doesn't feel right to pitch two community tools against each other here on > the mailing list - so i won't do that here. > > I would say that the deployment tool is a means to an end. So you should > look at the technology choices, upgrade paths, support for "day 2 > operations", how bugs get addressed, documentation, operator experience etc > etc. Only you can decide which is appropriate for the constraints and > requirements of your deployment. > My reply will obviously be biassed, as I am a big contributor to > openstack-ansible. My observation is that the operators that gain the most > out of any of these tools are the ones who engage with the community around > those tools, primarily in the case of openstack-ansible that would be > through our IRC channel, weekly meetings and bug reports on Launchpad. You > will gain insight and be able to leverage the knowledge of other operators > who in some cases have literally written the book on various aspects of > OpenStack. Trying to fight though every decision or problem on your own is > the worst way to approach using any of these community driven tools. > > If you instead want a more "shrink wrap" approach to an installer, or more > formal support, then it would be wise to look at the product oriented > offerings from the large vendors. > > Both openstack-ansible and kolla-ansible will expect you to make a good > number of decisions about the specifics of your deployment, for example > storage, networking and security concerns. Both would also expect you to > gain sufficient knowledge about how OpenStack itself works to be able to > make good use of the customization opportunities that both provide. This is > really the unique selling point of the community tooling, you get a very > high degree of customization potential but that can come at the cost of > some complexity. > > As you are already using openstack-ansible I would suggest that you > continue, but as I've already said I have an existing interest here and I > really don't want to start a tooling debate. Join us in IRC in > #openstack-ansible and discuss any pain points you have. This way we can > hopefully help you out, or address specific issues in the code - you may > have discovered legitimate bugs or a use case that is not straightforward > to fulfill. This is how all of the community tools get improved and evolved > over time. > > On one specific point I would recommend that you move entirely to Debian > 11 as Xena will be the last release that openstack-ansible supports Buster. > > I'm not sure there is a fool-proof installer really. Installing the code > is one thing, being effective though upgrades and applying bugfixes to a > production environment is a different and a more important concern in the > long term. Both openstack-ansible and kolla-ansible offer "all-in-one" > deployments which are intended as "should-just-work" demonstrators of how > things fit together and for lightweight testing. Scaling those out to > larger deployments is where the real work is, and neither tool sets out to > be particularly prescriptive about some parts of how you build your > environment. > > Hopefully this is helpful, > Jonathan. > > On 09/06/2022 15:58, Dave Hall wrote: > > Hello, > > My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am > sensitive to the effort that has been put into both of these projects, what > I really need right now is the most fool-proof way to deploy and manage a > small production cluster for academic instructional use. > > (BTW, I do understand that there are other differences between Kolla and > regular openstack.) > > I have been working for a couple months with OpenStack-Ansible to deploy > a small (3-node) Xena test cluster on VirtualBox VMs in preparation for a > larger deployment on real hardware - 6 to 10 nodes. My VirtualBox > deployment has been: > > Debian 11 deployment, Debian 10 infra, compute, and storage nodes > > > It has been slow going, at least partially due to some issues and > limitations with VirtualBox (and Vagrant early on). However, deploying a > test cluster on VMs still seems preferable to just diving into deployment > on real hardware and going through multiple scrubs/reinstalls. > > I've recently seen more posts in the list about Kolla-Ansible. So, as a > 'beginner', should I shift and look at Kolla-Ansible, or should I stay > the course and continue with Openstack-Ansible? What are the pros and > cons of each? > > For that matter, is there some other deployment mechanism that would be > more fool-proof for a first-timer? Although I'm more familiar with Ansible > than the other tools (Chef, Puppet, etc.) I'm most interested in how to get > a cluster up and running regardless of the underlying tools. > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.page at canonical.com Fri Jun 10 09:15:02 2022 From: james.page at canonical.com (James Page) Date: Fri, 10 Jun 2022 10:15:02 +0100 Subject: Error creating octavia amphora In-Reply-To: References: Message-ID: On Fri, Jun 10, 2022 at 7:06 AM Russell Stather < Russell.Stather at ignitiongroup.co.za> wrote: > Hi all > > Thank you for the assistance. > > checking octavia.conf I found that this: > > [service_auth] > > auth_section = keystone_authtoken > > > [keystone_authtoken] > > auth_uri = https://10.0.143.28:5000 > > auth_url = https://10.0.143.28:35357 > > auth_type = password > > project_domain_name = service_domain > > user_domain_name = service_domain > > project_name = services > > username = octavia > > password = FN9BztN8tYcWrtxbHYPR2myrpz5HgSJ4T7479ynkCNFyWPhbcfgdbfc5xSCxkPH6 > > memcached_servers = inet6:[::1]:11211 > > > So I rebult the security groups and put them in the services project. > > All working fine now. Thanks so much! > Glad to hear you have it working now. For future followers the octavia charm has a 'configure-resources' action that will either create all of the required resources (networking and security groups) in OpenStack for Octavia operation or discover them if they have been created by the user of the charm by hand - see the deployment guide doc on this topic: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-octavia.html#resource-configuration Cheers James -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim+openstack.org at coote.org Fri Jun 10 10:52:19 2022 From: tim+openstack.org at coote.org (tim+openstack.org at coote.org) Date: Fri, 10 Jun 2022 11:52:19 +0100 Subject: simple build fails to allocate network interface In-Reply-To: References: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> <0044A33F-0E87-433C-8130-77C66D79926A@coote.org> Message-ID: > On 9 Jun 2022, at 13:20, Rodolfo Alonso Hernandez wrote: > > Hello: > > This seems to be a problem in Neutron. Please provide what backend you are using, and the Neutron server logs. If the ML2 driver has agents (DHCP, ML2 agent), provide the agents logs too. > > Regards. > Hullo thanks for the response. I don?t know what to look for wrt ml2 driver - I?m using the default `stack.sh` installation script from the devstack repo. The neutron log has helped, I think. I found this useful as a starting point:https://bit.ly/3MG6jnR as it enables log extraction of events relating to the creation of a specific server (although that?s still quite large for a mailing list). I think that this may help with your question (but as I said, I don?t really understand the question). There may be a dhcp issue: I cannot ping either ipv4 or ipv6 dhcp addresses mentioned in the log from the vm. I?ve no idea where they come from, however. Jun 10 10:13:49 localhost.localdomain nova-compute[20725]: DEBUG nova.network.neutron [req-1c6a3f76-b810-4f78-89ab-ba8c0d1a2b39 req-172db7ab-dbda-4bad-912c-3bc0f26069a5 service nova] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1f be4f] Updated VIF entry in instance network info cache for port 5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc. {{(pid=20725) _build_network_info_model /opt/stack/nova/nova/network/neutron.py:3420}} Jun 10 10:13:49 localhost.localdomain nova-compute[20725]: DEBUG nova.network.neutron [req-1c6a3f76-b810-4f78-89ab-ba8c0d1a2b39 req-172db7ab-dbda-4bad-912c-3bc0f26069a5 service nova] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1f be4f] Updating instance_info_cache with network_info: [{"id": "5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc", "address": "fa:16:3e:02:7a:6a", "network": {"id": "2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63", "bridge": "br-int", "label": "private", "subnets": [{"cidr": "10.0.0.0/26", "dns": [], "gateway": {"address": "10.0.0.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "10.0.0.51", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"dhcp_server": "10.0.0.1"}}, {"cidr": "fd5b:4cf4:f7c1::/64", "dns": [], "gateway": {"address": "fd5b:4cf4:f7c1::1", "type": "gateway", "version": 6, "meta": {}}, "ips": [{"address": "fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a", "type": "fixed", "version": 6, "meta": {}, "floating_ips": []}], "routes": [], "version": 6, "meta": {"ipv6_address_mode": "slaac", "dhcp_server": "fd5b:4cf4:f7c1::1"}}], "meta": {"injected": false, "tenant_id": "6c0b876005f44086bd9bde5fef672dfe", "mtu": 1442, "physical_network": null, "tunneled": true}}, "type": "ovs", "details": {"port_filter": true, "connectivity": "l2", "bound_drivers": {"0": "ovn"}}, "devname": "tap5ae4c0d6-2a", "ovs_interfaceid": "5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc", "qbh_params": null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}] {{(pid=20725) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:117}} Jun 10 10:14:11 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} Jun 10 10:15:12 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} Jun 10 10:16:12 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} Jun 10 10:17:12 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} Jun 10 10:17:50 localhost.localdomain neutron-server[20918]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc, resource={'id': '5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc', 'name': '', 'network_id': '2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63', 'tenant_id': '6c0b876005f44086bd9bde5fef672dfe', 'mac_address': 'fa:16:3e:02:7a:6a', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f', 'device_owner': 'compute:nova', 'standard_attr_id': 41, 'fixed_ips': [{'subnet_id': 'e7377bed-3e22-4e8d-9c2d-ea7ba740fcfd', 'ip_address': '10.0.0.51'}, {'subnet_id': 'f915b88c-9988-4e1f-9060-a6295465699a', 'ip_address': 'fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['ed75ab0b-d9a4-4023-9204-93038729f6d3'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'localhost.localdomain', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2022-06-10T10:13:44Z', 'updated_at': '2022-06-10T10:13:46Z', 'revision_number': 3, 'project_id': '6c0b876005f44086bd9bde5fef672dfe'}, resource_type=ports, if_exists=True) {{(pid=20918) do_commit /usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}} Jun 10 10:17:50 localhost.localdomain neutron-server[20918]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLSwitchPortCommand(lport=5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc, columns={'external_ids': {'neutron:port_name': '', 'neutron:device_id': 'fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f', 'neutron:project_id': '6c0b876005f44086bd9bde5fef672dfe', 'neutron:cidrs': '10.0.0.51/26 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a/64', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63', 'neutron:security_group_ids': 'ed75ab0b-d9a4-4023-9204-93038729f6d3', 'neutron:revision_number': '3'}, 'parent_name': [], 'tag': [], 'options': {'requested-chassis': 'localhost.localdomain', 'mcast_flood_reports': 'true'}, 'enabled': True, 'port_security': ['fa:16:3e:02:7a:6a 10.0.0.51 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'], 'dhcpv4_options': [UUID('01cae053-612b-4c91-81e8-39884ef035dd')], 'dhcpv6_options': [], 'type': '', 'addresses': ['fa:16:3e:02:7a:6a 10.0.0.51 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'], 'ha_chassis_group': []}, if_exists=False) {{(pid=20918) do_commit /usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}} Jun 10 10:18:13 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} Jun 10 10:18:49 localhost.localdomain nova-compute[20725]: WARNING nova.compute.manager [None req-4a03d9e1-6053-4167-ab43-fba30072a79c demo admin] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Timeout waiting for ['network-vif-plugged-5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc'] for instance with vm_state building and task_state spawning. Event states are: network-vif-plugged-5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc: timed out after 300.00 seconds: eventlet.timeout.Timeout: 300 seconds Jun 08 15:24:48 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 205fa7d7-7067-4f63-b3d9-6a9b37b> -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal.arbet at ultimum.io Fri Jun 10 12:05:34 2022 From: michal.arbet at ultimum.io (Michal Arbet) Date: Fri, 10 Jun 2022 14:05:34 +0200 Subject: [cinder] Failed to delete volume In-Reply-To: References: Message-ID: Hi, Cinder is from victoria, backend is ceph, steps to reproduce ? Just delete volume. I think that it could be an error not in cinder but somehow a rabbitmq issue. :/ Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Po???? 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet at ultimum.io *https://ultimum.io * LinkedIn | Twitter | Facebook ?t 7. 6. 2022 v 19:27 odes?latel Sofia Enriquez napsal: > Hi Michal, > > I do not think we have such a bug on the Launchpad platform.[1] > Could you create a new bug report indicating which backend you are using, > which Cinder version you have, and if the error occurs after following some > steps? > > Thanks in advance, > Sofia > > [1] https://bugs.launchpad.net/cinder > > On Tue, Jun 7, 2022 at 4:28 AM Michal Arbet > wrote: > >> Hi, >> >> I would like to ask if someone saw issue when volume failed to delete, >> log below : >> This is happening from time to time. >> >> LOG : https://paste.opendev.org/show/b1jRCxuBFnnVzf64i9yV/ >> >> Thanks, >> >> Michal Arbet >> Openstack Engineer >> >> Ultimum Technologies a.s. >> Na Po???? 1047/26, 11000 Praha 1 >> Czech Republic >> >> +420 604 228 897 >> michal.arbet at ultimum.io >> *https://ultimum.io * >> >> LinkedIn | >> Twitter | Facebook >> >> > > > -- > > Sof?a Enriquez > > she/her > > Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > @RedHat Red Hat > Red Hat > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Greg.Waines at windriver.com Fri Jun 10 12:30:23 2022 From: Greg.Waines at windriver.com (Waines, Greg) Date: Fri, 10 Jun 2022 12:30:23 +0000 Subject: [doc] question/enhancement-request on Sphinx theme used by openstack docs Message-ID: Hey ... I am on the technical steering committee for the openstack starlingx team and also provide primary technical review for our docs at https://docs.starlingx.io/ . docs.starlingx.io basically uses the openstack docs team's Sphinx theme for building our html docs ... see https://opendev.org/starlingx/docs . docs.starlingx.io is a very large doc suite. We are interested in improvements to the Table-of-Contents (TOC) navigation and presentation ... but we have limited Sphinx expertise. Specifically we are looking for the TOC support to have - more subsection header levels in TOC // such that user has full view of where he currently is in the doc hierarchy - highlighting to indicate currently selected section - separate scroll bar for TOC // ... so you don't lose sight of selected TOC selection when you scroll down on the main page which has large content. Have you ever considered such doc improvements? Did you have Sphinx expertise to do such improvements ? let us know, Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal.arbet at ultimum.io Fri Jun 10 12:40:25 2022 From: michal.arbet at ultimum.io (Michal Arbet) Date: Fri, 10 Jun 2022 14:40:25 +0200 Subject: [kolla-ansible][xena] Cell database HA/redundancy? In-Reply-To: References: Message-ID: Hi, I think my patchsets are in good shape and ready for merge, just need some reviewers finally :(. I am glad that there is this type of question, who knows, maybe this will speed up the process of merge :) You can still backport it to your downstream branches, we are using it in this way for several kolla-ansible versions and it works. Thanks, Michal Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Po???? 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet at ultimum.io *https://ultimum.io * LinkedIn | Twitter | Facebook st 8. 6. 2022 v 16:37 odes?latel Norrie, Andrew napsal: > Hi Mark, > > > > Thanks for the reply. > > > > I?ll look into the HAProxy ?custom config? as suggested and using > different ports for the cell databases. > > Maybe that can be worked in. > > > > We are expanding our OpenStack/kolla-ansible personnel/knowledge so will > look into the possibility for > > us to contribute as reviewers/testers as a start. > > > > Best regards .... Andrew > > > > *From:* Mark Goddard > *Sent:* Monday, June 6, 2022 2:22 AM > *To:* Norrie, Andrew > *Cc:* openstack-discuss at lists.openstack.org > *Subject:* Re: [kolla-ansible][xena] Cell database HA/redundancy? > > > > > > > > On Fri, 3 Jun 2022 at 14:10, Norrie, Andrew wrote: > > > > Hi, > > > > We are currently planning some large OpenStack deployments utilizing > kolla-ansible and > > I?m curious what folks are doing with the cell database HA/redundancy. > > > > With kolla-ansible (xena) it appears that a loadbalancer setup is only > allowed for the > > default database shard (shard 0)... reference: > https://docs.openstack.org/kolla-ansible/latest/reference/databases/mariadb-guide.html > > > > > If we are setting up separate cell database shards with Galera, I?m > wondering if there is a convenient work-around or configuration for > implementation of HA for these cell databases. > > > > In the inventory group_vars directory you can specify the group variables > (for each cell database) as: > > > > nova_cell_database_address: > > nova_cell_database_group: > > > > but these aren?t virtual IP hosts/addresses (non default db shard). This > works perfectly fine with > > a single server cell database but not Galera. If that db server goes down > the cell instance information is lost. > > > > > > Hi Andrew, > > > > You are correct - only the first shard is load balanced. The sharding > feature was actually just the first patch in a series intended to support > proxysql. I believe this would add the functionality you are looking for. > In fact, the proposed test case for proxysql in our CI is a multi-cell > deployment. > > > > Here is the patch series: > > > > https://review.opendev.org/q/topic:proxysql > > > > > Unfortunately it has been stuck for a while, largely due to core reviewer > bandwidth. The author is Michael Arbet, who is often on IRC as kevko. I'd > suggest registering your interest in these patches via gerrit, and at one > of the weekly kolla IRC meetings (this week is cancelled due to the > summit). If you have time available, then testing and providing reviews > could help to get the patches moving. > > > > In the meantime, you can configure HAProxy to load balance additional > services, by placing an HAProxy config snippet in > /etc/kolla/config/haproxy/services.d/*.cfg. > > > > Regards, > > Mark > > > > Many thanks ... Andrew > ------------------------------ > > ?This e-mail and any accompanying attachments are confidential. The > information is intended solely for the use of the individual to whom it is > addressed. Any review, disclosure, copying, distribution, or use of the > email by others is strictly prohibited. If you are not the intended > recipient, you must not review, disclose, copy, distribute or use this > e-mail; please delete it from your system and notify the sender > immediately.? > > ------------------------------ > ?This e-mail and any accompanying attachments are confidential. The > information is intended solely for the use of the individual to whom it is > addressed. Any review, disclosure, copying, distribution, or use of the > email by others is strictly prohibited. If you are not the intended > recipient, you must not review, disclose, copy, distribute or use this > e-mail; please delete it from your system and notify the sender > immediately.? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.harrison at manchester.ac.uk Fri Jun 10 13:13:05 2022 From: paul.harrison at manchester.ac.uk (Paul Harrison) Date: Fri, 10 Jun 2022 13:13:05 +0000 Subject: [nova] local LVM volume on compute hosts In-Reply-To: <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> References: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> Message-ID: <92824CD0-05FB-4941-B9C0-C27296B958F0@manchester.ac.uk> > On 2022-06 -08, at 08:26, Paul Harrison wrote: > > > >> On 2022-06 -07, at 13:25, Sean Mooney > wrote: >> >> no there is noting else you need to configure but this option is not what you think it is. >> the images_type option contols what storage will be used for all non cinder storage. >> i.e. vms that are booted with out usign a boot volume. >> >> >> >> so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks >> but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that >> is what they will recive but if they use teh default storage provided by the flaovr then it will be local. > > thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools has never got enough votes for implementation. > > I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying ?no? to "create new volume?) - I think it was failing here https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b78d19ae6497/nova/virt/libvirt/imagebackend.py#L722 as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume. Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line. I had a look at what the GUI sends to the /api/nova/servers/ endpoint {"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud?} However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful. Thanks, Paul. p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2893 bytes Desc: not available URL: From vince.mulhollon at springcitysolutions.com Fri Jun 10 13:29:15 2022 From: vince.mulhollon at springcitysolutions.com (Vince Mulhollon) Date: Fri, 10 Jun 2022 08:29:15 -0500 Subject: [magnum] Docker Swarm errors on Yoga Ubuntu using Install Guide cut-n-paste Message-ID: I'm trying to use Magnum on a fresh Yoga install on Ubuntu to run some non-load balanced docker swarm workloads. Its a long story but the parts of the swarm workload are heterogeneous so a cluster would just get in the way. Everything else about the cluster is working fine, all is well, etc. I'm a small time VMware and OS admin with years of experience but absolute zero experience with Magnum. Also I have "some" experience with Docker Swarm clusters mostly created and assembled with Ansible. So I know quite well how to look at log files or configs in a general sense, but I have no idea what to expect from Magnum specifically as a "glue" technology. I've followed every detail of the "Magnum Installation Guide" for Yoga as seen at: https://docs.openstack.org/magnum/yoga/install/index.html When provisioning a docker swarm, I very literally cut and paste from: https://docs.openstack.org/magnum/yoga/install/launch-instance.html#provision-a-docker-swarm-cluster-and-create-a-container cluster creation fails every time with "Property allowed_cidrs not assigned" exactly as seen on this mailing list post from 2021-August-20 about ten months ago on a Wallaby install, they were also using Ubuntu. http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024273.html Advice at that point ranged from, try installing Octavia, which I was trying to avoid and I definitely do NOT want a load balancer in front of that particular workload, to Docker swarm is no longer tested for Magnum and it's in practice a K8S exclusive product. I played around for an afternoon and got nowhere WRT configuring the allowed cidrs for a k8s load balancer on a config that does not use a load balancer and does not use k8s. Heat works fine, its super fun, and one alternative is I could give up on Magnum and use some Heat templates, which I know how to do and seems simpler than getting Magnum to work at this point. Or I could brave the scary looking mostly-undocumented world of kuryr and zun, which I was attempting to avoid by using Magnum which is at least simple and easy to install. Or just keep spinning up manual images and letting ansible turn them into docker swarms for me. But, ideally, I'd like to use Magnum to spin up some Docker Swarms for personal curiosity reasons, if nothing else, it would be very convenient if it worked. So the point of this long ramble is, can anyone successfully running Docker Swarm on Yoga on Ubuntu using a very typical "Install Guide" installation provide a pointer or link or github or something that Google searches is not finding to get it to work? Also I hate to piggyback multiple questions but this paragraph also interests me, does anyone have any information and experience with this: https://docs.openstack.org/magnum/yoga/user/index.html#heat-stack-templates "Sample cluster driver To help developers in creating new COE drivers, a minimal cluster driver is provided as an example. The ?docker? cluster driver will simply deploy a single VM running Ubuntu with the latest Docker version installed. It is not a true cluster, but the simplicity will help to illustrate the key concepts. To be filled in" That "To be filled in" part would also be very interesting to me as worst case I could shove the Docker Swarm's workload onto a single Docker host, although then I'm back to orchestrating with Heat and presumably Ansible but via a different path. -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Fri Jun 10 13:43:55 2022 From: smooney at redhat.com (Sean Mooney) Date: Fri, 10 Jun 2022 14:43:55 +0100 Subject: [nova] local LVM volume on compute hosts In-Reply-To: <92824CD0-05FB-4941-B9C0-C27296B958F0@manchester.ac.uk> References: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> <92824CD0-05FB-4941-B9C0-C27296B958F0@manchester.ac.uk> Message-ID: On Fri, 2022-06-10 at 13:13 +0000, Paul Harrison wrote: > > > On 2022-06 -08, at 08:26, Paul Harrison wrote: > > > > > > > > > On 2022-06 -07, at 13:25, Sean Mooney > wrote: > > > > > > no there is noting else you need to configure but this option is not what you think it is. > > > the images_type option contols what storage will be used for all non cinder storage. > > > i.e. vms that are booted with out usign a boot volume. > > > > > > > > > > > > > so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks > > > but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that > > > is what they will recive but if they use teh default storage provided by the flaovr then it will be local. > > > > thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools has never got enough votes for implementation. > > > > > > I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying ?no? to "create new volume?) - I think it was failing here > > https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b78d19ae6497/nova/virt/libvirt/imagebackend.py#L722 > > as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume. > > Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line. so on the commandlien to use local storage assume you have a flavor with root disk not equal to 0 is as follows openstack server create --flavor --image --network you can try somethign like this export IMAGE_URL=https://github.com/cirros-dev/cirros/releases/download/0.5.1/ ARCH=$(uname -m) export IMAGE=cirros-0.5.1-${ARCH}-disk.img curl --fail -L -o cirros.qcow ${IMAGE_URL}/${IMAGE} image create --disk-format qcow2 --container-format bare --public --file cirros.qcow cirros openstack flavor create --public local-storage-tiny --id auto --ram 256 --disk 1 --vcpus 1 openstack network create lvm-test openstack subnet create --subnet-range 10.0.0.0/24 --network lvm-test \ --gateway 10.0.0.1 --dns-nameserver 8.8.8.8 demo-subnet openstack router create lvm-router openstack router add subnet lvm-router lvm-subnet # assuming your public external network is called public openstack router set --external-gateway public lvm-router openstack server create --flavor local-storage-tiny --image cirros --network lvm-test local-storage-vm this is losely based on https://github.com/openstack/kolla-ansible/blob/master/tools/init-runonce > > I had a look at what the GUI sends to the /api/nova/servers/ endpoint > > {"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud?} > > However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful. > > Thanks, > Paul. > > p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server > > From haleyb.dev at gmail.com Fri Jun 10 13:56:52 2022 From: haleyb.dev at gmail.com (Brian Haley) Date: Fri, 10 Jun 2022 09:56:52 -0400 Subject: simple build fails to allocate network interface In-Reply-To: References: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> <0044A33F-0E87-433C-8130-77C66D79926A@coote.org> Message-ID: <54ad0900-9524-013c-587b-50b0f00bf8f1@gmail.com> Hi, From your log below you have this: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 205fa7d7-7067-4f63-b3d9-6a9b37b That comes from nova/network/neutron.py: if net.get('router:external') and not net.get('shared'): raise exception.ExternalNetworkAttachForbidden( network_uuid=net['id']) And from your original email 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f is the "public" network, not the "shared" network you created. My guess if you don't have the "shared" attribute set on that network, or you're using the wrong network. What does 'openstack network show 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f' output look like? -Brian On 6/10/22 06:52, tim+openstack.org at coote.org wrote: > > >> On 9 Jun 2022, at 13:20, Rodolfo Alonso Hernandez > > wrote: >> >> Hello: >> >> This seems to be a problem in Neutron. Please provide what backend you >> are using, and the Neutron server logs. If the ML2 driver has agents >> (DHCP, ML2 agent), provide the agents logs too. >> >> Regards. >> > Hullo > thanks for the response. I don?t know what to look for wrt ml2 driver - > I?m using the default `stack.sh` installation script from the devstack repo. > > The neutron log has helped, I think. I found this useful as a starting > point:https://bit.ly/3MG6jnR as it enables log > extraction of events relating to the creation of a specific server > (although that?s still quite large for a mailing list). > > I think that this may help with your question (but as I said, I don?t > really understand the question). > > There may be a dhcp issue: I cannot ping either ipv4 or ipv6 dhcp > addresses mentioned in the log from the vm. I?ve no idea where they come > from, however. > Jun 10 10:13:49 localhost.localdomain nova-compute[20725]: DEBUG > nova.network.neutron [req-1c6a3f76-b810-4f78-89ab-ba8c0d1a2b39 > req-172db7ab-dbda-4bad-912c-3bc0f26069a5 service nova] [instance: > fc8c3558-e9f5-488f-96b6-8a2eeb1f > be4f] Updated VIF entry in instance network info cache for port > 5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc. {{(pid=20725) > _build_network_info_model /opt/stack/nova/nova/network/neutron.py:3420}} > Jun 10 10:13:49 localhost.localdomain nova-compute[20725]: DEBUG > nova.network.neutron [req-1c6a3f76-b810-4f78-89ab-ba8c0d1a2b39 > req-172db7ab-dbda-4bad-912c-3bc0f26069a5 service nova] [instance: > fc8c3558-e9f5-488f-96b6-8a2eeb1f > be4f] Updating instance_info_cache with network_info: [{"id": > "5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc", "address": "fa:16:3e:02:7a:6a", > "network": {"id": "2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63", "bridge": > "br-int", "label": "private", "subnets": [{"cidr": "10.0.0.0/26", "dns": > [], "gateway": {"address": "10.0.0.1", "type": "gateway", "version": 4, > "meta": {}}, "ips": [{"address": "10.0.0.51", "type": "fixed", > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": > 4, "meta": {"dhcp_server": "10.0.0.1"}}, {"cidr": "fd5b:4cf4:f7c1::/64", > "dns": [], "gateway": {"address": "fd5b:4cf4:f7c1::1", "type": > "gateway", "version": 6, "meta": {}}, "ips": [{"address": > "fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a", "type": "fixed", "version": 6, > "meta": {}, "floating_ips": []}], "routes": [], "version": 6, "meta": > {"ipv6_address_mode": "slaac", "dhcp_server": "fd5b:4cf4:f7c1::1"}}], > "meta": {"injected": false, "tenant_id": > "6c0b876005f44086bd9bde5fef672dfe", "mtu": 1442, "physical_network": > null, "tunneled": true}}, "type": "ovs", "details": {"port_filter": > true, "connectivity": "l2", "bound_drivers": {"0": "ovn"}}, "devname": > "tap5ae4c0d6-2a", "ovs_interfaceid": > "5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc", "qbh_params": null, > "qbg_params": null, "active": false, "vnic_type": "normal", "profile": > {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}] > {{(pid=20725) update_instance_cache_with_nw_info > /opt/stack/nova/nova/network/neutron.py:117}} > Jun 10 10:14:11 localhost.localdomain nova-compute[20725]: DEBUG > nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None > None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network > cache update for instance because it is Building. {{(pid=20725) > _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} > Jun 10 10:15:12 localhost.localdomain nova-compute[20725]: DEBUG > nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None > None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network > cache update for instance because it is Building. {{(pid=20725) > _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} > Jun 10 10:16:12 localhost.localdomain nova-compute[20725]: DEBUG > nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None > None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network > cache update for instance because it is Building. {{(pid=20725) > _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} > Jun 10 10:17:12 localhost.localdomain nova-compute[20725]: DEBUG > nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None > None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network > cache update for instance because it is Building. {{(pid=20725) > _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} > Jun 10 10:17:50 localhost.localdomain neutron-server[20918]: DEBUG > ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): > CheckRevisionNumberCommand(name=5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc, > resource={'id': '5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc', 'name': '', > 'network_id': '2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63', 'tenant_id': > '6c0b876005f44086bd9bde5fef672dfe', 'mac_address': 'fa:16:3e:02:7a:6a', > 'admin_state_up': True, 'status': 'DOWN', 'device_id': > 'fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f', 'device_owner': 'compute:nova', > 'standard_attr_id': 41, 'fixed_ips': [{'subnet_id': > 'e7377bed-3e22-4e8d-9c2d-ea7ba740fcfd', 'ip_address': '10.0.0.51'}, > {'subnet_id': 'f915b88c-9988-4e1f-9060-a6295465699a', 'ip_address': > 'fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'}], 'allowed_address_pairs': [], > 'extra_dhcp_opts': [], 'security_groups': > ['ed75ab0b-d9a4-4023-9204-93038729f6d3'], 'description': '', > 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': > 'localhost.localdomain', 'binding:vif_type': 'ovs', > 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', > 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': True, 'tags': > [], 'created_at': '2022-06-10T10:13:44Z', 'updated_at': > '2022-06-10T10:13:46Z', 'revision_number': 3, 'project_id': > '6c0b876005f44086bd9bde5fef672dfe'}, resource_type=ports, > if_exists=True) {{(pid=20918) do_commit > /usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}} > Jun 10 10:17:50 localhost.localdomain neutron-server[20918]: DEBUG > ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): > SetLSwitchPortCommand(lport=5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc, > columns={'external_ids': {'neutron:port_name': '', 'neutron:device_id': > 'fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f', 'neutron:project_id': > '6c0b876005f44086bd9bde5fef672dfe', 'neutron:cidrs': '10.0.0.51/26 > fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a/64', 'neutron:device_owner': > 'compute:nova', 'neutron:network_name': > 'neutron-2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63', > 'neutron:security_group_ids': 'ed75ab0b-d9a4-4023-9204-93038729f6d3', > 'neutron:revision_number': '3'}, 'parent_name': [], 'tag': [], > 'options': {'requested-chassis': 'localhost.localdomain', > 'mcast_flood_reports': 'true'}, 'enabled': True, 'port_security': > ['fa:16:3e:02:7a:6a 10.0.0.51 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'], > 'dhcpv4_options': [UUID('01cae053-612b-4c91-81e8-39884ef035dd')], > 'dhcpv6_options': [], 'type': '', 'addresses': ['fa:16:3e:02:7a:6a > 10.0.0.51 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'], 'ha_chassis_group': > []}, if_exists=False) {{(pid=20918) do_commit > /usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}} > Jun 10 10:18:13 localhost.localdomain nova-compute[20725]: DEBUG > nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None > None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network > cache update for instance because it is Building. {{(pid=20725) > _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} > Jun 10 10:18:49 localhost.localdomain nova-compute[20725]: WARNING > nova.compute.manager [None req-4a03d9e1-6053-4167-ab43-fba30072a79c demo > admin] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Timeout waiting > for ['network-vif-plugged-5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc'] for > instance with vm_state building and task_state spawning. Event states > are: network-vif-plugged-5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc: timed out > after 300.00 seconds: eventlet.timeout.Timeout: 300 seconds > Jun 08 15:24:48 localhost.localdomain nova-compute[87555]: ERROR > nova.compute.manager nova.exception.ExternalNetworkAttachForbidden: It > is not allowed to create an interface on external network > 205fa7d7-7067-4f63-b3d9-6a9b37b> > > From paul.harrison at manchester.ac.uk Fri Jun 10 14:38:15 2022 From: paul.harrison at manchester.ac.uk (Paul Harrison) Date: Fri, 10 Jun 2022 14:38:15 +0000 Subject: [nova] local LVM volume on compute hosts In-Reply-To: <92824CD0-05FB-4941-B9C0-C27296B958F0@manchester.ac.uk> References: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> <92824CD0-05FB-4941-B9C0-C27296B958F0@manchester.ac.uk> Message-ID: <7EE30BD6-929F-435D-8FB2-7456F852FECF@manchester.ac.uk> > On 2022-06 -10, at 14:13, Paul Harrison wrote: > > > >> On 2022-06 -08, at 08:26, Paul Harrison > wrote: >> >> >> >>> On 2022-06 -07, at 13:25, Sean Mooney > wrote: >>> >>> no there is noting else you need to configure but this option is not what you think it is. >>> the images_type option contols what storage will be used for all non cinder storage. >>> i.e. vms that are booted with out usign a boot volume. >>> >>> > >>> >>> so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks >>> but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that >>> is what they will recive but if they use teh default storage provided by the flaovr then it will be local. >> >> thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools has never got enough votes for implementation. >> >> > > I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying ?no? to "create new volume?) - I think it was failing here > > https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b78d19ae6497/nova/virt/libvirt/imagebackend.py#L722 > > as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume. > > Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line. > > I had a look at what the GUI sends to the /api/nova/servers/ endpoint > > {"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud?} > > However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful. > > Thanks, > Paul. > > p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server > > Have just realised, taking the hint from the GUI - if I do not try to force anything disk/volume related on the CLI, it does what I want! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2893 bytes Desc: not available URL: From smooney at redhat.com Fri Jun 10 14:52:25 2022 From: smooney at redhat.com (Sean Mooney) Date: Fri, 10 Jun 2022 15:52:25 +0100 Subject: [nova] local LVM volume on compute hosts In-Reply-To: <7EE30BD6-929F-435D-8FB2-7456F852FECF@manchester.ac.uk> References: <11D720A5-672B-4636-84C2-BFB4BED450C0@manchester.ac.uk> <215f5ae11529ce856ef6daaad7bc21d3bba3e525.camel@redhat.com> <52687CFB-C0D3-4E5B-A6F3-A35DA838C7B6@manchester.ac.uk> <92824CD0-05FB-4941-B9C0-C27296B958F0@manchester.ac.uk> <7EE30BD6-929F-435D-8FB2-7456F852FECF@manchester.ac.uk> Message-ID: <827b54689e61a89a53bfc2df45b14d327c40806c.camel@redhat.com> On Fri, 2022-06-10 at 14:38 +0000, Paul Harrison wrote: > > > On 2022-06 -10, at 14:13, Paul Harrison wrote: > > > > > > > > > On 2022-06 -08, at 08:26, Paul Harrison > wrote: > > > > > > > > > > > > > On 2022-06 -07, at 13:25, Sean Mooney > wrote: > > > > > > > > no there is noting else you need to configure but this option is not what you think it is. > > > > the images_type option contols what storage will be used for all non cinder storage. > > > > i.e. vms that are booted with out usign a boot volume. > > > > > > > > > > > > > > > > > > so your current config will make any non boot from volume nova instance use lvm storage to provision the vm root/swap/epmeral disks > > > > but will not prevent end users requesting cinder data voluems or boot volumes via the cli/api. if the opt in to cinder stoage that > > > > is what they will recive but if they use teh default storage provided by the flaovr then it will be local. > > > > > > thanks for the explanation - it is a shame that there is not a more direct way in config to force local storage - looks like https://blueprints.launchpad.net/nova/+spec/use-libvirt-storage-pools has never got enough votes for implementation. > > > > > > > > > > I have discovered the reason why my original setup was not working (it would fail if I tried to force in the GUI, by specifying ?no? to "create new volume?) - I think it was failing here > > > > https://opendev.org/openstack/nova/src/commit/d86916360858daa06164ebc0d012b78d19ae6497/nova/virt/libvirt/imagebackend.py#L722 > > > > as the volume group device does not appear when there are no logical volumes in the group (in Centos 8 stream at least). So I forced the creation of that device by adding a dummy logical volume. > > > > Anyway, the situation is now that I can create instances that will use compute node local LVM storage, if I start them from the GUI, but not from the command line. > > > > I had a look at what the GUI sends to the /api/nova/servers/ endpoint > > > > {"availability_zone":"nova","config_drive":false,"user_data":"","disk_config":"AUTO","instance_count":1,"name":"fromgui","scheduler_hints":{},"security_groups":["48648c10-ce91-4916-9347-d88fbdba9ce6"],"create_volume_default":true,"hide_create_volume":false,"source_id":"b2a3ca46-8b0b-4748-863f-6f3e11301872","flavor_id":"da28d141-3d05-4f0f-a188-229352ccf0a3","nics":[{"net-id":"5e7a171a-ceda-4051-abe6-0496e2e8e154","v4-fixed-ip":""}],"key_name":"cloud?} > > > > However, I have not been able to find the set of command line switches for "openstack server create" that achieve the same effect - if someone knows, I would be grateful. > > > > Thanks, > > Paul. > > > > p.s. I was not really able to match up what the GUI sends with the API spec either. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server > > > > > > Have just realised, taking the hint from the GUI - if I do not try to force anything disk/volume related on the CLI, it does what I want! yep the default behavior is what you were after. unfortunetly horizons default is differnt form nova. im not sure if you can tweak that in the horizon config so the gui also default to local sotrage but for cli users if they dont do anything fancy they will use the local lvm storage not cinder. > From tim+openstack.org at coote.org Fri Jun 10 15:06:00 2022 From: tim+openstack.org at coote.org (tim+openstack.org at coote.org) Date: Fri, 10 Jun 2022 16:06:00 +0100 Subject: simple build fails to allocate network interface In-Reply-To: <54ad0900-9524-013c-587b-50b0f00bf8f1@gmail.com> References: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> <0044A33F-0E87-433C-8130-77C66D79926A@coote.org> <54ad0900-9524-013c-587b-50b0f00bf8f1@gmail.com> Message-ID: Hi Brian You?re correct about that. The Attachment to public networks means, I think, that it needs to be a private network. In my follow up, I managed to get much more detail about a specific server creation, and showed, I think, that the vm had been trying to use a couple of dhcp servers (v4 and v6), that afaict didn?t exist - I couldn?t ping them and didn?t recognise the network segments. I couldn?t work out where they came from. I got that detail by reviewing this: https://bit.ly/3MG6jnR atm, I?m trying to see if I can get this to work: https://bit.ly/3mD60PU However, I immediately ran into some VirtuaBox changes that invalidated the Vagrantfile. I think that I can fix these. The experience highlights to me the need for a decent test environment. Even then, I suspect that it?s very easy to have some of the latent bugs that you get in distributed systems, which undergrad courses note (https://bit.ly/2OgyWiI: I?m particularly fond of the example that lurks in Cassandra from that lecture set), and which AWS had to resort to formal proofs with TLA+ to even spot. Tim > On 10 Jun 2022, at 14:56, Brian Haley wrote: > > Hi, > > From your log below you have this: > > nova.exception.ExternalNetworkAttachForbidden: It > is not allowed to create an interface on external network > 205fa7d7-7067-4f63-b3d9-6a9b37b > > That comes from nova/network/neutron.py: > > if net.get('router:external') and not net.get('shared'): > raise exception.ExternalNetworkAttachForbidden( > network_uuid=net['id']) > > And from your original email 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f is the "public" network, not the "shared" network you created. > > My guess if you don't have the "shared" attribute set on that network, or you're using the wrong network. What does 'openstack network show 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f' output look like? > > -Brian > > On 6/10/22 06:52, tim+openstack.org at coote.org wrote: >>> On 9 Jun 2022, at 13:20, Rodolfo Alonso Hernandez > wrote: >>> >>> Hello: >>> >>> This seems to be a problem in Neutron. Please provide what backend you are using, and the Neutron server logs. If the ML2 driver has agents (DHCP, ML2 agent), provide the agents logs too. >>> >>> Regards. >>> >> Hullo >> thanks for the response. I don?t know what to look for wrt ml2 driver - I?m using the default `stack.sh` installation script from the devstack repo. >> The neutron log has helped, I think. I found this useful as a starting point:https://bit.ly/3MG6jnR as it enables log extraction of events relating to the creation of a specific server (although that?s still quite large for a mailing list). >> I think that this may help with your question (but as I said, I don?t really understand the question). >> There may be a dhcp issue: I cannot ping either ipv4 or ipv6 dhcp addresses mentioned in the log from the vm. I?ve no idea where they come from, however. >> Jun 10 10:13:49 localhost.localdomain nova-compute[20725]: DEBUG nova.network.neutron [req-1c6a3f76-b810-4f78-89ab-ba8c0d1a2b39 req-172db7ab-dbda-4bad-912c-3bc0f26069a5 service nova] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1f >> be4f] Updated VIF entry in instance network info cache for port 5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc. {{(pid=20725) _build_network_info_model /opt/stack/nova/nova/network/neutron.py:3420}} >> Jun 10 10:13:49 localhost.localdomain nova-compute[20725]: DEBUG nova.network.neutron [req-1c6a3f76-b810-4f78-89ab-ba8c0d1a2b39 req-172db7ab-dbda-4bad-912c-3bc0f26069a5 service nova] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1f >> be4f] Updating instance_info_cache with network_info: [{"id": "5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc", "address": "fa:16:3e:02:7a:6a", "network": {"id": "2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63", "bridge": "br-int", "label": "private", "subnets": [{"cidr": "10.0.0.0/26", "dns": [], "gateway": {"address": "10.0.0.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "10.0.0.51", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"dhcp_server": "10.0.0.1"}}, {"cidr": "fd5b:4cf4:f7c1::/64", "dns": [], "gateway": {"address": "fd5b:4cf4:f7c1::1", "type": "gateway", "version": 6, "meta": {}}, "ips": [{"address": "fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a", "type": "fixed", "version": 6, "meta": {}, "floating_ips": []}], "routes": [], "version": 6, "meta": {"ipv6_address_mode": "slaac", "dhcp_server": "fd5b:4cf4:f7c1::1"}}], "meta": {"injected": false, "tenant_id": "6c0b876005f44086bd9bde5fef672dfe", "mtu": 1442, "physical_network": null, "tunneled": true}}, "type": "ovs", "details": {"port_filter": true, "connectivity": "l2", "bound_drivers": {"0": "ovn"}}, "devname": "tap5ae4c0d6-2a", "ovs_interfaceid": "5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc", "qbh_params": null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}] {{(pid=20725) update_instance_cache_with_nw_info /opt/stack/nova/nova/network/neutron.py:117}} >> Jun 10 10:14:11 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} >> Jun 10 10:15:12 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} >> Jun 10 10:16:12 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} >> Jun 10 10:17:12 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} >> Jun 10 10:17:50 localhost.localdomain neutron-server[20918]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc, resource={'id': '5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc', 'name': '', 'network_id': '2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63', 'tenant_id': '6c0b876005f44086bd9bde5fef672dfe', 'mac_address': 'fa:16:3e:02:7a:6a', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f', 'device_owner': 'compute:nova', 'standard_attr_id': 41, 'fixed_ips': [{'subnet_id': 'e7377bed-3e22-4e8d-9c2d-ea7ba740fcfd', 'ip_address': '10.0.0.51'}, {'subnet_id': 'f915b88c-9988-4e1f-9060-a6295465699a', 'ip_address': 'fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['ed75ab0b-d9a4-4023-9204-93038729f6d3'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'localhost.localdomain', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2022-06-10T10:13:44Z', 'updated_at': '2022-06-10T10:13:46Z', 'revision_number': 3, 'project_id': '6c0b876005f44086bd9bde5fef672dfe'}, resource_type=ports, if_exists=True) {{(pid=20918) do_commit /usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}} >> Jun 10 10:17:50 localhost.localdomain neutron-server[20918]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLSwitchPortCommand(lport=5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc, columns={'external_ids': {'neutron:port_name': '', 'neutron:device_id': 'fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f', 'neutron:project_id': '6c0b876005f44086bd9bde5fef672dfe', 'neutron:cidrs': '10.0.0.51/26 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a/64', 'neutron:device_owner': 'compute:nova', 'neutron:network_name': 'neutron-2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63', 'neutron:security_group_ids': 'ed75ab0b-d9a4-4023-9204-93038729f6d3', 'neutron:revision_number': '3'}, 'parent_name': [], 'tag': [], 'options': {'requested-chassis': 'localhost.localdomain', 'mcast_flood_reports': 'true'}, 'enabled': True, 'port_security': ['fa:16:3e:02:7a:6a 10.0.0.51 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'], 'dhcpv4_options': [UUID('01cae053-612b-4c91-81e8-39884ef035dd')], 'dhcpv6_options': [], 'type': '', 'addresses': ['fa:16:3e:02:7a:6a 10.0.0.51 fd5b:4cf4:f7c1:0:f816:3eff:fe02:7a6a'], 'ha_chassis_group': []}, if_exists=False) {{(pid=20918) do_commit /usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}} >> Jun 10 10:18:13 localhost.localdomain nova-compute[20725]: DEBUG nova.compute.manager [None req-66324ba4-aeda-4bbd-9711-9eafbf9c7bb7 None None] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Skipping network cache update for instance because it is Building. {{(pid=20725) _heal_instance_info_cache /opt/stack/nova/nova/compute/manager.py:9465}} >> Jun 10 10:18:49 localhost.localdomain nova-compute[20725]: WARNING nova.compute.manager [None req-4a03d9e1-6053-4167-ab43-fba30072a79c demo admin] [instance: fc8c3558-e9f5-488f-96b6-8a2eeb1fbe4f] Timeout waiting for ['network-vif-plugged-5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc'] for instance with vm_state building and task_state spawning. Event states are: network-vif-plugged-5ae4c0d6-2ad1-4f91-9a3b-5ed0d93239cc: timed out after 300.00 seconds: eventlet.timeout.Timeout: 300 seconds >> Jun 08 15:24:48 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 205fa7d7-7067-4f63-b3d9-6a9b37b> From marios at redhat.com Fri Jun 10 15:31:02 2022 From: marios at redhat.com (Marios Andreou) Date: Fri, 10 Jun 2022 18:31:02 +0300 Subject: [TripleO] move stable/victoria tripleo to End Of Life OK? In-Reply-To: References: Message-ID: As promised last week and since there were no further comments the proposal to move tripleo stable/victoria to EOL is at [1]. Before we can move ahead with that, we first need to get folks to stop posting patches against stable/victoria. So we should give some time to get any final patches merged (links to all the repos/gerrit open reviews at [2] for reference) and then we can re-spin [1] to pick up those final commits . I'll send a separate request to stop posting to stable/victoria next week for better visibility. thanks, marios [1] https://review.opendev.org/c/openstack/releases/+/845148 [2] https://gist.github.com/marios/97c7ac7dcdcdabb744a823b1a59ffd74?permalink_comment_id=4196484 On Fri, Jun 3, 2022 at 6:22 PM Marios Andreou wrote: > > Hello TripleO > > the tripleo-ci team proposes that we transition the stable/victoria > branch across all tripleo repos [1] to End Of Life [2]. > Hopefully this is not a suprise to anyone as it has been discussed for > a while (most recent discussion I can point to is at Zed PTG [3]). > > The stable/victoria branch was moved to Extended Maintenance with [4] > so we can no longer make any releases for these repos. Victoria is not > an active branch for TripleO (not an FFU branch) so we are just > merging patches there on the way back to train. > > Are there any concerns, objections or any other comment? I'll give > this a few days and if there are no further comments I'll post the EOL > proposal and update here towards the end of next week, > > regards, marios > > [1] https://releases.openstack.org/teams/tripleo.html#victoria > [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases > [3] https://etherpad.opendev.org/p/tripleo-zed-ci-load > [4] https://review.opendev.org/c/openstack/releases/+/837982 From artem.goncharov at gmail.com Fri Jun 10 15:31:30 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Fri, 10 Jun 2022 17:31:30 +0200 Subject: [doc] question/enhancement-request on Sphinx theme used by openstack docs In-Reply-To: References: Message-ID: <2C29D595-A175-4D81-ACE9-E7CD56A18746@gmail.com> Hi, Not in any way an advertisement, but we quite heavily reworked openstackdocstheme for our cloud (https://github.com/opentelekomcloud/otcdocstheme => https://docs-beta.otc.t-systems.com/otcdocstheme/ ). We have updated to the latest bootstrap, fixed some of the layout issues, added right sidebar with local top and some other tunings. Our there is surely not to be reused by anybody else, but it may give you some hints what can be improved. Regards, Artem > On 10. Jun 2022, at 14:30, Waines, Greg wrote: > > Hey ? I am on the technical steering committee for the openstack starlingx team and also provide primary technical review for our docs at https://docs.starlingx.io/ . > > docs.starlingx.io basically uses the openstack docs team?s Sphinx theme for building our html docs ? seehttps://opendev.org/starlingx/docs . > > docs.starlingx.io is a very large doc suite. > > We are interested in improvements to the Table-of-Contents (TOC) navigation and presentation ? but we have limited Sphinx expertise. > > Specifically we are looking for the TOC support to have > - more subsection header levels in TOC > // such that user has full view of where he currently is in the doc hierarchy > - highlighting to indicate currently selected section > - separate scroll bar for TOC > // ... so you don't lose sight of selected TOC selection when you scroll down on the main page which has large content. > > Have you ever considered such doc improvements? > Did you have Sphinx expertise to do such improvements ? > > let us know, > Greg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marios at redhat.com Fri Jun 10 16:06:41 2022 From: marios at redhat.com (Marios Andreou) Date: Fri, 10 Jun 2022 19:06:41 +0300 Subject: [TripleO] move stable/victoria tripleo to End Of Life OK? In-Reply-To: References: Message-ID: On Fri, Jun 10, 2022 at 6:31 PM Marios Andreou wrote: > > As promised last week and since there were no further comments the > proposal to move tripleo stable/victoria to EOL is at [1]. > > Before we can move ahead with that, we first need to get folks to stop > posting patches against stable/victoria. > > So we should give some time to get any final patches merged (links to > all the repos/gerrit open reviews at [2] for reference) and then we > can re-spin [1] to pick up those final commits . > > I'll send a separate request to stop posting to stable/victoria next > week for better visibility. > > thanks, marios > > [1] https://review.opendev.org/c/openstack/releases/+/845148 > [2] https://gist.github.com/marios/97c7ac7dcdcdabb744a823b1a59ffd74?permalink_comment_id=4196484 ^^^ that one is more like https://gist.github.com/marios/97c7ac7dcdcdabb744a823b1a59ffd74?permalink_comment_id=4196484#gistcomment-4196484 > > > On Fri, Jun 3, 2022 at 6:22 PM Marios Andreou wrote: > > > > Hello TripleO > > > > the tripleo-ci team proposes that we transition the stable/victoria > > branch across all tripleo repos [1] to End Of Life [2]. > > Hopefully this is not a suprise to anyone as it has been discussed for > > a while (most recent discussion I can point to is at Zed PTG [3]). > > > > The stable/victoria branch was moved to Extended Maintenance with [4] > > so we can no longer make any releases for these repos. Victoria is not > > an active branch for TripleO (not an FFU branch) so we are just > > merging patches there on the way back to train. > > > > Are there any concerns, objections or any other comment? I'll give > > this a few days and if there are no further comments I'll post the EOL > > proposal and update here towards the end of next week, > > > > regards, marios > > > > [1] https://releases.openstack.org/teams/tripleo.html#victoria > > [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases > > [3] https://etherpad.opendev.org/p/tripleo-zed-ci-load > > [4] https://review.opendev.org/c/openstack/releases/+/837982 From kdhall at binghamton.edu Fri Jun 10 17:01:47 2022 From: kdhall at binghamton.edu (Dave Hall) Date: Fri, 10 Jun 2022 13:01:47 -0400 Subject: [nova][openstack-ansible] "No Valid Host Found" when running on VirtualBox In-Reply-To: References: Message-ID: Dmitry, I do have 'nested VT-x/AMD-V' turned on on all of my VirtualBox VMs, but not 'PAE/NX'. Also, I have installed the VBox Guest Additions, but I disabled vboxadd-service.service due to clock synchronization issues. Lastly, I added nova_virt_type: qemu to /etc/openstack_deploy/user_variables.yml and re-ran the playbooks. If re-running the playbooks won't flip this setting, is there some other approach, like destroying the LXC containers for nova and then re-run the playbooks? Add another compute node? Side note: I also added # rsyslog server log_hosts: log1: ip: 172.29.236.14 to openstack_user_config.yml. After doing so, 'openstack-ansible setup-hosts' hangs during a task related to rsyslog, but the other two playbooks complete successfully. Thanks. -Dave -- Dave Hall Binghamton University kdhall at binghamton.edu On Thu, Jun 9, 2022 at 3:18 PM Dmitriy Rabotyagov wrote: > Hey, > > One question - have you set nova_virt_type to qemu in overrides? As > otherwise you indeed need to have nested virtualization enabled. > > ??, 9 ???. 2022 ?., 20:21 Dave Hall : > >> Hello, >> >> I'm trying to run a test/learning cluster on VirtualBox VMs deployed with >> Openstack-Ansible prior to deploying on real hardware. >> >> The playbooks succeed, and the basic functional checks all look good. >> However, I am unable to deploy an instance due to 'No Valid Host Found'. I >> have seen that this might be because VirtualBox does not support KVM. But >> I am also perplexed - I am able to deploy instances on an AIO running in >> VirtualBox on the same physical host. >> >> If there is an easy fix for this please point me in the right direction. >> If I need to destroy and re-clone my VMs I can try that as well. In an >> effort to learn, though, I'd really like to learn how to adjust the current >> deployment. >> >> Thanks. >> >> -Dave >> >> -- >> Dave Hall >> Binghamton University >> kdhall at binghamton.edu >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Fri Jun 10 17:27:40 2022 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Fri, 10 Jun 2022 19:27:40 +0200 Subject: [nova][openstack-ansible] "No Valid Host Found" when running on VirtualBox In-Reply-To: References: Message-ID: The only playbook needs rerunning is os-nova-install.yml. And it should target "compute" VM where nova-compute service runs. There's no reason to destroy containers to apply setting, but yes, we have playbooks for destroying/recreating containers . We actually don't use rsyslog as of today. We should likely fix documentation to reflect that. All loges are stored in journald. What does `openstack compute service list` return as a result? Is there any nova-compute in the list and it's alive? if yes, you should check logs for nova-scheduler in nova_api container and placement in it's container. Eventually logs are exposed to host also but can't recall from top of my head the command to check from host efficiently. ??, 10 ???. 2022 ?., 19:04 Dave Hall : > Dmitry, > > I do have 'nested VT-x/AMD-V' turned on on all of my VirtualBox VMs, but > not 'PAE/NX'. Also, I have installed the VBox Guest Additions, but I > disabled vboxadd-service.service due to clock synchronization issues. > > Lastly, I added > > nova_virt_type: qemu > > to /etc/openstack_deploy/user_variables.yml and re-ran the playbooks. If > re-running the playbooks won't flip this setting, is there some other > approach, like destroying the LXC containers for nova and then re-run the > playbooks? Add another compute node? > > Side note: I also added > > # rsyslog server > log_hosts: > log1: > ip: 172.29.236.14 > > to openstack_user_config.yml. After doing so, 'openstack-ansible > setup-hosts' hangs during a task related to rsyslog, but the other two > playbooks complete successfully. > > Thanks. > > -Dave > > > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > > > On Thu, Jun 9, 2022 at 3:18 PM Dmitriy Rabotyagov > wrote: > >> Hey, >> >> One question - have you set nova_virt_type to qemu in overrides? As >> otherwise you indeed need to have nested virtualization enabled. >> >> ??, 9 ???. 2022 ?., 20:21 Dave Hall : >> >>> Hello, >>> >>> I'm trying to run a test/learning cluster on VirtualBox VMs deployed >>> with Openstack-Ansible prior to deploying on real hardware. >>> >>> The playbooks succeed, and the basic functional checks all look good. >>> However, I am unable to deploy an instance due to 'No Valid Host Found'. I >>> have seen that this might be because VirtualBox does not support KVM. But >>> I am also perplexed - I am able to deploy instances on an AIO running in >>> VirtualBox on the same physical host. >>> >>> If there is an easy fix for this please point me in the right >>> direction. If I need to destroy and re-clone my VMs I can try that as >>> well. In an effort to learn, though, I'd really like to learn how to >>> adjust the current deployment. >>> >>> Thanks. >>> >>> -Dave >>> >>> -- >>> Dave Hall >>> Binghamton University >>> kdhall at binghamton.edu >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From adivya1.singh at gmail.com Fri Jun 10 19:10:08 2022 From: adivya1.singh at gmail.com (Adivya Singh) Date: Sat, 11 Jun 2022 00:40:08 +0530 Subject: Regarding Policy.json entries for glance image update not working for a user Message-ID: Hi Team, I have a use case where I have to give a user restriction on updating the image properties as a member. I have created a policy Json file and give the modify_image rule to the particular role, but still it is not working "modify_image": "role:user", This role is created in OpenStack. but still it is failing while updating properties with a particular user assigned to a role as "access denied" and unauthorized access Regards Adivya Singh -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Fri Jun 10 19:36:06 2022 From: satish.txt at gmail.com (Satish Patel) Date: Fri, 10 Jun 2022 21:36:06 +0200 Subject: [openstack-ansible] [kolla-ansible] Which deployment tool for first timer? In-Reply-To: References: Message-ID: Let me share my experience because I?m end user and cloud operator. I?m running 5 large production deployment on openstack-ansible (each cloud has more than 250+ computes nodes so you know scale). Openstack-ansible community is very supportive and helped me a lot to bring up massive deployment with scale. Few month back one of my customer ask me to use kolla-ansible so I use kolla-ansible to reply their cloud. For me both are great but I have more command on openstack-ansible and after deploying 5 large cloud it?s in my blood. Both ansible based so I don?t think you will have any difficulty to play. You can find me or others on IRC if you have more questions. Sent from my iPhone > On Jun 10, 2022, at 10:56 AM, Mark Goddard wrote: > > ? > This has been discussed before, but it would be nice if we as a community had an easier way to answer this question. I know many of us feel conflicted about aggressively promoting our own projects over alternatives, but also want to give them a fair chance. Maybe each project needs a "why pick me" page that could be linked to in a standard response. > > Mark > >> On Thu, 9 Jun 2022 at 16:40, Jonathan Rosser wrote: >> Hi Dave, >> >> I have been hesitating to reply to your mailing list post because it doesn't feel right to pitch two community tools against each other here on the mailing list - so i won't do that here. >> >> I would say that the deployment tool is a means to an end. So you should look at the technology choices, upgrade paths, support for "day 2 operations", how bugs get addressed, documentation, operator experience etc etc. Only you can decide which is appropriate for the constraints and requirements of your deployment. >> >> My reply will obviously be biassed, as I am a big contributor to openstack-ansible. My observation is that the operators that gain the most out of any of these tools are the ones who engage with the community around those tools, primarily in the case of openstack-ansible that would be through our IRC channel, weekly meetings and bug reports on Launchpad. You will gain insight and be able to leverage the knowledge of other operators who in some cases have literally written the book on various aspects of OpenStack. Trying to fight though every decision or problem on your own is the worst way to approach using any of these community driven tools. >> >> If you instead want a more "shrink wrap" approach to an installer, or more formal support, then it would be wise to look at the product oriented offerings from the large vendors. >> >> Both openstack-ansible and kolla-ansible will expect you to make a good number of decisions about the specifics of your deployment, for example storage, networking and security concerns. Both would also expect you to gain sufficient knowledge about how OpenStack itself works to be able to make good use of the customization opportunities that both provide. This is really the unique selling point of the community tooling, you get a very high degree of customization potential but that can come at the cost of some complexity. >> >> As you are already using openstack-ansible I would suggest that you continue, but as I've already said I have an existing interest here and I really don't want to start a tooling debate. Join us in IRC in #openstack-ansible and discuss any pain points you have. This way we can hopefully help you out, or address specific issues in the code - you may have discovered legitimate bugs or a use case that is not straightforward to fulfill. This is how all of the community tools get improved and evolved over time. >> >> On one specific point I would recommend that you move entirely to Debian 11 as Xena will be the last release that openstack-ansible supports Buster. >> >> I'm not sure there is a fool-proof installer really. Installing the code is one thing, being effective though upgrades and applying bugfixes to a production environment is a different and a more important concern in the long term. Both openstack-ansible and kolla-ansible offer "all-in-one" deployments which are intended as "should-just-work" demonstrators of how things fit together and for lightweight testing. Scaling those out to larger deployments is where the real work is, and neither tool sets out to be particularly prescriptive about some parts of how you build your environment. >> >> Hopefully this is helpful, >> Jonathan. >> >>> On 09/06/2022 15:58, Dave Hall wrote: >>> Hello, >>> >>> My question is about OpenStack-Ansible vs. Kolla-Ansible. While I am sensitive to the effort that has been put into both of these projects, what I really need right now is the most fool-proof way to deploy and manage a small production cluster for academic instructional use. >>> >>> (BTW, I do understand that there are other differences between Kolla and regular openstack.) >>> >>> I have been working for a couple months with OpenStack-Ansible to deploy a small (3-node) Xena test cluster on VirtualBox VMs in preparation for a larger deployment on real hardware - 6 to 10 nodes. My VirtualBox deployment has been: >>> >>> Debian 11 deployment, Debian 10 infra, compute, and storage nodes >>> >>> It has been slow going, at least partially due to some issues and limitations with VirtualBox (and Vagrant early on). However, deploying a test cluster on VMs still seems preferable to just diving into deployment on real hardware and going through multiple scrubs/reinstalls. >>> >>> I've recently seen more posts in the list about Kolla-Ansible. So, as a 'beginner', should I shift and look at Kolla-Ansible, or should I stay the course and continue with Openstack-Ansible? What are the pros and cons of each? >>> >>> For that matter, is there some other deployment mechanism that would be more fool-proof for a first-timer? Although I'm more familiar with Ansible than the other tools (Chef, Puppet, etc.) I'm most interested in how to get a cluster up and running regardless of the underlying tools. >>> >>> Thanks. >>> >>> -Dave >>> >>> -- >>> Dave Hall >>> Binghamton University >>> kdhall at binghamton.edu >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Sat Jun 11 01:51:01 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 10 Jun 2022 20:51:01 -0500 Subject: [all][tc] What's happening in Technical Committee: summary 10 June 2022: Reading: 5 min Message-ID: <18150751f5f.11e69cf6d82575.1368494779385694947@ghanshyammann.com> Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We cancelled this week's meeting. The next TC weekly meeting will be on 16 June Thursday at 15:00 UTC, feel free to add the topic to the agenda[1] by 15 June. 2. What we completed this week: ========================= * Added community infrastructure contributor help in upstream investment[2] 3. Activities In progress: ================== TC Tracker for Zed cycle ------------------------------ * Zed tracker etherpad includes the TC working items[3], many items are in-progress. Open Reviews ----------------- * Five open reviews for ongoing activities[4]. New release cadence "SLURP" open items -------------------------------------------------- 1. release notes strategy: Brian proposal for ""SLURP release notes approach is up for review[5]. Improve project governance --------------------------------- Slawek has the framework proposal up and is under review[6]. New ELK service dashboard: e-r service ----------------------------------------------- No update this week. Consistent and Secure Default RBAC ------------------------------------------- We called for the operator's feedback on RBAC new directions, I sent an email about it[7] and also created etherpad to collect feedback in a central place[8]. My humble request is to use this central etherpad for feedback/discussion otherwise it might be difficult for me to collect/know feedback conclusions from a lot of 100s places. We have feedback from the berlin summit but we still encourage operators/local events or so keep adding the feedback and your view on RBAC. Please note that operator feedback is very important for us to proceed on 'scope' in SRBAC. If you are operator or know anyone around you, we request you to provide feedback. 2021 User Survey TC Question Analysis ----------------------------------------------- No update on this. The survey summary is up for review[9]. Feel free to check and provide feedback. Zed cycle Leaderless projects ---------------------------------- No updates on this. Only Adjutant project is leaderless/maintainer-less. We will check Adjutant's the situation again on ML and hope Braden will be ready with their company side permission[10]. Fixing Zuul config error ---------------------------- Requesting projects with zuul config error to look into those and fix them which should not take much time[11][12]. Project updates ------------------- * None 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[13]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [14] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/843749 [3] https://etherpad.opendev.org/p/tc-zed-tracker [4] https://review.opendev.org/q/projects:openstack/governance+status:open [5] https://review.opendev.org/c/openstack/project-team-guide/+/843457 [6] https://review.opendev.org/c/openstack/governance/+/839880 [7] http://lists.openstack.org/pipermail/openstack-discuss/2022-June/028878.html [8] https://etherpad.opendev.org/p/rbac-operator-feedback [9] https://review.opendev.org/c/openstack/governance/+/836888 [10] http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027626.html [11] https://etherpad.opendev.org/p/zuul-config-error-openstack [12] http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html [13] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [14] http://eavesdrop.openstack.org/#Technical_Committee_Meeting -gmann From kdhall at binghamton.edu Sat Jun 11 22:28:01 2022 From: kdhall at binghamton.edu (Dave Hall) Date: Sat, 11 Jun 2022 18:28:01 -0400 Subject: [nova][openstack-ansible] "No Valid Host Found" when running on VirtualBox In-Reply-To: References: Message-ID: Dmitry, I reran os-nova-install.yml. I observed from the dashboard that my compute node is listed as running QEMU. `openstack compute service list` shows my compute node as up and enabled. I think I did find a way to get past 'No Valid Host'. But then the instance failed with a volume error. I think maybe I don't have all of the needed pieces in place - networks, ports, security groups, etc. Is there a tutorial that you could point me to? Thanks. -Dave -- Dave Hall Binghamton University kdhall at binghamton.edu On Fri, Jun 10, 2022 at 1:28 PM Dmitriy Rabotyagov wrote: > The only playbook needs rerunning is os-nova-install.yml. And it should > target "compute" VM where nova-compute service runs. There's no reason to > destroy containers to apply setting, but yes, we have playbooks for > destroying/recreating containers . > > We actually don't use rsyslog as of today. We should likely fix > documentation to reflect that. > All loges are stored in journald. > > What does `openstack compute service list` return as a result? Is there > any nova-compute in the list and it's alive? > > if yes, you should check logs for nova-scheduler in nova_api container and > placement in it's container. > Eventually logs are exposed to host also but can't recall from top of my > head the command to check from host efficiently. > > > > ??, 10 ???. 2022 ?., 19:04 Dave Hall : > >> Dmitry, >> >> I do have 'nested VT-x/AMD-V' turned on on all of my VirtualBox VMs, but >> not 'PAE/NX'. Also, I have installed the VBox Guest Additions, but I >> disabled vboxadd-service.service due to clock synchronization issues. >> >> Lastly, I added >> >> nova_virt_type: qemu >> >> to /etc/openstack_deploy/user_variables.yml and re-ran the playbooks. >> If re-running the playbooks won't flip this setting, is there some other >> approach, like destroying the LXC containers for nova and then re-run the >> playbooks? Add another compute node? >> >> Side note: I also added >> >> # rsyslog server >> log_hosts: >> log1: >> ip: 172.29.236.14 >> >> to openstack_user_config.yml. After doing so, 'openstack-ansible >> setup-hosts' hangs during a task related to rsyslog, but the other two >> playbooks complete successfully. >> >> Thanks. >> >> -Dave >> >> >> >> -- >> Dave Hall >> Binghamton University >> kdhall at binghamton.edu >> >> >> On Thu, Jun 9, 2022 at 3:18 PM Dmitriy Rabotyagov < >> noonedeadpunk at gmail.com> wrote: >> >>> Hey, >>> >>> One question - have you set nova_virt_type to qemu in overrides? As >>> otherwise you indeed need to have nested virtualization enabled. >>> >>> ??, 9 ???. 2022 ?., 20:21 Dave Hall : >>> >>>> Hello, >>>> >>>> I'm trying to run a test/learning cluster on VirtualBox VMs deployed >>>> with Openstack-Ansible prior to deploying on real hardware. >>>> >>>> The playbooks succeed, and the basic functional checks all look good. >>>> However, I am unable to deploy an instance due to 'No Valid Host Found'. I >>>> have seen that this might be because VirtualBox does not support KVM. But >>>> I am also perplexed - I am able to deploy instances on an AIO running in >>>> VirtualBox on the same physical host. >>>> >>>> If there is an easy fix for this please point me in the right >>>> direction. If I need to destroy and re-clone my VMs I can try that as >>>> well. In an effort to learn, though, I'd really like to learn how to >>>> adjust the current deployment. >>>> >>>> Thanks. >>>> >>>> -Dave >>>> >>>> -- >>>> Dave Hall >>>> Binghamton University >>>> kdhall at binghamton.edu >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.aminian.server at gmail.com Sun Jun 12 06:37:06 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Sun, 12 Jun 2022 11:07:06 +0430 Subject: snapshot problem Message-ID: When I snapshot from the instance , server will gone away and its not reachable until the snapshot is complete here is the logs : 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting 2022-06-12 09:42:34.755 7 INFO nova.compute.manager [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) 2022-06-12 09:42:34.995 7 INFO nova.compute.manager [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the instance has a pending task (image_pending_upload). Skip. 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver [req-e79e4177-4712-4795-91da-853bc524fac0 93fb420b3c604d4fae408b81135b58e9 ef940663426b4c87a751afaf13b758e0 - default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning image upload 2022-06-12 09:43:08.778 7 INFO nova.compute.manager [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Sun Jun 12 08:10:38 2022 From: eblock at nde.ag (Eugen Block) Date: Sun, 12 Jun 2022 08:10:38 +0000 Subject: snapshot problem In-Reply-To: Message-ID: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> Hi, can you share more details about your environment? Which openstack version is it? What is the storage backend? In earlier releases there was an option: #disable_libvirt_livesnapshot = false but this option has been deprecated. But if you're on an older version that could explain it. Zitat von Parsa Aminian : > When I snapshot from the instance , server will gone away and its not > reachable until the snapshot is complete here is the logs : > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the instance > has a pending task (image_pending_upload). Skip. > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver > [req-e79e4177-4712-4795-91da-853bc524fac0 93fb420b3c604d4fae408b81135b58e9 > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning image > upload > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) From noonedeadpunk at gmail.com Sun Jun 12 08:59:35 2022 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Sun, 12 Jun 2022 10:59:35 +0200 Subject: [nova][openstack-ansible] "No Valid Host Found" when running on VirtualBox In-Reply-To: References: Message-ID: What's the volume error? What does 'openstack volume show ' show for status? For networking good set of useful commands you can find here https://docs.openstack.org/neutron/yoga/admin/archives/use.html But all really depends on setup, as openstack is very flexible so it all depends. ??, 12 ???. 2022 ?., 00:28 Dave Hall : > Dmitry, > > I reran os-nova-install.yml. I observed from the dashboard that my > compute node is listed as running QEMU. > > `openstack compute service list` shows my compute node as up and enabled. > > I think I did find a way to get past 'No Valid Host'. But then the > instance failed with a volume error. > > I think maybe I don't have all of the needed pieces in place - networks, > ports, security groups, etc. Is there a tutorial that you could point me > to? > > Thanks. > > -Dave > > -- > Dave Hall > Binghamton University > kdhall at binghamton.edu > > > > On Fri, Jun 10, 2022 at 1:28 PM Dmitriy Rabotyagov < > noonedeadpunk at gmail.com> wrote: > >> The only playbook needs rerunning is os-nova-install.yml. And it should >> target "compute" VM where nova-compute service runs. There's no reason to >> destroy containers to apply setting, but yes, we have playbooks for >> destroying/recreating containers . >> >> We actually don't use rsyslog as of today. We should likely fix >> documentation to reflect that. >> All loges are stored in journald. >> >> What does `openstack compute service list` return as a result? Is there >> any nova-compute in the list and it's alive? >> >> if yes, you should check logs for nova-scheduler in nova_api container >> and placement in it's container. >> Eventually logs are exposed to host also but can't recall from top of my >> head the command to check from host efficiently. >> >> >> >> ??, 10 ???. 2022 ?., 19:04 Dave Hall : >> >>> Dmitry, >>> >>> I do have 'nested VT-x/AMD-V' turned on on all of my VirtualBox VMs, but >>> not 'PAE/NX'. Also, I have installed the VBox Guest Additions, but I >>> disabled vboxadd-service.service due to clock synchronization issues. >>> >>> Lastly, I added >>> >>> nova_virt_type: qemu >>> >>> to /etc/openstack_deploy/user_variables.yml and re-ran the playbooks. >>> If re-running the playbooks won't flip this setting, is there some other >>> approach, like destroying the LXC containers for nova and then re-run the >>> playbooks? Add another compute node? >>> >>> Side note: I also added >>> >>> # rsyslog server >>> log_hosts: >>> log1: >>> ip: 172.29.236.14 >>> >>> to openstack_user_config.yml. After doing so, 'openstack-ansible >>> setup-hosts' hangs during a task related to rsyslog, but the other two >>> playbooks complete successfully. >>> >>> Thanks. >>> >>> -Dave >>> >>> >>> >>> -- >>> Dave Hall >>> Binghamton University >>> kdhall at binghamton.edu >>> >>> >>> On Thu, Jun 9, 2022 at 3:18 PM Dmitriy Rabotyagov < >>> noonedeadpunk at gmail.com> wrote: >>> >>>> Hey, >>>> >>>> One question - have you set nova_virt_type to qemu in overrides? As >>>> otherwise you indeed need to have nested virtualization enabled. >>>> >>>> ??, 9 ???. 2022 ?., 20:21 Dave Hall : >>>> >>>>> Hello, >>>>> >>>>> I'm trying to run a test/learning cluster on VirtualBox VMs deployed >>>>> with Openstack-Ansible prior to deploying on real hardware. >>>>> >>>>> The playbooks succeed, and the basic functional checks all look good. >>>>> However, I am unable to deploy an instance due to 'No Valid Host Found'. I >>>>> have seen that this might be because VirtualBox does not support KVM. But >>>>> I am also perplexed - I am able to deploy instances on an AIO running in >>>>> VirtualBox on the same physical host. >>>>> >>>>> If there is an easy fix for this please point me in the right >>>>> direction. If I need to destroy and re-clone my VMs I can try that as >>>>> well. In an effort to learn, though, I'd really like to learn how to >>>>> adjust the current deployment. >>>>> >>>>> Thanks. >>>>> >>>>> -Dave >>>>> >>>>> -- >>>>> Dave Hall >>>>> Binghamton University >>>>> kdhall at binghamton.edu >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Sun Jun 12 11:02:08 2022 From: eblock at nde.ag (Eugen Block) Date: Sun, 12 Jun 2022 11:02:08 +0000 Subject: snapshot problem In-Reply-To: References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> Message-ID: <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> Have you tried with debug logs? Has it worked with live snapshots before for other instances or did it never work and all snapshots were "cold"? Zitat von Parsa Aminian : > Hi > kolla-ansible victoria version with ceph backend without volume > > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block wrote: > >> Hi, >> >> can you share more details about your environment? Which openstack >> version is it? What is the storage backend? In earlier releases there >> was an option: >> >> #disable_libvirt_livesnapshot = false >> >> but this option has been deprecated. But if you're on an older version >> that could explain it. >> >> Zitat von Parsa Aminian : >> >> > When I snapshot from the instance , server will gone away and its not >> > reachable until the snapshot is complete here is the logs : >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> instance >> > has a pending task (image_pending_upload). Skip. >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> 93fb420b3c604d4fae408b81135b58e9 >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning image >> > upload >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) >> >> >> >> >> From eblock at nde.ag Sun Jun 12 13:14:36 2022 From: eblock at nde.ag (Eugen Block) Date: Sun, 12 Jun 2022 13:14:36 +0000 Subject: snapshot problem In-Reply-To: References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> Message-ID: <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> You should respond to the list so other users can try to support you. So nova is trying to live snapshot the instance: > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > instance snapshotting > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning > live snapshot process But I don't see any 'rbd snap create' command. Either the rbd image doesn't support it or there is some setting to keep all rbd images "flat". Can you check any relevant configs you might have in nova? Also can you show the output of 'rbd info /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if the underlying rbd functions work as expected you could try to create a live snapshot manually: rbd snap create /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap And paste any relevant output here. Zitat von Parsa Aminian : > Its not working for any instances and all of them are paused . I enable > debug logs please check the logs : > > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > :: waited 0.000s inner > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute host > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > 63426b4c87a751afaf13b758e0 - default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process > default default] Lazy-loading 'pci_devices' on Instance uuid > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 > obj_load_attr > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event > => Paused> emit_event > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state > after lifecycle event "Paused"; current vm_state: active, current > task_state: image_pending_upload, current DB power_state: 1, VM > power_state: 3 handle_lifecycle_event > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the instance > has a pending task (image_pending_upload). Skip. > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd > (subprocess): qemu-img convert -t none -O raw -f raw > rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad > execute > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event > => Stopped> emit_event > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > INFO nova.compute.manager [-] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - - > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > _get_power_state > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - - > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing > instance power state after lifecycle event "Stopped"; current vm_state: > active, current task_state: image_pending_upload, current DB power_state: > 1, VM power_state: 4 handle_lifecycle_event > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - - - > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state > the instance has a pending task (image_pending_upload). Skip. > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock > "25c8d676-e20a-4238 > -a45c-d51daa62b941" acquired by > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > :: waited 0.000s inner > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the instance > has a pending task (image_pending_upload). Skip. > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock > "25c8d676-e20a-4238-a45c-d51daa62b941" released by > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > :: held 0.001s inner > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img convert > -t none -O raw -f raw > rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" > returned: 0 in 23.730s execute > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot > extracted, beginning image upload > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event > => Started> emit_event > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > _get_power_state > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state > after lifecycle event "Started"; current vm_state: active, current > task_state: image_pending_upload, current DB power_state: 1, VM > power_state: 1 handle_lifecycle_event > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network info > cache for instance _get_instance_nw_info > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache with > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "address": > "fa:16:3e:ca:00:d9", "network": {"id": > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label": > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": {}}], > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4, > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, > "meta": {}}], "meta": {"injected": true, "tenant_id": > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network": > "physnet1", "tunneled": false}}, "type": "ovs", "details": {"connectivity": > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "qbh_params": > null, "qbg_params": null, "active": true, "vnic_type": "normal", "profile": > {}, "preserve_on_delete": false, "meta": {}}] > update_instance_cache_with_nw_info > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > compute host and has allocations in placement: {'resources': {'VCPU': 1, > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot the > instance on the hypervisor. > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > :: waited 0.000s inner > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute host > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager > [req-e79e4177-4712-4795-91da-853bc524fac0 93fb420b3c604d4fae408b81135b58e9 > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: > >> Have you tried with debug logs? Has it worked with live snapshots >> before for other instances or did it never work and all snapshots were >> "cold"? >> >> Zitat von Parsa Aminian : >> >> > Hi >> > kolla-ansible victoria version with ceph backend without volume >> > >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block wrote: >> > >> >> Hi, >> >> >> >> can you share more details about your environment? Which openstack >> >> version is it? What is the storage backend? In earlier releases there >> >> was an option: >> >> >> >> #disable_libvirt_livesnapshot = false >> >> >> >> but this option has been deprecated. But if you're on an older version >> >> that could explain it. >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> > When I snapshot from the instance , server will gone away and its not >> >> > reachable until the snapshot is complete here is the logs : >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> instance >> >> > has a pending task (image_pending_upload). Skip. >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning >> image >> >> > upload >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From p.aminian.server at gmail.com Sun Jun 12 19:33:16 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Mon, 13 Jun 2022 00:03:16 +0430 Subject: snapshot problem In-Reply-To: <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> Message-ID: Hi kolla-ansible victoria version with ceph backend . rbd info output : rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': size 20 GiB in 5120 objects order 22 (4 MiB objects) snapshot_count: 0 id: b69aaf907284da block_name_prefix: rbd_data.b69aaf907284da format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Fri May 20 00:04:47 2022 access_timestamp: Sun Jun 12 16:26:02 2022 --------------- also live snapshot seems to work correctly without any error or any downtime : docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk SNAPID NAME SIZE PROTECTED TIMESTAMP 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 also on compute nova.conf, images_type is set on rbd . On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: > You should respond to the list so other users can try to support you. > > So nova is trying to live snapshot the instance: > > > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - > > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > > instance snapshotting > > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning > > live snapshot process > > But I don't see any 'rbd snap create' command. Either the rbd image > doesn't support it or there is some setting to keep all rbd images > "flat". Can you check any relevant configs you might have in nova? > Also can you show the output of 'rbd info > /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if > the underlying rbd functions work as expected you could try to create > a live snapshot manually: > > rbd snap create /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap > > And paste any relevant output here. > > Zitat von Parsa Aminian : > > > Its not working for any instances and all of them are paused . I enable > > debug logs please check the logs : > > > > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager > > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for > > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock > > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > > > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > > :: waited 0.000s inner > > > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker > > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute > host > > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': > > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager > > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > 4dbffaa9c14e401c8c210e23ebe0ab7b > > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > 4dbffaa9c14e401c8c210e23ebe0ab7b > > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > > 63426b4c87a751afaf13b758e0 - default default] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process > > default default] Lazy-loading 'pci_devices' on Instance uuid > > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance > > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > 4dbffaa9c14e401c8c210e23ebe0ab7b > > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading > > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 > > obj_load_attr > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event > > > => Paused> emit_event > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) > > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager > > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager > > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state > > after lifecycle event "Paused"; current vm_state: active, current > > task_state: image_pending_upload, current DB power_state: 1, VM > > power_state: 3 handle_lifecycle_event > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager > > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > instance > > has a pending task (image_pending_upload). Skip. > > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils > > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > 4dbffaa9c14e401c8c210e23ebe0ab7b > > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd > > (subprocess): qemu-img convert -t none -O raw -f raw > > > rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > > > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad > > execute > > > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event > > > => Stopped> emit_event > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > > INFO nova.compute.manager [-] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) > > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - > - > > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > > _get_power_state > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - > - > > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing > > instance power state after lifecycle event "Stopped"; current vm_state: > > active, current task_state: image_pending_upload, current DB power_state: > > 1, VM power_state: 4 handle_lifecycle_event > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - > - - > > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During > sync_power_state > > the instance has a pending task (image_pending_upload). Skip. > > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager > > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for > > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock > > "25c8d676-e20a-4238 > > -a45c-d51daa62b941" acquired by > > > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > > :: waited 0.000s inner > > > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > instance > > has a pending task (image_pending_upload). Skip. > > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock > > "25c8d676-e20a-4238-a45c-d51daa62b941" released by > > > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > > :: held 0.001s inner > > > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils > > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > 4dbffaa9c14e401c8c210e23ebe0ab7b > > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img convert > > -t none -O raw -f raw > > > rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > > > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" > > returned: 0 in 23.730s execute > > > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > Snapshot > > extracted, beginning image upload > > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver > > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event > > > => Started> emit_event > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager > > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) > > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > > _get_power_state > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state > > after lifecycle event "Started"; current vm_state: active, current > > task_state: image_pending_upload, current DB power_state: 1, VM > > power_state: 1 handle_lifecycle_event > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager > > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event > > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils > > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock > > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock > > > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 > > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network info > > cache for instance _get_instance_nw_info > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 > > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron > > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache with > > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "address": > > "fa:16:3e:ca:00:d9", "network": {"id": > > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label": > > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": > > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, > > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": {}}], > > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4, > > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", > > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": > 4, > > "meta": {}}], "meta": {"injected": true, "tenant_id": > > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network": > > "physnet1", "tunneled": false}}, "type": "ovs", "details": > {"connectivity": > > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": > > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", > > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "qbh_params": > > null, "qbg_params": null, "active": true, "vnic_type": "normal", > "profile": > > {}, "preserve_on_delete": false, "meta": {}}] > > update_instance_cache_with_nw_info > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 > > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > > compute host and has allocations in placement: {'resources': {'VCPU': 1, > > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager > > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > 4dbffaa9c14e401c8c210e23ebe0ab7b > > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot the > > instance on the hypervisor. > > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager > > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for > > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > > > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > > :: waited 0.000s inner > > > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker > > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute > host > > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': > > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager > > [req-e79e4177-4712-4795-91da-853bc524fac0 > 93fb420b3c604d4fae408b81135b58e9 > > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > > > > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: > > > >> Have you tried with debug logs? Has it worked with live snapshots > >> before for other instances or did it never work and all snapshots were > >> "cold"? > >> > >> Zitat von Parsa Aminian : > >> > >> > Hi > >> > kolla-ansible victoria version with ceph backend without volume > >> > > >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block wrote: > >> > > >> >> Hi, > >> >> > >> >> can you share more details about your environment? Which openstack > >> >> version is it? What is the storage backend? In earlier releases there > >> >> was an option: > >> >> > >> >> #disable_libvirt_livesnapshot = false > >> >> > >> >> but this option has been deprecated. But if you're on an older > version > >> >> that could explain it. > >> >> > >> >> Zitat von Parsa Aminian : > >> >> > >> >> > When I snapshot from the instance , server will gone away and its > not > >> >> > reachable until the snapshot is complete here is the logs : > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager > >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) > >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager > >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > >> >> instance > >> >> > has a pending task (image_pending_upload). Skip. > >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) > >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver > >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> >> 93fb420b3c604d4fae408b81135b58e9 > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning > >> image > >> >> > upload > >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager > >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> > >> > >> > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Sun Jun 12 22:10:19 2022 From: zigo at debian.org (Thomas Goirand) Date: Mon, 13 Jun 2022 00:10:19 +0200 Subject: Upgrading to a more recent version of jsonschema Message-ID: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> Hi, A few DDs are pushing me to upgrade jsonschema in Debian Unstable. However, OpenStack global requirements are still stuck at 3.2.0. Is there any reason for it, or should we attempt to upgrade to 4.6.0? I'd really appreciate if someone (else than me) was driving this... Cheers, Thomas Goirand (zigo) From eblock at nde.ag Mon Jun 13 07:33:00 2022 From: eblock at nde.ag (Eugen Block) Date: Mon, 13 Jun 2022 07:33:00 +0000 Subject: snapshot problem In-Reply-To: References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> Message-ID: <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> Could you share your whole nova.conf (only the uncommented options)? Is this option set in your env? #snapshot_image_format = Also when you manually created the snapshot did you do it as the nova user on the compute node? If not, could you retry? Zitat von Parsa Aminian : > Hi > kolla-ansible victoria version with ceph backend . > rbd info output : > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': > size 20 GiB in 5120 objects > order 22 (4 MiB objects) > snapshot_count: 0 > id: b69aaf907284da > block_name_prefix: rbd_data.b69aaf907284da > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, > deep-flatten > op_features: > flags: > create_timestamp: Fri May 20 00:04:47 2022 > access_timestamp: Sun Jun 12 16:26:02 2022 > --------------- > also live snapshot seems to work correctly without any error or any > downtime : > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk > SNAPID NAME SIZE PROTECTED TIMESTAMP > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 > > also on compute nova.conf, images_type is set on rbd . > > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: > >> You should respond to the list so other users can try to support you. >> >> So nova is trying to live snapshot the instance: >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> > instance snapshotting >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning >> > live snapshot process >> >> But I don't see any 'rbd snap create' command. Either the rbd image >> doesn't support it or there is some setting to keep all rbd images >> "flat". Can you check any relevant configs you might have in nova? >> Also can you show the output of 'rbd info >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if >> the underlying rbd functions work as expected you could try to create >> a live snapshot manually: >> >> rbd snap create /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap >> >> And paste any relevant output here. >> >> Zitat von Parsa Aminian : >> >> > Its not working for any instances and all of them are paused . I enable >> > debug logs please check the logs : >> > >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> > >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> > :: waited 0.000s inner >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute >> host >> > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> > 63426b4c87a751afaf13b758e0 - default default] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process >> > default default] Lazy-loading 'pci_devices' on Instance uuid >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading >> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 >> > obj_load_attr >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event >> > > > => Paused> emit_event >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state >> > after lifecycle event "Paused"; current vm_state: active, current >> > task_state: image_pending_upload, current DB power_state: 1, VM >> > power_state: 3 handle_lifecycle_event >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> instance >> > has a pending task (image_pending_upload). Skip. >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd >> > (subprocess): qemu-img convert -t none -O raw -f raw >> > >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> > >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad >> > execute >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event >> > > > => Stopped> emit_event >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> > INFO nova.compute.manager [-] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - >> - >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> > _get_power_state >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - >> - >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing >> > instance power state after lifecycle event "Stopped"; current vm_state: >> > active, current task_state: image_pending_upload, current DB power_state: >> > 1, VM power_state: 4 handle_lifecycle_event >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - >> - - >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During >> sync_power_state >> > the instance has a pending task (image_pending_upload). Skip. >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock >> > "25c8d676-e20a-4238 >> > -a45c-d51daa62b941" acquired by >> > >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> > :: waited 0.000s inner >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> instance >> > has a pending task (image_pending_upload). Skip. >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by >> > >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> > :: held 0.001s inner >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img convert >> > -t none -O raw -f raw >> > >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> > >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" >> > returned: 0 in 23.730s execute >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> Snapshot >> > extracted, beginning image upload >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event >> > > > => Started> emit_event >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> > _get_power_state >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state >> > after lifecycle event "Started"; current vm_state: active, current >> > task_state: image_pending_upload, current DB power_state: 1, VM >> > power_state: 1 handle_lifecycle_event >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network info >> > cache for instance _get_instance_nw_info >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache with >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "address": >> > "fa:16:3e:ca:00:d9", "network": {"id": >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label": >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, >> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": {}}], >> > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4, >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": >> 4, >> > "meta": {}}], "meta": {"injected": true, "tenant_id": >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network": >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": >> {"connectivity": >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "qbh_params": >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", >> "profile": >> > {}, "preserve_on_delete": false, "meta": {}}] >> > update_instance_cache_with_nw_info >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this >> > compute host and has allocations in placement: {'resources': {'VCPU': 1, >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot the >> > instance on the hypervisor. >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> > >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> > :: waited 0.000s inner >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute >> host >> > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> 93fb420b3c604d4fae408b81135b58e9 >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> > >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: >> > >> >> Have you tried with debug logs? Has it worked with live snapshots >> >> before for other instances or did it never work and all snapshots were >> >> "cold"? >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> > Hi >> >> > kolla-ansible victoria version with ceph backend without volume >> >> > >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block wrote: >> >> > >> >> >> Hi, >> >> >> >> >> >> can you share more details about your environment? Which openstack >> >> >> version is it? What is the storage backend? In earlier releases there >> >> >> was an option: >> >> >> >> >> >> #disable_libvirt_livesnapshot = false >> >> >> >> >> >> but this option has been deprecated. But if you're on an older >> version >> >> >> that could explain it. >> >> >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> >> >> > When I snapshot from the instance , server will gone away and its >> not >> >> >> > reachable until the snapshot is complete here is the logs : >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> >> instance >> >> >> > has a pending task (image_pending_upload). Skip. >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning >> >> image >> >> >> > upload >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From gael.therond at bitswalk.com Mon Jun 13 08:33:06 2022 From: gael.therond at bitswalk.com (=?UTF-8?Q?Ga=C3=ABl_THEROND?=) Date: Mon, 13 Jun 2022 10:33:06 +0200 Subject: [VICTORIA][IRONIC] - Can inspect but not deploy Message-ID: Hi everyone! I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA platform, we activated the dnsmasq pxe filtering option at the inspector level, it works great as only IRONIC listed hosts are then served by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir. BUT, I'm having a weird issue now. All my nodes are able to get an IP from the DHCP at the INSPECTION step, however, as soon as the inspection step is finished, the mac related file is once again filled with ",ignore" which prohibits further operations. This means as soon as I put that node as available and try to deploy an "instance" on it (provision the host), it doesn't work as dnsmasq reject the host boot DHCP requests. So, is there a way to instruct the ironic-conductor to edit/allow the host the way the inspector is able to manipulate this file? Did I missed something? Thanks a lot! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at stackhpc.com Mon Jun 13 09:40:48 2022 From: pierre at stackhpc.com (Pierre Riteau) Date: Mon, 13 Jun 2022 11:40:48 +0200 Subject: [VICTORIA][IRONIC] - Can inspect but not deploy In-Reply-To: References: Message-ID: Hello Ga?l, Which network_interface are you using for your nodes? Is your provisioning network different from the inspection network? Pierre On Mon, 13 Jun 2022 at 10:36, Ga?l THEROND wrote: > Hi everyone! > > I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA > platform, we activated the dnsmasq pxe filtering option at the inspector > level, it works great as only IRONIC listed hosts are then served by the > dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir. > > BUT, I'm having a weird issue now. > > All my nodes are able to get an IP from the DHCP at the INSPECTION step, > however, as soon as the inspection step is finished, the mac related file > is once again filled with ",ignore" which prohibits further operations. > > This means as soon as I put that node as available and try to deploy an > "instance" on it (provision the host), it doesn't work as dnsmasq reject > the host boot DHCP requests. > > So, is there a way to instruct the ironic-conductor to edit/allow the host > the way the inspector is able to manipulate this file? > > Did I missed something? > > Thanks a lot! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Mon Jun 13 10:06:04 2022 From: katonalala at gmail.com (Lajos Katona) Date: Mon, 13 Jun 2022 12:06:04 +0200 Subject: [neutron][networking-l2gw][puppet] Status of networking-l2gw project In-Reply-To: References: Message-ID: Hi Takashi Thanks for the headsup, I try to keep l2gw project maintained, but recently had no time to check its status. I check your patch. Lajos Takashi Kajinami ezt ?rta (id?pont: 2022. j?n. 8., Sze, 17:01): > Hello team, > > # I understand networking-l2gw is not under neutron stadium > # but adding [neutron] tag hoping it would attract more attention. > > We found a recent change in neutron[1] broke networking-l2gw[2], and > some jobs in puppet repos and rdo, with l2gw enabled, are failing. > > I proposed a potential fix for the bug[3], but I noticed the project > looks unmaintained according to the following points. > > - No change has been merged for more than 1 year > > - No release was created since Wallaby > > - The zuul setting still points the victoria job template > (openstack-python3-victoria-jobs-neutron) > > I'm wondering if anybody is actually interested in maintaining the project. > If not, I'll propose retiring the plugin support from puppet (and TripleO > because it requires puppet implementation). > > I appreciate any feedback about this. > > [1] https://review.opendev.org/c/openstack/neutron/+/837392 > [2] https://bugs.launchpad.net/networking-l2gw/+bug/1977980 > [3] https://review.opendev.org/c/x/networking-l2gw/+/845141 > > Thank you, > Takashi > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From iurygregory at gmail.com Mon Jun 13 10:15:40 2022 From: iurygregory at gmail.com (Iury Gregory) Date: Mon, 13 Jun 2022 12:15:40 +0200 Subject: [ironic] No weekly meeting today (June 13) Message-ID: Hello Ironicers, I'm still on my way back home from the OIS, I'll not be able to run the upstream meeting I've asked in the irc and people are ok with skipping today's meeting, see you next week! -- *Att[]'s* *Iury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Ironic PTL * *Senior Software Engineer at Red Hat Brazil* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory at gmail.com * -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.therond at bitswalk.com Mon Jun 13 10:41:01 2022 From: gael.therond at bitswalk.com (=?UTF-8?Q?Ga=C3=ABl_THEROND?=) Date: Mon, 13 Jun 2022 12:41:01 +0200 Subject: [VICTORIA][IRONIC] - Can inspect but not deploy In-Reply-To: References: Message-ID: Hi pierre, I?m using a dedicated interface but this interface is the same for all ironic networks inspection/provisioning/cleaning. This interface works fine for inspection, my only issue is the pxe_filter that the ironic inspector process allow during inspection correctly but then tag as disallowed again at the end of the inspection, shouldn?t the deploy process allow the Mac again before booting the node? I can correctly see the conductor instruct the node to boot up using pxe from the kvm console but the BootP process doesn?t load the IPA kernel/initramfs as the dnsmasq pxe discard the request (because of the mac being still tagged as ?,ignore ? within the hostdir file). I?m a bit disappointed by this behavior. Thanks a lot! Le lun. 13 juin 2022 ? 11:41, Pierre Riteau a ?crit : > Hello Ga?l, > > Which network_interface are you using for your nodes? Is your provisioning > network different from the inspection network? > > Pierre > > On Mon, 13 Jun 2022 at 10:36, Ga?l THEROND > wrote: > >> Hi everyone! >> >> I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA >> platform, we activated the dnsmasq pxe filtering option at the inspector >> level, it works great as only IRONIC listed hosts are then served by the >> dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir. >> >> BUT, I'm having a weird issue now. >> >> All my nodes are able to get an IP from the DHCP at the INSPECTION step, >> however, as soon as the inspection step is finished, the mac related file >> is once again filled with ",ignore" which prohibits further operations. >> >> This means as soon as I put that node as available and try to deploy an >> "instance" on it (provision the host), it doesn't work as dnsmasq reject >> the host boot DHCP requests. >> >> So, is there a way to instruct the ironic-conductor to edit/allow the >> host the way the inspector is able to manipulate this file? >> >> Did I missed something? >> >> Thanks a lot! >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at stackhpc.com Mon Jun 13 11:14:03 2022 From: pierre at stackhpc.com (Pierre Riteau) Date: Mon, 13 Jun 2022 13:14:03 +0200 Subject: [VICTORIA][IRONIC] - Can inspect but not deploy In-Reply-To: References: Message-ID: Hi Ga?l, I am not talking about the physical network interface, but about the `network_interface` field on on your Ironic nodes: https://docs.openstack.org/ironic/latest/admin/multitenancy.html#network-interfaces Pierre On Mon, 13 Jun 2022 at 12:41, Ga?l THEROND wrote: > Hi pierre, > > I?m using a dedicated interface but this interface is the same for all > ironic networks inspection/provisioning/cleaning. > > This interface works fine for inspection, my only issue is the pxe_filter > that the ironic inspector process allow during inspection correctly but > then tag as disallowed again at the end of the inspection, shouldn?t the > deploy process allow the Mac again before booting the node? > > I can correctly see the conductor instruct the node to boot up using pxe > from the kvm console but the BootP process doesn?t load the IPA > kernel/initramfs as the dnsmasq pxe discard the request (because of the mac > being still tagged as ?,ignore ? within the hostdir file). > > I?m a bit disappointed by this behavior. > > Thanks a lot! > > Le lun. 13 juin 2022 ? 11:41, Pierre Riteau a > ?crit : > >> Hello Ga?l, >> >> Which network_interface are you using for your nodes? Is your >> provisioning network different from the inspection network? >> >> Pierre >> >> On Mon, 13 Jun 2022 at 10:36, Ga?l THEROND >> wrote: >> >>> Hi everyone! >>> >>> I'm dealing with a strange issue today, we deployed IRONIC on a VICTORIA >>> platform, we activated the dnsmasq pxe filtering option at the inspector >>> level, it works great as only IRONIC listed hosts are then served by the >>> dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir. >>> >>> BUT, I'm having a weird issue now. >>> >>> All my nodes are able to get an IP from the DHCP at the INSPECTION step, >>> however, as soon as the inspection step is finished, the mac related file >>> is once again filled with ",ignore" which prohibits further operations. >>> >>> This means as soon as I put that node as available and try to deploy an >>> "instance" on it (provision the host), it doesn't work as dnsmasq reject >>> the host boot DHCP requests. >>> >>> So, is there a way to instruct the ironic-conductor to edit/allow the >>> host the way the inspector is able to manipulate this file? >>> >>> Did I missed something? >>> >>> Thanks a lot! >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.therond at bitswalk.com Mon Jun 13 11:43:00 2022 From: gael.therond at bitswalk.com (=?UTF-8?Q?Ga=C3=ABl_THEROND?=) Date: Mon, 13 Jun 2022 13:43:00 +0200 Subject: [VICTORIA][IRONIC] - Can inspect but not deploy In-Reply-To: References: Message-ID: Aaaah yes, sorry my bad, as we do use Kolla-ansible I forget about this one, so we're using Flat network interface. Le lun. 13 juin 2022 ? 13:14, Pierre Riteau a ?crit : > Hi Ga?l, > > I am not talking about the physical network interface, but about the > `network_interface` field on on your Ironic nodes: > https://docs.openstack.org/ironic/latest/admin/multitenancy.html#network-interfaces > > Pierre > > On Mon, 13 Jun 2022 at 12:41, Ga?l THEROND > wrote: > >> Hi pierre, >> >> I?m using a dedicated interface but this interface is the same for all >> ironic networks inspection/provisioning/cleaning. >> >> This interface works fine for inspection, my only issue is the pxe_filter >> that the ironic inspector process allow during inspection correctly but >> then tag as disallowed again at the end of the inspection, shouldn?t the >> deploy process allow the Mac again before booting the node? >> >> I can correctly see the conductor instruct the node to boot up using pxe >> from the kvm console but the BootP process doesn?t load the IPA >> kernel/initramfs as the dnsmasq pxe discard the request (because of the mac >> being still tagged as ?,ignore ? within the hostdir file). >> >> I?m a bit disappointed by this behavior. >> >> Thanks a lot! >> >> Le lun. 13 juin 2022 ? 11:41, Pierre Riteau a >> ?crit : >> >>> Hello Ga?l, >>> >>> Which network_interface are you using for your nodes? Is your >>> provisioning network different from the inspection network? >>> >>> Pierre >>> >>> On Mon, 13 Jun 2022 at 10:36, Ga?l THEROND >>> wrote: >>> >>>> Hi everyone! >>>> >>>> I'm dealing with a strange issue today, we deployed IRONIC on a >>>> VICTORIA platform, we activated the dnsmasq pxe filtering option at the >>>> inspector level, it works great as only IRONIC listed hosts are then served >>>> by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir. >>>> >>>> BUT, I'm having a weird issue now. >>>> >>>> All my nodes are able to get an IP from the DHCP at the INSPECTION >>>> step, however, as soon as the inspection step is finished, the mac related >>>> file is once again filled with ",ignore" which prohibits further >>>> operations. >>>> >>>> This means as soon as I put that node as available and try to deploy an >>>> "instance" on it (provision the host), it doesn't work as dnsmasq reject >>>> the host boot DHCP requests. >>>> >>>> So, is there a way to instruct the ironic-conductor to edit/allow the >>>> host the way the inspector is able to manipulate this file? >>>> >>>> Did I missed something? >>>> >>>> Thanks a lot! >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From adivya1.singh at gmail.com Mon Jun 13 12:29:10 2022 From: adivya1.singh at gmail.com (Adivya Singh) Date: Mon, 13 Jun 2022 17:59:10 +0530 Subject: Regarding Policy.json entries for glance image update not working for a user In-Reply-To: References: Message-ID: hi Team, Any thoughts on this Regards Adivya Singh On Sat, Jun 11, 2022 at 12:40 AM Adivya Singh wrote: > Hi Team, > > I have a use case where I have to give a user restriction on updating the > image properties as a member. > > I have created a policy Json file and give the modify_image rule to the > particular role, but still it is not working > > "modify_image": "role:user", This role is created in OpenStack. > > but still it is failing while updating properties with a particular user > assigned to a role as "access denied" and unauthorized access > > Regards > Adivya Singh > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From romain.chanu at univ-lyon1.fr Mon Jun 13 12:41:48 2022 From: romain.chanu at univ-lyon1.fr (CHANU ROMAIN) Date: Mon, 13 Jun 2022 12:41:48 +0000 Subject: [manila] CephFS NFS high availability cluster Message-ID: Hello, I added Manila service with CephFS NFS driver to my openstack cluster. Everything works fine but I would like to add 2 nfs-ganesha servers to ensure high availability to the service. I configured haproxy to forward 2049 to ganesha backend but Manila cephFS NFS provides only IP restriction and see only haproxy's IP address. To make it work you have to add haproxy to allowed ip but it means everyone can access the share. So currently the only way I found out is to use pacemaker to set public vip to a running nfs-ganesha node. Could you confirm that is not possible to provide an active/active nfs-ganesha cluster with manila cephfs NFS driver? Best regards, Romain -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4513 bytes Desc: not available URL: From gael.therond at bitswalk.com Mon Jun 13 12:44:08 2022 From: gael.therond at bitswalk.com (=?UTF-8?Q?Ga=C3=ABl_THEROND?=) Date: Mon, 13 Jun 2022 14:44:08 +0200 Subject: [VICTORIA][IRONIC] - Can inspect but not deploy In-Reply-To: References: Message-ID: Alright, I just tested to disable the pxe_filter option and it works, the host is then deployed using the glance image and neutron network settings correctly. I think there is either a bug or maybe something that I didn?t catch up but it?s pretty annoying. I?ll propose a patch to kolla-ansible as well as I discovered that you can?t disable this feature for now due to the template shape. Any ideas about why the conductor isn?t able to whitelist this node ? Thanks a lot! Le lun. 13 juin 2022 ? 13:43, Ga?l THEROND a ?crit : > Aaaah yes, sorry my bad, as we do use Kolla-ansible I forget about this > one, so we're using Flat network interface. > > Le lun. 13 juin 2022 ? 13:14, Pierre Riteau a > ?crit : > >> Hi Ga?l, >> >> I am not talking about the physical network interface, but about the >> `network_interface` field on on your Ironic nodes: >> https://docs.openstack.org/ironic/latest/admin/multitenancy.html#network-interfaces >> >> Pierre >> >> On Mon, 13 Jun 2022 at 12:41, Ga?l THEROND >> wrote: >> >>> Hi pierre, >>> >>> I?m using a dedicated interface but this interface is the same for all >>> ironic networks inspection/provisioning/cleaning. >>> >>> This interface works fine for inspection, my only issue is the >>> pxe_filter that the ironic inspector process allow during inspection >>> correctly but then tag as disallowed again at the end of the inspection, >>> shouldn?t the deploy process allow the Mac again before booting the node? >>> >>> I can correctly see the conductor instruct the node to boot up using pxe >>> from the kvm console but the BootP process doesn?t load the IPA >>> kernel/initramfs as the dnsmasq pxe discard the request (because of the mac >>> being still tagged as ?,ignore ? within the hostdir file). >>> >>> I?m a bit disappointed by this behavior. >>> >>> Thanks a lot! >>> >>> Le lun. 13 juin 2022 ? 11:41, Pierre Riteau a >>> ?crit : >>> >>>> Hello Ga?l, >>>> >>>> Which network_interface are you using for your nodes? Is your >>>> provisioning network different from the inspection network? >>>> >>>> Pierre >>>> >>>> On Mon, 13 Jun 2022 at 10:36, Ga?l THEROND >>>> wrote: >>>> >>>>> Hi everyone! >>>>> >>>>> I'm dealing with a strange issue today, we deployed IRONIC on a >>>>> VICTORIA platform, we activated the dnsmasq pxe filtering option at the >>>>> inspector level, it works great as only IRONIC listed hosts are then served >>>>> by the dnsmasq DHCP as those hosts are allowed using the dhcp hosts dir. >>>>> >>>>> BUT, I'm having a weird issue now. >>>>> >>>>> All my nodes are able to get an IP from the DHCP at the INSPECTION >>>>> step, however, as soon as the inspection step is finished, the mac related >>>>> file is once again filled with ",ignore" which prohibits further >>>>> operations. >>>>> >>>>> This means as soon as I put that node as available and try to deploy >>>>> an "instance" on it (provision the host), it doesn't work as dnsmasq reject >>>>> the host boot DHCP requests. >>>>> >>>>> So, is there a way to instruct the ironic-conductor to edit/allow the >>>>> host the way the inspector is able to manipulate this file? >>>>> >>>>> Did I missed something? >>>>> >>>>> Thanks a lot! >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaser at vexxhost.com Mon Jun 13 12:57:48 2022 From: mnaser at vexxhost.com (Mohammed Naser) Date: Mon, 13 Jun 2022 08:57:48 -0400 Subject: [qa][refstack][python-tempestconf] adding image Message-ID: Hi there, I'm wondering if the python-tempestconf team is open to publishing Docker images that contain `python-tempestconf`. This is something that we have a need for downstream but we'd be happy to contribute upstream. The main thing is that we'd need to setup an organization on Docker hub to publish things to. Is that something that the team is interested in? We can contribute it if that is the case. Thanks, Mohammed -- Mohammed Naser VEXXHOST, Inc. From rosmaita.fossdev at gmail.com Mon Jun 13 12:58:47 2022 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Mon, 13 Jun 2022 08:58:47 -0400 Subject: Regarding Policy.json entries for glance image update not working for a user In-Reply-To: References: Message-ID: On 6/13/22 8:29 AM, Adivya Singh wrote: > hi Team, > > Any thoughts on this H Adivya, Please supply some more information, for example: - which openstack release you are using - the full API request you are making to modify the image - the full API response you receive - whether the user with "role:user" is in the same project that owns the image - debug level log extract for this call if you have it - anything else that could be relevant, for example, have you modified any other policies, and if so, what values are you using now? cheers, brian > > Regards > Adivya Singh > > On Sat, Jun 11, 2022 at 12:40 AM Adivya Singh > wrote: > > Hi Team, > > I have a use case where I have to give a user restriction on > updating the image properties as a member. > > I have created a policy Json file and give the modify_image rule to > the particular role, but still it is not working > > "modify_image": "role:user", This role is created in OpenStack. > > but still it is failing while updating properties with a > particular?user assigned to a role as "access denied" and > unauthorized access > > Regards > Adivya Singh > From abishop at redhat.com Mon Jun 13 13:06:06 2022 From: abishop at redhat.com (Alan Bishop) Date: Mon, 13 Jun 2022 06:06:06 -0700 Subject: Regarding Policy.json entries for glance image update not working for a user In-Reply-To: References: Message-ID: On Mon, Jun 13, 2022 at 6:00 AM Brian Rosmaita wrote: > On 6/13/22 8:29 AM, Adivya Singh wrote: > > hi Team, > > > > Any thoughts on this > > H Adivya, > > Please supply some more information, for example: > > - which openstack release you are using > - the full API request you are making to modify the image > - the full API response you receive > - whether the user with "role:user" is in the same project that owns the > image > - debug level log extract for this call if you have it > - anything else that could be relevant, for example, have you modified > any other policies, and if so, what values are you using now? > Also bear in mind that the default policy_file name is "policy.yaml" (not .json). You either need to provide a policy.yaml file, or override the policy_file setting if you really want to use policy.json. Alan cheers, > brian > > > > > Regards > > Adivya Singh > > > > On Sat, Jun 11, 2022 at 12:40 AM Adivya Singh > > wrote: > > > > Hi Team, > > > > I have a use case where I have to give a user restriction on > > updating the image properties as a member. > > > > I have created a policy Json file and give the modify_image rule to > > the particular role, but still it is not working > > > > "modify_image": "role:user", This role is created in OpenStack. > > > > but still it is failing while updating properties with a > > particular user assigned to a role as "access denied" and > > unauthorized access > > > > Regards > > Adivya Singh > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adivya1.singh at gmail.com Mon Jun 13 13:07:52 2022 From: adivya1.singh at gmail.com (Adivya Singh) Date: Mon, 13 Jun 2022 18:37:52 +0530 Subject: 504 Gateway time out error while loading OpenStack Projects Message-ID: hi Team, We have Xena release installed on Ubuntu Openstack, Lately we are getting "504 Gateway Timeout error ", As the we moved from 1 node to 3 node Open Stack i do not see any error in haproxy , neither there is a time out when i ping external-VIP-IP configured. Apache 2 are also loaded, tried to delete the Horizon Container and created it again, but no errors, Still the same errors. Any guesses on that Regards Adivya Singh -------------- next part -------------- An HTML attachment was scrubbed... URL: From ricolin at ricolky.com Mon Jun 13 13:10:55 2022 From: ricolin at ricolky.com (Rico Lin) Date: Mon, 13 Jun 2022 21:10:55 +0800 Subject: Propose to add Takashi Kajinami as heat core reviewer Message-ID: Hi all I want to suggest adding Takashi Kajinami as heat core reviewer. Who help provide valid reviews and fixes for issues on heat. Will be great to have Takashi Kajinami join heat core reviewer team and help. *Rico Lin* -------------- next part -------------- An HTML attachment was scrubbed... URL: From nblkhan.pool at gmail.com Mon Jun 13 13:14:42 2022 From: nblkhan.pool at gmail.com (Nabeel Tariq Khan) Date: Mon, 13 Jun 2022 18:14:42 +0500 Subject: [Magnum resize Master node root volume/docker volume or resize flavor] Message-ID: Hi, Is it possible to resize the root volume or docker volume or resize the flavor of master or minion node on openstack magnum? regards, Nabeel Tariq Khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramishra at redhat.com Mon Jun 13 13:47:48 2022 From: ramishra at redhat.com (Rabi Mishra) Date: Mon, 13 Jun 2022 19:17:48 +0530 Subject: Propose to add Takashi Kajinami as heat core reviewer In-Reply-To: References: Message-ID: On Mon, Jun 13, 2022 at 6:43 PM Rico Lin wrote: > Hi all > > I want to suggest adding Takashi Kajinami as heat > core reviewer. > Who help provide valid reviews and fixes for issues on heat. > Will be great to have Takashi Kajinami join heat core reviewer team and > help. > +1, Good addition, he is actively involved for some time and contributes to keep the project healthy . > > *Rico Lin* > -- Regards, Rabi Mishra -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkajinam at redhat.com Wed Jun 8 14:54:31 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Wed, 8 Jun 2022 23:54:31 +0900 Subject: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version In-Reply-To: References: Message-ID: You'd need to check whether that oc_provisioning network is part of network_data and is mapped to the Controller role (might be a different role if you completely customize roles). If you want to use the provisioning network used by undercloud then it should be ctlplane instead. Although bugzilla is open, we maintain upstream bugs for OpenStack components not in bugzilla but in launchpad, so please report a bug in launchpad instead. https://bugs.launchpad.net/tripleo On Wed, Jun 8, 2022 at 11:48 PM Swogat Pradhan wrote: > Hi Lokendra, > I am not sure why this issue is caused, can you please check the > tripleo_dnsmasq service and the tripleo_ironic_pxe service in your director > node? IF the services are running fine then please report this here for > fast resolvement https://bugzilla.redhat.com/ > > With regards, > Swogat Pradhan > > On Tue, Jun 7, 2022 at 12:54 PM Lokendra Rathour < > lokendrarathour at gmail.com> wrote: > >> Hi Swogat, >> Yes after correcting the j2 template, as suggested by you, it started >> working. Thanks a lot once again. >> Now the Overcloud is deployed. >> >> Further, as a next requirement, we were also trying to add an *additional >> Node as a bare-metal Instance* on the overcloud. As it is the composable >> network approach, we add an entry in parameters default for ServiceNetMap, >> as like below in environments/network-environments.yaml >> >> parameter_defaults: >> # This section is where deployment-specific configuration is done >> ServiceNetMap: >> IronicApiNetwork: oc_provisioning >> IronicNetwork: oc_provisioning >> >> this is now passed as in the environment.yaml as parameter_defaults, but >> somehow we are not able to deploy the setup. it gives the error for >> containers. >> >> >> *2022-06-06 19:08:37.560800 | 525400fd-f24d-9703-b0e7-0000000084b2 | >> FATAL | Manage container systemd services and cleanup old systemd >> healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | >> overcloud-controller-0 | error={"changed": false, "msg": "Service >> ironic_pxe_tftp has not started yet"}* >> >> >> and if we remove this setting as stated above, the deployment is >> successful, but at the time of node provisioning it is not able to reach >> the TFTP server to pull files, mainly because of the composable network >> (oc_provisioning), we faced this issue in Train, so we did the above config >> changes and it working. But as it looks something more is needed in >> Wallaby for Baremetal provisioning as an instance. >> >> Please share your learnings with the reference to it. Thanks once again >> for your all-time support. >> reference link: Bare Metal Instances in Overcloud ? TripleO 3.0.0 >> documentation (openstack.org) >> >> >> -Lokendra >> >> >> On Wed, Jun 1, 2022 at 12:19 AM Lokendra Rathour < >> lokendrarathour at gmail.com> wrote: >> >>> Ok Swogat, >>> Thanks once again. >>> i will try this approach once and will let you know. >>> >>> >>> On Wed, 1 Jun 2022, 00:04 Swogat Pradhan, >>> wrote: >>> >>>> Hi Lokendra, >>>> Like i said, >>>> > NOTE: when passing --network-config parameter in node provision step, >>>> it creates a directory in /etc/os-net-config and in it creates a file >>>> > config.yaml, do check the indentation of that file. (in my case the >>>> indentation was wrong when i was using bonding everytime, so i had to >>>> > manually change the script and run a while loop and then my node >>>> provision step was successful) >>>> this parameter --network-config creates a config.yaml(network config) >>>> file in /etc/os-net-config directory in the overcloud nodes and then >>>> ansible tries to apply the network config from the generated config file. >>>> And in wallaby version if you are using bonding then the syntax in the >>>> generated conf is wrong (atleast was in my case) and then the ansible tries >>>> to apply the network config (with wrong syntax) so your overcloud nodes >>>> become unavailable. >>>> >>>> Please run node provision separately, and keep on monitoring >>>> "metalsmith list" command. Once IP is assigned to the overcloud nodes, ssh >>>> to the nodes and assign a password to heat-admin user so that even if the >>>> network becomes unavailable you still will be able to access the nodes via >>>> console access, then you can visit /etc/os-net-config directory and verify >>>> the config.yaml file. >>>> >>>> With regards, >>>> Swogat pradhan. >>>> >>>> On Tue, May 31, 2022 at 11:56 PM Lokendra Rathour < >>>> lokendrarathour at gmail.com> wrote: >>>> >>>>> Hi Swogat, >>>>> Thanks once again for your input it really helped much. >>>>> >>>>> instead of running mentioned those three provisioning steps i used >>>>> alternate method and passed directly in deploy command. now my current >>>>> deploy command is: >>>>> >>>>> openstack overcloud deploy --templates \ >>>>> --networks-file /home/stack/templates/custom_network_data.yaml \ >>>>> --vip-file /home/stack/templates/custom_vip_data.yaml \ >>>>> --baremetal-deployment >>>>> /home/stack/templates/overcloud-baremetal-deploy.yaml \ >>>>> --network-config \ >>>>> -e /home/stack/templates/environment.yaml \ >>>>> -e >>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml >>>>> \ >>>>> -e >>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml >>>>> \ >>>>> -e >>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml >>>>> \ >>>>> -e /home/stack/templates/ironic-config.yaml \ >>>>> -e >>>>> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ >>>>> -e >>>>> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ >>>>> -e /home/stack/containers-prepare-parameter.yaml >>>>> >>>>> The files as suggested by you are well created. >>>>> >>>>> But once we run the deployment I get this error for all nodes: >>>>> >>>>> 0:00:16.064781 | 1.16s >>>>> 2022-05-31 19:13:00.276954 | 525400ef-b928-9ded-fecc-000000000094 | >>>>> TASK | Run tripleo_os_net_config_module with network_config >>>>> 2022-05-31 19:40:30.061582 | 525400ef-b928-9ded-fecc-000000000094 | >>>>> FATAL | Run tripleo_os_net_config_module with network_config | >>>>> overcloud-controller-1 | error={"msg": "Data could not be sent to remote >>>>> host \"30.30.30.117\". Make sure this host can be reached over ssh: ssh: >>>>> connect to host 30.30.30.117 port 22: No route to host\r\n"} >>>>> 2022- >>>>> >>>>> >>>>> Baremetal node list are showing in as active. >>>>> >>>>> (undercloud) [stack at undercloud ~]$ openstack baremetal node list >>>>> /usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: >>>>> DeprecationWarning: The _yaml extension module is now located at yaml._yaml >>>>> and its location is subject to change. To use the LibYAML-based parser and >>>>> emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper >>>>> as Dumper`. >>>>> DeprecationWarning >>>>> >>>>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>>>> | UUID | Name | Instance UUID >>>>> | Power State | Provisioning State | Maintenance | >>>>> >>>>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>>>> | 1a4d873c-f9f7-4504-a3af-92c11f954171 | node-a | >>>>> 901453a1-183f-4de8-aaab-0f38be2be455 | power on | active | >>>>> False | >>>>> | d18610fc-9532-410c-918e-8efc326c89f8 | node-b | >>>>> d059b94a-8357-4f8e-a0d8-15a24b0c1afe | power on | active | >>>>> False | >>>>> | b69f2d5a-5b18-4453-8843-15c6af79aca0 | node-c | >>>>> f196ef3a-7950-47b9-a5ae-751f06b18f75 | power on | active | >>>>> False | >>>>> | 8a38c584-f812-4ebc-a0b1-4299f0917637 | node-d | >>>>> 1636517c-2ab2-43d7-8205-9f02c5290207 | power on | active | >>>>> False | >>>>> >>>>> +--------------------------------------+--------+--------------------------------------+-------------+--------------------+-------------+ >>>>> >>>>> Some config is missing it seems, please check once and advise. >>>>> >>>>> >>>>> On Tue, May 31, 2022 at 11:40 AM Swogat Pradhan < >>>>> swogatpradhan22 at gmail.com> wrote: >>>>> >>>>>> Hi Lokendra, >>>>>> You need to generate another file also in the following step: openstack >>>>>> overcloud node provision --stack overcloud --overcloud-ssh-key >>>>>> /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml also you >>>>>> need to pass another parameter --network-config. >>>>>> example: >>>>>> openstack overcloud node provision --stack overcloud >>>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa *--network-config* *--output >>>>>> overcloud-baremetal-deployed.yaml* overcloud-baremetal-deploy.yaml >>>>>> >>>>>> And then all these output files will be passed on to the openstack >>>>>> overcloud deploy command. >>>>>> NOTE: when passing --network-config parameter in node provision step, >>>>>> it creates a directory in /etc/os-net-config and in it creates a file >>>>>> config.yaml, do check the indentation of that file. (in my case the >>>>>> indentation was wrong when i was using bondind everytime, so i had to >>>>>> manually change the script and run a while loop and then my node provision >>>>>> step was successful) >>>>>> >>>>>> On Tue, May 31, 2022 at 8:59 AM Lokendra Rathour < >>>>>> lokendrarathour at gmail.com> wrote: >>>>>> >>>>>>> Hi Swogat, >>>>>>> I tried generating the scripts as used by you in your deployments >>>>>>> using the >>>>>>> >>>>>>> >>>>>>> #openstack overcloud network provision --stack overcloud --output >>>>>>> networks-deployed-environment.yaml custom_network_data.yaml >>>>>>> # openstack overcloud network vip provision --stack overcloud >>>>>>> --output vip-deployed-environment.yaml custom_vip_data.yaml >>>>>>> # openstack overcloud node provision --stack overcloud >>>>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa >>>>>>> overcloud-baremetal-deploy.yaml >>>>>>> >>>>>>> and used the first two in the final deployment script, but it gives >>>>>>> the error: >>>>>>> >>>>>>> heatclient.exc.HTTPInternalServerError: ERROR: Internal Error >>>>>>> 2022-05-30 14:14:39.772 479668 ERROR >>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent >>>>>>> call last):\n', ' File "/usr/lib/python3.6/ted_stack\n >>>>>>> nested_stack.validate()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in >>>>>>> wrapper\n result = f(*args, ine 969, in validate\n result = >>>>>>> res.validate()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/openstack/neutron/port.py", >>>>>>> line 454site-packages/heat/engine/resources/openstack/neutron/neutron.py", >>>>>>> line 43, in validate\n res = super(NeutronResource, self).validate()\n', >>>>>>> ' File "/un return self.validate_template()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 1882, in >>>>>>> validate_template\n self.t.rpy", line 200, in >>>>>>> _validate_service_availability\n raise ex\n', >>>>>>> 'heat.common.exception.ResourceTypeUnavailable: HEAT-E99001 Service neutron >>>>>>> is not avaieutron network endpoint is not in service catalog.\n', '\nDuring >>>>>>> handling of the above exception, another exception occurred:\n\n', >>>>>>> 'Traceback (most recens/stack_resource.py", line 75, in >>>>>>> validate_nested_stack\n nested_stack.validate()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py"thon3.6/site-packages/heat/engine/stack.py", >>>>>>> line 969, in validate\n result = res.validate()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/ateResource, >>>>>>> self).validate()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", >>>>>>> line 65, in validate\n self.validources/stack_resource.py", line 81, in >>>>>>> validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', >>>>>>> 'heat.common.exception.StackValidationFaileeploy/overcloud/tripleo-heat-templates/deployed-server/deployed-server.yaml>: >>>>>>> HEAT-E99001 Service neutron is not available for resource type >>>>>>> OS::TripleO::vice catalog.\n', '\nDuring handling of the above exception, >>>>>>> another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' >>>>>>> File "/usr/lib/pline 320, in validate_nested_stack\n >>>>>>> nested_stack.validate()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in >>>>>>> wrappe/heat/engine/stack.py", line 969, in validate\n result = >>>>>>> res.validate()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/template_relidate()\n', >>>>>>> ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", >>>>>>> line 65, in validate\n self.validate_nested_stack()\n'.py", line 81, in >>>>>>> validate_nested_stack\n ex, path=[self.stack.t.RESOURCES, path])\n', >>>>>>> 'heat.common.exception.StackValidationFailed: >>>>>>> ResourceTypeUnavaimplates/puppet/compute-role.yaml>.resources.NovaCompute>>>>>> reason: neutron network endpoint is not in service catalog.\n', '\nDuring >>>>>>> handling of the above exception, >>>>>>> another/lib/python3.6/site-packages/heat/common/context.py", line 416, in >>>>>>> wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File >>>>>>> "/usr/lib/python3.6/sirce_name, template_id)\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 756, in >>>>>>> _parse_template_and_validate_stack\n stack.v line 160, in wrapper\n >>>>>>> result = f(*args, **kwargs)\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 969, in >>>>>>> validate\n resesources/stack_resource.py", line 65, in validate\n >>>>>>> self.validate_nested_stack()\n', ' File >>>>>>> "/usr/lib/python3.6/site-packages/heat/engine/resources/oph=[self.stack.t.RESOURCES, >>>>>>> path])\n', 'heat.common.exception.StackValidationFailed: >>>>>>> ResourceTypeUnavailable: >>>>>>> resources.Compute.resources.0.resources.NovaCompute: >>>>>>> HEAT-E9900::ControlPlanePort, reason: neutron network endpoint is not in >>>>>>> service catalog.\n']. >>>>>>> >>>>>>> Request you to check once, please. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, May 30, 2022 at 11:06 AM Lokendra Rathour < >>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Swogat, >>>>>>>> Thanks once again. >>>>>>>> >>>>>>>> with the files as shown below I am running the overcloud deploy for >>>>>>>> wallaby using this command: >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud ~]$ cat >>>>>>>> deploy_overcloud_working_1.sh >>>>>>>> openstack overcloud deploy --templates \ >>>>>>>> -n /home/stack/templates/network_data.yaml \ >>>>>>>> -r /home/stack/templates/roles_data.yaml \ >>>>>>>> -e /home/stack/templates/environment.yaml \ >>>>>>>> -e /home/stack/templates/environments/network-isolation.yaml \ >>>>>>>> -e /home/stack/templates/environments/network-environment.yaml \ >>>>>>>> -e >>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-conductor.yaml >>>>>>>> \ >>>>>>>> -e >>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-inspector.yaml >>>>>>>> \ >>>>>>>> -e >>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/services/ironic-overcloud.yaml >>>>>>>> \ >>>>>>>> -e /home/stack/templates/ironic-config.yaml \ >>>>>>>> -e >>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ >>>>>>>> -e >>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ >>>>>>>> -e /home/stack/containers-prepare-parameter.yaml >>>>>>>> (undercloud) [stack at undercloud ~]$ >>>>>>>> >>>>>>>> >>>>>>>> This deployment is on ipv6 using triple0 wallaby, templates, as >>>>>>>> mentioned below, are generated using rendering steps and the >>>>>>>> network_data.yaml the roles_data.yaml >>>>>>>> Steps used to render the templates: >>>>>>>> cd /usr/share/openstack-tripleo-heat-templates/ >>>>>>>> ./tools/process-templates.py -o >>>>>>>> ~/openstack-tripleo-heat-templates-rendered_at_wallaby -n >>>>>>>> /home/stack/templates/network_data.yaml -r >>>>>>>> /home/stack/templates/roles_data.yaml >>>>>>>> >>>>>>>> *Now if i try adding the related to VIP port I do get the error as:* >>>>>>>> >>>>>>>> 2022-05-30 10:37:12.792 979387 WARNING >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template >>>>>>>> to file: >>>>>>>> /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/controller-role.yaml >>>>>>>> 2022-05-30 10:37:12.792 979387 WARNING >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template >>>>>>>> to file: >>>>>>>> /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/compute-role.yaml >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured >>>>>>>> while running the command: ValueError: The environment is not a valid YAML >>>>>>>> mapping data type. >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent >>>>>>>> call last): >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, >>>>>>>> self).run(parsed_args) >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in >>>>>>>> run >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, >>>>>>>> self).run(parsed_args) >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = >>>>>>>> self.take_action(parsed_args) or 0 >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>>>> line 1189, in take_action >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud stack, parsed_args, >>>>>>>> new_tht_root, user_tht_root) >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>>>> line 227, in create_env_files >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud created_env_files, >>>>>>>> parsed_args, new_tht_root, user_tht_root) >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", >>>>>>>> line 204, in build_image_params >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud cleanup=(not >>>>>>>> parsed_args.no_cleanup)) >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1929, in >>>>>>>> process_multiple_environments >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud env_path=env_path, >>>>>>>> include_env_in_files=include_env_in_files) >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", >>>>>>>> line 326, in process_environment_and_files >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud env = >>>>>>>> environment_format.parse(raw_env) >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud File >>>>>>>> "/usr/lib/python3.6/site-packages/heatclient/common/environment_format.py", >>>>>>>> line 50, in parse >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud raise >>>>>>>> ValueError(_('The environment is not a valid ' >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: The >>>>>>>> environment is not a valid YAML mapping data type. >>>>>>>> 2022-05-30 10:37:14.455 979387 ERROR >>>>>>>> tripleoclient.v1.overcloud_deploy.DeployOvercloud >>>>>>>> 2022-05-30 10:37:14.457 979387 ERROR openstack [-] The environment >>>>>>>> is not a valid YAML mapping data type. >>>>>>>> 2022-05-30 10:37:14.457 979387 INFO osc_lib.shell [-] END return >>>>>>>> value: 1 >>>>>>>> (undercloud) [stack at undercloud ~]$ >>>>>>>> >>>>>>>> This is more of a syntax error where it is not able to understand >>>>>>>> the passed VIP data file: >>>>>>>> >>>>>>>> undercloud) [stack at undercloud ~]$ cat >>>>>>>> /home/stack/templates/vip-data-default-network-isolation.yaml >>>>>>>> - >>>>>>>> dns_name: overcloud >>>>>>>> network: internal_api >>>>>>>> - >>>>>>>> dns_name: overcloud >>>>>>>> network: external >>>>>>>> - >>>>>>>> dns_name: overcloud >>>>>>>> network: ctlplane >>>>>>>> - >>>>>>>> dns_name: overcloud >>>>>>>> network: oc_provisioning >>>>>>>> - >>>>>>>> dns_name: overcloud >>>>>>>> network: j3mgmt >>>>>>>> >>>>>>>> >>>>>>>> Please advise, also please note that similar templates generated in >>>>>>>> prior releases such as train/ussuri works perfectly. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Please check the list of *templates *files: >>>>>>>> >>>>>>>> drwxr-xr-x. 2 stack stack 68 May 30 09:22 environments >>>>>>>> -rw-r--r--. 1 stack stack 265 May 27 13:47 environment.yaml >>>>>>>> -rw-rw-r--. 1 stack stack 297 May 27 13:47 init-repo.yaml >>>>>>>> -rw-r--r--. 1 stack stack 570 May 27 13:47 ironic-config.yaml >>>>>>>> drwxrwxr-x. 4 stack stack 4096 May 27 13:53 network >>>>>>>> -rw-r--r--. 1 stack stack 6370 May 27 14:26 network_data.yaml >>>>>>>> -rw-r--r--. 1 stack stack 11137 May 27 13:53 roles_data.yaml >>>>>>>> -rw-r--r--. 1 stack stack 234 May 30 09:23 >>>>>>>> vip-data-default-network-isolation.yaml >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ cat environment.yaml >>>>>>>> >>>>>>>> parameter_defaults: >>>>>>>> OvercloudControllerFlavor: control >>>>>>>> OvercloudComputeFlavor: compute >>>>>>>> ControllerCount: 3 >>>>>>>> ComputeCount: 1 >>>>>>>> TimeZone: 'Asia/Kolkata' >>>>>>>> NtpServer: ['30.30.30.3'] >>>>>>>> NeutronBridgeMappings: datacentre:br-tenant >>>>>>>> NeutronFlatNetworks: datacentre >>>>>>>> (undercloud) [stack at undercloud templates]$ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ cat ironic-config.yaml >>>>>>>> >>>>>>>> parameter_defaults: >>>>>>>> NovaSchedulerDefaultFilters: >>>>>>>> - AggregateInstanceExtraSpecsFilter >>>>>>>> - AvailabilityZoneFilter >>>>>>>> - ComputeFilter >>>>>>>> - ComputeCapabilitiesFilter >>>>>>>> - ImagePropertiesFilter >>>>>>>> IronicEnabledHardwareTypes: >>>>>>>> - ipmi >>>>>>>> - redfish >>>>>>>> IronicEnabledPowerInterfaces: >>>>>>>> - ipmitool >>>>>>>> - redfish >>>>>>>> IronicEnabledManagementInterfaces: >>>>>>>> - ipmitool >>>>>>>> - redfish >>>>>>>> IronicCleaningDiskErase: metadata >>>>>>>> IronicIPXEEnabled: true >>>>>>>> IronicInspectorSubnets: >>>>>>>> - ip_range: 172.23.3.100,172.23.3.150 >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ cat network_data.yaml >>>>>>>> >>>>>>>> - name: J3Mgmt >>>>>>>> name_lower: j3mgmt >>>>>>>> vip: true >>>>>>>> vlan: 400 >>>>>>>> ipv6: true >>>>>>>> ipv6_subnet: 'fd80:fd00:fd00:4000::/64' >>>>>>>> ipv6_allocation_pools: [{'start': 'fd80:fd00:fd00:4000::10', >>>>>>>> 'end': 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] >>>>>>>> mtu: 9000 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> - name: InternalApi >>>>>>>> name_lower: internal_api >>>>>>>> vip: true >>>>>>>> vlan: 418 >>>>>>>> ipv6: true >>>>>>>> ipv6_subnet: 'fd00:fd00:fd00:2000::/64' >>>>>>>> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', >>>>>>>> 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] >>>>>>>> mtu: 9000 >>>>>>>> >>>>>>>> >>>>>>>> - name: External >>>>>>>> vip: true >>>>>>>> name_lower: external >>>>>>>> vlan: 408 >>>>>>>> ipv6: true >>>>>>>> gateway_ipv6: 'fd00:fd00:fd00:9900::1' >>>>>>>> ipv6_subnet: 'fd00:fd00:fd00:9900::/64' >>>>>>>> ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:9900::10', >>>>>>>> 'end': 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}] >>>>>>>> mtu: 9000 >>>>>>>> >>>>>>>> >>>>>>>> - name: OCProvisioning >>>>>>>> vip: true >>>>>>>> name_lower: oc_provisioning >>>>>>>> vlan: 412 >>>>>>>> ip_subnet: '172.23.3.0/24' >>>>>>>> allocation_pools: [{'start': '172.23.3.10', 'end': '172.23.3.50'}] >>>>>>>> mtu: 9000 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ cat roles_data.yaml >>>>>>>> >>>>>>>> >>>>>>>> ############################################################################### >>>>>>>> # File generated by TripleO >>>>>>>> >>>>>>>> ############################################################################### >>>>>>>> >>>>>>>> ############################################################################### >>>>>>>> # Role: Controller >>>>>>>> # >>>>>>>> >>>>>>>> ############################################################################### >>>>>>>> - name: Controller >>>>>>>> description: | >>>>>>>> Controller role that has all the controller services loaded and >>>>>>>> handles >>>>>>>> Database, Messaging, and Network functions. >>>>>>>> CountDefault: 1 >>>>>>>> tags: >>>>>>>> - primary >>>>>>>> - controller >>>>>>>> # Create external Neutron bridge for SNAT (and floating IPs >>>>>>>> when using >>>>>>>> # ML2/OVS without DVR) >>>>>>>> - external_bridge >>>>>>>> networks: >>>>>>>> External: >>>>>>>> subnet: external_subnet >>>>>>>> InternalApi: >>>>>>>> subnet: internal_api_subnet >>>>>>>> OCProvisioning: >>>>>>>> subnet: oc_provisioning_subnet >>>>>>>> J3Mgmt: >>>>>>>> subnet: j3mgmt_subnet >>>>>>>> >>>>>>>> >>>>>>>> # For systems with both IPv4 and IPv6, you may specify a gateway >>>>>>>> network for >>>>>>>> # each, such as ['ControlPlane', 'External'] >>>>>>>> default_route_networks: ['External'] >>>>>>>> HostnameFormatDefault: '%stackname%-controller-%index%' >>>>>>>> RoleParametersDefault: >>>>>>>> OVNCMSOptions: "enable-chassis-as-gw" >>>>>>>> # Deprecated & backward-compatible values (FIXME: Make parameters >>>>>>>> consistent) >>>>>>>> # Set uses_deprecated_params to True if any deprecated params are >>>>>>>> used. >>>>>>>> uses_deprecated_params: True >>>>>>>> deprecated_param_extraconfig: 'controllerExtraConfig' >>>>>>>> deprecated_param_flavor: 'OvercloudControlFlavor' >>>>>>>> deprecated_param_image: 'controllerImage' >>>>>>>> deprecated_nic_config_name: 'controller.yaml' >>>>>>>> update_serial: 1 >>>>>>>> ServicesDefault: >>>>>>>> - OS::TripleO::Services::Aide >>>>>>>> - OS::TripleO::Services::AodhApi >>>>>>>> - OS::TripleO::Services::AodhEvaluator >>>>>>>> >>>>>>>> .. >>>>>>>> . >>>>>>>> >>>>>>>> >>>>>>>> ..############################################################################### >>>>>>>> # Role: Compute >>>>>>>> # >>>>>>>> >>>>>>>> ############################################################################### >>>>>>>> - name: Compute >>>>>>>> description: | >>>>>>>> Basic Compute Node role >>>>>>>> CountDefault: 1 >>>>>>>> # Create external Neutron bridge (unset if using ML2/OVS without >>>>>>>> DVR) >>>>>>>> tags: >>>>>>>> - compute >>>>>>>> - external_bridge >>>>>>>> networks: >>>>>>>> InternalApi: >>>>>>>> subnet: internal_api_subnet >>>>>>>> J3Mgmt: >>>>>>>> subnet: j3mgmt_subnet >>>>>>>> HostnameFormatDefault: '%stackname%-novacompute-%index%' >>>>>>>> RoleParametersDefault: >>>>>>>> FsAioMaxNumber: 1048576 >>>>>>>> TunedProfileName: "virtual-host" >>>>>>>> # Deprecated & backward-compatible values (FIXME: Make parameters >>>>>>>> consistent) >>>>>>>> # Set uses_deprecated_params to True if any deprecated params are >>>>>>>> used. >>>>>>>> # These deprecated_params only need to be used for existing roles >>>>>>>> and not for >>>>>>>> # composable roles. >>>>>>>> uses_deprecated_params: True >>>>>>>> deprecated_param_image: 'NovaImage' >>>>>>>> deprecated_param_extraconfig: 'NovaComputeExtraConfig' >>>>>>>> deprecated_param_metadata: 'NovaComputeServerMetadata' >>>>>>>> deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints' >>>>>>>> deprecated_param_ips: 'NovaComputeIPs' >>>>>>>> deprecated_server_resource_name: 'NovaCompute' >>>>>>>> >>>>>>>> deprecated_nic_config_name: 'compute.yaml' >>>>>>>> update_serial: 25 >>>>>>>> ServicesDefault: >>>>>>>> - OS::TripleO::Services::Aide >>>>>>>> - OS::TripleO::Services::AuditD >>>>>>>> - OS::TripleO::Services::BootParams >>>>>>>> >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ cat >>>>>>>> environments/network-environment.yaml >>>>>>>> >>>>>>>> #This file is an example of an environment file for defining the >>>>>>>> isolated >>>>>>>> #networks and related parameters. >>>>>>>> resource_registry: >>>>>>>> # Network Interface templates to use (these files must exist). >>>>>>>> You can >>>>>>>> # override these by including one of the net-*.yaml environment >>>>>>>> files, >>>>>>>> # such as net-bond-with-vlans.yaml, or modifying the list here. >>>>>>>> # Port assignments for the Controller >>>>>>>> OS::TripleO::Controller::Net::SoftwareConfig: OS::Heat::None >>>>>>>> # Port assignments for the Compute >>>>>>>> OS::TripleO::Compute::Net::SoftwareConfig: OS::Heat::None >>>>>>>> >>>>>>>> >>>>>>>> parameter_defaults: >>>>>>>> # This section is where deployment-specific configuration is done >>>>>>>> # >>>>>>>> ServiceNetMap: >>>>>>>> IronicApiNetwork: oc_provisioning >>>>>>>> IronicNetwork: oc_provisioning >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # This section is where deployment-specific configuration is done >>>>>>>> ControllerNetworkConfigTemplate: >>>>>>>> 'templates/bonds_vlans/bonds_vlans.j2' >>>>>>>> ComputeNetworkConfigTemplate: >>>>>>>> 'templates/bonds_vlans/bonds_vlans.j2' >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # Customize the IP subnet to match the local environment >>>>>>>> J3MgmtNetCidr: 'fd80:fd00:fd00:4000::/64' >>>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>>> J3MgmtAllocationPools: [{'start': 'fd80:fd00:fd00:4000::10', >>>>>>>> 'end': 'fd80:fd00:fd00:4000:ffff:ffff:ffff:fffe'}] >>>>>>>> # Customize the VLAN ID to match the local environment >>>>>>>> J3MgmtNetworkVlanID: 400 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # Customize the IP subnet to match the local environment >>>>>>>> InternalApiNetCidr: 'fd00:fd00:fd00:2000::/64' >>>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>>> InternalApiAllocationPools: [{'start': 'fd00:fd00:fd00:2000::10', >>>>>>>> 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}] >>>>>>>> # Customize the VLAN ID to match the local environment >>>>>>>> InternalApiNetworkVlanID: 418 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # Customize the IP subnet to match the local environment >>>>>>>> ExternalNetCidr: 'fd00:fd00:fd00:9900::/64' >>>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>>> # Leave room if the external network is also used for floating IPs >>>>>>>> ExternalAllocationPools: [{'start': 'fd00:fd00:fd00:9900::10', >>>>>>>> 'end': 'fd00:fd00:fd00:9900:ffff:ffff:ffff:fffe'}] >>>>>>>> # Gateway router for routable networks >>>>>>>> ExternalInterfaceDefaultRoute: 'fd00:fd00:fd00:9900::1' >>>>>>>> # Customize the VLAN ID to match the local environment >>>>>>>> ExternalNetworkVlanID: 408 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # Customize the IP subnet to match the local environment >>>>>>>> OCProvisioningNetCidr: '172.23.3.0/24' >>>>>>>> # Customize the IP range to use for static IPs and VIPs >>>>>>>> OCProvisioningAllocationPools: [{'start': '172.23.3.10', 'end': >>>>>>>> '172.23.3.50'}] >>>>>>>> # Customize the VLAN ID to match the local environment >>>>>>>> OCProvisioningNetworkVlanID: 412 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # List of Neutron network types for tenant networks (will be used >>>>>>>> in order) >>>>>>>> NeutronNetworkType: 'geneve,vlan' >>>>>>>> # Neutron VLAN ranges per network, for example >>>>>>>> 'datacentre:1:499,tenant:500:1000': >>>>>>>> NeutronNetworkVLANRanges: 'datacentre:1:1000' >>>>>>>> # Customize bonding options, e.g. "mode=4 lacp_rate=1 >>>>>>>> updelay=1000 miimon=100" >>>>>>>> # for Linux bonds w/LACP, or "bond_mode=active-backup" for OVS >>>>>>>> active/backup. >>>>>>>> BondInterfaceOvsOptions: "bond_mode=active-backup" >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ >>>>>>>> >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ cat >>>>>>>> environments/network-isolation.yaml >>>>>>>> >>>>>>>> # NOTE: This template is now deprecated, and is only included for >>>>>>>> compatibility >>>>>>>> # when upgrading a deployment where this template was originally >>>>>>>> used. For new >>>>>>>> # deployments, set "ipv6: true" on desired networks in >>>>>>>> network_data.yaml, and >>>>>>>> # include network-isolation.yaml. >>>>>>>> # >>>>>>>> # Enable the creation of Neutron networks for isolated Overcloud >>>>>>>> # traffic and configure each role to assign ports (related >>>>>>>> # to that role) on these networks. >>>>>>>> resource_registry: >>>>>>>> # networks as defined in network_data.yaml >>>>>>>> OS::TripleO::Network::J3Mgmt: ../network/j3mgmt_v6.yaml >>>>>>>> OS::TripleO::Network::InternalApi: ../network/internal_api_v6.yaml >>>>>>>> OS::TripleO::Network::External: ../network/external_v6.yaml >>>>>>>> OS::TripleO::Network::OCProvisioning: >>>>>>>> ../network/oc_provisioning.yaml >>>>>>>> >>>>>>>> >>>>>>>> # Port assignments for the VIPs >>>>>>>> OS::TripleO::Network::Ports::J3MgmtVipPort: >>>>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>>>> OS::TripleO::Network::Ports::InternalApiVipPort: >>>>>>>> ../network/ports/internal_api_v6.yaml >>>>>>>> OS::TripleO::Network::Ports::ExternalVipPort: >>>>>>>> ../network/ports/external_v6.yaml >>>>>>>> OS::TripleO::Network::Ports::OCProvisioningVipPort: >>>>>>>> ../network/ports/oc_provisioning.yaml >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> # Port assignments by role, edit role definition to assign >>>>>>>> networks to roles. >>>>>>>> # Port assignments for the Controller >>>>>>>> OS::TripleO::Controller::Ports::J3MgmtPort: >>>>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>>>> OS::TripleO::Controller::Ports::InternalApiPort: >>>>>>>> ../network/ports/internal_api_v6.yaml >>>>>>>> OS::TripleO::Controller::Ports::ExternalPort: >>>>>>>> ../network/ports/external_v6.yaml >>>>>>>> OS::TripleO::Controller::Ports::OCProvisioningPort: >>>>>>>> ../network/ports/oc_provisioning.yaml >>>>>>>> # Port assignments for the Compute >>>>>>>> OS::TripleO::Compute::Ports::J3MgmtPort: >>>>>>>> ../network/ports/j3mgmt_v6.yaml >>>>>>>> OS::TripleO::Compute::Ports::InternalApiPort: >>>>>>>> ../network/ports/internal_api_v6.yaml >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> parameter_defaults: >>>>>>>> # Enable IPv6 environment for Manila >>>>>>>> ManilaIPv6: True >>>>>>>> >>>>>>>> (undercloud) [stack at undercloud templates]$ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, May 24, 2022 at 5:04 PM Lokendra Rathour < >>>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks, I'll check them out. >>>>>>>>> will let you know in case it works out. >>>>>>>>> >>>>>>>>> On Tue, May 24, 2022 at 2:37 PM Swogat Pradhan < >>>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> Please find the below templates: >>>>>>>>>> These are for openstack wallaby release: >>>>>>>>>> >>>>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>>>> custom_network_data.yaml >>>>>>>>>> - name: Storage >>>>>>>>>> name_lower: storage >>>>>>>>>> vip: true >>>>>>>>>> mtu: 1500 >>>>>>>>>> subnets: >>>>>>>>>> storage_subnet: >>>>>>>>>> ip_subnet: 172.25.202.0/26 >>>>>>>>>> allocation_pools: >>>>>>>>>> - start: 172.25.202.6 >>>>>>>>>> end: 172.25.202.20 >>>>>>>>>> vlan: 1105 >>>>>>>>>> - name: StorageMgmt >>>>>>>>>> name_lower: storage_mgmt >>>>>>>>>> vip: true >>>>>>>>>> mtu: 1500 >>>>>>>>>> subnets: >>>>>>>>>> storage_mgmt_subnet: >>>>>>>>>> ip_subnet: 172.25.202.64/26 >>>>>>>>>> allocation_pools: >>>>>>>>>> - start: 172.25.202.72 >>>>>>>>>> end: 172.25.202.87 >>>>>>>>>> vlan: 1106 >>>>>>>>>> - name: InternalApi >>>>>>>>>> name_lower: internal_api >>>>>>>>>> vip: true >>>>>>>>>> mtu: 1500 >>>>>>>>>> subnets: >>>>>>>>>> internal_api_subnet: >>>>>>>>>> ip_subnet: 172.25.201.192/26 >>>>>>>>>> allocation_pools: >>>>>>>>>> - start: 172.25.201.198 >>>>>>>>>> end: 172.25.201.212 >>>>>>>>>> vlan: 1104 >>>>>>>>>> - name: Tenant >>>>>>>>>> vip: false # Tenant network does not use VIPs >>>>>>>>>> mtu: 1500 >>>>>>>>>> name_lower: tenant >>>>>>>>>> subnets: >>>>>>>>>> tenant_subnet: >>>>>>>>>> ip_subnet: 172.25.202.128/26 >>>>>>>>>> allocation_pools: >>>>>>>>>> - start: 172.25.202.135 >>>>>>>>>> end: 172.25.202.150 >>>>>>>>>> vlan: 1108 >>>>>>>>>> - name: External >>>>>>>>>> name_lower: external >>>>>>>>>> vip: true >>>>>>>>>> mtu: 1500 >>>>>>>>>> subnets: >>>>>>>>>> external_subnet: >>>>>>>>>> ip_subnet: 172.25.201.128/26 >>>>>>>>>> allocation_pools: >>>>>>>>>> - start: 172.25.201.135 >>>>>>>>>> end: 172.25.201.150 >>>>>>>>>> gateway_ip: 172.25.201.129 >>>>>>>>>> vlan: 1103 >>>>>>>>>> >>>>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>>>> custom_vip_data.yaml >>>>>>>>>> - network: ctlplane >>>>>>>>>> #dns_name: overcloud >>>>>>>>>> ip_address: 172.25.201.91 >>>>>>>>>> subnet: ctlplane-subnet >>>>>>>>>> - network: external >>>>>>>>>> #dns_name: overcloud >>>>>>>>>> ip_address: 172.25.201.150 >>>>>>>>>> subnet: external_subnet >>>>>>>>>> - network: internal_api >>>>>>>>>> #dns_name: overcloud >>>>>>>>>> ip_address: 172.25.201.250 >>>>>>>>>> subnet: internal_api_subnet >>>>>>>>>> - network: storage >>>>>>>>>> #dns_name: overcloud >>>>>>>>>> ip_address: 172.25.202.50 >>>>>>>>>> subnet: storage_subnet >>>>>>>>>> - network: storage_mgmt >>>>>>>>>> #dns_name: overcloud >>>>>>>>>> ip_address: 172.25.202.90 >>>>>>>>>> subnet: storage_mgmt_subnet >>>>>>>>>> >>>>>>>>>> (undercloud) [stack at hkg2director workplace]$ cat >>>>>>>>>> overcloud-baremetal-deploy.yaml >>>>>>>>>> - name: Controller >>>>>>>>>> count: 4 >>>>>>>>>> defaults: >>>>>>>>>> networks: >>>>>>>>>> - network: ctlplane >>>>>>>>>> vif: true >>>>>>>>>> - network: external >>>>>>>>>> subnet: external_subnet >>>>>>>>>> - network: internal_api >>>>>>>>>> subnet: internal_api_subnet >>>>>>>>>> - network: storage >>>>>>>>>> subnet: storage_subnet >>>>>>>>>> - network: storage_mgmt >>>>>>>>>> subnet: storage_mgmt_subnet >>>>>>>>>> - network: tenant >>>>>>>>>> subnet: tenant_subnet >>>>>>>>>> network_config: >>>>>>>>>> template: /home/stack/templates/controller.j2 >>>>>>>>>> default_route_network: >>>>>>>>>> - external >>>>>>>>>> instances: >>>>>>>>>> - hostname: overcloud-controller-0 >>>>>>>>>> name: dc1-controller2 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-controller-1 >>>>>>>>>> name: dc2-controller2 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-controller-2 >>>>>>>>>> name: dc1-controller1 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-controller-no-ceph-3 >>>>>>>>>> name: dc2-ceph2 >>>>>>>>>> #provisioned: false >>>>>>>>>> #- hostname: overcloud-controller-3 >>>>>>>>>> #name: dc2-compute3 >>>>>>>>>> #provisioned: false >>>>>>>>>> >>>>>>>>>> - name: Compute >>>>>>>>>> count: 5 >>>>>>>>>> defaults: >>>>>>>>>> networks: >>>>>>>>>> - network: ctlplane >>>>>>>>>> vif: true >>>>>>>>>> - network: internal_api >>>>>>>>>> subnet: internal_api_subnet >>>>>>>>>> - network: tenant >>>>>>>>>> subnet: tenant_subnet >>>>>>>>>> - network: storage >>>>>>>>>> subnet: storage_subnet >>>>>>>>>> network_config: >>>>>>>>>> template: /home/stack/templates/compute.j2 >>>>>>>>>> instances: >>>>>>>>>> - hostname: overcloud-novacompute-0 >>>>>>>>>> name: dc2-compute1 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-novacompute-1 >>>>>>>>>> name: dc2-ceph1 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-novacompute-2 >>>>>>>>>> name: dc1-compute1 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-novacompute-3 >>>>>>>>>> name: dc1-compute2 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-novacompute-4 >>>>>>>>>> name: dc2-compute3 >>>>>>>>>> #provisioned: false >>>>>>>>>> >>>>>>>>>> - name: CephStorage >>>>>>>>>> count: 4 >>>>>>>>>> defaults: >>>>>>>>>> networks: >>>>>>>>>> - network: ctlplane >>>>>>>>>> vif: true >>>>>>>>>> - network: internal_api >>>>>>>>>> subnet: internal_api_subnet >>>>>>>>>> - network: storage >>>>>>>>>> subnet: storage_subnet >>>>>>>>>> - network: storage_mgmt >>>>>>>>>> subnet: storage_mgmt_subnet >>>>>>>>>> network_config: >>>>>>>>>> template: /home/stack/templates/ceph-storage.j2 >>>>>>>>>> instances: >>>>>>>>>> - hostname: overcloud-cephstorage-0 >>>>>>>>>> name: dc2-controller1 >>>>>>>>>> #provisioned: false >>>>>>>>>> # - hostname: overcloud-cephstorage-1 >>>>>>>>>> # name: dc2-ceph2 >>>>>>>>>> - hostname: overcloud-cephstorage-1 >>>>>>>>>> name: dc1-ceph1 >>>>>>>>>> # provisioned: false >>>>>>>>>> - hostname: overcloud-cephstorage-2 >>>>>>>>>> name: dc1-ceph2 >>>>>>>>>> #provisioned: false >>>>>>>>>> - hostname: overcloud-cephstorage-3 >>>>>>>>>> name: dc2-compute2 >>>>>>>>>> #provisioned: false >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> You must use these templates to provision network, vip and nodes. >>>>>>>>>> You must use the output files generated during the provisioning >>>>>>>>>> step in openstack overcloud deploy command using -e parameter. >>>>>>>>>> >>>>>>>>>> With regards, >>>>>>>>>> Swogat Pradhan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, May 23, 2022 at 8:33 PM Lokendra Rathour < >>>>>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Swogat, >>>>>>>>>>> I tried checking your solution and my templates but could not >>>>>>>>>>> relate much. >>>>>>>>>>> But issue seems the same >>>>>>>>>>> >>>>>>>>>>> http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028401.html >>>>>>>>>>> >>>>>>>>>>> I tried somemore ways but looks like some issue with templates. >>>>>>>>>>> Can you please share the templates used to deploy the overcloud. >>>>>>>>>>> >>>>>>>>>>> Mysetup have 3 controller and 1 compute. >>>>>>>>>>> >>>>>>>>>>> Thanks once again for reading my mail. >>>>>>>>>>> >>>>>>>>>>> Waiting for your reply. >>>>>>>>>>> >>>>>>>>>>> -Lokendra >>>>>>>>>>> >>>>>>>>>>> On Fri, 20 May 2022, 08:25 Swogat Pradhan, < >>>>>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> Yes I was able to find the issue and fix it. >>>>>>>>>>>> The issue was with the overcloud-baremetal-deployed.yaml file i >>>>>>>>>>>> was trying to provision controller-0, controller-1 and controller-3 and >>>>>>>>>>>> kept controller-2 aside for later, but the tripleo scripts are built in >>>>>>>>>>>> such a way that they were taking controller- 0, 1 and 2 inplace of >>>>>>>>>>>> controller-3, so the network ports and vip were created for controller 0,1 >>>>>>>>>>>> and 2 but not for 3 , so this error was popping off. Also i would request >>>>>>>>>>>> you to check the jinja nic templates and once the node provisioning is done >>>>>>>>>>>> check the /etc/os-net-config/config.json/yaml file for syntax if using >>>>>>>>>>>> bonded nic template. >>>>>>>>>>>> If you need any more infor please let me know. >>>>>>>>>>>> >>>>>>>>>>>> With regards, >>>>>>>>>>>> Swogat Pradhan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, May 20, 2022 at 8:01 AM Lokendra Rathour < >>>>>>>>>>>> lokendrarathour at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Swogat, >>>>>>>>>>>>> Thanks for raising this issue. >>>>>>>>>>>>> Did you find any solution? to this problem ? >>>>>>>>>>>>> >>>>>>>>>>>>> Please let me know it might be helpful >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Apr 19, 2022 at 12:43 PM Swogat Pradhan < >>>>>>>>>>>>> swogatpradhan22 at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> I am currently trying to deploy openstack wallaby using >>>>>>>>>>>>>> tripleo arch. >>>>>>>>>>>>>> I created the network jinja templates, ran the following >>>>>>>>>>>>>> commands also: >>>>>>>>>>>>>> >>>>>>>>>>>>>> #openstack overcloud network provision --stack overcloud >>>>>>>>>>>>>> --output networks-deployed-environment.yaml custom_network_data.yaml >>>>>>>>>>>>>> # openstack overcloud network vip provision --stack >>>>>>>>>>>>>> overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml >>>>>>>>>>>>>> # openstack overcloud node provision --stack overcloud >>>>>>>>>>>>>> --overcloud-ssh-key /home/stack/sshkey/id_rsa >>>>>>>>>>>>>> overcloud-baremetal-deploy.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> and used the environment files in the openstack overcloud >>>>>>>>>>>>>> deploy command: >>>>>>>>>>>>>> >>>>>>>>>>>>>> (undercloud) [stack at hkg2director ~]$ cat deploy.sh >>>>>>>>>>>>>> #!/bin/bash >>>>>>>>>>>>>> THT=/usr/share/openstack-tripleo-heat-templates/ >>>>>>>>>>>>>> CNF=/home/stack/ >>>>>>>>>>>>>> openstack overcloud deploy --templates $THT \ >>>>>>>>>>>>>> -r $CNF/templates/roles_data.yaml \ >>>>>>>>>>>>>> -n $CNF/workplace/custom_network_data.yaml \ >>>>>>>>>>>>>> -e ~/containers-prepare-parameter.yaml \ >>>>>>>>>>>>>> -e $CNF/templates/node-info.yaml \ >>>>>>>>>>>>>> -e $CNF/templates/scheduler-hints.yaml \ >>>>>>>>>>>>>> -e $CNF/workplace/networks-deployed-environment.yaml \ >>>>>>>>>>>>>> -e $CNF/workplace/vip-deployed-environment.yaml \ >>>>>>>>>>>>>> -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ >>>>>>>>>>>>>> -e $CNF/workplace/custom-net-bond-with-vlans.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> Now when i run the ./deploy.sh script i encounter an error >>>>>>>>>>>>>> stating: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ERROR openstack [-] Resource >>>>>>>>>>>>>> OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type >>>>>>>>>>>>>> OS::Neutron::Port and the Neutron service is not available when using >>>>>>>>>>>>>> ephemeral Heat. The generated environments from 'openstack overcloud >>>>>>>>>>>>>> baremetal provision' and 'openstack overcloud network provision' must be >>>>>>>>>>>>>> included with the deployment command.: >>>>>>>>>>>>>> tripleoclient.exceptions.InvalidConfiguration: Resource >>>>>>>>>>>>>> OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type >>>>>>>>>>>>>> OS::Neutron::Port and the Neutron service is not available when using >>>>>>>>>>>>>> ephemeral Heat. The generated environments from 'openstack overcloud >>>>>>>>>>>>>> baremetal provision' and 'openstack overcloud network provision' must be >>>>>>>>>>>>>> included with the deployment command. >>>>>>>>>>>>>> 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END >>>>>>>>>>>>>> return value: 1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can someone tell me where the mistake is? >>>>>>>>>>>>>> >>>>>>>>>>>>>> With regards, >>>>>>>>>>>>>> Swogat Pradhan >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From christophe.sauthier at accenture.com Wed Jun 8 15:08:14 2022 From: christophe.sauthier at accenture.com (Sauthier, Christophe) Date: Wed, 8 Jun 2022 15:08:14 +0000 Subject: [External] [CLOUDKITTY] presentation used in the forum panels In-Reply-To: References: Message-ID: I just wanted to take a quick moment to thank you Rafael and Pierre for continuing the ongoing development on Cloudkitty ! Christophe ________________________________ De : Rafael Weing?rtner Envoy? : mercredi 8 juin 2022 09:59 ? : openstack-discuss Objet : [External] [CLOUDKITTY] presentation used in the forum panels This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments. ________________________________ Hello CloudKitters! It was a pleasure to meet you guys during the OpenStack summit. Also, it was a good surprise to see that so many people already use CloudKitty. I really hope that we can all collaborate together to keep improving CloudKitty, and making it more and more solid. Attached is the presentation we used during the Forum, so you all can consult this material. Also, as always, if you need some guidance/help/or anything else, let us know. Link for the presentation online: https://docs.google.com/presentation/d/15e37xGKCJoTMwbarrGcSlLpECUkcJ2exazW_TztCehk/edit?usp=sharing -- Rafael Weing?rtner ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy. ______________________________________________________________________________________ www.accenture.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim+openstack.org at coote.org Thu Jun 9 08:58:46 2022 From: tim+openstack.org at coote.org (tim+openstack.org at coote.org) Date: Thu, 9 Jun 2022 09:58:46 +0100 Subject: simple build fails to allocate network interface Message-ID: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> Hullo I?m trying to spin up a simple OS environment in a single vm and think that I?m making some silly mistake, but cannot spot it. Any hints would be greatly expected. I?ve tried this on a couple of laptops, with the same result, so I suspect that it?s something to do with how I?ve set up my vm, but I cannot identify it. (I think that there may be a few other devstack issues, but I?ve got to get it working to log any bugs). I?m using a vagrant box and VirtualBox on x86 Macs (Monterey), and the devstack install/build process. Everything seems to install, but when I try: `openstack server create --flavor 42 --image cirros-0.5.2-x86_64-disk --nic net-id=2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 --security-group default wibble` The network id is identified from this: ??" [vagrant at localhost devstack]$ openstack network list +--------------------------------------+---------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+---------+----------------------------------------------------------------------------+ | 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f | public | 1bbbfa3c-8e0a-483f-a084-4e38241b3315, eecfc603-c057-4b28-90bd-950a47345410 | | 2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 | private | e7377bed-3e22-4e8d-9c2d-ea7ba740fcfd, f915b88c-9988-4e1f-9060-a6295465699a | | fab3fb15-cbd2-45fe-8451-3f0063d59454 | shared | 97e6d8e6-0f57-48bc-8f1d-e30587086153 | +--------------------------------------+---------+??????????????????????????????????????+ ??? Using a public network fails with an error that this isn?t allowed. With the private network above, the failure throws out the following errors. ??" [vagrant at localhost devstack]$ sudo journalctl -f |grep ERROR Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7510, in _create_guest_with_network Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] guest = self._create_guest( Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] next(self.gen) Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 556, in wait_for_instance_event Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._wait_for_instance_events( Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 468, in _wait_for_instance_events Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] actual_event = event.wait() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 433, in wait Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] instance_event = self.event.wait() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] result = hub.switch() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] return self.greenlet.switch() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] eventlet.timeout.Timeout: 300 seconds Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] During handling of the above exception, another exception occurred: Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 2728, in _build_resources Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] yield resources Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 2487, in _build_and_run_instance Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self.driver.spawn(context, instance, image_meta, Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4344, in spawn Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._create_guest_with_network( Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7528, in _create_guest_with_network Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] raise exception.VirtualInterfaceCreateException() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Failed to allocate network(s): nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7510, in _create_guest_with_network Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] guest = self._create_guest( Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] next(self.gen) Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 556, in wait_for_instance_event Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._wait_for_instance_events( Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 468, in _wait_for_instance_events Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] actual_event = event.wait() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 433, in wait Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] instance_event = self.event.wait() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] result = hub.switch() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] return self.greenlet.switch() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] eventlet.timeout.Timeout: 300 seconds Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] During handling of the above exception, another exception occurred: Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 2487, in _build_and_run_instance Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self.driver.spawn(context, instance, image_meta, Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4344, in spawn Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._create_guest_with_network( Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7528, in _create_guest_with_network Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] raise exception.VirtualInterfaceCreateException() Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Build of instance 20e1b5bf-130a-48f9-af4e-9b4c7a475315 aborted: Failed to allocate the network(s), not rescheduling.: nova.exception.BuildAbortException: Build of instance 20e1b5bf-130a-48f9-af4e-9b4c7a475315 aborted: Failed to allocate the network(s), not rescheduling. ??" The Vagrantfile (commented out lines relate to trying this with Ubunutu, which failed the same way, and attempts to use multiple vms): ??" # -*- mode: ruby -*- # vi: set ft=ruby : $script = <<-SHELL dnf update -y #apt update #apt upgrade echo "updated" systemctl disable firewalld systemctl stop firewalld systemctl disable NetworkManager systemctl stop NetworkManager systemctl enable network systemctl start network #dnf config-manager --enable powertools #dnf install -y centos-release-openstack-yoga dnf update -y dnf install -y git python3-pip #dnf install -y openstack-packstack #packstack --provision-demo=n --os-neutron-ml2-type-drivers=vxlan,flat,vlan --gen-answer-file=packstack-answers.txt #sed -i -e 's:10.0.2.15:10.1.0.10:' packstack-answers.txt # not for centos 9 #dnf install -y network-scripts dnf install -y net-tools #cat /vagrant/answers.addon >> packstack-answers.txt # don't want this, but httpd won't start without it setenforce 0 pip3 install invoke #packstack --allinone cd /vagrant # permissions all wrong: ./go.sh # need su - vagrant cd /vagrant && ./go.sh (?) # cd devstack # not working: wrong user ./stack.sh SHELL Vagrant.configure(2) do |config| config.vm.box = "eurolinux-vagrant/centos-stream-9" #config.vm.box = "hashicorp/bionic64" # machines = { # 'node1.example.dd' #'node1.example.dd' => { :ip => '10.1.0.10'}, # 'node2.example.dd' => { :ip =>'10.1.0.12'}, # } config.hostmanager.enabled = true config.hostmanager.manage_host = true config.hostmanager.manage_guest = true config.hostmanager.ignore_private_ip = false config.hostmanager.include_offline = true config.ssh.pty = true config.vm.provision "shell", inline: $script config.vm.network "forwarded_port", guest: 80, host: 8080 # machines.each do | hostname, attrs| # config.vm.define hostname do |machine| # machine.vm.hostname = hostname # machine.vm.network :private_network, :ip => attrs[:ip] # machine.vm.provider "virtualbox" do | v | config.vm.provider "virtualbox" do | v | #v.memory = "4096" #v.memory = "8192" v.memory = "9216" v.cpus = "2" end #end # end end ??? I think that the vm has enough RAM, although there is minimal swap being used, but I think that this is not significant as there is much more RAM used for caching files. Any obvious hints as to why the spin up fails to create the NIC - or somewhere to look for further and better details? Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafaelweingartner at gmail.com Thu Jun 9 18:08:26 2022 From: rafaelweingartner at gmail.com (=?UTF-8?Q?Rafael_Weing=C3=A4rtner?=) Date: Thu, 9 Jun 2022 15:08:26 -0300 Subject: [CLOUDKITTY][OpenInfra summit 2022][June 9th] Presentation on Ceilometer, Gnocchi, and CloudKitty Message-ID: Hello CloudKitters! It was a great opportunity to be able to present this talk for you all guys. It was a pleasure to see that so many people are interested in CloudKitty, Ceilometer, and Gnocchi. I really hope that we can all collaborate together to keep improving CloudKitty, and other related systems. Attached is the presentation we used during the presentation, so you all can consult this material. Also, as always, if you need some guidance, let us know. Furthermore, as I mentioned in the presentation, I will provide a step-by-step process on how to create a new metric using Ceilometer Dynamic pollster that can collect/generate/derive (however you want to call it) IOPs information for volumes. The new metric name is going to be called "dynamic_pollster.volume.services.volume.iops". Also, the idea is that we collect some attributes, and we use the attribute called "bootable", which indicates if the volume is bootable or not. Then, if the volume is bootable, we will charge for its IOPS; otherwise, we do not charge it. As I mentioned, this is just a very simple use case. It is not targeting anyone's need in production, but it can be used to get you going quite easily with the pipeline. Requirements: - Ceilometer up and running. Version 15.0.0 (or higher is recommended); - Gnocchi up and running. Version 4.3.0 (or higher). If you want to run this in production, this patch might be very important ( https://github.com/gnocchixyz/gnocchi/pull/1059); it has not been merged yet; - CloudKitty up and running. Version 10.0.0 (or higher). With all systems up and running, you will need to do the following: - Ceilometer - Add the new metric collection in the "polling.yaml" file. You can add it in any of the sections you already have there. Or, you can create its own, as follows: ``` - name: volume_iops_metric interval: 600 meters: - dynamic_pollster.volume.services.volume.iops ``` - After declaring the metrics there, you can create the pollster in the "pollster.d" folder in Ceilometer. The name of the file can be anything, as long as it ends with ".yaml"; for instance, "pollsters.d/volume-iops.yaml". Then, you will need to add the following content. Remember to replace the contents of "<>" with the values of your environment. ``` - name: "dynamic_pollster.volume.services.volume.iops" response_entries_key: "volumes" next_sample_url_attribute: "volumes_links | filter(lambda v: v.get('rel') == 'next', value) | list(value) | value[0] | value.get('href')" sample_type: "gauge" unit: "GB" value_attribute: "volume_type" endpoint_type: "volume" url_path: "/v3//volumes/detail?all_tenants=true&limit=10" headers: "Openstack-API-Version": "latest" project_id_attribute: "os-vol-tenant-attr:tenant_id" metadata_fields: - "status" - "name" - "volume_type" - "availability_zone" - "user_id" - "bootable | str(value)" - "attachments | value or [ { 'server_id': '' } ] | value[0] | value['server_id']" metadata_mapping: "availability_zone": "dynamic_availability_zone" "name": "dynamic_display_name" "volume_type": "dynamic_volume_type" "attachments | value or [{ 'server_id': '' }] | value[0] | value['server_id']": "dynamic_instance_id" "bootable | str(value)": "bootable" value_mapping: "": "" ``` - After creating that, you will need to customize the file "gnocchi_resources.yaml", which is the file that maps metrics to resources, and attributes that are pushed to the backend. You can also create your own "gnocchi_resources.yaml" file, and then use it in the pipeline configuration in the "pipeline.yaml" file, in the configuration of "publishers". Then, you would have something like "gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml", assuming the file is called "custom_gnocchi_resources.yaml". In this file, you will need to add the new metric and attribute to be pushed to the metrics backend. The following is an example of "volume" resource declaration in the "gnocchi_resource.yaml" file with the new metric and attribute that we want to capture. ``` - resource_type: volume metrics: dynamic_pollster.volume.services.volume.iops: attributes: display_name: resource_metadata.(dynamic_display_name|notification_display_name) bootable: resource_metadata.bootable ``` - After the above configuration, you will need to create the new attribute in the Gnocchi resource type. To achieve that, you can issue the following command. ``` gnocchi resource-type update -a bootable:string:false volume ``` - Then, you will need to restart the Ceilometer. - Now, you can issue "gnocchi resource show ", and you should see the new metric in the resource, and the new attribute as well. With Ceilometer and Gnocchi configured for the new metric, we can move on to CloudKitty. - The new metric needs to be declared in the "metrics.yml" file. The content should be something similar to the following (adapt as you need). ``` dynamic_pollster.volume.services.volume.iops: unit: IOPS alt_name: disk-volume-iops-usage-hours groupby: - id - user_id - project_id - volume_type - display_name - revision_start - instance_id - availability_zone - bootable metadata: [] extra_args: aggregation_method: max resource_type: volume use_all_resource_revisions: false ``` - Restart CloudKitty processor - With the new metric introduced in CloudKitty processor, you will need to create the rating configuration. In our case, we wanted to rate the volumes marked as bootable for IOPs. The volumes that are not bootable would not be rated. The following are the steps one needs to execute to create the hashmap rating rules. - Create the service ``` openstack rating hashmap service create disk-volume-iops-usage-hours ``` - Then, create the field ``` openstack rating hashmap field create ``` - After that, we can create the rating rule ``` openstack rating hashmap mapping create --field --value 'true' 1 --type flat ``` - Having done that, you should be able to rate all volumes by IOPS if they have the flag "bootable" set to true. Otherwise, they will not be rated. That is all that one needs to execute to replicate the example I showed during the presentation. If you guys have any doubts or difficulties when using CloudKitty, do not hesitate to ping me. -- Rafael Weing?rtner -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Gnocchi, Ceilometer, and CloudKitty, a dynamic and agnostic cloud monitoring and billing stack [OpenInfra summit 2022]..pdf Type: application/pdf Size: 761249 bytes Desc: not available URL: From kkchn.in at gmail.com Fri Jun 10 13:01:52 2022 From: kkchn.in at gmail.com (KK CHN) Date: Fri, 10 Jun 2022 18:31:52 +0530 Subject: Devstack Installation Fails with Internal Server Error (HTTP 500) Message-ID: List, I am trying Devstack installation on a Virtual Machine to try TROVE DbaaS. All the time ./stack.sh command stops with the same error message at this particular point + functions:_upload_image:121 : openstack --os-cloud=devstack-admin --os-region-name=RegionOne image create cirros-0.5.2-x86_64-disk --public --container-format bare --disk-format qcow2 --property hw_rng_model=virtio Internal Server Error (HTTP 500) I have ./unstack.sh && ./clean.sh and re run ./stack.sh multiple time but no progress. Here always the script exit with error . Screenshot attached (using "stack" user created by the devstack/tool/ stack user script) my local.conf file as follows here 10.184.48.187 is my VMs IP Any hints to rectify this error most welcome!! ############################################################ [[local|localrc]] RECLONE=False HOST_IP=10.184.48.187 enable_plugin trove https://opendev.org/openstack/trove enable_plugin trove-dashboard https://opendev.org/openstack/trove-dashboard LIBS_FROM_GIT+=,python-troveclient DATABASE_PASSWORD=password ADMIN_PASSWORD=password SERVICE_PASSWORD=password SERVICE_TOKEN=password RABBIT_PASSWORD=password LOGFILE=$DEST/logs/stack.sh.log VERBOSE=True LOG_COLOR=False LOGDAYS=1 IPV4_ADDRS_SAFE_TO_USE=10.111.0.0/26 FIXED_RANGE=10.111.0.0/26 NETWORK_GATEWAY=10.111.0.1 FLOATING_RANGE=10.184.48.0/24 PUBLIC_NETWORK_GATEWAY=10.184.48.1 #pre requisites ENABLED_SERVICES=rabbit,mysql,key #nova enable_service horizon enable_service n-api enable_service n-cpu enable_service n-cond enable_service n-sch enable_service n-api-meta enable_service placement-api enable_service placement-client #Glance enable_service g-api enable_service g-reg #Cinder enable_service cinder enable_service c-api enable_service c-vol enable_service c-sch #Neutron enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta #enable DVR Q_AGENT=openvswitch Q_DVR_MODE=legacy Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch Q_ML2_TENANT_NETWORK_TYPE=vxlan Q_PLUGIN=ml2 #SWIFT ENABLED_SERVICES+=,swift SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5 SWIFT_REPLICAS=1 stack at XenaCtrl1:/home/cloud/trove_devstack/devstack$ ################################################################# -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: DevstackError1.png Type: image/png Size: 228221 bytes Desc: not available URL: From p.aminian.server at gmail.com Mon Jun 13 09:59:41 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Mon, 13 Jun 2022 14:29:41 +0430 Subject: snapshot problem In-Reply-To: <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> Message-ID: I attached nova.conf On Mon, Jun 13, 2022 at 12:15 PM Eugen Block wrote: > Could you share your whole nova.conf (only the uncommented options)? > Is this option set in your env? > > #snapshot_image_format = > > Also when you manually created the snapshot did you do it as the nova > user on the compute node? If not, could you retry? > > > Zitat von Parsa Aminian : > > > Hi > > kolla-ansible victoria version with ceph backend . > > rbd info output : > > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': > > size 20 GiB in 5120 objects > > order 22 (4 MiB objects) > > snapshot_count: 0 > > id: b69aaf907284da > > block_name_prefix: rbd_data.b69aaf907284da > > format: 2 > > features: layering, exclusive-lock, object-map, fast-diff, > > deep-flatten > > op_features: > > flags: > > create_timestamp: Fri May 20 00:04:47 2022 > > access_timestamp: Sun Jun 12 16:26:02 2022 > > --------------- > > also live snapshot seems to work correctly without any error or any > > downtime : > > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls > > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk > > SNAPID NAME SIZE PROTECTED TIMESTAMP > > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 > > > > also on compute nova.conf, images_type is set on rbd . > > > > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: > > > >> You should respond to the list so other users can try to support you. > >> > >> So nova is trying to live snapshot the instance: > >> > >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - > >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > >> > instance snapshotting > >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning > >> > live snapshot process > >> > >> But I don't see any 'rbd snap create' command. Either the rbd image > >> doesn't support it or there is some setting to keep all rbd images > >> "flat". Can you check any relevant configs you might have in nova? > >> Also can you show the output of 'rbd info > >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if > >> the underlying rbd functions work as expected you could try to create > >> a live snapshot manually: > >> > >> rbd snap create > /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap > >> > >> And paste any relevant output here. > >> > >> Zitat von Parsa Aminian : > >> > >> > Its not working for any instances and all of them are paused . I > enable > >> > debug logs please check the logs : > >> > > >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync > for > >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: waited 0.000s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute > >> host > >> > and has allocations in placement: {'resources': {'VCPU': 1, > 'MEMORY_MB': > >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> > 63426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process > >> > default default] Lazy-loading 'pci_devices' on Instance uuid > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading > >> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > obj_load_attr > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > => Paused> emit_event > >> > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) > >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power > state > >> > after lifecycle event "Paused"; current vm_state: active, current > >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> > power_state: 3 handle_lifecycle_event > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > >> instance > >> > has a pending task (image_pending_upload). Skip. > >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd > >> > (subprocess): qemu-img convert -t none -O raw -f raw > >> > > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> > > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad > >> > execute > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > => Stopped> emit_event > >> > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> > INFO nova.compute.manager [-] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) > >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > - - > >> - > >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> > _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > - - > >> - > >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing > >> > instance power state after lifecycle event "Stopped"; current > vm_state: > >> > active, current task_state: image_pending_upload, current DB > power_state: > >> > 1, VM power_state: 4 handle_lifecycle_event > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - > - > >> - - > >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During > >> sync_power_state > >> > the instance has a pending task (image_pending_upload). Skip. > >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync > for > >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> > "25c8d676-e20a-4238 > >> > -a45c-d51daa62b941" acquired by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: waited 0.000s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > >> instance > >> > has a pending task (image_pending_upload). Skip. > >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: held 0.001s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img > convert > >> > -t none -O raw -f raw > >> > > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> > > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" > >> > returned: 0 in 23.730s execute > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > >> Snapshot > >> > extracted, beginning image upload > >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > => Started> emit_event > >> > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) > >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> > _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power > state > >> > after lifecycle event "Started"; current vm_state: active, current > >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> > power_state: 1 handle_lifecycle_event > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event > >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock > >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network > info > >> > cache for instance _get_instance_nw_info > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 > >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache > with > >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > "address": > >> > "fa:16:3e:ca:00:d9", "network": {"id": > >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label": > >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": > >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, > >> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": > {}}], > >> > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4, > >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", > >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], > "version": > >> 4, > >> > "meta": {}}], "meta": {"injected": true, "tenant_id": > >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network": > >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": > >> {"connectivity": > >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": > >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", > >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > "qbh_params": > >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", > >> "profile": > >> > {}, "preserve_on_delete": false, "meta": {}}] > >> > update_instance_cache_with_nw_info > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 > >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > >> > compute host and has allocations in placement: {'resources': {'VCPU': > 1, > >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. > _remove_deleted_instances_allocations > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot > the > >> > instance on the hypervisor. > >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync > for > >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: waited 0.000s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute > >> host > >> > and has allocations in placement: {'resources': {'VCPU': 1, > 'MEMORY_MB': > >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager > >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> 93fb420b3c604d4fae408b81135b58e9 > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> > > >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: > >> > > >> >> Have you tried with debug logs? Has it worked with live snapshots > >> >> before for other instances or did it never work and all snapshots > were > >> >> "cold"? > >> >> > >> >> Zitat von Parsa Aminian : > >> >> > >> >> > Hi > >> >> > kolla-ansible victoria version with ceph backend without volume > >> >> > > >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block > wrote: > >> >> > > >> >> >> Hi, > >> >> >> > >> >> >> can you share more details about your environment? Which openstack > >> >> >> version is it? What is the storage backend? In earlier releases > there > >> >> >> was an option: > >> >> >> > >> >> >> #disable_libvirt_livesnapshot = false > >> >> >> > >> >> >> but this option has been deprecated. But if you're on an older > >> version > >> >> >> that could explain it. > >> >> >> > >> >> >> Zitat von Parsa Aminian : > >> >> >> > >> >> >> > When I snapshot from the instance , server will gone away and > its > >> not > >> >> >> > reachable until the snapshot is complete here is the logs : > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager > >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle > Event) > >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager > >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state > the > >> >> >> instance > >> >> >> > has a pending task (image_pending_upload). Skip. > >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] > [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle > Event) > >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver > >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> >> >> 93fb420b3c604d4fae408b81135b58e9 > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, > beginning > >> >> image > >> >> >> > upload > >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager > >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle > Event) > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> > >> > >> > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- for nova.conf yes I changed ips and passwords for security reason : [root at R3SG4 ~]# cat /etc/kolla/nova-compute/nova.conf [DEFAULT] debug = False log_dir = /var/log/kolla/nova state_path = /var/lib/nova allow_resize_to_same_host = true compute_driver = libvirt.LibvirtDriver my_ip = ip instance_usage_audit = True instance_usage_audit_period = hour transport_url = rabbit://openstack:iyuiyiuyu at ip:5672// injected_network_template = /usr/lib/python3.6/site-packages/nova/virt/interfaces.template flat_injected = true force_config_drive = true config_drive_cdrom = True enable_instance_password = True dhcp_domain = resume_guests_state_on_host_boot = true reclaim_instance_interval = 86400 cpu_allocation_ratio = 3.5 ram_allocation_ratio = 1.0 disk_allocation_ratio = 1.0 resize_confirm_window = 10 [conductor] workers = 5 [vnc] novncproxy_host = ip novncproxy_port = 6080 server_listen = ip server_proxyclient_address = ip novncproxy_base_url = http://ip:6080/vnc_auto.html [oslo_concurrency] lock_path = /var/lib/nova/tmp [glance] api_servers = http://ip:9292 cafile = cafile = num_retries = 3 debug = False [neutron] metadata_proxy_shared_secret = jklhiuy service_metadata_proxy = true auth_url = http://ip:35357 auth_type = password cafile = project_domain_name = Default user_domain_id = default project_name = service username = neutron password = iupouopiupou region_name = RegionThree valid_interfaces = internal [libvirt] connection_uri = qemu+tcp://ip/system live_migration_inbound_addr = ip images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = cinder disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 987798 virt_type = kvm inject_partition = -1 inject_password = true cpu_mode = custom cpu_model = Westmere [upgrade_levels] compute = auto [oslo_messaging_notifications] transport_url = rabbit://openstack:lkjhlkhlk at ip:5672// driver = messagingv2 topics = notifications [privsep_entrypoint] helper_command = sudo nova-rootwrap /etc/nova/rootwrap.conf privsep-helper --config-file /etc/nova/nova.conf [guestfs] debug = False [placement] auth_type = password auth_url = http://ip:35357 username = placement password = oiuoiuoiu user_domain_name = Default project_name = service project_domain_name = Default region_name = RegionThree cafile = valid_interfaces = internal [notifications] notify_on_state_change = vm_and_task_state notification_format = unversioned [keystone_authtoken] www_authenticate_uri = http://ip:5000 auth_url = http://ip:35357 ------------- From p.aminian.server at gmail.com Mon Jun 13 10:01:15 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Mon, 13 Jun 2022 14:31:15 +0430 Subject: snapshot problem In-Reply-To: <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> Message-ID: snapshot_image_format doesn't exist on compute or controller nova.conf For the manual snapshot on the previous reply I ran it directly from ceph, but it's interesting from nova compute there is an error about connecting to CEPH cluster !!! also ceph -s is showing similar error [root at R3SG4 ~]# docker exec -u root -it nova_compute rbd snap create vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap2022-06-13 13:42:18.061 7f03dac23200 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory2022-06-13 13:42:18.061 7f03dac23200 -1 AuthRegistry(0x561d7014a790) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling ceph2022-06-13 13:42:18.065 7f03dac23200 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication rbd: couldn't connect to the cluster! On Mon, Jun 13, 2022 at 12:15 PM Eugen Block wrote: > Could you share your whole nova.conf (only the uncommented options)? > Is this option set in your env? > > #snapshot_image_format = > > Also when you manually created the snapshot did you do it as the nova > user on the compute node? If not, could you retry? > > > Zitat von Parsa Aminian : > > > Hi > > kolla-ansible victoria version with ceph backend . > > rbd info output : > > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': > > size 20 GiB in 5120 objects > > order 22 (4 MiB objects) > > snapshot_count: 0 > > id: b69aaf907284da > > block_name_prefix: rbd_data.b69aaf907284da > > format: 2 > > features: layering, exclusive-lock, object-map, fast-diff, > > deep-flatten > > op_features: > > flags: > > create_timestamp: Fri May 20 00:04:47 2022 > > access_timestamp: Sun Jun 12 16:26:02 2022 > > --------------- > > also live snapshot seems to work correctly without any error or any > > downtime : > > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls > > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk > > SNAPID NAME SIZE PROTECTED TIMESTAMP > > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 > > > > also on compute nova.conf, images_type is set on rbd . > > > > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: > > > >> You should respond to the list so other users can try to support you. > >> > >> So nova is trying to live snapshot the instance: > >> > >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - > >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > >> > instance snapshotting > >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning > >> > live snapshot process > >> > >> But I don't see any 'rbd snap create' command. Either the rbd image > >> doesn't support it or there is some setting to keep all rbd images > >> "flat". Can you check any relevant configs you might have in nova? > >> Also can you show the output of 'rbd info > >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if > >> the underlying rbd functions work as expected you could try to create > >> a live snapshot manually: > >> > >> rbd snap create > /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap > >> > >> And paste any relevant output here. > >> > >> Zitat von Parsa Aminian : > >> > >> > Its not working for any instances and all of them are paused . I > enable > >> > debug logs please check the logs : > >> > > >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync > for > >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: waited 0.000s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute > >> host > >> > and has allocations in placement: {'resources': {'VCPU': 1, > 'MEMORY_MB': > >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> > 63426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process > >> > default default] Lazy-loading 'pci_devices' on Instance uuid > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading > >> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > obj_load_attr > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > => Paused> emit_event > >> > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) > >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power > state > >> > after lifecycle event "Paused"; current vm_state: active, current > >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> > power_state: 3 handle_lifecycle_event > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > >> instance > >> > has a pending task (image_pending_upload). Skip. > >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd > >> > (subprocess): qemu-img convert -t none -O raw -f raw > >> > > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> > > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad > >> > execute > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > => Stopped> emit_event > >> > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> > INFO nova.compute.manager [-] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) > >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > - - > >> - > >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> > _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > - - > >> - > >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing > >> > instance power state after lifecycle event "Stopped"; current > vm_state: > >> > active, current task_state: image_pending_upload, current DB > power_state: > >> > 1, VM power_state: 4 handle_lifecycle_event > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - > - > >> - - > >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During > >> sync_power_state > >> > the instance has a pending task (image_pending_upload). Skip. > >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync > for > >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> > "25c8d676-e20a-4238 > >> > -a45c-d51daa62b941" acquired by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: waited 0.000s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > >> instance > >> > has a pending task (image_pending_upload). Skip. > >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: held 0.001s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img > convert > >> > -t none -O raw -f raw > >> > > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> > > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" > >> > returned: 0 in 23.730s execute > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > >> Snapshot > >> > extracted, beginning image upload > >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 > >> > => Started> emit_event > >> > > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) > >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> > _get_power_state > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power > state > >> > after lifecycle event "Started"; current vm_state: active, current > >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> > power_state: 1 handle_lifecycle_event > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager > >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event > >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock > >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network > info > >> > cache for instance _get_instance_nw_info > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 > >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache > with > >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > "address": > >> > "fa:16:3e:ca:00:d9", "network": {"id": > >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label": > >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": > >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, > >> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": > {}}], > >> > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4, > >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", > >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], > "version": > >> 4, > >> > "meta": {}}], "meta": {"injected": true, "tenant_id": > >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network": > >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": > >> {"connectivity": > >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": > >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", > >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > "qbh_params": > >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", > >> "profile": > >> > {}, "preserve_on_delete": false, "meta": {}}] > >> > update_instance_cache_with_nw_info > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 > >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > >> > compute host and has allocations in placement: {'resources': {'VCPU': > 1, > >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. > _remove_deleted_instances_allocations > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager > >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot > the > >> > instance on the hypervisor. > >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync > for > >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> > > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> > :: waited 0.000s inner > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker > >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute > >> host > >> > and has allocations in placement: {'resources': {'VCPU': 1, > 'MEMORY_MB': > >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> > > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager > >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> 93fb420b3c604d4fae408b81135b58e9 > >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> > > >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: > >> > > >> >> Have you tried with debug logs? Has it worked with live snapshots > >> >> before for other instances or did it never work and all snapshots > were > >> >> "cold"? > >> >> > >> >> Zitat von Parsa Aminian : > >> >> > >> >> > Hi > >> >> > kolla-ansible victoria version with ceph backend without volume > >> >> > > >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block > wrote: > >> >> > > >> >> >> Hi, > >> >> >> > >> >> >> can you share more details about your environment? Which openstack > >> >> >> version is it? What is the storage backend? In earlier releases > there > >> >> >> was an option: > >> >> >> > >> >> >> #disable_libvirt_livesnapshot = false > >> >> >> > >> >> >> but this option has been deprecated. But if you're on an older > >> version > >> >> >> that could explain it. > >> >> >> > >> >> >> Zitat von Parsa Aminian : > >> >> >> > >> >> >> > When I snapshot from the instance , server will gone away and > its > >> not > >> >> >> > reachable until the snapshot is complete here is the logs : > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager > >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle > Event) > >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager > >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state > the > >> >> >> instance > >> >> >> > has a pending task (image_pending_upload). Skip. > >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] > [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle > Event) > >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver > >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> >> >> 93fb420b3c604d4fae408b81135b58e9 > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, > beginning > >> >> image > >> >> >> > upload > >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager > >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle > Event) > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> > >> > >> > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Mon Jun 13 16:44:11 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Mon, 13 Jun 2022 11:44:11 -0500 Subject: [all][tc] Technical Committee next weekly meeting on 16 June 2022 at 1500 UTC Message-ID: <1815df38dec.b4d59475189195.5924549091605681735@ghanshyammann.com> Hello Everyone, The technical Committee's next weekly meeting is scheduled for 16 June 2022, at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, 15 June at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann From eblock at nde.ag Mon Jun 13 17:03:42 2022 From: eblock at nde.ag (Eugen Block) Date: Mon, 13 Jun 2022 17:03:42 +0000 Subject: snapshot problem In-Reply-To: References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> Message-ID: <20220613170342.Horde.qRaR_2eecVXaGrvWHuz6tC7@webmail.nde.ag> Hi, I don't have a containerized deployment so I don't have anything to compare, but my compute nodes have a ceph.conf and all required keyrings in /etc/ceph/, maybe someone else can confirm that it's missing and should be present in a kolla-ansible deployment. In the meantime you could provide those files to the container and retry the live snapshot. Zitat von Parsa Aminian : > snapshot_image_format doesn't exist on compute or controller nova.conf > For the manual snapshot on the previous reply I ran it directly from ceph, > but it's interesting from nova compute there is an error about connecting > to CEPH cluster !!! also ceph -s is showing similar error > [root at R3SG4 ~]# docker exec -u root -it nova_compute rbd snap create > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap2022-06-13 > 13:42:18.061 7f03dac23200 -1 auth: unable to find a keyring on > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory2022-06-13 13:42:18.061 7f03dac23200 -1 > AuthRegistry(0x561d7014a790) no keyring found at > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling ceph2022-06-13 13:42:18.065 7f03dac23200 -1 monclient: > authenticate NOTE: no keyring found; disabled cephx authentication > rbd: couldn't connect to the cluster! > > On Mon, Jun 13, 2022 at 12:15 PM Eugen Block wrote: > >> Could you share your whole nova.conf (only the uncommented options)? >> Is this option set in your env? >> >> #snapshot_image_format = >> >> Also when you manually created the snapshot did you do it as the nova >> user on the compute node? If not, could you retry? >> >> >> Zitat von Parsa Aminian : >> >> > Hi >> > kolla-ansible victoria version with ceph backend . >> > rbd info output : >> > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': >> > size 20 GiB in 5120 objects >> > order 22 (4 MiB objects) >> > snapshot_count: 0 >> > id: b69aaf907284da >> > block_name_prefix: rbd_data.b69aaf907284da >> > format: 2 >> > features: layering, exclusive-lock, object-map, fast-diff, >> > deep-flatten >> > op_features: >> > flags: >> > create_timestamp: Fri May 20 00:04:47 2022 >> > access_timestamp: Sun Jun 12 16:26:02 2022 >> > --------------- >> > also live snapshot seems to work correctly without any error or any >> > downtime : >> > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls >> > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk >> > SNAPID NAME SIZE PROTECTED TIMESTAMP >> > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 >> > >> > also on compute nova.conf, images_type is set on rbd . >> > >> > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: >> > >> >> You should respond to the list so other users can try to support you. >> >> >> >> So nova is trying to live snapshot the instance: >> >> >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> >> > instance snapshotting >> >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning >> >> > live snapshot process >> >> >> >> But I don't see any 'rbd snap create' command. Either the rbd image >> >> doesn't support it or there is some setting to keep all rbd images >> >> "flat". Can you check any relevant configs you might have in nova? >> >> Also can you show the output of 'rbd info >> >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if >> >> the underlying rbd functions work as expected you could try to create >> >> a live snapshot manually: >> >> >> >> rbd snap create >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap >> >> >> >> And paste any relevant output here. >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> > Its not working for any instances and all of them are paused . I >> enable >> >> > debug logs please check the logs : >> >> > >> >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync >> for >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> > :: waited 0.000s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute >> >> host >> >> > and has allocations in placement: {'resources': {'VCPU': 1, >> 'MEMORY_MB': >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> > 63426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process >> >> > default default] Lazy-loading 'pci_devices' on Instance uuid >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading >> >> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > obj_load_attr >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event >> >> > > 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > => Paused> emit_event >> >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) >> >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power >> state >> >> > after lifecycle event "Paused"; current vm_state: active, current >> >> > task_state: image_pending_upload, current DB power_state: 1, VM >> >> > power_state: 3 handle_lifecycle_event >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> instance >> >> > has a pending task (image_pending_upload). Skip. >> >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd >> >> > (subprocess): qemu-img convert -t none -O raw -f raw >> >> > >> >> >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> >> > >> >> >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad >> >> > execute >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 >> >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event >> >> > > 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > => Stopped> emit_event >> >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> > INFO nova.compute.manager [-] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) >> >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa >> - - >> >> - >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> >> > _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa >> - - >> >> - >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing >> >> > instance power state after lifecycle event "Stopped"; current >> vm_state: >> >> > active, current task_state: image_pending_upload, current DB >> power_state: >> >> > 1, VM power_state: 4 handle_lifecycle_event >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - >> - >> >> - - >> >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During >> >> sync_power_state >> >> > the instance has a pending task (image_pending_upload). Skip. >> >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync >> for >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> > "25c8d676-e20a-4238 >> >> > -a45c-d51daa62b941" acquired by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> > :: waited 0.000s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> instance >> >> > has a pending task (image_pending_upload). Skip. >> >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> > :: held 0.001s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 >> >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img >> convert >> >> > -t none -O raw -f raw >> >> > >> >> >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> >> > >> >> >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" >> >> > returned: 0 in 23.730s execute >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> >> Snapshot >> >> > extracted, beginning image upload >> >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event >> >> > > 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > => Started> emit_event >> >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) >> >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> >> > _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power >> state >> >> > after lifecycle event "Started"; current vm_state: active, current >> >> > task_state: image_pending_upload, current DB power_state: 1, VM >> >> > power_state: 1 handle_lifecycle_event >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event >> >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock >> >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network >> info >> >> > cache for instance _get_instance_nw_info >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 >> >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache >> with >> >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", >> "address": >> >> > "fa:16:3e:ca:00:d9", "network": {"id": >> >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label": >> >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": >> >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, >> >> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": >> {}}], >> >> > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4, >> >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", >> >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], >> "version": >> >> 4, >> >> > "meta": {}}], "meta": {"injected": true, "tenant_id": >> >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network": >> >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": >> >> {"connectivity": >> >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": >> >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", >> >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", >> "qbh_params": >> >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", >> >> "profile": >> >> > {}, "preserve_on_delete": false, "meta": {}}] >> >> > update_instance_cache_with_nw_info >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 >> >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this >> >> > compute host and has allocations in placement: {'resources': {'VCPU': >> 1, >> >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. >> _remove_deleted_instances_allocations >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot >> the >> >> > instance on the hypervisor. >> >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync >> for >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> > :: waited 0.000s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute >> >> host >> >> > and has allocations in placement: {'resources': {'VCPU': 1, >> 'MEMORY_MB': >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> > >> >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: >> >> > >> >> >> Have you tried with debug logs? Has it worked with live snapshots >> >> >> before for other instances or did it never work and all snapshots >> were >> >> >> "cold"? >> >> >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> >> >> > Hi >> >> >> > kolla-ansible victoria version with ceph backend without volume >> >> >> > >> >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block >> wrote: >> >> >> > >> >> >> >> Hi, >> >> >> >> >> >> >> >> can you share more details about your environment? Which openstack >> >> >> >> version is it? What is the storage backend? In earlier releases >> there >> >> >> >> was an option: >> >> >> >> >> >> >> >> #disable_libvirt_livesnapshot = false >> >> >> >> >> >> >> >> but this option has been deprecated. But if you're on an older >> >> version >> >> >> >> that could explain it. >> >> >> >> >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> >> >> >> >> > When I snapshot from the instance , server will gone away and >> its >> >> not >> >> >> >> > reachable until the snapshot is complete here is the logs : >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle >> Event) >> >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state >> the >> >> >> >> instance >> >> >> >> > has a pending task (image_pending_upload). Skip. >> >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] >> [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle >> Event) >> >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver >> >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, >> beginning >> >> >> image >> >> >> >> > upload >> >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle >> Event) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From cboylan at sapwetik.org Mon Jun 13 17:14:57 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 13 Jun 2022 10:14:57 -0700 Subject: Devstack Installation Fails with Internal Server Error (HTTP 500) In-Reply-To: References: Message-ID: <559999b7-f47d-4c0b-ac48-65ff3d45437d@www.fastmail.com> On Fri, Jun 10, 2022, at 6:01 AM, KK CHN wrote: > List, > > I am trying Devstack installation on a Virtual Machine to try TROVE DbaaS. > > All the time ./stack.sh command stops with the same error > message at this particular point > > + functions:_upload_image:121 : openstack > --os-cloud=devstack-admin --os-region-name=RegionOne image create > cirros-0.5.2-x86_64-disk --public --container-format bare --disk-format > qcow2 --property hw_rng_model=virtio > Internal Server Error (HTTP 500) You should examine the glance api logs to see why the service had an error. > > I have ./unstack.sh && ./clean.sh and re run ./stack.sh > multiple time but no progress. Here always the script exit with error > . Screenshot attached > (using "stack" user created by the devstack/tool/ stack user script) > > my local.conf file as follows > here 10.184.48.187 is my VMs IP > > Any hints to rectify this error most welcome!! > > ############################################################ > [[local|localrc]] > RECLONE=False > HOST_IP=10.184.48.187 > > enable_plugin trove https://opendev.org/openstack/trove > enable_plugin trove-dashboard https://opendev.org/openstack/trove-dashboard > > LIBS_FROM_GIT+=,python-troveclient > DATABASE_PASSWORD=password > ADMIN_PASSWORD=password > SERVICE_PASSWORD=password > SERVICE_TOKEN=password > RABBIT_PASSWORD=password > LOGFILE=$DEST/logs/stack.sh.log > VERBOSE=True > LOG_COLOR=False > LOGDAYS=1 > > IPV4_ADDRS_SAFE_TO_USE=10.111.0.0/26 > FIXED_RANGE=10.111.0.0/26 > NETWORK_GATEWAY=10.111.0.1 > FLOATING_RANGE=10.184.48.0/24 > PUBLIC_NETWORK_GATEWAY=10.184.48.1 > > #pre requisites > ENABLED_SERVICES=rabbit,mysql,key > > #nova > enable_service horizon > > enable_service n-api > > enable_service n-cpu > > enable_service n-cond > > enable_service n-sch > enable_service n-api-meta > enable_service placement-api > enable_service placement-client > > #Glance > > > enable_service g-api > enable_service g-reg > > #Cinder > enable_service cinder > enable_service c-api > enable_service c-vol > > enable_service c-sch > > #Neutron > > > enable_service q-svc > enable_service q-agt > > enable_service q-dhcp > enable_service q-l3 > enable_service q-meta > > #enable DVR > Q_AGENT=openvswitch > Q_DVR_MODE=legacy > Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch > Q_ML2_TENANT_NETWORK_TYPE=vxlan > Q_PLUGIN=ml2 > > #SWIFT > ENABLED_SERVICES+=,swift > SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5 > SWIFT_REPLICAS=1 > stack at XenaCtrl1:/home/cloud/trove_devstack/devstack$ > > ################################################################# > > > > Attachments: > * DevstackError1.png From gouthampravi at gmail.com Mon Jun 13 18:12:28 2022 From: gouthampravi at gmail.com (Goutham Pacha Ravi) Date: Mon, 13 Jun 2022 23:42:28 +0530 Subject: [manila] CephFS NFS high availability cluster In-Reply-To: References: Message-ID: Hi Romain, On Mon, Jun 13, 2022 at 6:20 PM CHANU ROMAIN wrote: > > Hello, > > I added Manila service with CephFS NFS driver to my openstack cluster. > Everything works fine but I would like to add 2 nfs-ganesha servers to > ensure high availability to the service. > > I configured haproxy to forward 2049 to ganesha backend but Manila > cephFS NFS provides only IP restriction and see only haproxy's IP > address. To make it work you have to add haproxy to allowed ip but it > means everyone can access the share. True; HAProxy terminates client connections and NFS Ganesha sees the HAProxy's IP address instead of the client's IP address. This causes the client's mount operations to be denied since manila explicitly requests client restrictions to exports according to the share's access rules. Presumably, setting up haproxy in "transparent" mode may allow the client source IP to be preserved. We have found that this is infeasible in deployments such as the Red Hat OpenStack Platform. We're discussing with the nfs-ganesha community if they would support the PROXY protocol [1][2]. How are you orchestrating your nfs-ganesha solution with haproxy? Via a custom script? Or are you using cephadm? [3]. I ask also because the CephFS-NFS driver in manila currently communicates with NFS-Ganesha via dbus, and we're looking to support the use of new Ceph Manager APIs to setup, update and teardown exports in the Zed release - this should make configuring multiple NFS-Ganesha servers much more efficient and easy. [1] https://www.haproxy.com/blog/haproxy/proxy-protocol/ [2] https://github.com/nfs-ganesha/ntirpc/issues/252 [3] https://docs.ceph.com/en/latest/cephadm/services/nfs/ [4] https://docs.ceph.com/en/octopus/cephfs/fs-nfs-exports/#create-cephfs-export > > So currently the only way I found out is to use pacemaker to set public > vip to a running nfs-ganesha node. Could you confirm that is not > possible to provide an active/active nfs-ganesha cluster with manila > cephfs NFS driver? > > Best regards, > Romain From dms at danplanet.com Mon Jun 13 19:46:26 2022 From: dms at danplanet.com (Dan Smith) Date: Mon, 13 Jun 2022 12:46:26 -0700 Subject: Devstack Installation Fails with Internal Server Error (HTTP 500) In-Reply-To: <559999b7-f47d-4c0b-ac48-65ff3d45437d@www.fastmail.com> (Clark Boylan's message of "Mon, 13 Jun 2022 10:14:57 -0700") References: <559999b7-f47d-4c0b-ac48-65ff3d45437d@www.fastmail.com> Message-ID: > You should examine the glance api logs to see why the service had an > error. This is actually not a glance problem, but a case of missing the dbcounter plugin when glance connects to the DB -- uploading an image is just the first thing we try to do, other than keystone stuff. I've been working with them on IRC and their devstack log clearly shows it is installed successfully, so I suspect something else is going on. They dropped out before we finished, but I'll continue chasing it down. --Dan From smooney at redhat.com Tue Jun 14 00:26:50 2022 From: smooney at redhat.com (Sean Mooney) Date: Tue, 14 Jun 2022 01:26:50 +0100 Subject: Devstack Installation Fails with Internal Server Error (HTTP 500) In-Reply-To: References: <559999b7-f47d-4c0b-ac48-65ff3d45437d@www.fastmail.com> Message-ID: <7592ce705d219818f7574c961f22b2acde834358.camel@redhat.com> On Mon, 2022-06-13 at 12:46 -0700, Dan Smith wrote: > > You should examine the glance api logs to see why the service had an > > error. > > This is actually not a glance problem, but a case of missing the > dbcounter plugin when glance connects to the DB -- uploading an image is > just the first thing we try to do, other than keystone stuff. I've been > working with them on IRC and their devstack log clearly shows it is > installed successfully, so I suspect something else is going on. They > dropped out before we finished, but I'll continue chasing it down. i hit the same issue the other day if you recall me asking about it. didnt check my devstack logs to see if it installed or not but i can see if i can repoduce it tomorrow. in theory my ansibel role should hit it if its repoduceable reliable. i havent commit the change to my repo to disabel the perfromace counters so i shoudl be able to repoduce it either on my home server or laptop which ever does not have the change locally. ill let you know if i can and i can proably give you access to the vm if you want to take a look. > > --Dan > From adivya1.singh at gmail.com Tue Jun 14 04:56:32 2022 From: adivya1.singh at gmail.com (Adivya Singh) Date: Tue, 14 Jun 2022 10:26:32 +0530 Subject: Regarding Policy.json entries for glance image update not working for a user In-Reply-To: References: Message-ID: Hi Brian, Please find the response > 1> i am using Xena release version 24.0.1 > > Now the scenario is line below, my customer wants to have their login > access on setting up the properties of an image to the public. now what i > did is > > 1> i created a role in openstack using the admin credential name as "user" > 2> i assigned that user to a role user. > 3> i assigned those user to those project id, which they want to access as > a user role > > Then i went to Glance container which is controller by lxc and made a > policy.yaml file as below > > root at aio1-glance-container-724aa778:/etc/glance# cat policy.yaml > > "modify_image": "role:user" > > then i went to utility container and try to set the properties of a image > using openstack command > > openstack image set --public > > and then i got this error > > HTTP 403 Forbidden: You are not authorized to complete publicize_image > action. > > Even when i am trying the upload image with this user , i get the above > error only > > export OS_ENDPOINT_TYPE=internalURL > export OS_INTERFACE=internalURL > export OS_USERNAME=adsingh > export OS_PASSWORD='adsingh' > export OS_PROJECT_NAME=adsingh > export OS_TENANT_NAME=adsingh > export OS_AUTH_TYPE=password > export OS_AUTH_URL=https://:5000/v3 > export OS_NO_CACHE=1 > export OS_USER_DOMAIN_NAME=Default > export OS_PROJECT_DOMAIN_NAME=Default > export OS_REGION_NAME=RegionOne > > Regards > Adivya Singh > > > > > On Mon, Jun 13, 2022 at 6:41 PM Alan Bishop wrote: > >> >> >> On Mon, Jun 13, 2022 at 6:00 AM Brian Rosmaita < >> rosmaita.fossdev at gmail.com> wrote: >> >>> On 6/13/22 8:29 AM, Adivya Singh wrote: >>> > hi Team, >>> > >>> > Any thoughts on this >>> >>> H Adivya, >>> >>> Please supply some more information, for example: >>> >>> - which openstack release you are using >>> - the full API request you are making to modify the image >>> - the full API response you receive >>> - whether the user with "role:user" is in the same project that owns the >>> image >>> - debug level log extract for this call if you have it >>> - anything else that could be relevant, for example, have you modified >>> any other policies, and if so, what values are you using now? >>> >> >> Also bear in mind that the default policy_file name is "policy.yaml" (not >> .json). You either >> need to provide a policy.yaml file, or override the policy_file setting >> if you really want to >> use policy.json. >> >> Alan >> >> cheers, >>> brian >>> >>> > >>> > Regards >>> > Adivya Singh >>> > >>> > On Sat, Jun 11, 2022 at 12:40 AM Adivya Singh >> > > wrote: >>> > >>> > Hi Team, >>> > >>> > I have a use case where I have to give a user restriction on >>> > updating the image properties as a member. >>> > >>> > I have created a policy Json file and give the modify_image rule to >>> > the particular role, but still it is not working >>> > >>> > "modify_image": "role:user", This role is created in OpenStack. >>> > >>> > but still it is failing while updating properties with a >>> > particular user assigned to a role as "access denied" and >>> > unauthorized access >>> > >>> > Regards >>> > Adivya Singh >>> > >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkajinam at redhat.com Tue Jun 14 05:24:33 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Tue, 14 Jun 2022 14:24:33 +0900 Subject: Regarding Policy.json entries for glance image update not working for a user In-Reply-To: References: Message-ID: Glance has a separate policy rule (publicize_image) for creating/updating public images., and you should define that policy rule instead of modify_image. https://docs.openstack.org/glance/xena/admin/policies.html ~~~ publicize_image - Create or update public images ~~~ AFAIK The modify_image policy defaults to rule:default and is allowed for any users as long as the target image is owned by that user. On Tue, Jun 14, 2022 at 2:01 PM Adivya Singh wrote: > Hi Brian, > > Please find the response > > >> 1> i am using Xena release version 24.0.1 >> >> Now the scenario is line below, my customer wants to have their login >> access on setting up the properties of an image to the public. now what i >> did is >> >> 1> i created a role in openstack using the admin credential name as "user" >> 2> i assigned that user to a role user. >> 3> i assigned those user to those project id, which they want to access >> as a user role >> >> Then i went to Glance container which is controller by lxc and made a >> policy.yaml file as below >> >> root at aio1-glance-container-724aa778:/etc/glance# cat policy.yaml >> >> "modify_image": "role:user" >> >> then i went to utility container and try to set the properties of a image >> using openstack command >> >> openstack image set --public >> >> and then i got this error >> >> HTTP 403 Forbidden: You are not authorized to complete publicize_image >> action. >> >> Even when i am trying the upload image with this user , i get the above >> error only >> >> export OS_ENDPOINT_TYPE=internalURL >> export OS_INTERFACE=internalURL >> export OS_USERNAME=adsingh >> export OS_PASSWORD='adsingh' >> export OS_PROJECT_NAME=adsingh >> export OS_TENANT_NAME=adsingh >> export OS_AUTH_TYPE=password >> export OS_AUTH_URL=https://:5000/v3 >> export OS_NO_CACHE=1 >> export OS_USER_DOMAIN_NAME=Default >> export OS_PROJECT_DOMAIN_NAME=Default >> export OS_REGION_NAME=RegionOne >> >> Regards >> Adivya Singh >> >> >> >> >> On Mon, Jun 13, 2022 at 6:41 PM Alan Bishop wrote: >> >>> >>> >>> On Mon, Jun 13, 2022 at 6:00 AM Brian Rosmaita < >>> rosmaita.fossdev at gmail.com> wrote: >>> >>>> On 6/13/22 8:29 AM, Adivya Singh wrote: >>>> > hi Team, >>>> > >>>> > Any thoughts on this >>>> >>>> H Adivya, >>>> >>>> Please supply some more information, for example: >>>> >>>> - which openstack release you are using >>>> - the full API request you are making to modify the image >>>> - the full API response you receive >>>> - whether the user with "role:user" is in the same project that owns >>>> the >>>> image >>>> - debug level log extract for this call if you have it >>>> - anything else that could be relevant, for example, have you modified >>>> any other policies, and if so, what values are you using now? >>>> >>> >>> Also bear in mind that the default policy_file name is "policy.yaml" >>> (not .json). You either >>> need to provide a policy.yaml file, or override the policy_file setting >>> if you really want to >>> use policy.json. >>> >>> Alan >>> >>> cheers, >>>> brian >>>> >>>> > >>>> > Regards >>>> > Adivya Singh >>>> > >>>> > On Sat, Jun 11, 2022 at 12:40 AM Adivya Singh < >>>> adivya1.singh at gmail.com >>>> > > wrote: >>>> > >>>> > Hi Team, >>>> > >>>> > I have a use case where I have to give a user restriction on >>>> > updating the image properties as a member. >>>> > >>>> > I have created a policy Json file and give the modify_image rule >>>> to >>>> > the particular role, but still it is not working >>>> > >>>> > "modify_image": "role:user", This role is created in OpenStack. >>>> > >>>> > but still it is failing while updating properties with a >>>> > particular user assigned to a role as "access denied" and >>>> > unauthorized access >>>> > >>>> > Regards >>>> > Adivya Singh >>>> > >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From kamil.madac at gmail.com Tue Jun 14 06:29:49 2022 From: kamil.madac at gmail.com (Kamil Madac) Date: Tue, 14 Jun 2022 08:29:49 +0200 Subject: [cinder] Failed to delete volume In-Reply-To: References: Message-ID: Hi Michal, We had similar issue in our kolla-ansible deployment. In our case we were not able to attach volume to the VM in 1 from 5 cases and it sometimes caused also inconsistencies in DB tables. We are also running Victoria release with Ceph backend. We found out that there were many disconnects/reconnects between nova and rabbitmq because of missing heartbeats which happened every 3 minutes. If attachment of volume happened in the time of reconnection, attachment was unsuccessful. What helped in our case was adding following lines to nova.conf: [oslo_messaging_rabbit] heartbeat_timeout_threshold = 0 In your case I would check cinder logs files for messaging errors whether you do not see regular reconnects to rabbitmq. Kamil Madac On Fri, Jun 10, 2022 at 2:08 PM Michal Arbet wrote: > Hi, > > Cinder is from victoria, backend is ceph, steps to reproduce ? Just delete > volume. > I think that it could be an error not in cinder but somehow a rabbitmq > issue. > > :/ > > Michal Arbet > Openstack Engineer > > Ultimum Technologies a.s. > Na Po???? 1047/26, 11000 Praha 1 > Czech Republic > > +420 604 228 897 > michal.arbet at ultimum.io > *https://ultimum.io * > > LinkedIn | Twitter > | Facebook > > > > ?t 7. 6. 2022 v 19:27 odes?latel Sofia Enriquez > napsal: > >> Hi Michal, >> >> I do not think we have such a bug on the Launchpad platform.[1] >> Could you create a new bug report indicating which backend you are using, >> which Cinder version you have, and if the error occurs after following some >> steps? >> >> Thanks in advance, >> Sofia >> >> [1] https://bugs.launchpad.net/cinder >> >> On Tue, Jun 7, 2022 at 4:28 AM Michal Arbet >> wrote: >> >>> Hi, >>> >>> I would like to ask if someone saw issue when volume failed to delete, >>> log below : >>> This is happening from time to time. >>> >>> LOG : https://paste.opendev.org/show/b1jRCxuBFnnVzf64i9yV/ >>> >>> Thanks, >>> >>> Michal Arbet >>> Openstack Engineer >>> >>> Ultimum Technologies a.s. >>> Na Po???? 1047/26, 11000 Praha 1 >>> Czech Republic >>> >>> +420 604 228 897 >>> michal.arbet at ultimum.io >>> *https://ultimum.io * >>> >>> LinkedIn | >>> Twitter | Facebook >>> >>> >> >> >> -- >> >> Sof?a Enriquez >> >> she/her >> >> Software Engineer >> >> Red Hat PnT >> >> IRC: @enriquetaso >> @RedHat Red Hat >> Red Hat >> >> >> >> -- Kamil Madac -------------- next part -------------- An HTML attachment was scrubbed... URL: From geguileo at redhat.com Tue Jun 14 07:38:46 2022 From: geguileo at redhat.com (Gorka Eguileor) Date: Tue, 14 Jun 2022 09:38:46 +0200 Subject: [cinder] Failed to delete volume In-Reply-To: References: Message-ID: <20220614073846.fd7pdx4hwe7joqpz@localhost> On 07/06, Michal Arbet wrote: > Hi, > > I would like to ask if someone saw issue when volume failed to delete, log > below : > This is happening from time to time. > > LOG : https://paste.opendev.org/show/b1jRCxuBFnnVzf64i9yV/ > > Thanks, > > Michal Arbet > Openstack Engineer > Hi, Error seems to be happening acquiring a lock with the Redis driver in Tooz, so it must be configured to be used by Cinder as the DLM. This is only necessary when the Cinder Volume service is configured as Active-Active. Are you running the volume service as Active-Active? If you are not you should be able to fix the issue by commenting the backend_url configuration option in the [coordination] section of the cinder.conf file. Cheers, Gorka. > Ultimum Technologies a.s. > Na Po???? 1047/26, 11000 Praha 1 > Czech Republic > > +420 604 228 897 > michal.arbet at ultimum.io > *https://ultimum.io * > > LinkedIn | Twitter > | Facebook > From fereshtehloghmani at gmail.com Tue Jun 14 07:50:44 2022 From: fereshtehloghmani at gmail.com (fereshteh loghmani) Date: Tue, 14 Jun 2022 12:20:44 +0430 Subject: evacuate problem Message-ID: hello i use masakari for migrate servers when the compute being down. i install these container on the region: masakari-monitors masakari-engine masakari-api hacluster-pacemaker hacluster-corosync and on the compute server i install "hacluster-pacemaker-remote". In openstack i created segment and I added 2 hosts (in this case the name of that 2 hosts are: R3SG5 & R3SG12). for testing evacuate function i shut off one of that compute that i added in host. (in this case i shutoff R3SG5) i attached the log that i found in this directory: /var/log/kolla/masakari/masakari-hostmonitor.log ********* INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'R3SG5' is 'online' (current: 'online'). INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'R3SG12' is 'online' (current: 'online'). WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication using 'eth0' is failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. ERROR masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication is failed. INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'R3SG5' is 'offline' (current: 'offline'). INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'type': 'COMPUTE_HOST', 'hostname': 'R3SG5', 'generated_time': datetime.datetime(2022, 6, 14, 7, 6, 46, 138867), 'payload': {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'}}} INFO masakarimonitors.ha.masakari [-] Response: openstack.instance_ha.v1.notification.Notification(type=COMPUTE_HOST, hostname=R3SG5, generated_time=2022-06-14T07:06:46.138867, payload={'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'}, id=105, notification_uuid=a7364095-cc7d-48f8-b963-c64ba147897c, source_host_uuid=6328f08c-c752-43d5-4689-801d91dd67ec, status=new, created_at=2022-06-14T07:06:47.000000, updated_at=None, location=Munch({'cloud': 'controller', 'region_name': 'RegionThree', 'zone': None, 'project': Munch({'id': 'a75a951b4537478e8cea39a932f830da', 'name': None, 'domain_id': None, 'domain_name': None})})) INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'R3SG12' is 'online' (current: 'online'). WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication using 'eth0' is failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. ERROR masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication is failed. INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'R3SG5' is 'offline' (current: 'offline'). INFO masakarimonitors.hostmonitor.host_handler.handle_host [-] 'R3SG12' is 'online' (current: 'online'). WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] Corosync communication using 'eth0' is failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. ***************** also i checked nova_scheduler logs and on that directory i receive error: *** ERROR oslo_messaging.rpc.server nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. ******* finally in OpenStack dashboard in the notification section tatus change from running to failed. after the error state that shows in the notification section, my VM that was on R3SG5 became to ERROR state and the VM still exists on R3SG5 and it doenst been migrated to R3SG12. could you please help me why evacuate function doesn't work correctly? -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Tue Jun 14 08:07:26 2022 From: skaplons at redhat.com (Slawek Kaplonski) Date: Tue, 14 Jun 2022 10:07:26 +0200 Subject: simple build fails to allocate network interface In-Reply-To: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> References: <9A546D4D-7637-4021-9BBB-EDF756E54EAE@coote.org> Message-ID: <12175423.SmF1yDmi0e@p1> Hi, Dnia czwartek, 9 czerwca 2022 10:58:46 CEST tim+openstack.org at coote.org pisze: > Hullo > I?m trying to spin up a simple OS environment in a single vm and think that I?m making some silly mistake, but cannot spot it. Any hints would be greatly expected. I?ve tried this on a couple of laptops, with the same result, so I suspect that it?s something to do with how I?ve set up my vm, but I cannot identify it. (I think that there may be a few other devstack issues, but I?ve got to get it working to log any bugs). > > I?m using a vagrant box and VirtualBox on x86 Macs (Monterey), and the devstack install/build process. Everything seems to install, but when I try: > > `openstack server create --flavor 42 --image cirros-0.5.2-x86_64-disk --nic net-id=2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 --security-group default wibble` > > The network id is identified from this: > ??" > [vagrant at localhost devstack]$ openstack network list > +--------------------------------------+---------+----------------------------------------------------------------------------+ > | ID | Name | Subnets | > +--------------------------------------+---------+----------------------------------------------------------------------------+ > | 205fa7d7-7067-4f63-b3d9-6a9b37b4f11f | public | 1bbbfa3c-8e0a-483f-a084-4e38241b3315, eecfc603-c057-4b28-90bd-950a47345410 | > | 2a1a4a3a-a47b-48bd-a7df-c90fc75a1c63 | private | e7377bed-3e22-4e8d-9c2d-ea7ba740fcfd, f915b88c-9988-4e1f-9060-a6295465699a | > | fab3fb15-cbd2-45fe-8451-3f0063d59454 | shared | 97e6d8e6-0f57-48bc-8f1d-e30587086153 | > +--------------------------------------+---------+??????????????????????????????????????+ > ??? > > Using a public network fails with an error that this isn?t allowed. > > With the private network above, the failure throws out the following errors. > ??" > [vagrant at localhost devstack]$ sudo journalctl -f |grep ERROR > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7510, in _create_guest_with_network > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] guest = self._create_guest( > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] next(self.gen) > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 556, in wait_for_instance_event > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._wait_for_instance_events( > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 468, in _wait_for_instance_events > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] actual_event = event.wait() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 433, in wait > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] instance_event = self.event.wait() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] result = hub.switch() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] return self.greenlet.switch() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] eventlet.timeout.Timeout: 300 seconds > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] During handling of the above exception, another exception occurred: > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 2728, in _build_resources > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] yield resources > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 2487, in _build_and_run_instance > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self.driver.spawn(context, instance, image_meta, > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4344, in spawn > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._create_guest_with_network( > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7528, in _create_guest_with_network > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] raise exception.VirtualInterfaceCreateException() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Failed to allocate network(s): nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7510, in _create_guest_with_network > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] guest = self._create_guest( > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__ > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] next(self.gen) > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 556, in wait_for_instance_event > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._wait_for_instance_events( > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 468, in _wait_for_instance_events > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] actual_event = event.wait() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 433, in wait > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] instance_event = self.event.wait() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] result = hub.switch() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] return self.greenlet.switch() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] eventlet.timeout.Timeout: 300 seconds > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] During handling of the above exception, another exception occurred: > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Traceback (most recent call last): > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/compute/manager.py", line 2487, in _build_and_run_instance > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self.driver.spawn(context, instance, image_meta, > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4344, in spawn > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] self._create_guest_with_network( > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7528, in _create_guest_with_network > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] raise exception.VirtualInterfaceCreateException() > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] > Jun 08 15:33:52 localhost.localdomain nova-compute[87555]: ERROR nova.compute.manager [None req-01294678-b54a-4e40-964c-488ccf03d66c demo demo] [instance: 20e1b5bf-130a-48f9-af4e-9b4c7a475315] Build of instance 20e1b5bf-130a-48f9-af4e-9b4c7a475315 aborted: Failed to allocate the network(s), not rescheduling.: nova.exception.BuildAbortException: Build of instance 20e1b5bf-130a-48f9-af4e-9b4c7a475315 aborted: Failed to allocate the network(s), not rescheduling. > ??" > > The Vagrantfile (commented out lines relate to trying this with Ubunutu, which failed the same way, and attempts to use multiple vms): > ??" > # -*- mode: ruby -*- > # vi: set ft=ruby : > > $script = <<-SHELL > dnf update -y > #apt update > #apt upgrade > echo "updated" > systemctl disable firewalld > systemctl stop firewalld > systemctl disable NetworkManager > systemctl stop NetworkManager > systemctl enable network > systemctl start network > #dnf config-manager --enable powertools > #dnf install -y centos-release-openstack-yoga > dnf update -y > dnf install -y git python3-pip > #dnf install -y openstack-packstack > #packstack --provision-demo=n --os-neutron-ml2-type-drivers=vxlan,flat,vlan --gen-answer-file=packstack-answers.txt > #sed -i -e 's:10.0.2.15:10.1.0.10:' packstack-answers.txt > # not for centos 9 > #dnf install -y network-scripts > dnf install -y net-tools > #cat /vagrant/answers.addon >> packstack-answers.txt > # don't want this, but httpd won't start without it > setenforce 0 > pip3 install invoke > #packstack --allinone > cd /vagrant > # permissions all wrong: ./go.sh # need su - vagrant cd /vagrant && ./go.sh (?) > # cd devstack > # not working: wrong user ./stack.sh > SHELL > > Vagrant.configure(2) do |config| > > config.vm.box = "eurolinux-vagrant/centos-stream-9" > #config.vm.box = "hashicorp/bionic64" > > # machines = { > # 'node1.example.dd' > #'node1.example.dd' => { :ip => '10.1.0.10'}, > # 'node2.example.dd' => { :ip =>'10.1.0.12'}, > # } > > config.hostmanager.enabled = true > config.hostmanager.manage_host = true > config.hostmanager.manage_guest = true > config.hostmanager.ignore_private_ip = false > config.hostmanager.include_offline = true > > config.ssh.pty = true > > config.vm.provision "shell", inline: $script > > config.vm.network "forwarded_port", guest: 80, host: 8080 > > # machines.each do | hostname, attrs| > # config.vm.define hostname do |machine| > # machine.vm.hostname = hostname > # machine.vm.network :private_network, :ip => attrs[:ip] > > # machine.vm.provider "virtualbox" do | v | > config.vm.provider "virtualbox" do | v | > #v.memory = "4096" > #v.memory = "8192" > v.memory = "9216" > v.cpus = "2" > end > > #end > # end > end > ??? > > I think that the vm has enough RAM, although there is minimal swap being used, but I think that this is not significant as there is much more RAM used for caching files. > > Any obvious hints as to why the spin up fails to create the NIC - or somewhere to look for further and better details? > > Tim > This is some issue in neutron while provisioning port. Please check neutron server and neutron agents logs (depending on You backend it can be neutron-ovs-agent and neutron-dhcp-agent, or in case of OVN most of the things You will find in the neutron-server logs). -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From marios at redhat.com Tue Jun 14 10:33:35 2022 From: marios at redhat.com (Marios Andreou) Date: Tue, 14 Jun 2022 13:33:35 +0300 Subject: [TripleO] no more patches for tripleo stable/victoria please (going EOL) Message-ID: Hello o/ as proposed at [1][2] we are moving ahead with transitioning stable/victoria tripleo repos to End Of Life. In order to do this we need to have no open patches against stable/victoria. So please stop posting new patches for tripleo stable/victoria and if you have open patches please merge or abandon these by next Tuesday 21st June (for reference, list of repos with currently open patches at [3]). No problem if someone needs a few more days we can wait just please reach out here or on irc. If there are no such requests then I will abandon all open reviews on 21st and respin [2] to pick up any new commits between now and then. thanks for your help, marios [1] http://lists.openstack.org/pipermail/openstack-discuss/2022-June/028829.html [2] https://review.opendev.org/c/openstack/releases/+/845148 [3] (list of repos with open reviews): * https://review.opendev.org/q/project:openstack%252Fos-net-config+status:open+branch:stable/victoria * https://review.opendev.org/q/project:openstack%252Fpython-tripleoclient+status:open+branch:stable/victoria * https://review.opendev.org/q/project:openstack%252Ftripleo-ansible+status:open+branch:stable/victoria * https://review.opendev.org/q/project:openstack%252Ftripleo-common+status:open+branch:stable/victoria * https://review.opendev.org/q/project:openstack%252Ftripleo-heat-templates+status:open+branch:stable/victoria * https://review.opendev.org/q/project:openstack%252Ftripleo-image-elements+status:open+branch:stable/victoria * https://review.opendev.org/q/project:openstack%252Ftripleo-puppet-elements+status:open+branch:stable/victoria * https://review.opendev.org/q/project:openstack%252Ftripleo-validations+status:open+branch:stable/victoria From tkajinam at redhat.com Tue Jun 14 11:40:30 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Tue, 14 Jun 2022 20:40:30 +0900 Subject: [TripleO] no more patches for tripleo stable/victoria please (going EOL) In-Reply-To: References: Message-ID: I'd like to remind you that we should check *whether that patch has been backported to any older branches (ussuri, train)* we abandon any victoria backport. In the past, we abandoned backport without checking this and this resulted in broken commit id in some backports. (stable/train backports point to the commit id of ussuri backport, which was never merged). If the patch is already merged in ussuri or train then IMO we should try landing the victoria backport. If not, then please make sure you update the stable/train backport to remove any reference to the victoria backport. On Tue, Jun 14, 2022 at 7:41 PM Marios Andreou wrote: > Hello o/ > > as proposed at [1][2] we are moving ahead with transitioning > stable/victoria tripleo repos to End Of Life. > > In order to do this we need to have no open patches against > stable/victoria. > > So please stop posting new patches for tripleo stable/victoria and if > you have open patches please merge or abandon these by next Tuesday > 21st June (for reference, list of repos with currently open patches at > [3]). > > No problem if someone needs a few more days we can wait just please > reach out here or on irc. If there are no such requests then I will > abandon all open reviews on 21st and respin [2] to pick up any new > commits between now and then. > > thanks for your help, > > marios > > [1] > http://lists.openstack.org/pipermail/openstack-discuss/2022-June/028829.html > [2] https://review.opendev.org/c/openstack/releases/+/845148 > [3] (list of repos with open reviews): > * > https://review.opendev.org/q/project:openstack%252Fos-net-config+status:open+branch:stable/victoria > * > https://review.opendev.org/q/project:openstack%252Fpython-tripleoclient+status:open+branch:stable/victoria > * > https://review.opendev.org/q/project:openstack%252Ftripleo-ansible+status:open+branch:stable/victoria > * > https://review.opendev.org/q/project:openstack%252Ftripleo-common+status:open+branch:stable/victoria > * > https://review.opendev.org/q/project:openstack%252Ftripleo-heat-templates+status:open+branch:stable/victoria > * > https://review.opendev.org/q/project:openstack%252Ftripleo-image-elements+status:open+branch:stable/victoria > * > https://review.opendev.org/q/project:openstack%252Ftripleo-puppet-elements+status:open+branch:stable/victoria > * > https://review.opendev.org/q/project:openstack%252Ftripleo-validations+status:open+branch:stable/victoria > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Tue Jun 14 11:59:42 2022 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 14 Jun 2022 11:59:42 +0000 Subject: [TripleO] no more patches for tripleo stable/victoria please (going EOL) In-Reply-To: References: Message-ID: <20220614115942.wqiaqk7hpa2hzdlk@yuggoth.org> On 2022-06-14 20:40:30 +0900 (+0900), Takashi Kajinami wrote: [...] > In the past, we abandoned backport without checking this and this > resulted in broken commit id in some backports. (stable/train > backports point to the commit id of ussuri backport, which was > never merged). [...] Not sure if it's too heavy-handed, but one workaround would be to add a Depends-On footer in the commit message of each backport referencing the change for the branch ahead of it. Then, Zuul would only ever allow the backport to merge if the same change on all newer branches had already merged successfully. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From marios at redhat.com Tue Jun 14 12:22:03 2022 From: marios at redhat.com (Marios Andreou) Date: Tue, 14 Jun 2022 15:22:03 +0300 Subject: [TripleO] no more patches for tripleo stable/victoria please (going EOL) In-Reply-To: References: Message-ID: On Tue, Jun 14, 2022 at 2:40 PM Takashi Kajinami wrote: > > I'd like to remind you that we should check whether that patch has been backported to any older branches (ussuri, train) > we abandon any victoria backport. > > In the past, we abandoned backport without checking this and this resulted in broken commit id in some backports. > (stable/train backports point to the commit id of ussuri backport, which was never merged). > > If the patch is already merged in ussuri or train then IMO we should try landing the victoria backport. > If not, then please make sure you update the stable/train backport to remove any reference to the victoria backport. > > OK thanks for the reminder. Noted, I'll check this next week and ideally folks will be reading this thread and checking their patches ;) regards > On Tue, Jun 14, 2022 at 7:41 PM Marios Andreou wrote: >> >> Hello o/ >> >> as proposed at [1][2] we are moving ahead with transitioning >> stable/victoria tripleo repos to End Of Life. >> >> In order to do this we need to have no open patches against stable/victoria. >> >> So please stop posting new patches for tripleo stable/victoria and if >> you have open patches please merge or abandon these by next Tuesday >> 21st June (for reference, list of repos with currently open patches at >> [3]). >> >> No problem if someone needs a few more days we can wait just please >> reach out here or on irc. If there are no such requests then I will >> abandon all open reviews on 21st and respin [2] to pick up any new >> commits between now and then. >> >> thanks for your help, >> >> marios >> >> [1] http://lists.openstack.org/pipermail/openstack-discuss/2022-June/028829.html >> [2] https://review.opendev.org/c/openstack/releases/+/845148 >> [3] (list of repos with open reviews): >> * https://review.opendev.org/q/project:openstack%252Fos-net-config+status:open+branch:stable/victoria >> * https://review.opendev.org/q/project:openstack%252Fpython-tripleoclient+status:open+branch:stable/victoria >> * https://review.opendev.org/q/project:openstack%252Ftripleo-ansible+status:open+branch:stable/victoria >> * https://review.opendev.org/q/project:openstack%252Ftripleo-common+status:open+branch:stable/victoria >> * https://review.opendev.org/q/project:openstack%252Ftripleo-heat-templates+status:open+branch:stable/victoria >> * https://review.opendev.org/q/project:openstack%252Ftripleo-image-elements+status:open+branch:stable/victoria >> * https://review.opendev.org/q/project:openstack%252Ftripleo-puppet-elements+status:open+branch:stable/victoria >> * https://review.opendev.org/q/project:openstack%252Ftripleo-validations+status:open+branch:stable/victoria >> >> From tkajinam at redhat.com Tue Jun 14 12:22:37 2022 From: tkajinam at redhat.com (Takashi Kajinami) Date: Tue, 14 Jun 2022 21:22:37 +0900 Subject: [TripleO] no more patches for tripleo stable/victoria please (going EOL) In-Reply-To: <20220614115942.wqiaqk7hpa2hzdlk@yuggoth.org> References: <20220614115942.wqiaqk7hpa2hzdlk@yuggoth.org> Message-ID: On Tue, Jun 14, 2022 at 9:04 PM Jeremy Stanley wrote: > On 2022-06-14 20:40:30 +0900 (+0900), Takashi Kajinami wrote: > [...] > > In the past, we abandoned backport without checking this and this > > resulted in broken commit id in some backports. (stable/train > > backports point to the commit id of ussuri backport, which was > > never merged). > [...] > > Not sure if it's too heavy-handed, but one workaround would be to > add a Depends-On footer in the commit message of each backport > referencing the change for the branch ahead of it. Then, Zuul would > only ever allow the backport to merge if the same change on all > newer branches had already merged successfully. > Yeah I think that would be a solution though it requires relatively big overhead, because we need to manually add that Depends-on line. I sometimes hope zuul can block backports depending on the "cherry picked from commit ..." line but it might be worth discussing as a potential enhancement with the zuul devs. > -- > Jeremy Stanley > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.aminian.server at gmail.com Tue Jun 14 06:50:37 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Tue, 14 Jun 2022 11:20:37 +0430 Subject: snapshot problem In-Reply-To: <20220613170342.Horde.qRaR_2eecVXaGrvWHuz6tC7@webmail.nde.ag> References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> <20220613170342.Horde.qRaR_2eecVXaGrvWHuz6tC7@webmail.nde.ag> Message-ID: yes you are right , I added ceph.client.admin.keyring to nova compute container and ceph cluster error gone i also try rbd snap create from nova compute container and its work without problem and vm is up . maybe the problem is related to glance .my glance backend is not on ceph I think maybe its the reason On Mon, Jun 13, 2022 at 9:38 PM Eugen Block wrote: > Hi, > > I don't have a containerized deployment so I don't have anything to > compare, but my compute nodes have a ceph.conf and all required > keyrings in /etc/ceph/, maybe someone else can confirm that it's > missing and should be present in a kolla-ansible deployment. In the > meantime you could provide those files to the container and retry the > live snapshot. > > > Zitat von Parsa Aminian : > > > snapshot_image_format doesn't exist on compute or controller nova.conf > > For the manual snapshot on the previous reply I ran it directly from > ceph, > > but it's interesting from nova compute there is an error about connecting > > to CEPH cluster !!! also ceph -s is showing similar error > > [root at R3SG4 ~]# docker exec -u root -it nova_compute rbd snap create > > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap2022-06-13 > > 13:42:18.061 7f03dac23200 -1 auth: unable to find a keyring on > > > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > > (2) No such file or directory2022-06-13 13:42:18.061 7f03dac23200 -1 > > AuthRegistry(0x561d7014a790) no keyring found at > > > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > > disabling ceph2022-06-13 13:42:18.065 7f03dac23200 -1 monclient: > > authenticate NOTE: no keyring found; disabled cephx authentication > > rbd: couldn't connect to the cluster! > > > > On Mon, Jun 13, 2022 at 12:15 PM Eugen Block wrote: > > > >> Could you share your whole nova.conf (only the uncommented options)? > >> Is this option set in your env? > >> > >> #snapshot_image_format = > >> > >> Also when you manually created the snapshot did you do it as the nova > >> user on the compute node? If not, could you retry? > >> > >> > >> Zitat von Parsa Aminian : > >> > >> > Hi > >> > kolla-ansible victoria version with ceph backend . > >> > rbd info output : > >> > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': > >> > size 20 GiB in 5120 objects > >> > order 22 (4 MiB objects) > >> > snapshot_count: 0 > >> > id: b69aaf907284da > >> > block_name_prefix: rbd_data.b69aaf907284da > >> > format: 2 > >> > features: layering, exclusive-lock, object-map, fast-diff, > >> > deep-flatten > >> > op_features: > >> > flags: > >> > create_timestamp: Fri May 20 00:04:47 2022 > >> > access_timestamp: Sun Jun 12 16:26:02 2022 > >> > --------------- > >> > also live snapshot seems to work correctly without any error or any > >> > downtime : > >> > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls > >> > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk > >> > SNAPID NAME SIZE PROTECTED TIMESTAMP > >> > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 > >> > > >> > also on compute nova.conf, images_type is set on rbd . > >> > > >> > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: > >> > > >> >> You should respond to the list so other users can try to support you. > >> >> > >> >> So nova is trying to live snapshot the instance: > >> >> > >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - > >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > >> >> > instance snapshotting > >> >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning > >> >> > live snapshot process > >> >> > >> >> But I don't see any 'rbd snap create' command. Either the rbd image > >> >> doesn't support it or there is some setting to keep all rbd images > >> >> "flat". Can you check any relevant configs you might have in nova? > >> >> Also can you show the output of 'rbd info > >> >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if > >> >> the underlying rbd functions work as expected you could try to create > >> >> a live snapshot manually: > >> >> > >> >> rbd snap create > >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap > >> >> > >> >> And paste any relevant output here. > >> >> > >> >> Zitat von Parsa Aminian : > >> >> > >> >> > Its not working for any instances and all of them are paused . I > >> enable > >> >> > debug logs please check the logs : > >> >> > > >> >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager > >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering > sync > >> for > >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> >> > > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> > :: waited 0.000s inner > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker > >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > compute > >> >> host > >> >> > and has allocations in placement: {'resources': {'VCPU': 1, > >> 'MEMORY_MB': > >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager > >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > _get_power_state > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> > 63426b4c87a751afaf13b758e0 - default default] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot > process > >> >> > default default] Lazy-loading 'pci_devices' on Instance uuid > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance > >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading > >> >> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> > obj_load_attr > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event > >> >> > >> 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> > => Paused> emit_event > >> >> > > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) > >> >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager > >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > _get_power_state > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager > >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power > >> state > >> >> > after lifecycle event "Paused"; current vm_state: active, current > >> >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> >> > power_state: 3 handle_lifecycle_event > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager > >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > >> >> instance > >> >> > has a pending task (image_pending_upload). Skip. > >> >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils > >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd > >> >> > (subprocess): qemu-img convert -t none -O raw -f raw > >> >> > > >> >> > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> >> > > >> >> > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad > >> >> > execute > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > >> >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event > >> >> > >> 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> > => Stopped> emit_event > >> >> > > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> >> > INFO nova.compute.manager [-] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) > >> >> > DEBUG nova.compute.manager > [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > >> - - > >> >> - > >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking > state > >> >> > _get_power_state > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> > DEBUG nova.compute.manager > [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > >> - - > >> >> - > >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing > >> >> > instance power state after lifecycle event "Stopped"; current > >> vm_state: > >> >> > active, current task_state: image_pending_upload, current DB > >> power_state: > >> >> > 1, VM power_state: 4 handle_lifecycle_event > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> >> > INFO nova.compute.manager > [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - > >> - > >> >> - - > >> >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During > >> >> sync_power_state > >> >> > the instance has a pending task (image_pending_upload). Skip. > >> >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager > >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering > sync > >> for > >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> >> > "25c8d676-e20a-4238 > >> >> > -a45c-d51daa62b941" acquired by > >> >> > > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> > :: waited 0.000s inner > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the > >> >> instance > >> >> > has a pending task (image_pending_upload). Skip. > >> >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock > >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by > >> >> > > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> > :: held 0.001s inner > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > >> >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils > >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img > >> convert > >> >> > -t none -O raw -f raw > >> >> > > >> >> > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> >> > > >> >> > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" > >> >> > returned: 0 in 23.730s execute > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > >> >> Snapshot > >> >> > extracted, beginning image upload > >> >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver > >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event > >> >> > >> 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> > => Started> emit_event > >> >> > > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager > >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) > >> >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> >> > _get_power_state > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power > >> state > >> >> > after lifecycle event "Started"; current vm_state: active, current > >> >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> >> > power_state: 1 handle_lifecycle_event > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager > >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event > >> >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils > >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock > >> >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network > >> info > >> >> > cache for instance _get_instance_nw_info > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 > >> >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron > >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache > >> with > >> >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > >> "address": > >> >> > "fa:16:3e:ca:00:d9", "network": {"id": > >> >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", > "label": > >> >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": > >> >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, > >> >> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": > >> {}}], > >> >> > "gateway": {"address": "141.11.42.1", "type": "gateway", > "version": 4, > >> >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", > >> >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], > >> "version": > >> >> 4, > >> >> > "meta": {}}], "meta": {"injected": true, "tenant_id": > >> >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, > "physical_network": > >> >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": > >> >> {"connectivity": > >> >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, > "datapath_type": > >> >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", > >> >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > >> "qbh_params": > >> >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", > >> >> "profile": > >> >> > {}, "preserve_on_delete": false, "meta": {}}] > >> >> > update_instance_cache_with_nw_info > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 > >> >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on > this > >> >> > compute host and has allocations in placement: {'resources': > {'VCPU': > >> 1, > >> >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. > >> _remove_deleted_instances_allocations > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager > >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to > snapshot > >> the > >> >> > instance on the hypervisor. > >> >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager > >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering > sync > >> for > >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> >> > > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> > :: waited 0.000s inner > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker > >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > compute > >> >> host > >> >> > and has allocations in placement: {'resources': {'VCPU': 1, > >> 'MEMORY_MB': > >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> >> > > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager > >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> >> 93fb420b3c604d4fae408b81135b58e9 > >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> > > >> >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: > >> >> > > >> >> >> Have you tried with debug logs? Has it worked with live snapshots > >> >> >> before for other instances or did it never work and all snapshots > >> were > >> >> >> "cold"? > >> >> >> > >> >> >> Zitat von Parsa Aminian : > >> >> >> > >> >> >> > Hi > >> >> >> > kolla-ansible victoria version with ceph backend without volume > >> >> >> > > >> >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block > >> wrote: > >> >> >> > > >> >> >> >> Hi, > >> >> >> >> > >> >> >> >> can you share more details about your environment? Which > openstack > >> >> >> >> version is it? What is the storage backend? In earlier releases > >> there > >> >> >> >> was an option: > >> >> >> >> > >> >> >> >> #disable_libvirt_livesnapshot = false > >> >> >> >> > >> >> >> >> but this option has been deprecated. But if you're on an older > >> >> version > >> >> >> >> that could explain it. > >> >> >> >> > >> >> >> >> Zitat von Parsa Aminian : > >> >> >> >> > >> >> >> >> > When I snapshot from the instance , server will gone away and > >> its > >> >> not > >> >> >> >> > reachable until the snapshot is complete here is the logs : > >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager > >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] > [instance: > >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle > >> Event) > >> >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager > >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] > [instance: > >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state > >> the > >> >> >> >> instance > >> >> >> >> > has a pending task (image_pending_upload). Skip. > >> >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] > >> [instance: > >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle > >> Event) > >> >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver > >> >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> >> >> >> 93fb420b3c604d4fae408b81135b58e9 > >> >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] > [instance: > >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, > >> beginning > >> >> >> image > >> >> >> >> > upload > >> >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager > >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] > [instance: > >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle > >> Event) > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> > >> > >> > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Tue Jun 14 12:53:30 2022 From: eblock at nde.ag (Eugen Block) Date: Tue, 14 Jun 2022 12:53:30 +0000 Subject: snapshot problem In-Reply-To: References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> <20220613170342.Horde.qRaR_2eecVXaGrvWHuz6tC7@webmail.nde.ag> Message-ID: <20220614125330.Horde.u5KxkrplHvtLIS5bmxnzeIM@webmail.nde.ag> You don't need the admin keyring on a compute node but for tests it can help. Basically it depends on your ceph auth config which keyring you need on the openstack nodes. If you already have ceph configured why don't you use it for all services? Of course live snapshots are not possible if your glance images are not within ceph (I don't think you mentioned that before). Zitat von Parsa Aminian : > yes you are right , I added ceph.client.admin.keyring to nova compute > container and ceph cluster error gone i also try rbd snap create from nova > compute container and its work without problem and vm is up . > maybe the problem is related to glance .my glance backend is not on ceph I > think maybe its the reason > > On Mon, Jun 13, 2022 at 9:38 PM Eugen Block wrote: > >> Hi, >> >> I don't have a containerized deployment so I don't have anything to >> compare, but my compute nodes have a ceph.conf and all required >> keyrings in /etc/ceph/, maybe someone else can confirm that it's >> missing and should be present in a kolla-ansible deployment. In the >> meantime you could provide those files to the container and retry the >> live snapshot. >> >> >> Zitat von Parsa Aminian : >> >> > snapshot_image_format doesn't exist on compute or controller nova.conf >> > For the manual snapshot on the previous reply I ran it directly from >> ceph, >> > but it's interesting from nova compute there is an error about connecting >> > to CEPH cluster !!! also ceph -s is showing similar error >> > [root at R3SG4 ~]# docker exec -u root -it nova_compute rbd snap create >> > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap2022-06-13 >> > 13:42:18.061 7f03dac23200 -1 auth: unable to find a keyring on >> > >> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> > (2) No such file or directory2022-06-13 13:42:18.061 7f03dac23200 -1 >> > AuthRegistry(0x561d7014a790) no keyring found at >> > >> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, >> > disabling ceph2022-06-13 13:42:18.065 7f03dac23200 -1 monclient: >> > authenticate NOTE: no keyring found; disabled cephx authentication >> > rbd: couldn't connect to the cluster! >> > >> > On Mon, Jun 13, 2022 at 12:15 PM Eugen Block wrote: >> > >> >> Could you share your whole nova.conf (only the uncommented options)? >> >> Is this option set in your env? >> >> >> >> #snapshot_image_format = >> >> >> >> Also when you manually created the snapshot did you do it as the nova >> >> user on the compute node? If not, could you retry? >> >> >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> > Hi >> >> > kolla-ansible victoria version with ceph backend . >> >> > rbd info output : >> >> > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': >> >> > size 20 GiB in 5120 objects >> >> > order 22 (4 MiB objects) >> >> > snapshot_count: 0 >> >> > id: b69aaf907284da >> >> > block_name_prefix: rbd_data.b69aaf907284da >> >> > format: 2 >> >> > features: layering, exclusive-lock, object-map, fast-diff, >> >> > deep-flatten >> >> > op_features: >> >> > flags: >> >> > create_timestamp: Fri May 20 00:04:47 2022 >> >> > access_timestamp: Sun Jun 12 16:26:02 2022 >> >> > --------------- >> >> > also live snapshot seems to work correctly without any error or any >> >> > downtime : >> >> > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls >> >> > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk >> >> > SNAPID NAME SIZE PROTECTED TIMESTAMP >> >> > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 >> >> > >> >> > also on compute nova.conf, images_type is set on rbd . >> >> > >> >> > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: >> >> > >> >> >> You should respond to the list so other users can try to support you. >> >> >> >> >> >> So nova is trying to live snapshot the instance: >> >> >> >> >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> >> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - >> >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> >> >> > instance snapshotting >> >> >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning >> >> >> > live snapshot process >> >> >> >> >> >> But I don't see any 'rbd snap create' command. Either the rbd image >> >> >> doesn't support it or there is some setting to keep all rbd images >> >> >> "flat". Can you check any relevant configs you might have in nova? >> >> >> Also can you show the output of 'rbd info >> >> >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if >> >> >> the underlying rbd functions work as expected you could try to create >> >> >> a live snapshot manually: >> >> >> >> >> >> rbd snap create >> >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap >> >> >> >> >> >> And paste any relevant output here. >> >> >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> >> >> > Its not working for any instances and all of them are paused . I >> >> enable >> >> >> > debug logs please check the logs : >> >> >> > >> >> >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering >> sync >> >> for >> >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> >> >> > >> >> >> >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> >> > :: waited 0.000s inner >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this >> compute >> >> >> host >> >> >> > and has allocations in placement: {'resources': {'VCPU': 1, >> >> 'MEMORY_MB': >> >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> _get_power_state >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> >> > 63426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot >> process >> >> >> > default default] Lazy-loading 'pci_devices' on Instance uuid >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> >> >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading >> >> >> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> >> > obj_load_attr >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> >> >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event >> >> >> > > >> 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> >> > => Paused> emit_event >> >> >> > >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) >> >> >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> _get_power_state >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power >> >> state >> >> >> > after lifecycle event "Paused"; current vm_state: active, current >> >> >> > task_state: image_pending_upload, current DB power_state: 1, VM >> >> >> > power_state: 3 handle_lifecycle_event >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> >> instance >> >> >> > has a pending task (image_pending_upload). Skip. >> >> >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd >> >> >> > (subprocess): qemu-img convert -t none -O raw -f raw >> >> >> > >> >> >> >> >> >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> >> >> > >> >> >> >> >> >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad >> >> >> > execute >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 >> >> >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event >> >> >> > > >> 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> >> > => Stopped> emit_event >> >> >> > >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> >> > INFO nova.compute.manager [-] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) >> >> >> > DEBUG nova.compute.manager >> [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa >> >> - - >> >> >> - >> >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking >> state >> >> >> > _get_power_state >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> >> > DEBUG nova.compute.manager >> [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa >> >> - - >> >> >> - >> >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing >> >> >> > instance power state after lifecycle event "Stopped"; current >> >> vm_state: >> >> >> > active, current task_state: image_pending_upload, current DB >> >> power_state: >> >> >> > 1, VM power_state: 4 handle_lifecycle_event >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> >> > INFO nova.compute.manager >> [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - >> >> - >> >> >> - - >> >> >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During >> >> >> sync_power_state >> >> >> > the instance has a pending task (image_pending_upload). Skip. >> >> >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering >> sync >> >> for >> >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> >> > "25c8d676-e20a-4238 >> >> >> > -a45c-d51daa62b941" acquired by >> >> >> > >> >> >> >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> >> > :: waited 0.000s inner >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> >> instance >> >> >> > has a pending task (image_pending_upload). Skip. >> >> >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by >> >> >> > >> >> >> >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> >> > :: held 0.001s inner >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 >> >> >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img >> >> convert >> >> >> > -t none -O raw -f raw >> >> >> > >> >> >> >> >> >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> >> >> > >> >> >> >> >> >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" >> >> >> > returned: 0 in 23.730s execute >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 >> >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> >> >> Snapshot >> >> >> > extracted, beginning image upload >> >> >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event >> >> >> > > >> 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> >> > => Started> emit_event >> >> >> > >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) >> >> >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> >> >> > _get_power_state >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power >> >> state >> >> >> > after lifecycle event "Started"; current vm_state: active, current >> >> >> > task_state: image_pending_upload, current DB power_state: 1, VM >> >> >> > power_state: 1 handle_lifecycle_event >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event >> >> >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock >> >> >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network >> >> info >> >> >> > cache for instance _get_instance_nw_info >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 >> >> >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache >> >> with >> >> >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", >> >> "address": >> >> >> > "fa:16:3e:ca:00:d9", "network": {"id": >> >> >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", >> "label": >> >> >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": >> >> >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, >> >> >> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": >> >> {}}], >> >> >> > "gateway": {"address": "141.11.42.1", "type": "gateway", >> "version": 4, >> >> >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", >> >> >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], >> >> "version": >> >> >> 4, >> >> >> > "meta": {}}], "meta": {"injected": true, "tenant_id": >> >> >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, >> "physical_network": >> >> >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": >> >> >> {"connectivity": >> >> >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, >> "datapath_type": >> >> >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", >> >> >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", >> >> "qbh_params": >> >> >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", >> >> >> "profile": >> >> >> > {}, "preserve_on_delete": false, "meta": {}}] >> >> >> > update_instance_cache_with_nw_info >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 >> >> >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on >> this >> >> >> > compute host and has allocations in placement: {'resources': >> {'VCPU': >> >> 1, >> >> >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. >> >> _remove_deleted_instances_allocations >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to >> snapshot >> >> the >> >> >> > instance on the hypervisor. >> >> >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering >> sync >> >> for >> >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> >> >> > >> >> >> >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" >> >> >> > :: waited 0.000s inner >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this >> compute >> >> >> host >> >> >> > and has allocations in placement: {'resources': {'VCPU': 1, >> >> 'MEMORY_MB': >> >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> >> >> > >> >> >> >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> >> > >> >> >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block wrote: >> >> >> > >> >> >> >> Have you tried with debug logs? Has it worked with live snapshots >> >> >> >> before for other instances or did it never work and all snapshots >> >> were >> >> >> >> "cold"? >> >> >> >> >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> >> >> >> >> > Hi >> >> >> >> > kolla-ansible victoria version with ceph backend without volume >> >> >> >> > >> >> >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block >> >> wrote: >> >> >> >> > >> >> >> >> >> Hi, >> >> >> >> >> >> >> >> >> >> can you share more details about your environment? Which >> openstack >> >> >> >> >> version is it? What is the storage backend? In earlier releases >> >> there >> >> >> >> >> was an option: >> >> >> >> >> >> >> >> >> >> #disable_libvirt_livesnapshot = false >> >> >> >> >> >> >> >> >> >> but this option has been deprecated. But if you're on an older >> >> >> version >> >> >> >> >> that could explain it. >> >> >> >> >> >> >> >> >> >> Zitat von Parsa Aminian : >> >> >> >> >> >> >> >> >> >> > When I snapshot from the instance , server will gone away and >> >> its >> >> >> not >> >> >> >> >> > reachable until the snapshot is complete here is the logs : >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager >> >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] >> [instance: >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle >> >> Event) >> >> >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager >> >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] >> [instance: >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state >> >> the >> >> >> >> >> instance >> >> >> >> >> > has a pending task (image_pending_upload). Skip. >> >> >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] >> >> [instance: >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle >> >> Event) >> >> >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver >> >> >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> >> >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] >> [instance: >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, >> >> beginning >> >> >> >> image >> >> >> >> >> > upload >> >> >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager >> >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] >> [instance: >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle >> >> Event) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From fungi at yuggoth.org Tue Jun 14 12:56:21 2022 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 14 Jun 2022 12:56:21 +0000 Subject: [TripleO] no more patches for tripleo stable/victoria please (going EOL) In-Reply-To: References: <20220614115942.wqiaqk7hpa2hzdlk@yuggoth.org> Message-ID: <20220614125620.nqngnpqchlrg5zgb@yuggoth.org> On 2022-06-14 21:22:37 +0900 (+0900), Takashi Kajinami wrote: [...] > I sometimes hope zuul can block backports depending on the "cherry > picked from commit ..." line but it might be worth discussing as a > potential enhancement with the zuul devs. Definitely worth further discussion, but I suspect the possibility for cherry-picks from other repositories could make that challenging to differentiate. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From amotoki at gmail.com Tue Jun 14 13:22:57 2022 From: amotoki at gmail.com (Akihiro Motoki) Date: Tue, 14 Jun 2022 22:22:57 +0900 Subject: [neutron] bug deputy (week of Jun 6) Message-ID: Hi, I was a bug deputy in neutron. Here is a summary. ## Untriaged * https://bugs.launchpad.net/neutron/+bug/1978088 After ovs-agent restart, table=21 and table=22 on br-tun openflow table is missing This happens when a large number of agents are restarted, but it looks like deeper investigations are needed to understand the situation. (liuyolong and ralonsoh already helped) ## In Progress * https://bugs.launchpad.net/neutron/+bug/1977819 [OVS][QoS] "delete_minimum_bandwidth_queue" uses the incorrect Port UUID High, In Progress * https://bugs.launchpad.net/neutron/+bug/1978519 Create auto allocated topology encounter error when enable ndp_proxy service plugin ## RFE * https://bugs.launchpad.net/neutron/+bug/1978039 [ovn] Floating IP adds distributed attributes ## Fix Released * https://bugs.launchpad.net/neutron/+bug/1977752 [OVS][QoS] Minimum bandwidth backend enforcement does not provide max-rate values * https://bugs.launchpad.net/neutron/+bug/1977969 [OVN] DB sync tool: local variable 'members_to_verify' referenced before assignment" * https://bugs.launchpad.net/neutron/+bug/1978035 remove unused updated_at parameter for AgentCache.update * https://bugs.launchpad.net/neutron/+bug/1978389 Basic networking in Neutron - max VLAN should be 4094, not 4095 * https://bugs.launchpad.net/neutron/+bug/1977831 Response time increasing on new subnets over same network Thanks, Akihiro Motoki (amotoki) From dms at danplanet.com Tue Jun 14 13:37:47 2022 From: dms at danplanet.com (Dan Smith) Date: Tue, 14 Jun 2022 06:37:47 -0700 Subject: Devstack Installation Fails with Internal Server Error (HTTP 500) In-Reply-To: <7592ce705d219818f7574c961f22b2acde834358.camel@redhat.com> (Sean Mooney's message of "Tue, 14 Jun 2022 01:26:50 +0100") References: <559999b7-f47d-4c0b-ac48-65ff3d45437d@www.fastmail.com> <7592ce705d219818f7574c961f22b2acde834358.camel@redhat.com> Message-ID: > i hit the same issue the other day if you recall me asking about it. > didnt check my devstack logs to see if it installed or not but i can > see if i can repoduce it tomorrow. I checked with the OP and they have it installed properly, system-wide. I was assuming your issue might have been venv (or ordering) related, but theirs was not. There are some other weird things going on with their setup, so I'm not so sure. --Dan From JRACE at augusta.edu Tue Jun 14 13:42:11 2022 From: JRACE at augusta.edu (Race, Jonathan) Date: Tue, 14 Jun 2022 13:42:11 +0000 Subject: [Nova] Hardware Architecture Emulation - Summit forum - Emulation beyond just Local QEMU -outcomes Message-ID: Great to see everyone at the summit! For those not in attendance and interested, see the below etherpad that is being used to collect input and requirements as our team moves forward with work. I have also included the outcomes that I have identified. https://etherpad.opendev.org/p/Emulation_going_beyond_local_QEMU Berlin Forum Outcomes I appreciate all the feedback during the forum session, and it was great to have active participation on how to grow and shape this unique feature. We discussed the feature, what it allows, how to actually leverage it, and discussed about further refining the applicable use cases so that operators and users can effectively use it and to help drive interest. A prime example was of a company wanting to shift their infra from x86_64 to arm for the sake of power consumption, and whether this feature could lend its hand to perform some initial fact finding and testing. The answer is yes, you can spin up some of the infra (in your desired arch) on top of your existing deployment and perform your tests and validations. One of the great items that came out of the discussion was that this can not only be to support education, research, and training but it can act as a great 1st step within planning. Meaning when you are going from the drawing board to implementation, you can now model and design your network environments within the cloud for IT and OT systems. This is where refining the use cases really came to light, as our organization leverages OpenStack as a private cloud for hosting extremely customizable network environments (i.e. enterprise iT, IOT, Critical infrastructure, medical devices, etc.) While the feature itself is overall in the experimental phase, it is a great start. This etherpad will remain available and feel free to add to it, as we leverage this year to fact find and identify all further requirements going into the AA release cycle. A snapshot of the discussion points can be found below, and ill be posting the recorded session within the week. * Most operators and providers are running a few releases behind so getting accurate feedback at this time is still ongoing due to this feature only being available in yoga and newer. * Pursuit of additional architectures to be supported (riscv, etc.) * This will require custom firmware support * evaluate current AMI/AKI/ARI functionalities (add ability for leveraging glance if able, and add additional security) * Adding capability for trusted signers of OOVMF file descriptors * evaluating current work related to edk2 oovmf file descriptors * https://github.com/riscv-admin/riscv-uefi-edk2-docs * Expansion and documentation of a valid support matrix * something similar to: (arch x | feat 1 | supported | temptest test 1) * Evaluate requirements for leveraging virtio in relation to scsi/sata devices similar to pci passthrough capabilities Jonathan Race Cyber Range Engineer | Georgia Cyber Range GEORGIA CYBER CENTER 100 Grace Hopper Lane | Augusta, Georgia | 30901 (c) 762-716-7733 www.gacyberrange.org [cid:image001.png at 01D87FD1.CF558390] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 5865 bytes Desc: image001.png URL: From hjensas at redhat.com Tue Jun 14 14:13:19 2022 From: hjensas at redhat.com (Harald Jensas) Date: Tue, 14 Jun 2022 16:13:19 +0200 Subject: [VICTORIA][IRONIC] - Can inspect but not deploy In-Reply-To: References: Message-ID: <181eb30b-4389-e852-7b0b-da7c6c0ffe25@redhat.com> On 6/13/22 12:41, Ga?l THEROND wrote: > Hi pierre, > > I?m using a dedicated interface but this interface is the same for all > ironic networks inspection/provisioning/cleaning. > > This interface works fine for inspection, my only issue is the > pxe_filter that the ironic inspector process allow during inspection > correctly but then tag as disallowed again at the end of the inspection, > shouldn?t the deploy process allow the Mac again before booting the node? > The ironic dnsmasq pxe filter is only supposed to allow DHCP requests when a node is inspected. There are two DHCP services, one for inspector and the other one is typically hosted by neutron for provisioning/cleaning/rescue. Only the dnsmasq for inspector uses the hostdir with ',ignore' files. > I can correctly see the conductor instruct the node to boot up using pxe > from the kvm console but the BootP process doesn?t load the IPA > kernel/initramfs as the dnsmasq pxe discard the request (because of the > mac being still tagged as ?,ignore ? within the hostdir file). > When a node is provisioned/cleaned/rescued the DHCP instance in neutron should be the one providing the DHCP service, not the dnsmasq instance for inspector service. i.e the ',ignore' entries should remain as 'ignore' to ensure the node does not get a DHCP reply from inspector's DHCP service which would deliver the wrong DHCP options. > I?m a bit disappointed by this behavior. > > Thanks a lot! > > Le?lun. 13 juin 2022 ? 11:41, Pierre Riteau > a ?crit?: > > Hello Ga?l, > > Which network_interface are you using for your nodes? Is your > provisioning network different from the inspection network? > > Pierre > > On Mon, 13 Jun 2022 at 10:36, Ga?l THEROND > > wrote: > > Hi everyone! > > I'm dealing with a strange issue today, we deployed IRONIC on a > VICTORIA platform, we activated the dnsmasq pxe filtering option > at the inspector level, it works great as only IRONIC listed > hosts are then served by the dnsmasq DHCP as those hosts are > allowed using the dhcp hosts dir. > > BUT, I'm having a weird issue now. > > All my nodes are able to get an IP from the DHCP at the > INSPECTION step, however, as soon as the inspection step is > finished, the mac related file is once again filled with > ",ignore" which prohibits further operations. > > This means as soon as I put that node as available and try to > deploy an "instance" on it (provision the host), it doesn't work > as dnsmasq reject the host boot DHCP requests. > > So, is there a way to instruct the ironic-conductor to > edit/allow the host the way the inspector is able to manipulate > this file? > > Did I missed?something? > > Thanks a lot! > From smooney at redhat.com Tue Jun 14 15:15:57 2022 From: smooney at redhat.com (Sean Mooney) Date: Tue, 14 Jun 2022 16:15:57 +0100 Subject: Devstack Installation Fails with Internal Server Error (HTTP 500) In-Reply-To: References: <559999b7-f47d-4c0b-ac48-65ff3d45437d@www.fastmail.com> <7592ce705d219818f7574c961f22b2acde834358.camel@redhat.com> Message-ID: <378ca4888789313a17e790c05369772728c8b207.camel@redhat.com> On Tue, 2022-06-14 at 06:37 -0700, Dan Smith wrote: > > i hit the same issue the other day if you recall me asking about it. > > didnt check my devstack logs to see if it installed or not but i can > > see if i can repoduce it tomorrow. > > I checked with the OP and they have it installed properly, > system-wide. I was assuming your issue might have been venv (or > ordering) related, but theirs was not. There are some other weird things > going on with their setup, so I'm not so sure. nope, was not useing a vnev or anything special, it just devstack installed normaly more or less but driven by the ansible roles. the run devstack rull litrally just opens a shell and invokes stack.sh so i dont think there was anything specific ot how i was invoking it. > > --Dan > From cboylan at sapwetik.org Tue Jun 14 17:13:57 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 14 Jun 2022 10:13:57 -0700 Subject: [TripleO] no more patches for tripleo stable/victoria please (going EOL) In-Reply-To: <20220614125620.nqngnpqchlrg5zgb@yuggoth.org> References: <20220614115942.wqiaqk7hpa2hzdlk@yuggoth.org> <20220614125620.nqngnpqchlrg5zgb@yuggoth.org> Message-ID: On Tue, Jun 14, 2022, at 5:56 AM, Jeremy Stanley wrote: > On 2022-06-14 21:22:37 +0900 (+0900), Takashi Kajinami wrote: > [...] >> I sometimes hope zuul can block backports depending on the "cherry >> picked from commit ..." line but it might be worth discussing as a >> potential enhancement with the zuul devs. > > Definitely worth further discussion, but I suspect the possibility > for cherry-picks from other repositories could make that challenging > to differentiate. I agree. I'm not sure this makes sense as a built in Zuul feature. However, you can always implement a backport sanity check job that fails if certain criteria are not met. > -- > Jeremy Stanley > > Attachments: > * signature.asc From adivya1.singh at gmail.com Tue Jun 14 18:18:02 2022 From: adivya1.singh at gmail.com (Adivya Singh) Date: Tue, 14 Jun 2022 23:48:02 +0530 Subject: Regarding Policy.json entries for glance image update not working for a user In-Reply-To: References: Message-ID: Hi Takashi, when a user upload images which is a member , The image will be set to private. This is what he is asking for access to make it public, The above rule applies for only public images regards Adivya Singh On Tue, Jun 14, 2022 at 10:54 AM Takashi Kajinami wrote: > Glance has a separate policy rule (publicize_image) for creating/updating > public images., > and you should define that policy rule instead of modify_image. > > https://docs.openstack.org/glance/xena/admin/policies.html > ~~~ > publicize_image - Create or update public images > ~~~ > > AFAIK The modify_image policy defaults to rule:default and is allowed for > any users > as long as the target image is owned by that user. > > > On Tue, Jun 14, 2022 at 2:01 PM Adivya Singh > wrote: > >> Hi Brian, >> >> Please find the response >> >> >>> 1> i am using Xena release version 24.0.1 >>> >>> Now the scenario is line below, my customer wants to have their login >>> access on setting up the properties of an image to the public. now what i >>> did is >>> >>> 1> i created a role in openstack using the admin credential name as >>> "user" >>> 2> i assigned that user to a role user. >>> 3> i assigned those user to those project id, which they want to access >>> as a user role >>> >>> Then i went to Glance container which is controller by lxc and made a >>> policy.yaml file as below >>> >>> root at aio1-glance-container-724aa778:/etc/glance# cat policy.yaml >>> >>> "modify_image": "role:user" >>> >>> then i went to utility container and try to set the properties of a >>> image using openstack command >>> >>> openstack image set --public >>> >>> and then i got this error >>> >>> HTTP 403 Forbidden: You are not authorized to complete publicize_image >>> action. >>> >>> Even when i am trying the upload image with this user , i get the above >>> error only >>> >>> export OS_ENDPOINT_TYPE=internalURL >>> export OS_INTERFACE=internalURL >>> export OS_USERNAME=adsingh >>> export OS_PASSWORD='adsingh' >>> export OS_PROJECT_NAME=adsingh >>> export OS_TENANT_NAME=adsingh >>> export OS_AUTH_TYPE=password >>> export OS_AUTH_URL=https://:5000/v3 >>> export OS_NO_CACHE=1 >>> export OS_USER_DOMAIN_NAME=Default >>> export OS_PROJECT_DOMAIN_NAME=Default >>> export OS_REGION_NAME=RegionOne >>> >>> Regards >>> Adivya Singh >>> >>> >>> >>> >>> On Mon, Jun 13, 2022 at 6:41 PM Alan Bishop wrote: >>> >>>> >>>> >>>> On Mon, Jun 13, 2022 at 6:00 AM Brian Rosmaita < >>>> rosmaita.fossdev at gmail.com> wrote: >>>> >>>>> On 6/13/22 8:29 AM, Adivya Singh wrote: >>>>> > hi Team, >>>>> > >>>>> > Any thoughts on this >>>>> >>>>> H Adivya, >>>>> >>>>> Please supply some more information, for example: >>>>> >>>>> - which openstack release you are using >>>>> - the full API request you are making to modify the image >>>>> - the full API response you receive >>>>> - whether the user with "role:user" is in the same project that owns >>>>> the >>>>> image >>>>> - debug level log extract for this call if you have it >>>>> - anything else that could be relevant, for example, have you modified >>>>> any other policies, and if so, what values are you using now? >>>>> >>>> >>>> Also bear in mind that the default policy_file name is "policy.yaml" >>>> (not .json). You either >>>> need to provide a policy.yaml file, or override the policy_file setting >>>> if you really want to >>>> use policy.json. >>>> >>>> Alan >>>> >>>> cheers, >>>>> brian >>>>> >>>>> > >>>>> > Regards >>>>> > Adivya Singh >>>>> > >>>>> > On Sat, Jun 11, 2022 at 12:40 AM Adivya Singh < >>>>> adivya1.singh at gmail.com >>>>> > > wrote: >>>>> > >>>>> > Hi Team, >>>>> > >>>>> > I have a use case where I have to give a user restriction on >>>>> > updating the image properties as a member. >>>>> > >>>>> > I have created a policy Json file and give the modify_image rule >>>>> to >>>>> > the particular role, but still it is not working >>>>> > >>>>> > "modify_image": "role:user", This role is created in OpenStack. >>>>> > >>>>> > but still it is failing while updating properties with a >>>>> > particular user assigned to a role as "access denied" and >>>>> > unauthorized access >>>>> > >>>>> > Regards >>>>> > Adivya Singh >>>>> > >>>>> >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From adivya1.singh at gmail.com Tue Jun 14 18:20:45 2022 From: adivya1.singh at gmail.com (Adivya Singh) Date: Tue, 14 Jun 2022 23:50:45 +0530 Subject: 504 Gateway time out error while loading OpenStack Projects In-Reply-To: References: Message-ID: hi Team, Any input on this Regards Adivya Singh On Mon, Jun 13, 2022 at 6:37 PM Adivya Singh wrote: > hi Team, > > We have Xena release installed on Ubuntu Openstack, Lately we are getting > "504 Gateway Timeout error ", As the we moved from 1 node to 3 node Open > Stack > i do not see any error in haproxy , neither there is a time out when i > ping external-VIP-IP configured. > > Apache 2 are also loaded, tried to delete the Horizon Container and > created it again, but no errors, Still the same errors. > > Any guesses on that > > Regards > Adivya Singh > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haiwu.us at gmail.com Tue Jun 14 18:28:05 2022 From: haiwu.us at gmail.com (hai wu) Date: Tue, 14 Jun 2022 13:28:05 -0500 Subject: [keystone] Getting a token from existing token Message-ID: Is it possible to get a new token from an existing token, and at the same time, to have the new token expiration time extended? (token renewal from existing token) It seems the token expiration time is not extended when trying the same curl command as shown in section "Getting a token from a token" in this doc: https://docs.openstack.org/keystone/train/api_curl_examples.html. Quote: curl -i \ -H "Content-Type: application/json" \ -d ' { "auth": { "identity": { "methods": ["token"], "token": { "id": "'$OS_TOKEN'" } } } }' \ "http://localhost:5000/v3/auth/tokens" ; echo From ken at jots.org Tue Jun 14 18:59:09 2022 From: ken at jots.org (Ken D'Ambrosio) Date: Tue, 14 Jun 2022 14:59:09 -0400 Subject: New install on CentOS: what versions, etc.? Message-ID: <7ec0e14593137adf8a91193628d4e301@jots.org> Hey, all. We're currently running (I hope you're sitting) Juno, and we're thinking about rolling out a new OpenStack version. My boss has asked me if CentOS 7.9 a) would be supported for control/compute nodes, and, if so, if there are any long-term potential issues, and b) if there would be any concerns about versions of client OSes. Thanks! -Ken From noonedeadpunk at gmail.com Tue Jun 14 19:08:41 2022 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Tue, 14 Jun 2022 21:08:41 +0200 Subject: New install on CentOS: what versions, etc.? In-Reply-To: <7ec0e14593137adf8a91193628d4e301@jots.org> References: <7ec0e14593137adf8a91193628d4e301@jots.org> Message-ID: Well, it depends of what new version you're thinking. As iirc last openstack version which would support CentOS 7 is Ussuri. After that you will have troubles with supported libvirt version and python bindings. As on CentsOS 7 you don't have python3 libvirt bindings, so likely you would need to run on python 2.7 which is already eol for couple of years. CentOS 8 Stream though will be working on Yoga but for Zed you will need CentOS 9 Stream. So all depends of what you consider new after Juno :D ??, 14 ???. 2022 ?., 21:02 Ken D'Ambrosio : > Hey, all. We're currently running (I hope you're sitting) Juno, and > we're thinking about rolling out a new OpenStack version. My boss has > asked me if CentOS 7.9 > a) would be supported for control/compute nodes, and, if so, if there > are any long-term potential issues, and > b) if there would be any concerns about versions of client OSes. > > Thanks! > > -Ken > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkopec at redhat.com Tue Jun 14 19:30:16 2022 From: mkopec at redhat.com (Martin Kopec) Date: Tue, 14 Jun 2022 21:30:16 +0200 Subject: [interop][refstack][summit] Interoperability testing feedback Message-ID: Hi everyone, we got great feedback during the summit, if you wanna check the topics we discussed, see this etherpad [1]. Feel also free to add additional concerns with the current interop processes. I was happy to see that many of you are thinking about interoperability and want to improve the testing in this regard. IWG has been seeking feedback from operators as well as users for some time now to make sure it helps with interoperability as much as possible. If you haven't already, please fill out this short (2-3 min) questionnaire [2]. Public Cloud SIG hosted an interesting session in terms of interoperability testing as well, see their etherpad [3]. I'm sure that you'll see further communication in that regard soon. I look forward to future collaboration between IWG and Public Cloud SIG. [1] https://etherpad.opendev.org/p/Berlin2022-Forum-InteropWG [2] https://forms.gle/YW95Hb2LoppaiLxv8 [3] https://etherpad.opendev.org/p/berlin-summit-future-public-cloud-sig Regards, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA IM: kopecmartin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkopec at redhat.com Tue Jun 14 20:10:06 2022 From: mkopec at redhat.com (Martin Kopec) Date: Tue, 14 Jun 2022 22:10:06 +0200 Subject: [qa][refstack][python-tempestconf] adding image In-Reply-To: References: Message-ID: Hi Mohammed, we are glad to hear that python-tempestconf is useful. We don't mind having python-tempestconf included in Docker images, although what exactly would we need to do? I'm afraid we can't commit to regular maintenance of the images. Cheers, On Mon, 13 Jun 2022 at 15:03, Mohammed Naser wrote: > Hi there, > > I'm wondering if the python-tempestconf team is open to publishing > Docker images that contain `python-tempestconf`. This is something > that we have a need for downstream but we'd be happy to contribute > upstream. > > The main thing is that we'd need to setup an organization on Docker > hub to publish things to. Is that something that the team is > interested in? We can contribute it if that is the case. > > Thanks, > Mohammed > > -- > Mohammed Naser > VEXXHOST, Inc. > > -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA IM: kopecmartin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Tue Jun 14 20:27:01 2022 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Tue, 14 Jun 2022 16:27:01 -0400 Subject: Regarding Policy.json entries for glance image update not working for a user In-Reply-To: References: Message-ID: On 6/14/22 2:18 PM, Adivya Singh wrote: > Hi Takashi, > > when a user upload images which is a member , The image?will be set to > private. > > This is what he is asking for access to make it public,? The above rule > applies for only public images Alan and Takashi have both given you good advice: - By default, Glance assumes that your custom policy file is named "policy.yaml". If it doesn't have that name, Glance will assume it does not exist and will use the defaults defined in code. You can change the filename glance will look for in your glance-api.conf -- look for [oslo_policy]/policy_file - We recommend that you use YAML instead of JSON to write your policy file because YAML allows comments, which you will find useful in documenting any changes you make to the file - You want to keep the permissions on modify_image at their default value, because otherwise users won't be able to do simple things like add image properties to their own images - Some image properties can affect the system or other users. Glance will not allow *any* user to modify some system properties (for example, 'id', 'status'), and it requires additional permission along with modify_image to set 'public' or 'community' for image visibility. - It's also possible to configure property protections to require additional permission to CRUD specific properties (the default setting is *not* to do this). For your particular use case, where you want a specific user to be able to publicize_image, I would encourage you to think more carefully about what exactly you want to accomplish. Traditionally, images with 'public' visibility are provided by the cloud operator, and this gives image consumers some confidence that there's nothing malicious on the image. Public images are accessible to all users, and they will show up in the default image-list call for all users, so if a public image contains something nasty, it can spread very quickly. Glance provides four levels of image visibility: - private: only visible to users in the project that owns the image - shared: visible to users in the project that owns the image *plus* any projects that are added to the image as "members". (A shared image with no members is effectively a private image.) See [0] for info about how image sharing is designed and what API calls are associated with it. There are a bunch of policies around this; the defaults are basically what you'd expect, with the image owner being able to add and delete members, and image members being able to 'accept' or 'reject' shared images. - community: accessible to everyone, but only visible if you look for them. See [1] for an explanation of what that means. The ability to set 'community' visibility on an image is controlled by the "communitize_image" policy (default is admin-or-owner). - public: accessible to everyone, and easily visible to all users. Controlled by the "publicize_image" policy (default is admin-only). You're running your own cloud, so you can configure things however you like, but I encourage you to think carefully before handing out publicize_image permission, and consider whether one of the other visibilities can accomplish what you want. For more info, the introductory section on "Images" in the api-ref [2] has a useful discussion of image properties and image visibility. The final thing I want to stress is that you should be sure to test carefully any policies you define in a custom policy file. You are actually having a good problem, that is, someone can't do something you would like them to. The way worse problem happens when in addition to that someone being able to do what you want them to, a whole bunch of other users can also do that same thing. OK, so to get to your particular issue: - you don't want to change the "modify_image" policy in the way you proposed in your email, because no one (other than the person having the 'user role) will be able to do any kind of image updates. - if you decide to give that user publicize_image permissions, be careful how you do it. For example, "publicize_image": "role:user" won't allow an admin to make images public (unless you also give each admin the 'user' role). If you look at most of the policies in the Xena policy.yaml.sample, they begin "role:admin or ...". - the reason you were seeing the 403 when you tried to do openstack image set --public as the user with the 'user' property is that you were allowed to modify_image but when you tried to change the visibility, you did not have permission (because the default for that is role:admin) Hope this helps! Once you get this figured out, you may want to put up a patch to update the Glance documentation around policies. I think everything said above is in there somewhere, but it may not be in the most obvious places. Actually, there is one more thing. The above all applies to Xena, but there's been some work around policies in Yoga and more happening in Zed, so be sure to read the Glance release notes when you eventually upgrade. [0] https://specs.openstack.org/openstack/glance-specs/specs/api/v2/sharing-image-api-v2.html [1] https://specs.openstack.org/openstack/glance-specs/specs/api/v2/sharing-image-api-v2.html#sharing-images-with-all-users [2] https://docs.openstack.org/api-ref/image/v2/index.html#images > > regards > Adivya Singh > > On Tue, Jun 14, 2022 at 10:54 AM Takashi Kajinami > wrote: > > Glance has a separate policy rule (publicize_image) for > creating/updating public images., > and you should define that policy rule instead of modify_image. > > https://docs.openstack.org/glance/xena/admin/policies.html > > ~~~ > |publicize_image| - Create or update public images > ~~~ > > AFAIK The modify_image policy defaults to rule:default and is > allowed for any users > as long as the target image is owned by that user. > > > On Tue, Jun 14, 2022 at 2:01 PM Adivya Singh > > wrote: > > ?Hi Brian, > > Please find the response > > 1> i am using Xena release version 24.0.1 > > Now the scenario?is line below, my customer wants to have > their login access on setting up the properties of an image > to the public. now what i did is > > 1> i created a role in openstack using the admin credential > name as "user" > 2> i assigned that user to a role user. > 3> i assigned those user to those project id, which they > want to access as a user role > > Then i went to Glance container which is controller by lxc > and made a policy.yaml file as below > > root at aio1-glance-container-724aa778:/etc/glance# cat policy.yaml > > ?"modify_image": "role:user" > > then i went to utility container and try to set the > properties of a image using openstack command > > openstack image set --public > > and then i got this error > > HTTP 403 Forbidden: You are not authorized to complete > publicize_image action. > > Even when i am trying the upload image with this user , i > get the above error only > > export OS_ENDPOINT_TYPE=internalURL > export OS_INTERFACE=internalURL > export OS_USERNAME=adsingh > export OS_PASSWORD='adsingh' > export OS_PROJECT_NAME=adsingh > export OS_TENANT_NAME=adsingh > export OS_AUTH_TYPE=password > export OS_AUTH_URL=https://:5000/v3 > export OS_NO_CACHE=1 > export OS_USER_DOMAIN_NAME=Default > export OS_PROJECT_DOMAIN_NAME=Default > export OS_REGION_NAME=RegionOne > > Regards > Adivya Singh > > > > On Mon, Jun 13, 2022 at 6:41 PM Alan Bishop > > wrote: > > > > On Mon, Jun 13, 2022 at 6:00 AM Brian Rosmaita > > wrote: > > On 6/13/22 8:29 AM, Adivya Singh wrote: > > hi Team, > > > > Any thoughts on this > > H Adivya, > > Please supply some more information, for example: > > - which openstack release you are using > - the full API request you are making to modify the > image > - the full API response you receive > - whether the user with "role:user" is in the same > project that owns the > image > - debug level log extract for this call if you have it > - anything else that could be relevant, for example, > have you modified > any other policies, and if so, what values are you > using now? > > > Also bear in mind that the default policy_file name is > "policy.yaml" (not .json). You either > need to provide a policy.yaml file, or override the > policy_file setting if you really want to > use policy.json. > > Alan > > cheers, > brian > > > > > Regards > > Adivya Singh > > > > On Sat, Jun 11, 2022 at 12:40 AM Adivya Singh > > > >> wrote: > > > >? ? ?Hi Team, > > > >? ? ?I have a use case where I have to give a user > restriction on > >? ? ?updating the image properties as a member. > > > >? ? ?I have created a policy Json file and give > the modify_image rule to > >? ? ?the particular role, but still it is not working > > > >? ? ?"modify_image": "role:user", This role is > created in OpenStack. > > > >? ? ?but still it is failing while updating > properties with a > >? ? ?particular?user assigned to a role as "access > denied" and > >? ? ?unauthorized access > > > >? ? ?Regards > >? ? ?Adivya Singh > > > > From anyrude10 at gmail.com Tue Jun 14 12:54:20 2022 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Tue, 14 Jun 2022 18:24:20 +0530 Subject: [Tripleo] - IPv6 Wallaby Undercloud Installation failure Message-ID: Hi Team, I am trying to deploy Openstack Wallaby Undercloud on IPv6, but facing the below error: 2022-06-14 05:01:23.213708 | 52540083-cfa2-3f20-e9dc-00000000286f | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 2022-06-14 05:03:22.912816 | 52540083-cfa2-3f20-e9dc-00000000286f | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_4 | undercloud | error={"changed": false, "msg": "Service ironic_pxe_tftp has not started yet"} 2022-06-14 05:03:22.914400 | 52540083-cfa2-3f20-e9dc-00000000286f | TIMING | tripleo_container_manage : Manage container systemd Sample Undercloud.conf is as follows: [DEFAULT] clean_nodes = true cleanup = false container_cli = podman container_healthcheck_disabled = true container_images_file = /home/stack/containers-prepare-parameter.yaml deployment_user = stack enable_ironic = true enable_ironic_inspector = true enable_neutron = true enable_routed_networks = false generate_service_certificate = false ipv6_address_mode = dhcpv6-stateful ipxe_enabled = true local_interface = enp8s0 local_ip = aaaa:aaaa:aaaa::1/64 subnets = ctlplane-subnet undercloud_admin_host = aaaa:aaaa:aaaa::1 undercloud_hostname = undercloud.com undercloud_ntp_servers = 30.30.30.3 undercloud_public_host = aaaa:aaaa:aaaa::1 undercloud_timezone = UTC [ctlplane-subnet] cidr = aaaa:aaaa:aaaa::/64 dhcp_end = aaaa:aaaa:aaaa::f dhcp_start = aaaa:aaaa:aaaa::a gateway = aaaa:aaaa:aaaa::1 inspection_iprange = aaaa:aaaa:aaaa::3,aaaa:aaaa:aaaa::9 Can someone please help in this regard. Anirudh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.aminian.server at gmail.com Tue Jun 14 17:27:21 2022 From: p.aminian.server at gmail.com (Parsa Aminian) Date: Tue, 14 Jun 2022 21:57:21 +0430 Subject: snapshot problem In-Reply-To: <20220614125330.Horde.u5KxkrplHvtLIS5bmxnzeIM@webmail.nde.ag> References: <20220612081038.Horde._xfxaaTmZ0fovOR01idXLyP@webmail.nde.ag> <20220612110208.Horde.CpZ6qNn0k0fnPwT4SwMk118@webmail.nde.ag> <20220612131436.Horde.n_8Sdwn8Jxu79Lbvu4WV3PH@webmail.nde.ag> <20220613073300.Horde.zitXQVnixZSoeq2NOodZs9l@webmail.nde.ag> <20220613170342.Horde.qRaR_2eecVXaGrvWHuz6tC7@webmail.nde.ag> <20220614125330.Horde.u5KxkrplHvtLIS5bmxnzeIM@webmail.nde.ag> Message-ID: sorry for my mistake yes you are right thanks for your help and Cooperation On Tue, Jun 14, 2022 at 5:28 PM Eugen Block wrote: > You don't need the admin keyring on a compute node but for tests it > can help. Basically it depends on your ceph auth config which keyring > you need on the openstack nodes. > If you already have ceph configured why don't you use it for all > services? Of course live snapshots are not possible if your glance > images are not within ceph (I don't think you mentioned that before). > > > Zitat von Parsa Aminian : > > > yes you are right , I added ceph.client.admin.keyring to nova compute > > container and ceph cluster error gone i also try rbd snap create from > nova > > compute container and its work without problem and vm is up . > > maybe the problem is related to glance .my glance backend is not on ceph > I > > think maybe its the reason > > > > On Mon, Jun 13, 2022 at 9:38 PM Eugen Block wrote: > > > >> Hi, > >> > >> I don't have a containerized deployment so I don't have anything to > >> compare, but my compute nodes have a ceph.conf and all required > >> keyrings in /etc/ceph/, maybe someone else can confirm that it's > >> missing and should be present in a kolla-ansible deployment. In the > >> meantime you could provide those files to the container and retry the > >> live snapshot. > >> > >> > >> Zitat von Parsa Aminian : > >> > >> > snapshot_image_format doesn't exist on compute or controller nova.conf > >> > For the manual snapshot on the previous reply I ran it directly from > >> ceph, > >> > but it's interesting from nova compute there is an error about > connecting > >> > to CEPH cluster !!! also ceph -s is showing similar error > >> > [root at R3SG4 ~]# docker exec -u root -it nova_compute rbd snap create > >> > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap2022-06-13 > >> > 13:42:18.061 7f03dac23200 -1 auth: unable to find a keyring on > >> > > >> > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > >> > (2) No such file or directory2022-06-13 13:42:18.061 7f03dac23200 -1 > >> > AuthRegistry(0x561d7014a790) no keyring found at > >> > > >> > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > >> > disabling ceph2022-06-13 13:42:18.065 7f03dac23200 -1 monclient: > >> > authenticate NOTE: no keyring found; disabled cephx authentication > >> > rbd: couldn't connect to the cluster! > >> > > >> > On Mon, Jun 13, 2022 at 12:15 PM Eugen Block wrote: > >> > > >> >> Could you share your whole nova.conf (only the uncommented options)? > >> >> Is this option set in your env? > >> >> > >> >> #snapshot_image_format = > >> >> > >> >> Also when you manually created the snapshot did you do it as the nova > >> >> user on the compute node? If not, could you retry? > >> >> > >> >> > >> >> Zitat von Parsa Aminian : > >> >> > >> >> > Hi > >> >> > kolla-ansible victoria version with ceph backend . > >> >> > rbd info output : > >> >> > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': > >> >> > size 20 GiB in 5120 objects > >> >> > order 22 (4 MiB objects) > >> >> > snapshot_count: 0 > >> >> > id: b69aaf907284da > >> >> > block_name_prefix: rbd_data.b69aaf907284da > >> >> > format: 2 > >> >> > features: layering, exclusive-lock, object-map, fast-diff, > >> >> > deep-flatten > >> >> > op_features: > >> >> > flags: > >> >> > create_timestamp: Fri May 20 00:04:47 2022 > >> >> > access_timestamp: Sun Jun 12 16:26:02 2022 > >> >> > --------------- > >> >> > also live snapshot seems to work correctly without any error or any > >> >> > downtime : > >> >> > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls > >> >> > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk > >> >> > SNAPID NAME SIZE PROTECTED TIMESTAMP > >> >> > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 > >> >> > > >> >> > also on compute nova.conf, images_type is set on rbd . > >> >> > > >> >> > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block wrote: > >> >> > > >> >> >> You should respond to the list so other users can try to support > you. > >> >> >> > >> >> >> So nova is trying to live snapshot the instance: > >> >> >> > >> >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> >> > 4dbffaa9c14e401c8c210e23ebe0ab7b > ef940663426b4c87a751afaf13b758e0 - > >> >> >> > default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] > >> >> >> > instance snapshotting > >> >> >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning > >> >> >> > live snapshot process > >> >> >> > >> >> >> But I don't see any 'rbd snap create' command. Either the rbd > image > >> >> >> doesn't support it or there is some setting to keep all rbd images > >> >> >> "flat". Can you check any relevant configs you might have in nova? > >> >> >> Also can you show the output of 'rbd info > >> >> >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test > if > >> >> >> the underlying rbd functions work as expected you could try to > create > >> >> >> a live snapshot manually: > >> >> >> > >> >> >> rbd snap create > >> >> /25c8d676-e20a-4238-a45c-d51daa62b941_disk at test-snap > >> >> >> > >> >> >> And paste any relevant output here. > >> >> >> > >> >> >> Zitat von Parsa Aminian : > >> >> >> > >> >> >> > Its not working for any instances and all of them are paused . I > >> >> enable > >> >> >> > debug logs please check the logs : > >> >> >> > > >> >> >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager > >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering > >> sync > >> >> for > >> >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> >> >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] > Lock > >> >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> >> >> > > >> >> >> > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> >> > :: waited 0.000s inner > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> >> >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker > >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > >> compute > >> >> >> host > >> >> >> > and has allocations in placement: {'resources': {'VCPU': 1, > >> >> 'MEMORY_MB': > >> >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> >> >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager > >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> _get_power_state > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager > >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> >> > 63426b4c87a751afaf13b758e0 - default default] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot > >> process > >> >> >> > default default] Lazy-loading 'pci_devices' on Instance uuid > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> >> >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance > >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading > >> >> >> > 'pci_devices' on Instance uuid > 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> >> > obj_load_attr > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 > >> >> >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting > event > >> >> >> > >> >> 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> >> > => Paused> emit_event > >> >> >> > > >> >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> >> >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] > [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle > Event) > >> >> >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager > >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> _get_power_state > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager > >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance > power > >> >> state > >> >> >> > after lifecycle event "Paused"; current vm_state: active, > current > >> >> >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> >> >> > power_state: 3 handle_lifecycle_event > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> >> >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager > >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state > the > >> >> >> instance > >> >> >> > has a pending task (image_pending_upload). Skip. > >> >> >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils > >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd > >> >> >> > (subprocess): qemu-img convert -t none -O raw -f raw > >> >> >> > > >> >> >> > >> >> > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad > >> >> >> > execute > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 > >> >> >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting > event > >> >> >> > >> >> 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> >> > => Stopped> emit_event > >> >> >> > > >> >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> >> >> > INFO nova.compute.manager [-] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle > Event) > >> >> >> > DEBUG nova.compute.manager > >> [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > >> >> - - > >> >> >> - > >> >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking > >> state > >> >> >> > _get_power_state > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> >> > DEBUG nova.compute.manager > >> [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa > >> >> - - > >> >> >> - > >> >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] > Synchronizing > >> >> >> > instance power state after lifecycle event "Stopped"; current > >> >> vm_state: > >> >> >> > active, current task_state: image_pending_upload, current DB > >> >> power_state: > >> >> >> > 1, VM power_state: 4 handle_lifecycle_event > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> >> >> > INFO nova.compute.manager > >> [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - > >> >> - > >> >> >> - - > >> >> >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During > >> >> >> sync_power_state > >> >> >> > the instance has a pending task (image_pending_upload). Skip. > >> >> >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager > >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering > >> sync > >> >> for > >> >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> >> >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] > Lock > >> >> >> > "25c8d676-e20a-4238 > >> >> >> > -a45c-d51daa62b941" acquired by > >> >> >> > > >> >> >> > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> >> > :: waited 0.000s inner > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> >> >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] > [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state > the > >> >> >> instance > >> >> >> > has a pending task (image_pending_upload). Skip. > >> >> >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] > Lock > >> >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by > >> >> >> > > >> >> >> > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> >> > :: held 0.001s inner > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 > >> >> >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils > >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD > "qemu-img > >> >> convert > >> >> >> > -t none -O raw -f raw > >> >> >> > > >> >> >> > >> >> > >> > rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" > >> >> >> > returned: 0 in 23.730s execute > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 > >> >> >> > default default] [instance: > 25c8d676-e20a-4238-a45c-d51daa62b941] > >> >> >> Snapshot > >> >> >> > extracted, beginning image upload > >> >> >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver > >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting > event > >> >> >> > >> >> 25c8d676-e20a-4238-a45c-d51daa62b941 > >> >> >> > => Started> emit_event > >> >> >> > > >> >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 > >> >> >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager > >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle > Event) > >> >> >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state > >> >> >> > _get_power_state > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance > power > >> >> state > >> >> >> > after lifecycle event "Started"; current vm_state: active, > current > >> >> >> > task_state: image_pending_upload, current DB power_state: 1, VM > >> >> >> > power_state: 1 handle_lifecycle_event > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 > >> >> >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager > >> >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle > Event > >> >> >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils > >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired > lock > >> >> >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing > network > >> >> info > >> >> >> > cache for instance _get_instance_nw_info > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 > >> >> >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron > >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating > instance_info_cache > >> >> with > >> >> >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > >> >> "address": > >> >> >> > "fa:16:3e:ca:00:d9", "network": {"id": > >> >> >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", > >> "label": > >> >> >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", > "dns": > >> >> >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": > {}}, > >> >> >> > {"address": "217.218.127.127", "type": "dns", "version": 4, > "meta": > >> >> {}}], > >> >> >> > "gateway": {"address": "141.11.42.1", "type": "gateway", > >> "version": 4, > >> >> >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": > "fixed", > >> >> >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], > >> >> "version": > >> >> >> 4, > >> >> >> > "meta": {}}], "meta": {"injected": true, "tenant_id": > >> >> >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, > >> "physical_network": > >> >> >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": > >> >> >> {"connectivity": > >> >> >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, > >> "datapath_type": > >> >> >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", > >> >> >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", > >> >> "qbh_params": > >> >> >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", > >> >> >> "profile": > >> >> >> > {}, "preserve_on_delete": false, "meta": {}}] > >> >> >> > update_instance_cache_with_nw_info > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 > >> >> >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on > >> this > >> >> >> > compute host and has allocations in placement: {'resources': > >> {'VCPU': > >> >> 1, > >> >> >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. > >> >> _remove_deleted_instances_allocations > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> >> >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager > >> >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 > >> >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to > >> snapshot > >> >> the > >> >> >> > instance on the hypervisor. > >> >> >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager > >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering > >> sync > >> >> for > >> >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 > >> >> >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by > >> >> >> > > >> >> >> > >> >> > >> > "nova.compute.manager.ComputeManager._sync_power_states.._sync..query_driver_power_state_and_sync" > >> >> >> > :: waited 0.000s inner > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 > >> >> >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker > >> >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this > >> compute > >> >> >> host > >> >> >> > and has allocations in placement: {'resources': {'VCPU': 1, > >> >> 'MEMORY_MB': > >> >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations > >> >> >> > > >> >> >> > >> >> > >> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 > >> >> >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager > >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> >> >> 93fb420b3c604d4fae408b81135b58e9 > >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: > >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting > >> >> >> > > >> >> >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block > wrote: > >> >> >> > > >> >> >> >> Have you tried with debug logs? Has it worked with live > snapshots > >> >> >> >> before for other instances or did it never work and all > snapshots > >> >> were > >> >> >> >> "cold"? > >> >> >> >> > >> >> >> >> Zitat von Parsa Aminian : > >> >> >> >> > >> >> >> >> > Hi > >> >> >> >> > kolla-ansible victoria version with ceph backend without > volume > >> >> >> >> > > >> >> >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block > >> >> wrote: > >> >> >> >> > > >> >> >> >> >> Hi, > >> >> >> >> >> > >> >> >> >> >> can you share more details about your environment? Which > >> openstack > >> >> >> >> >> version is it? What is the storage backend? In earlier > releases > >> >> there > >> >> >> >> >> was an option: > >> >> >> >> >> > >> >> >> >> >> #disable_libvirt_livesnapshot = false > >> >> >> >> >> > >> >> >> >> >> but this option has been deprecated. But if you're on an > older > >> >> >> version > >> >> >> >> >> that could explain it. > >> >> >> >> >> > >> >> >> >> >> Zitat von Parsa Aminian : > >> >> >> >> >> > >> >> >> >> >> > When I snapshot from the instance , server will gone away > and > >> >> its > >> >> >> not > >> >> >> >> >> > reachable until the snapshot is complete here is the logs > : > >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance > snapshotting > >> >> >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager > >> >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] > >> [instance: > >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle > >> >> Event) > >> >> >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager > >> >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] > >> [instance: > >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During > sync_power_state > >> >> the > >> >> >> >> >> instance > >> >> >> >> >> > has a pending task (image_pending_upload). Skip. > >> >> >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] > >> >> [instance: > >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped > (Lifecycle > >> >> Event) > >> >> >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver > >> >> >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 > >> >> >> >> >> 93fb420b3c604d4fae408b81135b58e9 > >> >> >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] > >> [instance: > >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, > >> >> beginning > >> >> >> >> image > >> >> >> >> >> > upload > >> >> >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager > >> >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] > >> [instance: > >> >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started > (Lifecycle > >> >> Event) > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> > >> > >> > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Tue Jun 14 22:47:59 2022 From: zigo at debian.org (Thomas Goirand) Date: Wed, 15 Jun 2022 00:47:59 +0200 Subject: Upgrading to a more recent version of jsonschema In-Reply-To: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> References: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> Message-ID: <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> On 6/13/22 00:10, Thomas Goirand wrote: > Hi, > > A few DDs are pushing me to upgrade jsonschema in Debian Unstable. > However, OpenStack global requirements are still stuck at 3.2.0. Is > there any reason for it, or should we attempt to upgrade to 4.6.0? > > I'd really appreciate if someone (else than me) was driving this... > > Cheers, > > Thomas Goirand (zigo) > FYI, Nova fails with it: https://ci.debian.net/data/autopkgtest/unstable/amd64/n/nova/22676760/log.gz Can someone from the Nova team investigate? Cheers, Thomas Goirand (zigo) From smooney at redhat.com Tue Jun 14 23:35:04 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 15 Jun 2022 00:35:04 +0100 Subject: Upgrading to a more recent version of jsonschema In-Reply-To: <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> References: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> Message-ID: On Wed, 2022-06-15 at 00:47 +0200, Thomas Goirand wrote: > On 6/13/22 00:10, Thomas Goirand wrote: > > Hi, > > > > A few DDs are pushing me to upgrade jsonschema in Debian Unstable. > > However, OpenStack global requirements are still stuck at 3.2.0. Is > > there any reason for it, or should we attempt to upgrade to 4.6.0? > > > > I'd really appreciate if someone (else than me) was driving this... > > > > Cheers, > > > > Thomas Goirand (zigo) > > > > FYI, Nova fails with it: > https://ci.debian.net/data/autopkgtest/unstable/amd64/n/nova/22676760/log.gz > > Can someone from the Nova team investigate? nova is not pinning this https://github.com/openstack/nova/blob/master/requirements.txt#L25= we test with whatever is allowed by uppper constriats so we need to look a this form a requirements team point of view anyjson seams to deplped on use2to3 which is breaking the script that updates requiremtns for me locally. if i remvoe that temporally and run .venv/bin/generate-constraints -b blacklist.txt -p python3.8 -r global-requirements.txt > upper-constraints.txt i get futher but then fail because i dont have all the bindep of every python lib in gloal-requiremetns install on my laptop ill try and check that again in a devstack vm which should have many of those deps installed. but something seams to be broken in the automation that updates the upper-constratis. the zuul periodic is at least sometime passing https://zuul.openstack.org/builds?job_name=propose-updates&project=openstack%2Frequirements&skip=0 but im not sure how many old deps we have and why the requiremnts repo is still pinning to 3.2.0 form november 2019 > > Cheers, > > Thomas Goirand (zigo) > From gmann at ghanshyammann.com Tue Jun 14 23:49:14 2022 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 14 Jun 2022 18:49:14 -0500 Subject: Upgrading to a more recent version of jsonschema In-Reply-To: <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> References: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> Message-ID: <181649f0df6.11d045b0f280764.1056849246214160471@ghanshyammann.com> ---- On Tue, 14 Jun 2022 17:47:59 -0500 Thomas Goirand wrote ---- > On 6/13/22 00:10, Thomas Goirand wrote: > > Hi, > > > > A few DDs are pushing me to upgrade jsonschema in Debian Unstable. > > However, OpenStack global requirements are still stuck at 3.2.0. Is > > there any reason for it, or should we attempt to upgrade to 4.6.0? > > > > I'd really appreciate if someone (else than me) was driving this... > > > > Cheers, > > > > Thomas Goirand (zigo) > > > > FYI, Nova fails with it: > https://ci.debian.net/data/autopkgtest/unstable/amd64/n/nova/22676760/log.gz > > Can someone from the Nova team investigate? Nova failures are due to the error message change (it happens regularly and they change them in most versions) in jsonschema new version. I remember we faced this type of issue previously also and we updated nova tests not to assert the error message but seems like there are a few left which is failing with jsonschema 4.6.0. Along with these error message failures fixes and using the latest jsonschema, in nova, we use Draft4Validator from jsonschema and the latest validator is Draft7Validator so we should check what are backward-incompatible changes in Draft7Validator and bump this too? -gmann > > Cheers, > > Thomas Goirand (zigo) > > From smooney at redhat.com Tue Jun 14 23:54:23 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 15 Jun 2022 00:54:23 +0100 Subject: New install on CentOS: what versions, etc.? In-Reply-To: <7ec0e14593137adf8a91193628d4e301@jots.org> References: <7ec0e14593137adf8a91193628d4e301@jots.org> Message-ID: <1057e63b912984f4c2b5f712cbe56106de86cd8a.camel@redhat.com> On Tue, 2022-06-14 at 14:59 -0400, Ken D'Ambrosio wrote: > Hey, all. We're currently running (I hope you're sitting) Juno, and > we're thinking about rolling out a new OpenStack version. My boss has > asked me if CentOS 7.9 > a) would be supported for control/compute nodes, and, if so, if there > are any long-term potential issues, and most openstack testing in the centos family has move to centos 9 stream i woudl not presonally recommend deploying a new install in 7.9 google tells me censot 7 is eol on June 30, 2024, realisticaly that give you only 2 years of "supprot" but centos 7 woudl be useing python 2.7 or 3.6.8 python 3.6 is eol upstream so it will not recive any security fixes for any of these issues https://repology.org/project/python/cves?version=3.6.8 so i would start with at least centos 8 stream or 9 stream if you want to move to anything modern if you do use 7.9 the last openstack version redhat maintained on rhel 7 was queens thats proably the best tested/maitained version as a reult but if its a new install i really dont think you shoudl consider anything older then wallaby personally if i was rooling my own deployment or useign say the opentstack ansible or simialr to deploy i would go to the most recent stable openstack version avaiabel that is supproted by my installer tool of chice. so yoga in this case. there are far to many issues that have been adressed in the interveing time for me to really recommend deploying old releases unless you are going to pay a vendor to supprot the openstack distorbution for you i which case i would recoment the newest release they offer. > b) if there would be any concerns about versions of client OSes. if you are on centos 7 that likely means that you will not be able to deploy an openstack release with supprot for vtpm that will mean you will not be able to run windows 11 or windows server 2022 guest as they requrie vtpm 2.0 support i also belive that rhel 9 is not supprot on 7.9 host form a readhat point of view https://access.redhat.com/articles/973133 https://access.redhat.com/articles/973163 so yes using centos 7.9 release will limit the guests you can run > > Thanks! > > -Ken > From smooney at redhat.com Tue Jun 14 23:58:05 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 15 Jun 2022 00:58:05 +0100 Subject: Upgrading to a more recent version of jsonschema In-Reply-To: <181649f0df6.11d045b0f280764.1056849246214160471@ghanshyammann.com> References: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> <181649f0df6.11d045b0f280764.1056849246214160471@ghanshyammann.com> Message-ID: <7fda4e895d6bb1d325c8b72522650c809bcc87f9.camel@redhat.com> On Tue, 2022-06-14 at 18:49 -0500, Ghanshyam Mann wrote: > ---- On Tue, 14 Jun 2022 17:47:59 -0500 Thomas Goirand wrote ---- > > On 6/13/22 00:10, Thomas Goirand wrote: > > > Hi, > > > > > > A few DDs are pushing me to upgrade jsonschema in Debian Unstable. > > > However, OpenStack global requirements are still stuck at 3.2.0. Is > > > there any reason for it, or should we attempt to upgrade to 4.6.0? > > > > > > I'd really appreciate if someone (else than me) was driving this... > > > > > > Cheers, > > > > > > Thomas Goirand (zigo) > > > > > > > FYI, Nova fails with it: > > https://ci.debian.net/data/autopkgtest/unstable/amd64/n/nova/22676760/log.gz > > > > Can someone from the Nova team investigate? > > Nova failures are due to the error message change (it happens regularly and they change them in most versions) > in jsonschema new version. I remember we faced this type of issue previously also and we updated > nova tests not to assert the error message but seems like there are a few left which is failing with > jsonschema 4.6.0. > > Along with these error message failures fixes and using the latest jsonschema, in nova, we use > Draft4Validator from jsonschema and the latest validator is Draft7Validator so we should check > what are backward-incompatible changes in Draft7Validator and bump this too? well i think the reason this is on 3.2.0 iis nothing to do with nova i bumpt it manually in https://review.opendev.org/c/openstack/requirements/+/845859/ The conflict is caused by: The user requested jsonschema===4.0.1 tripleo-common 16.4.0 depends on jsonschema>=3.2.0 rsd-lib 1.2.0 depends on jsonschema>=2.6.0 taskflow 4.7.0 depends on jsonschema>=3.2.0 zvmcloudconnector 1.4.1 depends on jsonschema>=2.3.0 os-net-config 15.2.0 depends on jsonschema>=3.2.0 task-core 0.2.1 depends on jsonschema>=3.2.0 python-zaqarclient 2.3.0 depends on jsonschema>=2.6.0 warlock 1.3.3 depends on jsonschema<4 and >=0.7 https://zuul.opendev.org/t/openstack/build/06ed295bb8244c16b48e2698c1049be9 it looks like warlock is clamping it ot less then 4 which is why we are stokc on 3.2.0 > > -gmann > > > > > Cheers, > > > > Thomas Goirand (zigo) > > > > > From smooney at redhat.com Wed Jun 15 00:01:47 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 15 Jun 2022 01:01:47 +0100 Subject: Upgrading to a more recent version of jsonschema In-Reply-To: <7fda4e895d6bb1d325c8b72522650c809bcc87f9.camel@redhat.com> References: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> <181649f0df6.11d045b0f280764.1056849246214160471@ghanshyammann.com> <7fda4e895d6bb1d325c8b72522650c809bcc87f9.camel@redhat.com> Message-ID: On Wed, 2022-06-15 at 00:58 +0100, Sean Mooney wrote: > On Tue, 2022-06-14 at 18:49 -0500, Ghanshyam Mann wrote: > > ---- On Tue, 14 Jun 2022 17:47:59 -0500 Thomas Goirand wrote ---- > > > On 6/13/22 00:10, Thomas Goirand wrote: > > > > Hi, > > > > > > > > A few DDs are pushing me to upgrade jsonschema in Debian Unstable. > > > > However, OpenStack global requirements are still stuck at 3.2.0. Is > > > > there any reason for it, or should we attempt to upgrade to 4.6.0? > > > > > > > > I'd really appreciate if someone (else than me) was driving this... > > > > > > > > Cheers, > > > > > > > > Thomas Goirand (zigo) > > > > > > > > > > FYI, Nova fails with it: > > > https://ci.debian.net/data/autopkgtest/unstable/amd64/n/nova/22676760/log.gz > > > > > > Can someone from the Nova team investigate? > > > > Nova failures are due to the error message change (it happens regularly and they change them in most versions) > > in jsonschema new version. I remember we faced this type of issue previously also and we updated > > nova tests not to assert the error message but seems like there are a few left which is failing with > > jsonschema 4.6.0. > > > > Along with these error message failures fixes and using the latest jsonschema, in nova, we use > > Draft4Validator from jsonschema and the latest validator is Draft7Validator so we should check > > what are backward-incompatible changes in Draft7Validator and bump this too? > > well i think the reason this is on 3.2.0 iis nothing to do with nova > i bumpt it manually in https://review.opendev.org/c/openstack/requirements/+/845859/ > > The conflict is caused by: > The user requested jsonschema===4.0.1 > tripleo-common 16.4.0 depends on jsonschema>=3.2.0 > rsd-lib 1.2.0 depends on jsonschema>=2.6.0 > taskflow 4.7.0 depends on jsonschema>=3.2.0 > zvmcloudconnector 1.4.1 depends on jsonschema>=2.3.0 > os-net-config 15.2.0 depends on jsonschema>=3.2.0 > task-core 0.2.1 depends on jsonschema>=3.2.0 > python-zaqarclient 2.3.0 depends on jsonschema>=2.6.0 > warlock 1.3.3 depends on jsonschema<4 and >=0.7 > > https://zuul.opendev.org/t/openstack/build/06ed295bb8244c16b48e2698c1049be9 > > it looks like warlock is clamping it ot less then 4 which is why we are stokc on 3.2.0 it look like there is already an issue open https://github.com/bcwaldon/warlock/issues/64 but they have not merged any comit sine 2020-02-09 so this looks like its at least parly unmaintaied > > > > > -gmann > > > > > > > > Cheers, > > > > > > Thomas Goirand (zigo) > > > > > > > > > From smooney at redhat.com Wed Jun 15 00:04:46 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 15 Jun 2022 01:04:46 +0100 Subject: Upgrading to a more recent version of jsonschema In-Reply-To: <7fda4e895d6bb1d325c8b72522650c809bcc87f9.camel@redhat.com> References: <74f5fdba-8225-5f6a-a6f6-68853875d4f8@debian.org> <3a6170d4-e1fb-2988-e980-e8c152cb852b@debian.org> <181649f0df6.11d045b0f280764.1056849246214160471@ghanshyammann.com> <7fda4e895d6bb1d325c8b72522650c809bcc87f9.camel@redhat.com> Message-ID: On Wed, 2022-06-15 at 00:58 +0100, Sean Mooney wrote: > On Tue, 2022-06-14 at 18:49 -0500, Ghanshyam Mann wrote: > > ---- On Tue, 14 Jun 2022 17:47:59 -0500 Thomas Goirand wrote ---- > > > On 6/13/22 00:10, Thomas Goirand wrote: > > > > Hi, > > > > > > > > A few DDs are pushing me to upgrade jsonschema in Debian Unstable. > > > > However, OpenStack global requirements are still stuck at 3.2.0. Is > > > > there any reason for it, or should we attempt to upgrade to 4.6.0? > > > > > > > > I'd really appreciate if someone (else than me) was driving this... > > > > > > > > Cheers, > > > > > > > > Thomas Goirand (zigo) > > > > > > > > > > FYI, Nova fails with it: > > > https://ci.debian.net/data/autopkgtest/unstable/amd64/n/nova/22676760/log.gz > > > > > > Can someone from the Nova team investigate? > > > > Nova failures are due to the error message change (it happens regularly and they change them in most versions) > > in jsonschema new version. I remember we faced this type of issue previously also and we updated > > nova tests not to assert the error message but seems like there are a few left which is failing with > > jsonschema 4.6.0. > > > > Along with these error message failures fixes and using the latest jsonschema, in nova, we use > > Draft4Validator from jsonschema and the latest validator is Draft7Validator so we should check > > what are backward-incompatible changes in Draft7Validator and bump this too? > > well i think the reason this is on 3.2.0 iis nothing to do with nova > i bumpt it manually in https://review.opendev.org/c/openstack/requirements/+/845859/ > > The conflict is caused by: > The user requested jsonschema===4.0.1 > tripleo-common 16.4.0 depends on jsonschema>=3.2.0 > rsd-lib 1.2.0 depends on jsonschema>=2.6.0 > taskflow 4.7.0 depends on jsonschema>=3.2.0 > zvmcloudconnector 1.4.1 depends on jsonschema>=2.3.0 > os-net-config 15.2.0 depends on jsonschema>=3.2.0 > task-core 0.2.1 depends on jsonschema>=3.2.0 > python-zaqarclient 2.3.0 depends on jsonschema>=2.6.0 > warlock 1.3.3 depends on jsonschema<4 and >=0.7 > > https://zuul.opendev.org/t/openstack/build/06ed295bb8244c16b48e2698c1049be9 > > it looks like warlock is clamping it ot less then 4 which is why we are stokc on 3.2.0 glance client seams to be the only real user of this https://codesearch.opendev.org/?q=warlock&i=nope&literal=nope&files=&excludeFiles=&repos= perhaps we could jsut remove the dependcy? > > > > > -gmann > > > > > > > > Cheers, > > > > > > Thomas Goirand (zigo) > > > > > > > > > From bshephar at redhat.com Wed Jun 15 05:22:12 2022 From: bshephar at redhat.com (Brendan Shephard) Date: Wed, 15 Jun 2022 15:22:12 +1000 Subject: Propose to add Takashi Kajinami as heat core reviewer In-Reply-To: References: Message-ID: +1 for sure. Takashi is an asset to all of OpenStack! I would happily endorse his inclusion as a Core reviewer to th