- openstack-discuss - lists.openstack.org

[nova] Clean up "building" instances
by Eugen Block 22 Feb '23

22 Feb '23

Hi, we had a network issue two weeks ago in a HA Victoria cloud resulting in a couple of stale resources (in pending state). Most of them I could cleanup relatively easy, but two instances are left in "building" state, but not yet in the nova database so I can't just remove them via 'openstack server delete <UUID>'. I've been looking through the various nova databases where traces have been left to get an impression where I could intervene (although I don't like manipulating the database). The VMs are two amphora instances: control01:~ # openstack server list --project service | grep -v ACTIVE +--------------------------------------+----------------------------------------------+--------+-------------------------------------------------------------------------------------+---------------------------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------------------------------------------+--------+-------------------------------------------------------------------------------------+---------------------------+---------+ | 0453a7e5-e4f9-419b-ad71-d837a20ef6bb | amphora-0ee32901-0c59-4752-8253-35b66da176ea | BUILD | | amphora-x64-haproxy_1.0.0 | amphora | | dc8cdc3a-f6b2-469b-af6f-ba2aa130ea9b | amphora-4990a47b-fe8a-431a-90ec-5ac2368a5251 | BUILD | | amphora-x64-haproxy_1.0.0 | amphora |+--------------------------------------+----------------------------------------------+--------+-------------------------------------------------------------------------------------+---------------------------+---------+ The database tables referring to the UUID 0453a7e5-e4f9-419b-ad71-d837a20ef6bb are these: nova_cell0/instance_id_mappings.ibd nova_cell0/instance_info_caches.ibd nova_cell0/instance_extra.ibd nova_cell0/instances.ibd nova_cell0/instance_system_metadata.ibd octavia/amphora.ibd nova_api/instance_mappings.ibd nova_api/request_specs.ibd My first approach would be to update the nova_cell0.instances table and edit the fields 'vm_state' and 'task_state', or even remove the intire row. But I don't know about the implications this would have on the other tables, so I'd like to know how you would recommend to deal with these orphans. Any comment is appreciated! Thanks, Eugen

3 7

[all][tc] Technical Committee next weekly meeting on 2023 Feb 22 at 1600 UTC
by Ghanshyam Mann 22 Feb '23

22 Feb '23

Hello Everyone, The technical Committee's next weekly meeting is scheduled for 2023 Feb 22, at 1600 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Tuesday, Feb 21 at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[openstack-dev][CLI] How to set NoneType in openstack CLI's json format string?
by Sam 22 Feb '23

22 Feb '23

Hi all, I'm using openstack command like this, and got error: ``` gyw@c1:~$ openstack port create --network selfservice --vnic-type remote-managed \ > --binding-profile '{"pci_vendor_info": None, "pci_slot": None, "physical_network": None, "card_serial_number": "AB0123XX0042", "pf_mac_address": "08:c0:eb:8e:bd:f8", "vf_num":1, "vnic_type": "remote-managed"}' \ > pf0vf0 Expected '<key>=<value>' or JSON data for option --binding-profile, but encountered JSON parsing error: Expecting value: line 1 column 21 (char 20) ``` So how to set "pci_vendor_info" to NoneType of python in openstack CLI's json format string? BTW, "NoneType of python" means the NoneType in python code, like these comment in https://github.com/openstack/neutron/blob/stable/yoga/neutron/common/ovn/ut… ``` # With this example param_set: # # param_set = { # 'do_not_check_this_key': None, # 'pci_slot': [str], # 'physical_network': [str, type(None)] # } # # We confirm that each binding_profile key is of one of the listed types, # allowing validation of polymorphic entries. # # 'physical_network' is polymorphic because: When a VNIC_REMOTE_MANAGED or # VNIC_DIRECT with PORT_CAP_SWITCHDEV capability port is attached to a # project network backed by an overlay (tunneled) network the value will be # 'None'. For the case of ports attached to a project network backed by # VLAN the value will be of type ``str``. This comes from Nova and is # provided in the ``physical_network`` tag in the Nova PCI Passthrough # configuration. # # In the above example the type of the value behind 'do_not_check_this_key' # will not be checked, 'pci_slot' must be ``str``, 'physical_network must # be either ``str`` or ``NoneType``. ``` Thank you~ -- Best Regards， ------------------------------------------------ Sam Green Home Page: http://blog.csdn.net/batmancn Email:batmanzg@163.com

1 1

[PTL][release] 2023.1 Antelope Cycle Highlights
by Elõd Illés 21 Feb '23

21 Feb '23

Hi, This is a reminder, that *this week* is the week for Cycle highlights [1][2]! They need to be added to deliverable yamls so that they can be included in release marketing preparations. (See the details about how to add them at the project team guide [3]) Note that the deadline was the Feature Freeze date in previous cycles, but due to requests of PTLs it was decided to postpone one week in the future (to R-4 week). This is better for PTLs as they are usually very busy around Feature Freeze, on the other hand gives less time to marketing team to process the highlights. [1] https://releases.openstack.org/antelope/schedule.html [2] https://releases.openstack.org/antelope/schedule.html#a-cycle-highlights [3] https://docs.openstack.org/project-team-guide/release-management.html#cycle… Thanks, Előd Illés irc: elodilles @ #openstack-release

2 1

[requirements][oslo] RFE for oslo* deliverables
by Herve Beraud 21 Feb '23

21 Feb '23

1 0

Volume can not be detached/deleted properly & Available volume can not be attached with new instance
by sudeb ghosh 21 Feb '23

21 Feb '23

Hi Team, we are using opensource Openstack (Openstack foundation) in our environment. Facing following issues to fix. Need some guidance or suggestion to resolve the same. Problem Detail: 1. We are struggling to delete the volume which are in 'available' state. After deleting from Dashboard/CLI its showing 'deleting' only. 2. Can not detach the volume from any instance. Showing 'detaching' itself. 3. Once status of volume made available by updating table of Database, the volumes are showing 'available' but could not attach further with any existing instance. 4. Once status of volume made deleted by updating table of Database or by deleting the entries of table the volumes are showing status as 'deleted' in CLI(mysql query) & disappeared from Dashboard but reclaiming of space is not happening in storage or quota point of view. Request your guidance with steps to fix the issue pl. BR// Sudeb Ghosh 7044064878 9332034788

2 1

[neutron] Etherpad for Bobcat release, now available
by Rodolfo Alonso Hernandez 21 Feb '23

21 Feb '23

Hello Neutrinos: Please check the link [1]. This is the etherpad for the Bobcat release PTG [2]. If you have any request, RFE, bug, question or topic you want to discuss during the virtual PTG, don't hesitate to add it in the "Topics" section. Once we have collected all of them, we'll create the agenda, adding the corresponding links to the virtual meeting rooms. If you have any question, please contact me (ralonsoh) in the Neutron channel (#openstack-neutron) or via mail. You can also talk to any Neutron core reviewer present in this channel. Regards. [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg [2]https://openinfra.dev/ptg/

1 0

[ironic][dib] Ubuntu ramdisk images victoria ironic-python-agent run error
by 韩光宇 21 Feb '23

21 Feb '23

Hi, I made a victoria ramdisk image of ubuntu using the following command, ```shell export DIB_DEV_USER_USERNAME=ipa export DIB_DEV_USER_PWDLESS_SUDO=yes export DIB_DEV_USER_PASSWORD='123' export DIB_REPOLOCATION_ironic_python_agent=/root/ironic-python-agent export DIB_REPOREF_ironic_python_agent=victoria-self export DIB_PYPI_MIRROR_URL=https://pypi.tuna.tsinghua.edu.cn/simple export DIB_NO_PYPI_PIP=1 ironic-python-agent-builder -o my-ssh-ubuntu-ipa-victoria -e selinux-permissive -e devuser ubuntu -e pypi -e pip-cache -e dynamic-login ``` But, in baremtal node clean step, I got the error log from ironic-python-agent （The error.png[1] is a screenshot of the log） ``` Started Ironic Python Agent. ... ImportError: canot import name 'json' from 'werkzeug.wrappers' ``` Is the werkzeug version installed by default too high and incompatible? If this is the reason, can I manually specify a lower version? I have tried to specify Werkzeug in the requirements.txt in the local ironic_python_agent git warehouse as a lower version, but the following error is reported when making image ``` 2023-02-15 13:05:02.689 | ERROR: Cannot install ironic-python-agent==6.4.5.dev3 because these package versions have conflicting dependencies. 2023-02-15 13:05:02.690 | 2023-02-15 13:05:02.690 | The conflict is caused by: 2023-02-15 13:05:02.690 | ironic-python-agent 6.4.5.dev3 depends on Werkzeug==1.0.1 2023-02-15 13:05:02.690 | The user requested (constraint) werkzeug===2.2.2 2023-02-15 13:05:02.690 | 2023-02-15 13:05:02.690 | To fix this you could try to: 2023-02-15 13:05:02.691 | 1. loosen the range of package versions you've specified 2023-02-15 13:05:02.691 | 2. remove package versions to allow pip attempt to solve the dependency conflict 2023-02-15 13:05:02.691 | 2023-02-15 13:05:02.691 | ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies ``` I would appreciate any kind of guidance or help. Thank you so so much. [1]: https://photos.app.goo.gl/pimvYyzh3iRXxt5B9

1 0

PTG March 2023 Teams List
by Kendall Nelson 21 Feb '23

21 Feb '23

Hello Everyone! The March 2023 Project Teams List is official! *Projects + Teams* - OpenInfra Foundation Teams - Diversity and Inclusion WG - Edge Computing Group - Environmental Sustainability Working Group - Interop WG - StarlingX - OpenStack Teams - Service Teams - Barbican - Blazar - Cinder - CloudKitty - Designate - Glance - Horizon - Ironic - Keystone - Kolla - Kuryr - Magnum - Manila - Neutron - Nova - Octavia - OpenaStackSDK/CLI/Ansible-Collections - OpenStack-Ansible - OpenStackCharms - QA - Tacker - Other Teams - Scientific SIG - PublicCloud SIG OpenStack - Security SIG - OpenStack Technical Committee - RBAC - I18n SIG If your team was planning to meet and isn’t on this list, please contact ptg(a)openinfra.dev IMMEDIATELY. Soon I will be contacting moderators to sign your team up for time via the PTGBot[1] once I have it configured for the teams that are signed up to attend the event. Otherwise, please don't forget to register[2] — *its FREE!* As usual, feel free to let us know if you have any questions! Thanks! -Kendall (diablo_rojo) [1] PTGBot Docs: https://github.com/openstack/ptgbot#open-infrastructure-ptg-bot [2] PTG Registration: https://openinfra-ptg.eventbrite.com

1 0

[kolla] [train] [designate] Terraform "repave" causes DNS records to become orphaned
by Albert Braden 21 Feb '23

21 Feb '23

We have customers who use Terraform to build their clusters. They do a thing that they call “repave” where they run an ansible playbook that calls “terraform destroy” and then immediately calls “terraform apply” to rebuild the cluster. It looks like Designate is not able to keep up, and it fails too delete one or more of the DNS records. We have 3 records, IPv4 forward (A) and reverse (PTR) and IPv6 forward (AAAA). When Designate fails to delete a record, it becomes orphaned. On the next “repave” the record is not deleted, because it’s not associated with the new VM, and we see errors in designate-sink.log: 2023-02-13 02:49:40.824 27 ERROR oslo_messaging.notify.dispatcher [parameters: {'id': '1282a6780f2f493c81ed20bc62ef370f', 'version': 1, 'created_at': datetime.datetime(2023, 2, 13, 2, 49, 40, 814726), 'zone_shard': 97, 'tenant_id': '130b797392d24b408e73c2be545d0a20', 'zone_id': '0616b8e0852540e59fd383cfb678af32', 'recordset_id': '1fc5a9eaea824d0f8b53eb91ea9ff6e2', 'data': '10.22.0.210', 'hash': 'e3270256501fceb97a14d4133d394880', 'managed': 1, 'managed_plugin_type': 'handler', 'managed_plugin_name': 'our_nova_fixed', 'managed_resource_type': 'instance', 'managed_resource_id': '842833cb9410404bbd5009eb6e0bf90a', 'status': 'PENDING', 'action': 'UPDATE', 'serial': 1676256582}] … 2023-02-13 02:49:40.824 27 ERROR oslo_messaging.notify.dispatcher designate.exceptions.DuplicateRecord: Duplicate Record The orphaned record is causing a mariadb collision because a record with that name and IP already exists. When this happens with an IPv6 record, it looks like Designate tries to create the IPv6 record, and fails, and then does not try to create an IPv4 record, which causes trouble because Terraform waits for the name resolution to work. The obvious solution is to tell TF users to introduce a delay between “destroy” and “apply” but that would be non-trivial for them, and we would prefer to fix it on our end. What can I do, to make Designate gracefully manage cases where a cluster is deleted and then immediately rebuilt with the same names and IPs? Also, how can I clean up these orphaned records. I’ve been asking the customer to destroy, and then deleting the record, and then asking them to rebuild, but that is a manual process for them. Is it possible to link the orphaned record to the new VM so that it will be deleted on the next “repave?” Example: This VM was built today: $ os server show f5e75688-5fa9-41b6-876f-289e0ebc04b9|grep launched_at | OS-SRV-USG:launched_at | 2023-02-16T02:48:49.000000 The A record was created in January: $ os recordset show 0616b8e0852540e59fd383cfb678af32 1fc5a9ea-ea82-4d0f-8b53-eb91ea9ff6e2|grep created_at | created_at | 2023-01-25T02:48:52.000000 |

3 5