January 2021 - openstack-discuss - lists.openstack.org

[nova] [Ussuri] Unable to perform live block migrations on ephemeral VMs without shared storage
by Justin Goetz 09 Jan '21

09 Jan '21

Hello! Happy new year to everyone! I'm experiencing some trouble in my attempts to get live block migrations working on machines who's boot drives live on ephemeral (local) host storage. Offline (cold) migrations appear to work fine, however, I need live migrations to function with no downtime to the VMs. We're running on OpenStack Ussuri, with Ubuntu 18.04.5 LTS as our hypervisor OS, running libvirt 6.0.0. Below is the following command I'm utilizing to perform the live block migration: openstack server migrate <id> --debug --verbose --live-migration --block-migration The verbose logging of the above command shows no obvious issues, however digging deeper on the hypervisor side, we see some libvirtd errors, and the most telling is this segment here: Jan 08 16:53:02 h5 libvirtd[4118]: Unsafe migration: Migration without shared storage is unsafe Jan 08 16:53:02 h5 nova-compute[130270]: 2021-01-08 16:53:02.860 130270 ERROR nova.virt.libvirt.driver [-] [instance: 72bc9feb-b8f0-45f2-bad4-9cb61c3abff3] Live Migration failure: Unsafe migration: Migration without shared storage is unsafe: libvirt.libvirtError: Unsafe migration: Migration without shared storage is unsafe Full log output is available here: http://paste.openstack.org/show/801525/ Perhaps I'm severely mistaken, but native libvirt should have no problem whatsoever performing live migrations without shared storage on it's own. I've dug into the libvirt side for hours to try and troubleshoot what the system is seeing as "unsafe" (I confirmed disk caching is disabled, as well as no strange custom settings are present), but was unsuccessful. The full output of my VM's QEMU XML file can be viewed here: http://paste.openstack.org/show/801526/ As I couldn't figure out why libvirtd was not seeing the migration as "safe", I attempted to override the libvirt message by attempting to utilize the live_migration_flag and block_migration_flag options in nova.conf to invoke the --unsafe parameter when running the libvirt commands (with the flags VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_UNSAFE,VIR_MIGRATE_NON_SHARED_INC), however I later learned from a previous developers thread on the mailing list from 2016 that both of these options have been depreciated in the later versions of Nova and replaced by the "live_migration_tunnelled" setting in nova.conf. I've ensured my nova.conf files have the live_migration_tunnelled parameter set to "True". However, I have not found any method for passing along the "--unsafe" flag (or any other libvirt flag, in this case) to libvirt from nova.conf from this point on after the depreciation of the live_migration_flag parameters. Any help would be greatly appreciated, from reading around online this seems to be a fairly common setup, I'm baffled at how I've run into this issue across several separate OpenStack clusters at this point. Thank you for reading. -- Justin Goetz Systems Engineer, TeraSwitch Inc. jgoetz(a)teraswitch.com 412-945-7045 (NOC) | 412-459-7945 (Direct)

1 0

how to read ipa source code#ironic-python-agent
by Ankele zhang 09 Jan '21

09 Jan '21

Hi~ I have an OpenStack platform on Rocky, and I used Nova/Keystone/Cinder/Glance/Ceph/Ironic. Recently, I have build my own ironic-python-agent deployment ramdisk and kernel images by ironic-python-agent-builder, the following work is to add my customized raid configuration method to HardwareManager. Before this, I want to read the ipa's source code. But I don't know where is the 'main()' entry. Looking forward to your help. Ankele

3 2

Re: [Release-job-failures] Release of openstack/kolla for ref refs/tags/9.3.0 failed
by Herve Beraud 09 Jan '21

09 Jan '21

Hello, FYI it seems that kolla met docker's new limitation during its release jobs, especially with your publish jobs, I saw that you (the kolla team) already discussed this limitation [1] on the ML. ``` 2021-01-07 17:21:03.396355 | primary | ERROR:kolla.common.utils.base:Error'd with the following message 2021-01-07 17:21:03.396465 | primary | ERROR:kolla.common.utils.base:toomanyrequests: You have reached your pull rate limit. You may increase ``` Three jobs here failed for the same reason. I don't think that reenqueue the failing jobs without a specific action to manage this limitation will help us here. Let us know if we can help us in some manner. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019148… Le jeu. 7 janv. 2021 à 20:41, <zuul(a)openstack.org> a écrit : > Build failed. > > - openstack-upload-github-mirror > https://zuul.opendev.org/t/openstack/build/a64b4c4bc398481f9afffa2ad465a012 > : SUCCESS in 1m 03s > - release-openstack-python > https://zuul.opendev.org/t/openstack/build/2ed64dfcbcaf483abf003cfdf4f25837 > : SUCCESS in 3m 11s > - announce-release > https://zuul.opendev.org/t/openstack/build/84d803d496a2485c983f6233dafcfd71 > : SUCCESS in 4m 07s > - propose-update-constraints > https://zuul.opendev.org/t/openstack/build/d0cac6a077054bc8ba9eb92e56c21799 > : SUCCESS in 4m 15s > - kolla-publish-centos-source > https://zuul.opendev.org/t/openstack/build/5e8f6aa2f56940a7be77a8dfe1c8ecc6 > : SUCCESS in 2h 20m 47s > - kolla-publish-centos-binary > https://zuul.opendev.org/t/openstack/build/8a8a021d9f9c4ca79755b06309710cc7 > : SUCCESS in 1h 56m 32s (non-voting) > - kolla-publish-centos8-source > https://zuul.opendev.org/t/openstack/build/fb6891d3f5e4493b880fce263a92e086 > : SUCCESS in 1h 50m 57s > - kolla-publish-centos8-binary > https://zuul.opendev.org/t/openstack/build/c312c05e5d084fdbb3f372755221f186 > : SUCCESS in 1h 13m 12s (non-voting) > - kolla-publish-debian-source > https://zuul.opendev.org/t/openstack/build/e24c12751b8c4aba881adb6c9ae8dc07 > : SUCCESS in 1h 27m 27s (non-voting) > - kolla-publish-debian-source-aarch64 > https://zuul.opendev.org/t/openstack/build/1ff3b02df53847d0aa54bf12ea7fa666 > : FAILURE in 1h 51m 49s (non-voting) > - kolla-publish-debian-binary > https://zuul.opendev.org/t/openstack/build/012c2de475fe45ea83f1cd8a7420aa6d > : SUCCESS in 1h 15m 15s (non-voting) > - kolla-publish-ubuntu-source > https://zuul.opendev.org/t/openstack/build/88ca21d972514cce954ecb586324fa29 > : FAILURE in 4m 14s > - kolla-publish-ubuntu-binary > https://zuul.opendev.org/t/openstack/build/f3d44e4d1b6c4e9799161b156290b238 > : FAILURE in 4m 19s (non-voting) > > _______________________________________________ > Release-job-failures mailing list > Release-job-failures(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures > -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

3 3

[ironic] Resuming meetings next week
by Julia Kreger 09 Jan '21

09 Jan '21

Greetings everyone, Now that the Holidays are over, it is time for us to resume meeting on a regular basis. Ironic will resume it's weekly meeting next Monday at 1500 UTC in #openstack-ironic on irc.freenode.net. The agenda can be found on the OpenStack wiki[0]. See you all there! -Julia [0]: https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting

1 0

[kolla] [train] Changing default quotas
by Braden, Albert 08 Jan '21

08 Jan '21

I'm trying to change default quotas on a Train kolla POC, and I can change compute, but when I try to change a network quota such as subnets I get an error: (openstack) [root@adjutant-poc openstack]# openstack quota set --class --subnets 2 default Network quotas are ignored since quota class is not supported. When I google around I find some old documents talking about setting neutron quotas in /etc/neutron/neutron.conf but I can't find anything recent, and I don't see any mention of quotas when I look at /etc/neutron/neutron.conf in our neutron_server containers. What is the best way to change neutron default quotas in kolla? I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

[cinder][tripleo][release] Victoria Cycle-Trailing Release Deadline
by Herve Beraud 08 Jan '21

08 Jan '21

Hello teams with trailing projects, Next week is the Victoria cycle-trailing release deadline [1], and all projects following the cycle-trailing release model must release their Victoria deliverables by 14 January, 2021. The following trailing projects haven't been yet released for Victoria. Cinder: - cinderlib Tripleo: - os-collect-config - os-refresh-config - tripleo-ipsec This is just a friendly reminder to allow you to release these projects in time. Do not hesitate to ping us if you have any questions or concerns. [1] https://releases.openstack.org/wallaby/schedule.html#w-cycle-trail Hervé Beraud (hberaud) and the Release Management Team -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 3

[cinder] review priorities for the next 2 weeks
by Brian Rosmaita 08 Jan '21

08 Jan '21

As discussed at yesterday's cinder weekly meeting, the new driver merge deadline will soon be upon us at the Wallaby-2 milestone in two weeks [0]. Thus, all members of the cinder community should make driver reviews the top priority over the next 2 weeks. There are four drivers proposed for Wallaby that haven't merged yet: 1 - Ceph ISCSI driver https://review.opendev.org/c/openstack/cinder/+/662829 2 - Dell/EMC PowerVault ME Series driver: https://review.opendev.org/c/openstack/cinder/+/758684/ 3 - TOYOU ACS5000 driver: https://review.opendev.org/c/openstack/cinder/+/767290/ 4 - Kioxia KumoScale NVMeoF volume driver: https://review.opendev.org/c/openstack/cinder/+/768574 nvmeof connector mdraid support: https://review.opendev.org/c/openstack/os-brick/+/768575 healing agent: https://review.opendev.org/c/openstack/os-brick/+/768576 Don't forget that the cinder docs contain a helpful checklist for reviewing drivers: https://docs.openstack.org/cinder/latest/contributor/new_driver_checklist.h… If you are waiting for reviews of your driver, it will be helpful for you to review other drivers. You may notice something that you missed or could code differently in your driver, or you may be able to suggest changes based on issues you've come across implementing your driver. Happy reviewing! brian [0] https://releases.openstack.org/wallaby/schedule.html#w-cinder-driver-deadli…

1 0

[cinder] Cinder & Ceph Integration Error: No Valid Backend
by SSelf＠performair.com 08 Jan '21

08 Jan '21

All; We're having problems with our Openstack/Ceph integration. The versions we're using are Ussuri & Nautilus. When trying to create a volume, the volume is created, though the status is stuck at 'ERROR'. This appears to be the most relevant line from the Cinder scheduler.log: 2021-01-07 14:00:38.473 140686 ERROR cinder.scheduler.flows.create_volume [req-f86556b5-cb2e-4b2d-b556-ed07e632289d 824c26c133b34d8b8e84a7acabbe6f91 a983323b5ffc47e18660794cd9344869 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available Here is the 'cinder.conf' from our Controller Node: [DEFAULT] # define own IP address my_ip = 10.0.80.40 log_dir = /var/log/cinder state_path = /var/lib/cinder auth_strategy = keystone enabled_backends = ceph glance_api_version = 2 debug = true # RabbitMQ connection info transport_url = rabbit://openstack:<password>@10.0.80.40:5672 enable_v3_api = True # MariaDB connection info [database] connection = mysql+pymysql://cinder:<password>@10.0.80.40/cinder # Keystone auth info [keystone_authtoken] www_authenticate_uri = http://10.0.80.40:5000 auth_url = http://10.0.80.40:5000 memcached_servers = 10.0.80.40:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = cinder password = <password> [oslo_concurrency] lock_path = $state_path/tmp [ceph] volume_driver = cinder.volume.drivers.rbd.RBDDriver volume_backend_name = ceph rbd_pool = rbd_os_volumes rbd_ceph_conf = /etc/ceph/463/ceph.conf rbd_flatten_volume_from_snapshot = false rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1 rbd_user = cinder rbd_exclusive_cinder_pool = true backup_driver = cinder.backup.drivers.ceph backup_ceph_conf = /etc/ceph/300/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = rbd_os_backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true Does anyone have any ideas as to what is going wrong? Thank you, Stephen Self IT Manager Perform Air International sself(a)performair.com www.performair.com

1 1

OpenStack Ansible - Telemetry
by Golasowski Martin 08 Jan '21

08 Jan '21

Dear All, I would like to know which monitoring solution is currently supported by OSA? We are operating a small cloud (~ 6 nodes) and we are interested in collecting performance metrics, events and logs. So, as far as I know, the official OSA solution is ceilometer/aodh/panko with Gnocchi as DB backend. However Gnocchi project seems abandoned at the moment and the grafana plugin is not compatible with latest Grafana. Then there is solution based on collectd with this plugin (https://github.com/signalfx/collectd-openstack <https://github.com/signalfx/collectd-openstack>) with Graphite or InfluxDB as backend. This supports only performance metrics and not the events. Then there are also some Prometheus exporters available, again, metrics only. What do you guys use these days? What would you recommend? Thanks! Best regards, Martin

2 2

[ironic] SPUC in 2021
by Dmitry Tantsur 08 Jan '21

08 Jan '21

Hi folks! I feel like SPUC (our Sanity Preserving Un-Conference) has been very successful in 2020, what do you think about continuing it as before: Friday, 10am UTC (in https://bluejeans.com/643711802) Friday, 5pm UTC (in https://bluejeans.com/313987753) ? Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

2 2