October 2019 - openstack-discuss - lists.openstack.org

[stable][oslo] Supporting qemu 4.1.0 on stein and older
by Ben Nemec 19 Oct '19

19 Oct '19

Hi, This is related to the FFE for train, but I wanted to discuss it separately because I think the circumstances are a bit different. Qemu 4.1.0 did not exist during the Stein cycle, so it's not clear to me that backporting bug fixes for it is valid. The original author of the patch actually wants it for Rocky, which is basically in the same situation as Stein. I should note he's willing to carry the patch downstream if necessary. On the one hand, it sounds like this is something at least one operator wants, but on the other I'm not sure the stable policy supports backporting patches to support a version of a dependency that didn't exist when the release was initially cut. I'm soliciting opinions on how to proceed here. Reference: https://review.opendev.org/#/c/686532 Thanks. -Ben

4 7

[openstack-operators] RBD problems after data center power outage
by Dinçer Çelik 18 Oct '19

18 Oct '19

Greetings, Today I had a data center power outage, and the OpenStack cluster went down. After taking the cluster up again, I cannot start some VMs due to error below. I've tried "rbd object-map rebuild" but it didn't work. What's the proper way to re-create the missing "_disk.config" files? Thanks. [instance: c2b54eac-179b-4907-9d61-8e075edc21cf] Failed to start libvirt guest: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-10-17T23:19:41.103720Z qemu-system-x86_64: -drive file=rbd:vms/c2b54eac-179b-4907-9d61-8e075edc21cf_disk.config:id=nova:auth_supported=cephx\;none:mon_host=10.250.129.10\:6789\;10.250.129.11\:6789\;10.250.129.12\:6789\;10.250.129.15\:6789,file.password-secret=ide0-0-0-secret0,format=raw,if=none,id=drive-ide0-0-0,readonly=on,cache=writeback,discard=unmap: error reading header from c2b54eac-179b-4907-9d61-8e075edc21cf_disk.config: No such file or directory /* Please encrypt every message you can. Privacy is your right, don't let anyone take it from you. */ /* My fingerprint is: 5E50 ABB0 F108 24DA 10CC BD43 D2AE DD2A 7893 0EAA */

3 5

[nova] Stance on trivial features for driver configs without integration testing
by Matt Riedemann 18 Oct '19

18 Oct '19

This was brought up in the nova meeting today [1] as: "Do we have a particular stance on features to the libvirt driver for non-integration tested configurations, e.g. lxc [2] and xen [3], meaning if they are trivial enough do we just say the driver's quality warning on startup is sufficient to let them land since these are changes from casual contributors scratching an itch?" We agreed to move this to the mailing list. We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter) and for the changes referenced they are from part-time contributors, minor and self-contained, and therefore I wouldn't expect them to build CI jobs for those configurations or stand up 3rd party CI. There are cases in the past where we've held features out due to lack of CI, e.g. live migration support in the vSphere driver. That's quite a bit different in my opinion because (1) it's a much more complicated feature, (2) there already was 3rd party CI for the vSphere driver and (3) there is a big rich corporation maintaining the driver so I figured they could pony up the resources to make that testing happen (and it eventually did). For these other small changes are we OK with letting them in knowing that the libvirt driver already logs a quality warning on startup for these configs [4]? In this case I am but wanted to ask and I don't think this sets a precedent as not all changes are equal. [1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log… [2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4] https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f6… -- Thanks, Matt

7 8

[metrics] [telemetry] [stein] cpu_util
by Budai Laszlo 18 Oct '19

18 Oct '19

Hello all, the release release announce of ceilometer rocky is deprecating the cpu_util and *.rate metrics "* cpu_util and *.rate meters are deprecated and will be removed in future release in favor of the Gnocchi rate calculation equivalent." so we don't have them in Stein. Can you direct me to some document that describes how to achieve these with Gnocchi rate calculation? Thank you, Laszlo

5 4

[all][ceilometer][aodh][[docs] Possible error in Stein's Aodh documentation and how to configure cpu_util and pass on value to Aodh
by Gauri Sindhu 18 Oct '19

18 Oct '19

Hi all, As per the Rocky release notes <https://docs.openstack.org/releasenotes/ceilometer/rocky.html>, *cpu_util and *.rate meters are deprecated and will be removed in future release in favor of the Gnocchi rate calculation equivalent.* I have two doubts regarding this. Firstly, if the 'cpu_util' metric has been deprecated then why has it been used as an example in the documentation <https://docs.openstack.org/aodh/stein/admin/telemetry-alarms.html> in the 'Using Alarms' section? I've attached an image of the same. Secondly, I'm using OpenStack Stein and want to use the cpu_util or its equivalent to create an alarm. If this metric is no longer available then what do I pass onto Aodh to create the alarm? There seems to be no documentation that to help me out with this. Additionally, even if Gnocchi rate calculation is to be used, how am I supposed to transfer the result to Aodh? I also cannot seem to find the Gnocchi documentation. Regards, Gauri Sindhu

2 4

[aodh] [heat] Stein: How to create alarms based on rate metrics like CPU utilization?
by Bernd Bausch 18 Oct '19

18 Oct '19

Prior to Stein, Ceilometer issued a metric named /cpu_util/, which I could use to trigger alarms and autoscaling when CPU utilization was too high. cpu_util doesn't exist anymore. Instead, we are asked to use Gnocchi's /rate/ feature. However, when using rates, alarms on a group of resources require more parameters than just one metric: Both an aggregation and a reaggregation method are needed. For example, a group of instances that implement "myapp": gnocchi measures aggregation -m cpu --reaggregation mean --aggregation rate:mean --query server_group=myapp --resource-type instance Actually, this command uses a deprecated API (but from what I can see, Aodh still uses it). The new way is like this: gnocchi aggregates --resource-type instance '(aggregate rate:mean (metric cpu mean))' server_group=myapp If rate:mean is in the archive policy, it also works the other way around: gnocchi aggregates --resource-type instance '(aggregate mean (metric cpu rate:mean))' server_group=myapp Without reaggregation, I get quite unexpected numbers, including negative CPU rates. If you want to understand why, see this discussion with one of the Gnocchi maintainers [1]. *My problem*: Aodh allows me to set an aggregation method, but not a reaggregation method. How can I create alarms based on rates? The problem extends to Heat and autoscaling. Thanks much, Bernd. [1] https://github.com/gnocchixyz/gnocchi/issues/1044

4 3

PostgreSQL driver has been removed from DevStack
by Stephen Finucane 18 Oct '19

18 Oct '19

Just a heads up that the PostgreSQL DB driver has been removed from DevStack as of today [1]. This was deprecated in Pike due to lack of maintenance and has been on the chopping block since then. If you have jobs that relied on this, it would be advisable to drop these jobs now. I don't think the DevStack team have the resources to maintain this so if someone *really* needed to keep it around, they should probably put together a plugin and consume it that way. Stephen [1] https://review.opendev.org/#/c/678496/

4 5

[tripleo] owls at ptg
by Wesley Hayutin 17 Oct '19

17 Oct '19

Greetings, A number of folks from TripleO will be at the OpenDev PTG. If you would like to discuss anything and collaborate please list your topic on this etherpad [1] Thank you! [1] https://etherpad.openstack.org/p/tripleo-ussuri-topics

3 2

[Neutron] Metering agent issues
by Eric K. Miller 17 Oct '19

17 Oct '19

Hi, We have a Stein deployment using Kolla Ansible 8.0.1 (the latest) and have enabled the deployment of the neutron metering agent. We have separate network nodes and compute nodes and use DVR. DVR is working properly, where ingress and egress traffic to floating IPs binded to fixed IPs flows directly to our upstream routers, not through the network nodes, whereas SNAT traffic flows through the network nodes. I configured the agent using the simplest of rules to test the agent (as cloud admin): openstack network meter create --description Shared --share bandwidth openstack network meter rule create --egress --remote-ip-prefix 0.0.0.0/0 bandwidth openstack network meter rule create --ingress --remote-ip-prefix 0.0.0.0/0 bandwidth None of the compute nodes appeared to be metering. So I checked the metering agent log (see below my signature for a snippet of the respective log entries), which shows that no routers are being found. Note that the Debug flag is set to True (in metering_agent.ini). On the primary network node, the "get_traffic_counters" method returns a list of all routers for every project, and runs a respective meter retrieval for each, such as the following, for each router: 2019-10-17 15:29:30.437 6 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-e0d2ee4f-76ec-4043-93f7-a47fae488d59', 'iptables', '-t', 'filter', '-L', 'neutron-meter-l-fb2c912d-9ce', '-n', '-v', '-x', '-w', '10', '-Z'] execute_rootwrap_daemon /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/util s.py:103 However, a single metering report is being created. Only one debug row appears in the log, and only one "bandwidth metric" measure is added to Gnocchi: 2019-10-17 15:38:15.494 6 DEBUG neutron.services.metering.agents.metering_agent [-] Send metering report: {'label_id': u'fb2c912d-9ced-433b-914e-df6526e034d1', 'time': 30, 'pkts': 0, 'tenant_id': u'96b2e5246a7744219e2af51f0e7b91d6', 'first_update': 1571300562, 'bytes': 0, 'host': 'networknode002', 'last_update': 1571344695} _metering_notification /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/services/meterin g/agents/metering_agent.py:103 No errors are being reported in the logs. Am I missing something? Thanks! Eric 2019-10-17 15:25:12.418 6 DEBUG oslo_concurrency.lockutils [-] Lock "metering-agent" acquired by "neutron.services.metering.agents.metering_agent._invoke_driver" :: waited 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockuti ls.py:327 2019-10-17 15:25:12.418 6 DEBUG neutron.services.metering.drivers.iptables.iptables_driver [-] neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeter ingDriver method get_traffic_counters called with arguments (<neutron_lib.context.ContextBase object at 0x7f661dfa9c90>, []) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66 2019-10-17 15:25:12.418 6 DEBUG oslo_concurrency.lockutils [-] Lock "metering-agent" released by "neutron.services.metering.agents.metering_agent._invoke_driver" :: held 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockuti ls.py:339 2019-10-17 15:25:27.421 6 DEBUG neutron.services.metering.agents.metering_agent [-] Get router traffic counters _get_traffic_counters /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/services/meterin g/agents/metering_agent.py:215 2019-10-17 15:25:27.421 6 DEBUG oslo_concurrency.lockutils [-] Lock "metering-agent" acquired by "neutron.services.metering.agents.metering_agent._invoke_driver" :: waited 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockuti ls.py:327 2019-10-17 15:25:27.421 6 DEBUG neutron.services.metering.drivers.iptables.iptables_driver [-] neutron.services.metering.drivers.iptables.iptables_driver.IptablesMeter ingDriver method get_traffic_counters called with arguments (<neutron_lib.context.ContextBase object at 0x7f661dfa9c90>, []) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66 2019-10-17 15:25:27.421 6 DEBUG oslo_concurrency.lockutils [-] Lock "metering-agent" released by "neutron.services.metering.agents.metering_agent._invoke_driver" :: held 0.000s inner /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockuti ls.py:339

1 0

[tc] Feedback on Airship pilot project
by Jeremy Stanley 17 Oct '19

17 Oct '19

Hi TC members, The Airship project will start its confirmation process with the OSF Board of Directors at the Board meeting[1] Tuesday next week. A draft of the slide deck[2] they plan to present is available for reference. Per the confirmation guidelines[3], the OSF Board of directors will take into account the feedback from representative bodies of existing confirmed Open Infrastructure Projects (OpenStack, Zuul and Kata) when evaluating Airship for confirmation. Particularly worth calling out, guideline #4 "Open collaboration" asserts the following: Project behaves as a good neighbor to other confirmed and pilot projects. If you (our community at large, not just TC members) have any observations/interactions with the Airship project which could serve as useful examples for how these projects do or do not meet this and other guidelines, please provide them on the etherpad[4] ASAP. If possible, include a citation with links to substantiate your feedback. If a TC representative can assemble this feedback and send it to the Board (for example, to the foundation mailing list) for consideration before the meeting next week, that would be appreciated. Apologies for the short notice. [1] http://lists.openstack.org/pipermail/foundation/2019-October/002800.html [2] https://www.airshipit.org/collateral/AirshipConfirmation-Review-for-the-OSF… [3] https://wiki.openstack.org/wiki/Governance/Foundation/OSFProjectConfirmatio… [4] https://etherpad.openstack.org/p/openstack-tc-airship-confirmation-feedback -- Jeremy Stanley

3 4