April 2019 - openstack-discuss - lists.openstack.org

[FYI][all] PySnooper, a poor man's debugger
by Artom Lifshitz 23 Apr '19

23 Apr '19

tl;dr Instead of debugging with LOG.debug and/or print() statements all over the place, use the pysnooper decorator [1]. I came across this on Hacker News this morning - if you're like me and are too lazy to invest in learning a real debugger, and instead do it with LOG.debug/print()s all over the place, pysnooper might be useful to you. This line from the README feels particularly relevant to me ;) > What makes PySnooper stand out from all other code intelligence tools? > You can use it in your sh***y, sprawling enterprise codebase without having > to do any setup. I tried it out with a random Nova unit test by decorating the function being tested, and it gave me the expected analysis of how the function executed. Just sharing this, I hope it might be useful to someone :) [1] https://github.com/cool-RR/pysnooper

5 4

[api-sig][neutron] Question on oslo policy assertion when supplied attribute equals the default
by Nate Johnston 23 Apr '19

23 Apr '19

API SIG and Neutron communities, TL;DR ----------------------- For API requests that specify boolean attributes to the request, we check "are you allowed to specify this attribute" before we check "would specifying this argument make any difference". I am proposing to change that so users can specify things forbidden by policy as long as the attribute specified equals the default and therefore the result would be the same as if it had not been specified. Are there any security problems with that? Long-winded explanation ----------------------- I would like your opinion on the potential security impact of a change I am proposing to the Neutron mechanism for assertion of oslo.policy. In response to an operator concern I proposed a change [1] that would exempt oslo.policy checks under one specific condition: if the request specifies a boolean attribute that matches a configured default. The way Neutron is currently coded, API requests have the oslo.policy check - "are you allowed to ask for X" before anything else about the request is processed. For those not familiar with it, oslo.polcy additionally provides the ability to control arguments within that request, so that you can permit only some users to use those arguments. For example, usually permissions to create a network are pretty broad but you must me admin to create a network and specify if it will be shared. The feedback that I got was that this was frustrating for users and some operators. The distinction that we make as programmers - asserting the policy check before the processing of the request - is counterintuitive from a user perspective. The preferred behavior is to say "if supplying the argument results in the same end condition as not supplying any argument (because the argument supplied is the default anyway) then allow it". Proposed change ----------------------- The proposed change I submitted to neutron is to bypass oslo.policy checks for method arguments when all boolean method arguments equal the configured default. If any boolean arguments differ from the default, or if there are any other arguments specified that are controlled by policy, then execute the oslo.policy checks as normal. Here are a few examples: - Regular project user creates a network specifying shared=false. - Current behavior: regular user cannot specify the shared attribute, denied. - Proposed behavior: "shared=false" is the default when creating networks, so allow it. - Regular project user creates a router specifying that it be non-HA. - Current behavior: regular user cannot specify the "ha" attribute, denied. - Proposed behavior: "ha=false" is the default when creating routers, so allow it. In neutron the oslo.policy check is done in the API base [2] almost immediately upon receipt of an API request, not as a decorator in the keystone style. So comparison of supplied boolean attributes against defaults could be done at that point centrally. The Concern ----------------------- The concern raised - and what I hope to get the API SIG's guidance on - is whether this raises any kind of security concern. I don't think it will. Since we are comparing equivalence of the user supplied attribute to the default, the end result of a request permitted by this proposal should always be the same as a request where the user did not supply that argument. But your wisdom would be appreciated in validating that assumption. Thanks for your feedback, Nate [1] https://bugs.launchpad.net/neutron/+bug/1821208 [2] https://opendev.org/openstack/neutron/src/branch/master/neutron/api/v2/base…

6 9

[nova] Running multinode resize tests in the gate
by Artom Lifshitz 23 Apr '19

23 Apr '19

Yo Novans, In trying to understand why my partial fix to bug 1813789 [1] - specifically the part where reverting a resize fails because we get the network-vif-plugged event from Neutron before we start listening for it - I noticed that we don't test multinode resize (and revert) in the gate. As far as I can tell, the only place where the revert tests are run are in tempest-full-py3, and that's on an allinone single node. In order to test a theory of mine about revert resize on the same node vs source to dest and back again, I do have a DNM patch up [2] to run the resize tests in the nova-live-migration job - I think that's the only multinode tempest job we have, right? I think that job also runs evacuation tests [3], correct? So, should we just rename it nova-multinode and add resize to the list of tests it runs? [1] https://bugs.launchpad.net/nova/+bug/1813789 [2] https://review.openstack.org/#/c/653498/ [3] https://github.com/openstack/nova/blob/54d8b40471e6f7b36dbe45966c1c605f94d4…

2 2

[nova][cinder] An API to replace volume-update (aka swap-volume)
by Matthew Booth 23 Apr '19

23 Apr '19

Consider this a straw-man proposal for the upcoming cross-project discussion at PTG. I'm not (currently, at any rate) proposing to do this work myself, which is also why I haven't put this in a spec. Volume-update is a problematic interface. There are likely many more than this, but problems which come to mind are: * It shares state between cinder and nova while running. A failure of either leaves an inconsistent state which is not easily recoverable. * It is fragile: it requires an instance is running. Stopping an instance while it is in progress is undefined at best. It will fail if an instance is not running. * It is mis-used by end-users to copy data between different volumes, which results in an unsupportable strange instance state. * It is slow. * It results in data corruption for multiattached volumes I'd like to propose the following api as a replacement. I'll describe the contract up top and put some example implementations below. This new flow for volume migration would be driven entirely by cinder, with no callbacks from nova, so all of the following are new nova apis. * volume-migration-start(src_attachment_id) After returning, nova will not write any data to src until completion or cancelled. Nova will return an error if it is not able to suspend writes to src. Start is a synchronous operation. Nova will not return until it can guarantee that there will be no further writes to src. * volume-migration-cancel(src_attachment_id) Nova will resume writing to src. A subsequent complete call will return an error. cancel is an asynchronous operation. Nova assumes it can resume writing to src immediately, but does not guarantee when it will finish the cancel operation. This has no impact on cinder, but nova instance operations may continue to be restricted until the cancel completes. * volume-migration-complete(src_attachment_id, dst_attachment_id) Nova expects that dst is a copy of src at the time that volume-migration-start() was called. Nova will detach from src and attach dst in its place. Nova will resume reading and writing to dst immediately. Nova will not return until it is safe for cinder to remove src, but the complete operation is not guaranteed to have finished at this time. This has no impact on cinder, but nova instance operations may continue to be restricted until it completes asynchronously. The implementation on the cinder side would be: volume-migration-start(src_attachment_id) copy(src, dst) volume-migration-complete(src, dst) Cinder doesn't need to be concerned with whether the instance or running or not, and it always does the copy itself. Using driver-specific optimisations this has the potential to be very fast, but even the fallback implementation of a full copy between 2 arrays should be substantially faster than a qemu block job on the compute host. It is entirely up to cinder to ensure this operation is safe wrt multiattach. I expect 'safe' here to mean it will refuse to do the operation. There may also be details to be worked out wrt who creates the dst attachment. If at all possible, I don't want to expose to nova the quirk that under cinder's covers a new volume is created which temporarily has the wrong volume id. I'd prefer that, from nova's POV, the operation is just a managed switch between 2 attachments to apparently the same volume. 2 possible implementations on the Nova side: 1. Hypervisor does not support live storage migration Summary: instance must be shutdown. volume-migration-start() returns success iff the instance is shutdown. Nova will set a task state such that instance cannot be started until cancel or complete. volume-migration-complete() reconfigures the specific volume attachment and unsets the task state so the instance can be started. 2. libvirt driver Summary: libvirt allows writes to continue during the copy by sending them to a local snapshot, which it commits to dst on complete(). volume-migration-start() sets task state such that instance cannot be started or stopped until cancel or completion. If instance is not running, does nothing else. If instance is running, creates local writeable qcow2 backed by read-only src volume. Swaps volume to local qcow2. volume-migration-cancel() does nothing if instance is not running except unset task state. If instance is running returns success to cinder immediately before commiting qcow2 to src volume and then unsetting task state. volume-migration-complete() updates instance to dst if it's not running, unsets task state and returns immediately. If instance is running, rebases local qcow2 from src to dst (cinder guarantees us that these are identical), then returns success to cinder because src can now be released. Nova starts a job to commit qcow2 to dst. On completion it updates instance to use dst and unsets task state. This API does not handle the case where the hypervisor is only capable of live storage migration where it manages the entire copy itself. If we had a requirement to support this we might change volume-migration-start() to return something indicating that nova will do the copy, or perhaps indicating that cinder should try again with a different entry point. However, I believe this approach will always be slower than a cinder-native copy, and I can't see any way around it requiring a callback to cinder on completion of the copy. I'd very much like some feedback from non-libvirt hypervisor folks around whether they're interested in this functionality at all, and if so what their capabilities are. Matt -- Matthew Booth Red Hat OpenStack Engineer, Compute DFG Phone: +442070094448 (UK)

2 3

Scuba Diving at the Denver Summit
by Monty Taylor 23 Apr '19

23 Apr '19

Hey everybody! tl;dr - Let me know if you'd like to go diving in the Rocky Mountains. Of all of the places we've had a Summit where one might reasonably expect there to be a possibility of Scuba Diving, the Denver Summit is probably REALLY low on the list. But it turns out that there is a Dive Shop that does dives at the Denver Aquarium [0] (with all the fish and sharks and whatnot) The Denver Aquarium is also very close to the Convention Center. I'm going to see about putting together a private group dive event for those of us who think that sounds like a fun idea. Due to cross-contamination concerns for the health and safety of the fish, the dive shop provides all equipment - so you don't have to bring gear. They will allow bringing your own mask - and I'm assuming if a well-scrubbed mask is ok then a well-scrubbed dive computer would be ok - but I'll be checking with them when I call to arrange things. If you are Open Water certified and would like to join, please send me a response back (reply directly to me rather than the list is fine) so I can judge general interest level when talking to the lovely humans. If you are **NOT** Open Water certified but are thinking to yourself "wow, that sounds like fun" - the dive shop does certification dives at the aquarium on the weekends. That's a bit out of scope for me to arrange, but if you want to show up in Denver early and do an Open Water class then join us, that's awesome. You can also get certified at home before coming ... but you'll need an Open Water certification to join us. BUT WAIT - THERE'S MORE! There is another interesting diving opportunity in the general Mountain West area - although not immediately **close** to Denver - that Sandy and I are going to be going to before the Summit - Homestead Crater. We're arranging to meet a Scuba instructor there to do our second Altitude cert dive, and also to do a 2-dive "Hot Springs Diver" specialty. We'd be happy to have folks join us. Homestead Crater [1] is outside of Salt Lake City. It's a hot spring - the water is 96F/35.5C and it's also an altitude dive. If people are interested, we can meet at the crater on Thursday morning pre-summit, dive the crater, then drive to Denver on Friday/Saturday. ** If only a small number of people are interested in any of this - we'll likely just do the Denver dives the weekend before during one of the normally scheduled dive times instead of arranging a group outing, and shift Homestead, if anyone is interested, a few days earlier ** I will not have space enough in my car to get you from the crater to Denver, so if you want to fly in to SLC ahead of time and meet there, awesome. Unlike Denver, the crater will require you have equipment, including scuba tanks. For anyone interested in coming, there are dive shops in Salt Lake where you can rent gear. There is a dive shop at the crater to do air fills. Anyway - if anyone is interested in some or all of these things, let me know this week so I can get a headcount and then we can talk more off-list about logistics and figure out how to deal with things for people coming from far off (carpools, equipment, etc) Monty PS. If any of you aren't already using Subsurface for dive logging, I'll be happy to help you get set up. [0] https://divedowntown.com/ [1] https://homesteadresort.com/utah-resort-things-to-do/homestead-crater/

4 5

[magnum] kubernests cinder deploying wordpress
by Ionut Biru 23 Apr '19

23 Apr '19

Hello, I'm trying to deploy a simple wordpress installation following the documentation from kubernets. https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-p… kubernets was deployed using magnum on openstack rocky using the following template: https://paste.xinu.at/sRk/ Then i'm using the cinder storage class that is described in kubernets documentation: https://kubernetes.io/docs/concepts/storage/storage-classes/#openstack-cind… and I also made it default. I did the following test: cinder volume created before hand and used in pod directly with the id, the volume is attached and mounted correctly like is described here: https://docs.openstack.org/magnum/latest/user/#storage Now if I try to deploy wordpress using the first link or other helms that use volume claim, the volume is created automatically by kubernets but they are never attached or mounted to the instances. On master I see the following error: pv_protection_controller.go:116] PV pvc-48fed935-5b61-11e9-9901-fa163e5ebeae failed with : Operation cannot be fulfilled on persistentvolumes "pvc-48fed935-5b61-11e9-9901-fa163e5ebea> the object has been modified; please apply your changes to the latest version and try again Is there something I miss? I'm using the wrong version of fedora-atomic? It's the latest version available for download. I did find that someone else had this error but not with cinder: https://github.com/kubernetes/kubernetes/issues/72905

1 1

[octavia][openstack-ansible] network layout example
by Ionut Biru 23 Apr '19

23 Apr '19

Hello guys, I have hard time understanding how to create the right layout infrastructure of octavia. Does anyone have octavia running in production with OSA? For now I have created a new tagged interface. Is this the right approach? auto bond0.40 iface bond0.40 inet manual vlan-raw-device bond0 auto br-lbaas iface br-lbaas inet manual bridge_stp off bridge_waitport 0 bridge_fd 0 pre-up ip link add br-lbaas-veth type veth peer name eth14 || true pre-up ip link set br-lbaas-veth up pre-up ip link set eth14 up post-down ip link del br-lbaas-veth || true bridge_ports bond0.40 br-lbaas-veth

1 0

[Third Party CI][Cinder] Changes from review.openstack.org to review.opedev.org
by Sean McGinnis 23 Apr '19

23 Apr '19

Attention operators of third party CI. It may have been easy to miss some of the changes that have been done last week to migrate from our Gerrit instance from review.openstack.org to review.opendev.org. For a lot of things, this should be seamless as there are pretty robust HHTP(s) redirects in place, making it easy to not even notice right away that things have changed. One thing that may impact third party CI operators though, is there is no way to automatically redirect SSH traffic. http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005423.ht… I think many CI systems SSH in and watch the event stream with something like: >> ssh -p 24198 review.openstack.org gerrit stream-events That will now fail with the move to opendev.org. Please make sure your CI configuration is updated to change that to review.opendev.org to continue to capture the event stream for triggering job execution. Thanks! Sean

3 3

[Glance] Weekly meetings
by Erno Kuvaja 23 Apr '19

23 Apr '19

Hi all, It's the Summit/PTG disruptions again. So This week Glance will be having weekly meeting normally at Thu (25th) 14:00, next week we will be having our meeting face to face in Denver so IRC will be quiet, the week after Summit, I might not be available to chair meeting if one is needed but I'll see if Abhishek or Brian can cover me. From Thu 16th of May you can again expect business as usual from us. Best, Erno

1 0

[Watcher] tomorrows meeting and agenda
by info＠dantalion.nl 23 Apr '19

23 Apr '19

Hello everyone, Tomorrow Watcher will have a meeting again at 08:00 UTC in the #openstack-meeting-alt channel. This is a reminder since we only recently started to have meetings again. The agenda is available on https://wiki.openstack.org/wiki/Watcher_Meeting_Agenda feel free to add any additional items. Kind Regards, Corne Lukken (Dantali0n)

1 0