December 2018 - openstack-discuss

Proposal to move cinder backup tests out of the integrated gate
by Matt Riedemann 11 Apr '19

11 Apr '19

I wanted to send this separate from the latest gate status update [1] since it's primarily about latent cinder bugs causing failures in the gate for which no one is really investigating. Running down our tracked gate bugs [2] there are several related to cinder-backup testing: * http://status.openstack.org/elastic-recheck/#1483434 * http://status.openstack.org/elastic-recheck/#1745168 * http://status.openstack.org/elastic-recheck/#1739482 * http://status.openstack.org/elastic-recheck/#1635643 All of those bugs were reported a long time ago. I've done some investigation into them (at least at the time of reporting) and some are simply due to cinder-api using synchronous RPC calls to cinder-volume (or cinder-backup) and that doesn't scale. This bug isn't a backup issue, but it's definitely related to using RPC call rather than cast: http://status.openstack.org/elastic-recheck/#1763712 Regarding the backup tests specifically, I don't see a reason why they need to be run in the integrated gate jobs, e.g. tempest-full(-py3). They don't involve other services, so in my opinion we should move the backup tests to a separate job which only runs on cinder changes to alleviate these latent bugs failing jobs for unrelated changes and resetting the entire gate. I would need someone from the cinder team that is more involved in knowing what their job setup looks like to identify a candidate job for these tests if this is something everyone can agree on doing. [1] http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000867… [2] http://status.openstack.org/elastic-recheck/ -- Thanks, Matt

6 12

[stable][kolla] EOL kolla ocata branch
by Eduardo Gonzalez 03 Apr '19

03 Apr '19

Hi stable team, Kolla project is ready to tag EOL the stable/ocata branch. We would like to tag the following repositories as EOL: - openstack/kolla - openstack/kolla-ansible Let me know if need anything else. Regards

4 5

[Interop-wg] [dev] [cinder] [qa] Strict Validation for Volume API using JSON Schema
by Ghanshyam Mann 08 Mar '19

08 Mar '19

Hello everyone, Tempest is planning to add the strict API validation using JSON schema for Volume test [1]. We do the same for all the compute test. With *Strict* JSON schema validation, all the API response will be validated with predefined schema with additionalProperties=False. additionalProperties=False will not allow any additional attribute in API response than what upstream APIs has in their current version. For example: If any vendor has modified the API response and returning additional attributes then Tempest tests going to fail. This will help: - To improve the OpenStack interoperability. Strict validation of API response is always helpful to maintain the interoperability. - To improve the volume API testing to avoid the backward compatible changes. Sometime we accidentally change the API in backward incompatible way and strict validation with JSON schema help to block those. We want to hear from cinder and interop team about any impact of this change to them. [1] https://blueprints.launchpad.net/tempest/+spec/volume-response-schema-valid… -gmann

4 6

[tc][all] Train Community Goals
by Lance Bragstad 15 Feb '19

15 Feb '19

Hi all, The purpose of this thread is to have a more focused discussion about what we'd like to target for Train community goals, bootstrapped with the outcomes from the session in Berlin [0]. During the session, we went through each item as a group and let the person who added it share why they thought it would be a good community goal candidate for the next release. Most goals have feedback captured in etherpad describing next steps, but the following stuck out as top contenders from the session (rated by upvotes): 1. Moving legacy clients to python-openstackclient 2. Cleaning up resources when deleting a project 3. Service-side health checks I don't think I missed any goals from the session, but if I did, please let me know and I'll add it to the list so that we can discuss it here. Does anyone have strong opinions either way about the goals listed above? [0] https://etherpad.openstack.org/p/BER-t-series-goals

15 26

[dev] How to develop changes in a series
by Eric Fried 31 Jan '19

31 Jan '19

This started off as a response to some questions Sundar asked, but I thought it might be interesting/useful for new[er] OpenStack developers at large, so broadcasting. I'm sure there's a document somewhere that covers this, but here's a quick run-down on how to develop multiple changes in a series. (Or at least, how I do it.) Start on a freshly-pull'd master branch: efried@efried-ThinkPad-W520:~/Neo/nova$ git checkout master Switched to branch 'master' Your branch is up-to-date with 'origin/master'. efried@efried-ThinkPad-W520:~/Neo/nova$ git pull --all <snip> When you're working on a blueprint, you want to name your local branch after the blueprint. So in this case, bp/nova-cyborg-interaction. efried@efried-ThinkPad-W520:~/Neo/nova$ git checkout -b bp/nova-cyborg-interaction Switched to a new branch 'bp/nova-cyborg-interaction' efried@efried-ThinkPad-W520:~/Neo/nova$ git log --oneline -1 --decorate 5bf6f63 (HEAD, origin/master, origin/HEAD, gerrit/master, master, bp/nova-cyborg-interaction) Merge "Deprecate the nova-xvpvncproxy service" When you `git commit` (without `--amend`), you're creating a new commit on top of whatever commit you started at. If you started with a clean, freshly pull'd master branch, that'll be whatever the most recently merged commit in the master branch was. In this example, that's commit 5bf6f63. So let's say I make an edit for my first patch and commit it: efried@efried-ThinkPad-W520:~/Neo/nova$ echo 'python-cyborgclient>=1.0' >> requirements.txt efried@efried-ThinkPad-W520:~/Neo/nova$ echo 'python-cyborgclient==1.1' >> lower-constraints.txt efried@efried-ThinkPad-W520:~/Neo/nova$ git commit -a -m "Add cyborg client to requirements" [bp/nova-cyborg-interaction 1b2c453] Add cyborg client to requirements 2 files changed, 2 insertions(+) efried@efried-ThinkPad-W520:~/Neo/nova$ git log --oneline -2 --decorate 1b2c453 (HEAD, bp/nova-cyborg-interaction) Add cyborg client to requirements 5bf6f63 (origin/master, origin/HEAD, gerrit/master, master) Merge "Deprecate the nova-xvpvncproxy service" I just made commit 1b2c453 on top of 5bf6f63. You'll notice my branch name (bp/nova-cyborg-interaction) came along with me. Now I'm going to make another change, but just part of it, a work-in-progress commit: efried@efried-ThinkPad-W520:~/Neo/nova$ mkdir nova/pci/cyborg efried@efried-ThinkPad-W520:~/Neo/nova$ touch nova/pci/cyborg/__init__.py efried@efried-ThinkPad-W520:~/Neo/nova$ git add nova/pci/cyborg/__init__.py efried@efried-ThinkPad-W520:~/Neo/nova$ git commit -m "WIP: Cyborg PCI handling" [bp/nova-cyborg-interaction ebb3505] WIP: Cyborg PCI handling 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 nova/pci/cyborg/__init__.py efried@efried-ThinkPad-W520:~/Neo/nova$ git log --oneline -3 --decorate ebb3505 (HEAD, bp/nova-cyborg-interaction) WIP: Cyborg PCI handling 1b2c453 Add cyborg client to requirements 5bf6f63 (origin/master, origin/HEAD, gerrit/master, master) Merge "Deprecate the nova-xvpvncproxy service" Now commit ebb3505 is on top of 1b2c453, which is still on top of 5bf6f63 (the master). Note that my branch name came with me again. At this point, I push my series up to gerrit. Note that it makes me confirm that I really want to push two commits at once. efried@efried-ThinkPad-W520:~/Neo/nova$ git review You are about to submit multiple commits. This is expected if you are submitting a commit that is dependent on one or more in-review commits, or if you are submitting multiple self-contained but dependent changes. Otherwise you should consider squashing your changes into one commit before submitting (for indivisible changes) or submitting from separate branches (for independent changes). The outstanding commits are: ebb3505 (HEAD, bp/nova-cyborg-interaction) WIP: Cyborg PCI handling 1b2c453 Add cyborg client to requirements Do you really want to submit the above commits? Type 'yes' to confirm, other to cancel: yes remote: remote: Processing changes: new: 2, refs: 2 (\) remote: Processing changes: new: 2, refs: 2 (\) remote: Processing changes: new: 2, refs: 2 (\) remote: Processing changes: new: 2, refs: 2, done remote: remote: New Changes: remote: https://review.openstack.org/623026 Add cyborg client to requirements remote: https://review.openstack.org/623027 WIP: Cyborg PCI handling remote: To ssh://efried@review.openstack.org:29418/openstack/nova.git * [new branch] HEAD -> refs/publish/master/bp/nova-cyborg-interaction Now if you go to either of those links - e.g. https://review.openstack.org/#/c/623026/ - you'll see that the patches are stacked up in series on the top right. But oops, I made a mistake in my first commit. My lower constraint can't be higher than my minimum in requirements.txt. If I still had my branch locally, I could skip this next step, but as a matter of rigor to avoid some common pratfalls, I will pull the whole series afresh from gerrit by asking git review to grab the *top* change: efried@efried-ThinkPad-W520:~/Neo/nova$ git review -d 623027 Downloading refs/changes/27/623027/1 from gerrit Switched to branch "review/eric_fried/bp/nova-cyborg-interaction" Now I'm sitting on the top change (which you'll notice happens to be exactly the same as before I pushed it - again, meaning I could technically have just worked from where I was, but see above): efried@efried-ThinkPad-W520:~/Neo/nova$ git log --oneline -3 --decorate ebb3505 (HEAD, review/eric_fried/bp/nova-cyborg-interaction, bp/nova-cyborg-interaction) WIP: Cyborg PCI handling 1b2c453 Add cyborg client to requirements 5bf6f63 (origin/master, origin/HEAD, gerrit/master, master) Merge "Deprecate the nova-xvpvncproxy service" But I want to edit 1b2c453, while leaving ebb3505 properly stacked on top of it. Here I use a tool called `git restack` (run `pip install git-restack` to install it). efried@efried-ThinkPad-W520:~/Neo/nova$ git restack This pops me into an editor showing me all the commits between wherever I am and the main branch (now they're in top-first order): pick 1b2c453 Add cyborg client to requirements pick ebb3505 WIP: Cyborg PCI handling <snip> I want to fix the first one, so I change to: edit 1b2c453 Add cyborg client to requirements pick ebb3505 WIP: Cyborg PCI handling <snip> Save and quit the editor, and I see: Stopped at 1b2c453453242a3fa57f2d4fdc80c837b02b804f... Add cyborg client to requirements You can amend the commit now, with git commit --amend Once you are satisfied with your changes, run git rebase --continue I fix lower-constraints: efried@efried-ThinkPad-W520:~/Neo/nova$ sed -i 's/cyborgclient==1.1/cyborgclient==1.0/' lower-constraints.txt ...and *amend* the current commit efried@efried-ThinkPad-W520:~/Neo/nova$ git commit -a --amend --no-edit [detached HEAD 6b3455f] Add cyborg client to requirements Date: Wed Dec 5 09:43:15 2018 -0600 2 files changed, 2 insertions(+) ...and tell `git restack` to proceed efried@efried-ThinkPad-W520:~/Neo/nova$ git rebase --continue Successfully rebased and updated refs/heads/review/eric_fried/bp/nova-cyborg-interaction. If I had a taller series, and I had changed 'pick' to 'edit' for more than one commit, I would now be sitting on the next one I needed to edit. As it is, that was the only thing I needed to do, so I'm done and sitting on the top of my series again. 124b612 (HEAD, review/eric_fried/bp/nova-cyborg-interaction) WIP: Cyborg PCI handling 6b3455f Add cyborg client to requirements 5bf6f63 (origin/master, origin/HEAD, gerrit/master, master) Merge "Deprecate the nova-xvpvncproxy service" Notice that the commit hashes have changed for *both* commits (but not the master). The top one changed because it got rebased onto the new version of the middle one. Now if I push the series back up to gerrit, I get the same confirmation prompt, and both changes get a new patch set. If you look at the top patch in gerrit, you'll see that PS2 shows up as just a rebase. ==================== That ought to be enough for now. There's a couple of gotchas when restacking and the automatic rebase results in merge conflicts, but we can save that for another time. To answer your specific question: > * If you have a patch sequence A followed by B, where patch B depends on patch A, > how do you communicate that in the submission? If they're in the same commit series, like the example above, It Just Happens. Zuul and the CIs know to pull the whole series when they run; gerrit won't merge N+1 until N is merged; etc. If they're *not* in the same commit series, or if they're not in the same repository, you can use Depends-On in the commit message, to the same effect. (But I'm not sure if/how Depends-On works for feature branches.) HTH, efried .

7 8

[ptl][goals][python3] please update your team's status in the wiki
by Doug Hellmann 24 Jan '19

24 Jan '19

I noticed this morning that the last time the Python 3 status page [1] was updated in the wiki was August. I hope we've had changes to the level of support since then. Keeping that page up to date is part of fulfilling the goal this cycle. Please take a few minutes to review the content today and update it if necessary. Thanks! [1] https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projec… -- Doug

2 2

[nova] review guide for the bandwidth patches
by Balázs Gibizer 21 Jan '19

21 Jan '19

Hi, After discussing generic concerns about the current proposed patch series[1] for the bandwidth-resource-provider bp [2] with Matt, I decided to write a summary that might help other reviewers. Grouping of the pathes ---------------------- The whole nova implementation is one long patch series that can be splitted into smaller logical steps * Already merged patches added the two new standard resource classes and added a field to RequestSpec to store the resource request of each neutron port. * The open patch series start at [3] now and the patches between [3] [4] introduce the ability to boot a server with neutron ports that has resource request. These patches implement handling the extra resources during boot, re-schedule and delete. These patches only work if a server has maximum one neutron port that has resource request. * The patches after [4] until [5] implement the calculation and communiction of the mapping between ports and resource providers that are fulfilling the given port resource request. (The reason why this is done in nova described below.) So these patches allow having more than one neutron port per server having resource request. * the next two patches [6][7] implement rejecting use cases that was decided to out of scope for the feature (interface attach with resource requst, booting with network having resource request) * The next patch [8] add support for removing resource allocation during interface detach if the detached port has resource request * The rest of the series up until [9] implement support for selecting VFs form the same PF that the bandwidth was allocated from. This allows handling bandwidth and VF resources properly even if there are more than one PFs on the same compute connected to the same physnet. What is still missing from the patch series: * Support for any kind of server move operations (resize, migrate, evacuate, live-migrate, unshelve) * Upgrade support: Bound SRIOV ports might already have QoS policy that would need healing resource allocation in placement. * Supporting separation of QoS aware and non aware ports on the same host. This can be done with host aggregates today, but the bp described possible ways to support a better separation. * Supporting multisegment network that has more than one segment having physnet name. For this we would need any-trait support from placement. The spec for it is approved but the implementation is on hold due to placement extraction. Order of the patches -------------------- One comment I got from Matt is that the patches are in wrong order. The current order allows booting server with bandwidth request _before_ every other server operation is supported (or rejected). This could lead to the violation of the resource view of the system. For example booting a server with a QoS aware port will result in bandwidth allocation on the compute host. But then later on migrating that server to another host does not properly handle moving the allocation to the target host. In my view reordering the patches to only allow booting the first server with bandwdith after every other operation is supported is not feasible. The current chain is already 17 patches long and it does not touch server moving operations at all. Also if I cannot boot a server with bandwidth then I cannot test that such server can be moved properly. So all the functional test would end up in the last patch of the series. However I think we can use the feature flag approach. We can implement and verify everything in the current order without the end user seeing or causing any kind of inconsistencies. The nova implementation is only activated if a neutron port has a resource_request field. This field is added to the neutron API as an API extension. We can mark this extension experimental while we are working through the missing pieces of the feature. But we can also turn on that extension in our test envirnoment to verify each and every step during the implementation process. Which RP fulfills the request of a neutron port ----------------------------------------------- Neutron expects a single RP uuid sent by nova in the port binding that tells neutron about which RP providers the resources for that port. This is the famous problem to find which request group is fulfilled by which resource provider in an allocation candidate. There are 3 possible solutions: * Neutron does the mapping. This would require Neutron to see each port binding for a given server as a single request. This is not the case today and not feasible to hack around in neutron. * Placement does the mapping. This is technically feasible and in my oppinion this is the good solution. But placement is in feature freeze so the spec [10] describing this change is put on hold. * Nova does the mapping. This is what pathes after [4] until [5] implement. It is a temporary solution until the placement based solution can be provided. The current patches are structured in a way that all ovo and neutronv2/api impacts in nova are reusable for the placement based solution later without change. There is only one patch [11] that will be thrown away when the placement based solution is available. I hope this mail helps reviewing the series. Cheers, gibi [1] https://review.openstack.org/#/q/topic:bp/bandwidth-resource-provider+(stat… [2] https://blueprints.launchpad.net/nova/+spec/bandwidth-resource-provider [3] https://review.openstack.org/#/c/625941/ [4] https://review.openstack.org/#/c/567268/ [5] https://review.openstack.org/#/c/573317/ [6] https://review.openstack.org/#/c/570078 [7] https://review.openstack.org/#/c/570079 [8] https://review.openstack.org/#/c/622421 [9] https://review.openstack.org/#/c/623543 [10] https://review.openstack.org/#/c/597601 [11] https://review.openstack.org/#/c/616239

7 25

[dev] [infra] [devstack] [qa] [kuryr] DevStack's etcd performance on gate VM's
by Michał Dulko 18 Jan '19

18 Jan '19

Hi, In Kuryr-Kubernetes we're using the DevStack-installed etcd as a backend store for Kubernetes that we run on our gates. For some time we can see its degraded performance manifesting like this [1] in the logs. Later on this leads to various K8s errors [2], [3], up to missing notifications from the API, which causes failures in Kuryr-Kubernetes tempest tests. From what I've seen those etcd warnings normally mean that disk latency is high. This seems to be mostly happening on OVH and RAX hosts. I've looked at this with OVH folks and there isn't anything immediately alarming about their hosts running gate VM's. Upgrading the etcd version doesn't seem to help, as well as patch [4] which increases IO priority for etcd process. Any ideas of what I can try next? I think we're the only project that puts so much pressure on the DevStack's etcd. Help would really be welcomed, getting rid of this issue will greatly increase our gates stability. Thanks, Michał [1] http://logs.openstack.org/49/624749/1/check/kuryr-kubernetes-tempest-daemon… [2] http://logs.openstack.org/49/624749/1/check/kuryr-kubernetes-tempest-daemon… [3] http://logs.openstack.org/49/624749/1/check/kuryr-kubernetes-tempest-daemon… [4] https://review.openstack.org/#/c/624730/

3 5

[all] Etcd as DLM
by Ben Nemec 17 Jan '19

17 Jan '19

Hi, I wanted to revisit this topic because it has come up in some downstream discussions around Cinder A/A HA and the last time we talked about it upstream was a year and a half ago[1]. There have certainly been changes since then so I think it's worth another look. For context, the conclusion of that session was: "Let's use etcd 3.x in the devstack CI, projects that are eventlet based an use the etcd v3 http experimental API and those that don't can use the etcd v3 gRPC API. Dims will submit a patch to tooz for the new driver with v3 http experimental API. Projects should feel free to use the DLM based on tooz+etcd3 from now on. Others projects can figure out other use cases for etcd3." The main question that has come up is whether this is still the best practice or if we should revisit the preferred drivers for etcd. Gorka has gotten the grpc-based driver working in a Cinder driver that needs etcd[2], so there's a question as to whether we still need the HTTP etcd-gateway or if everything should use grpc. I will admit I'm nervous about trying to juggle eventlet and grpc, but if it works then my only argument is general misgivings about doing anything clever that involves eventlet. :-) It looks like the HTTP API for etcd has moved out of experimental status[3] at this point, so that's no longer an issue. There was some vague concern from a downstream packaging perspective that the grpc library might use a funky build system, whereas the etcd3-gateway library only depends on existing OpenStack requirements. On the other hand, I don't know how much of a hassle it is to deploy and manage a grpc-gateway. I'm kind of hoping someone has already been down this road and can advise about what they found. Thanks. -Ben 1: https://etherpad.openstack.org/p/BOS-etcd-base-service 2: https://github.com/embercsi/ember-csi/blob/5bd4dffe9107bc906d14a45cd819d9a6… 3: https://github.com/grpc-ecosystem/grpc-gateway

11 20

[edge] Zero Touch Provisioning
by Curtis 15 Jan '19

15 Jan '19

Hi, I've been looking through the docs I can find related to the edge working group, and I'm wondering if there has been any discussion/documentation of a Zero Touch Provisioning use case. I can't seem to find anything, but I may not be looking in the right place. Just wanted to double check and see what the current state is, if any. Thanks, Curtis

4 12