- openstack-discuss - lists.openstack.org

[nova] NUMA live migration - mostly how it's tested
by Artom Lifshitz 05 Mar '19

05 Mar '19

Hey all, There won't be much new here for those who've reviewed the patches [1] already, but I wanted to address the testing situation. Until recently, the last patch was WIP because I had functional tests but no unit tests. Even without NUMA anywhere, the claims part of the new code could be tested in functional tests. With the new and improved implementation proposed by Dan Smith [2], this is no longer the case. Any test more involved than unit testing will need "real" NUMA instances on "real" NUMA hosts to trigger the new code. Because of that, I've dropped functional testing altogether, have added unit tests, and have taken the WIP tag off. What I've been using for testing is this: [3]. It's a series of patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch of us Nova Red Hatters to automate testing that's outside of Tempest's scope. Same idea as the intel-nfv-ci plugin [4]. The tests I currently have check that: * CPU pin mapping is updated if the destination has an instance pinned to the same CPUs as the incoming instance * emulator thread pins are updated if the destination has a different cpu_shared_set value and the instance has the hw:emulator_threads_policy set to `share` * NUMA node pins are updated for a hugepages instance if the destination has a hugepages instances consuming the same NUMA node as the incoming instance It's not exhaustive by any means, but I've made sure that all iterations pass those 3 tests. It should be fairly easy to add new tests, as most of the necessary scaffolding is already in place. [1] https://review.openstack.org/#/c/634606/ [2] https://review.openstack.org/#/c/634828/28/nova/virt/driver.py@1147 [3] https://review.rdoproject.org/r/#/c/18832/ [4] https://github.com/openstack/intel-nfv-ci-tests/

5 14

Re: [nova] Support adding/changing delete_on_termination in volume attach api
by Brin Zhang(张百林) 05 Mar '19

05 Mar '19

As Matt said, attach the discussion results at the Forum in Berlin(https://etherpad.openstack.org/p/BER-bfv-improvements #52), you can also read it yourself on the etherpad.: 1.Allow updating delete_on_termination for existing attachments: https://review.openstack.org/#/c/580336/ 1.1 How to model this in the API? -- PUT /servers/{server_id}/os-volume_attachments/{volume_id} body={'volumeAttachment': {'delete_on_termination': True}}? -- Note that the PUT route is already used for swap volume. 1.2 This could break Heat if Heat created the volume, the user changed the flag and then nova deletes the volume but Heat expected it to exist. 1.3 Alternative: if you really love the pet, snapshot it. 1.4 Alternative: if we do the spec above, detach the volume and then re-attach with the delete_on_termination flag set to what you want. For the above [1.2], it has been verified in our R&D environment and customer production environment, no abnormalities have occurred, and everything is operating well. I wonder if any friends have encountered the scene mentioned here, does it have a certain impact on this? Changes to the current nova API: a. I want to add the change of [spec 1 ] to update a volume attachment API, and add "block_device_mapping_v2": [{ "volume_id": "763b894f-af35-4e44-bdb8-2ca45db9ecd8", "delete_on_termination": False }] in the request body. b. Add the change of [spec 2] to attach a volume to an instance API, add the "delete_on_termination" field to request body. [spec 1] https://review.openstack.org/#/c/580336/ Support for changing deleted_on_termination after boot [spec 2] https://review.openstack.org/#/c/612949/ Support delete_on_termination in server attach volume Brin Zhang -----邮件原件----- 发件人: Matt Riedemann [mailto:mriedemos@gmail.com] 发送时间: 2019年3月5日 2:04 收件人: openstack-discuss(a)lists.openstack.org 主题: [lists.openstack.org代发]Re: [nova] Support adding/changing delete_on_termination in volume attach api On 3/3/2019 9:20 PM, Brin Zhang wrote: >> Hi all: >> >> Currently, you can set the "delete_on_termination" field to >> the rood disk when creating the server. You can delete the root disk >> when you delete the server. However, this configuration item cannot be >> updated, so here we propose a scheme that can modify the >> "delete_on_termination" configuration item, such as [1]. >> >> In addition, the data volume of the instance should be allowed to be >> set whether it can be deleted when it is deleted to clean up the >> environment and free up storage space, such as [2]. >> >> These are the more consistent and strongly added requirements from our >> various OpenStack users. >> >> Thank you, looking forward to your reply. >> >> [1] https://review.openstack.org/#/c/580336/ >> >> [2] https://review.openstack.org/#/c/612949/ >> >> Brin Zhang >> >As you know we discussed this at the Forum in Berlin (I know you were not there but you know about the notes in the etherpad). Can you summarize the output of those discussions and what, if anything, it changed about the proposed specs? > >The specs are targeted at Stein, and we are closed for new specs in Stein so those will have to be re-targeted to Train. > >Finally, I was thinking about this recently in regards to the root volume attach/detach change that is proposed because that does not change delete_on_termination when attaching a new root volume, so could be a case for [2] above. In other words, I could boot from volume and say delete_on_termination=True for the root volume (maybe because nova creates it and I do not care to preserve it when the server is gone), detach the root volume and attach a new volume, and now want that delete_on_termination flag to be False. So that might further justify this change. > >-- > >Thanks, > >Matt

1 0

[neutron] Multi-segment per host support for routed networks
by David G. Bingham 05 Mar '19

05 Mar '19

Neutron land, Problem: Neutron currently only allows a single network segment per host. This becomes a problem when networking teams want to limit the number of IPs it supports on a segment. This means that at times the number of IPs available to the host is the limiting factor for the number of instances we can deploy on a host. Ref: https://bugs.launchpad.net/neutron/+bug/1764738 Ongoing Work: We are excited in our work add "multi-segment support for routed networks". We currently have a proof of concept here https://review.openstack.org/#/c/623115 that for routed networks effectively: * Removes validation preventing multiple segments. * Injects segment_id into fixed IP records. * Uses the segment_id when creating a bridge (rather than network_id). In effect, it gives each segment its own bridge. It works pretty well for new networks and deployments. For existing routed networks, however, it breaks networking. Please use *caution* if you decide to try it. TODOs: Things TODO before this before it is fully baked: * Need to add code to handle ensuring bridges are also updated/deleted using the segment_id (rather than network_id). * Need to add something (a feature flag?) that prevents this from breaking routed networks when a cloud admin updates to master and is configured for routed networks. * Need to create checker and upgrade migration code that will convert existing bridges from network_id based to segment_id based (ideally live or with little network traffic downtime). Once converted, the feature flag could enable the feature and start using the new code. Need: 1. How does one go about adding a migration tool? Maybe some examples? 2. Will nova need to be notified/upgraded to have bridge related files updated? 3. Is there a way to migrate without (or minimal) downtime? 4. How to repeatably test this migration code? Grenade? Looking for any ideas that can keep this moving :) Thanks a ton, David Bingham (wwriverrat on irc) Kris Lindgren (klindgren on irc) Cloud Engineers at GoDaddy

2 2

[all] Three months with openstack-discuss (one last report)
by Jeremy Stanley 05 Mar '19

05 Mar '19

Here's a final (hopefully) review of uptake for the openstack-discuss mailing list, now that the old lists it replaced have been closed for three months. See my previous reports[0][1] from the one- and two-month anniversaries. The volume of posts from non-subscribers landing in the moderation queue has continued to dwindle to only a few each week now. We've reached 934 subscribers to openstack-discuss (up from 830 in the previous report). 76% of the addresses used to send 10 or more messages to the old lists in 2018 are now subscribed to the new one (it was 75% a month ago so we're probably in the long tail now). As anticipated, posting volume rose in February to 1083 messages as compared to January's 958; for reference that's an 11% drop from the 1196 we had in Februafy 2018 (and again recall that right at 10% of the messages on the old lists were duplicates from cross-posting). We're continuing to investigate causes of messages through the list failing DKIM signature validation: * The most frequent cause was Mailman's recipient deduplication, since it works by filtering list subscribers out of the Cc header on receipt prior to processing for redelivery. As an experiment we've removed[2] it from Mailman's module pipeline, which has significantly reduced the number of invalid DKIM signatures (but at the expense of an increase in people receiving duplicate copies of list replies on which they were also Cc'd directly). * For the next most common cause, we've observed some mail servers using the much more strict "simple" header canonicalization algorithm[3] which makes whitespace in message headers significant (as opposed to the more popular "relaxed" canonicalization), and Mailman has a penchant for normalizing whitespace in message headers prior to forwarding which invalidates these particular signatures. We're not sure what the best option for addressing this is yet, but are considering reaching out to the affected individuals to see if they'll lobby their respective organizations to reconsider their choice of canonicalization. * The third and least common cause we've identified is a participant whose References header incorrectly adds a comma separator (RFC 822 parser spec[4] says space/tab/CRLF are the only allowed separators here), so when Mailman reformats the references list prior to forwarding it concatenates some elements. We don't have a great solution for this one either aside from asking the affected individual to work out whether the problem is being introduced by their MUA or their organization's MTA. [0] http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001386.… [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002335… [2] https://review.openstack.org/635078 [3] http://dkim.org/specs/rfc4871-dkimbase.html#canonicalization [4] https://tools.ietf.org/html/rfc822#section-3.3 -- Jeremy Stanley

1 0

[neutron] [bugs]
by Bhatia, Manjeet S 05 Mar '19

05 Mar '19

Hi Neutrinos, Below are the bugs I observed last week, two of them I marked critical because one was affecting gate and other Was issue related to vlan unbinding in production deployment which is very strange. There were many RFE's reported this week and are under [new] section below. I'd request someone expert to look over bug 1817548 and confirm it. [critical] 1. https://bugs.launchpad.net/neutron/+bug/1817541 (pbr version broke lower constraints jobs) (fix released.) 2. https://bugs.launchpad.net/neutron/+bug/1818015 (Removed vlan binding in use, this bug was seen in production deployment based on queens). [high] 3. https://bugs.launchpad.net/neutron/+bug/1817548 (macvtap: possible race of interface scan/deletion at migration) temporary fixed commented but no permanent solution. 4. https://bugs.launchpad.net/neutron/+bug/1818224 IPv6 forwarding disabled on L3 HA routers without gateway [medium] 5. https://bugs.launchpad.net/neutron/+bug/1817953 (oslo policy generator not working for neutron) 6. https://bugs.launchpad.net/neutron/+bug/1817956 (Metadata not reachable when dvr_snat L3 agent is used on compute node) 7. https://bugs.launchpad.net/neutron/+bug/1818318 (Network filtering by dns_domain in dns-integration not supported) (looks like issue is observed in queens ) 8. https://bugs.launchpad.net/neutron/+bug/1818335 (Fullstack test test_dscp_marking_packet fails) [Low] 9. https://bugs.launchpad.net/neutron/+bug/1818233 (ping_ip_address in tests succeeds even though ping fails ) [new] 10. https://bugs.launchpad.net/neutron/+bug/1817697 [FwaasV1] updating ip address in firewall rule demands ip-version instead of using ipv4 as defaut (more details asked) Like what fwaas api version is being used. Fwaas folks can confirm this. 11. https://bugs.launchpad.net/neutron/+bug/1817872 (RFE neutron resource health check) 12. https://bugs.launchpad.net/neutron/+bug/1817881 (RFE L3 Ips monitor vis current Qos Functionality) 13. https://bugs.launchpad.net/neutron/+bug/1817886 (RFE cluster maximum capacity limitation) 14. https://bugs.launchpad.net/neutron/+bug/1818317 (Network filtering not supported by mtu's) 15. https://bugs.launchpad.net/neutron/+bug/1818479 (Decouple placement reporting service from Ml2 plugin) Thanks and Regards ! Manjeet Singh Bhatia

1 0

[docs] Implementation of the api-ref consolidation under doc/source/
by Luigi Toscano 05 Mar '19

05 Mar '19

Hi all, During the last PTG it was decided to move forward with the migration of the api-ref documentation together with the rest of the documentation [1]. This is one of the item still open after the (not so recent anymore) massive documentation restructuring [2]. (most likely anything below applies to releasenotes/ as well.) I asked about this item few weeks ago on the documentation channels. So far no one seems against moving forward with this, but, you know, resources :) I think that the process itself shouldn't be too complicated on the technical side, but more on the definition of the desired outcome. I can help with the technical part (the moving), but it would be better if someone from the doc team with the required knowledge and background on the doc process would start with at least a draft of a spec, which can be used to start the discussion. If implemented, this change would also fix an inconsistency in the guidelines [2]: the content of reference/ seems to overlap with the content of api- ref("Library projects should place their automatically generated class documentation here."), but then having api-ref there would allow us to always use api-ref. That's where the entire discussion started in the QA session [3]: some client libraries document their API in different places. [1] https://etherpad.openstack.org/p/docs-i18n-ptg-stein line 144 [2] https://docs.openstack.org/doc-contrib-guide/project-guides.html [3] https://etherpad.openstack.org/p/clean-up-the-tempest-documentation Ciao -- Luigi

8 15

[nova] Support adding/changing delete_on_termination in volume attach api
by Brin Zhang(张百林) 05 Mar '19

05 Mar '19

Hi all: Currently, you can set the "delete_on_termination" field to the rood disk when creating the server. You can delete the root disk when you delete the server. However, this configuration item cannot be updated, so here we propose a scheme that can modify the "delete_on_termination" configuration item, such as [1]. In addition, the data volume of the instance should be allowed to be set whether it can be deleted when it is deleted to clean up the environment and free up storage space, such as [2]. These are the more consistent and strongly added requirements from our various OpenStack users. Thank you, looking forward to your reply. [1] https://review.openstack.org/#/c/580336/ [2] https://review.openstack.org/#/c/612949/ Brin Zhang

2 1

[oslo] Denver topic etherpad
by Ben Nemec 05 Mar '19

05 Mar '19

Hi, As we discussed in the meeting today, I've started an etherpad[0] to collect topics for Denver. I'm expecting most of our topics to be more PTG-oriented, but if you have something you think would benefit from the Forum audience please feel free to propose it and make a note in the description that it should be a Forum session. I would also like to use this to collect topics for the Oslo project update session. I've added a topic for that and seeded the list with a few things I came up with quickly, but I expect there's quite a bit more so please add to it. Thanks. -Ben 0: https://etherpad.openstack.org/p/oslo-train-topics

1 0

[all] Last call for "Train" TC Election voting
by Jeremy Stanley 05 Mar '19

05 Mar '19

We are coming down to the last hours for voting in the TC election. Voting ends Mar 05, 2019 23:45 UTC. Search your gerrit preferred email address[0] for the following subject: Poll: 'Train' TC Election (note the unfortunate escaping of quotes around the release cycle name, we'll strive to do better for the PTL polls) That is your ballot and links you to the voting application. Please vote. If you have voted, please encourage your colleagues to vote. Candidate statements are linked to the names of all confirmed candidates: https://governance.openstack.org/election/#train-tc-candidates What to do if you don't see the email and have a commit in at least one of the official programs projects[1]: * check the trash of your gerrit Preferred Email address[0], in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from the program project repos[1] and email the election officials[2]. If we can confirm that you are entitled to vote, we will add you to the voters list and you will be emailed a ballot. Please vote! Thank you, [0] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your Preferred Email. That is where the ballot has been sent. [1] https://git.openstack.org/cgit/openstack/governance/plain/reference/project… [2] https://governance.openstack.org/election/#election-officials -- Jeremy Stanley

1 0

[placement] Placement and Storyboard
by Chris Dent 05 Mar '19

05 Mar '19

As discussed in today's scheduler meeting [1]: Placement now has a home in Storyboard for all four of placement, osc-placement, os-resource-classes and os-traits. https://storyboard.openstack.org/#!/project_group/placement For the rest of Stein we will be using both existing launchpad projects and storyboard for reporting and managing placement-related bugs. The reason for the transition is simply to not surprise anyone nor lose bugs while we are nearing a release. In Train placement-related projects will solely use Storyboard for bugs and feature tracking. For the rest of Stein, placement developers should watch both places for bugs. For remembering features or other work that is planned for Train, make a story in Storyboard. Based on discussion we've had thus far, "stories" seem the best choice for a replacement for blueprints. People who experience placement bugs and report them to launchpad, that's okay. We'll notice and try to do our best to deal with them, starting at least in Train, but maybe sooner, we may respond with "could you please report this in Storyboard instead". Note that because placement bugs lived in the nova project, there's no way to "shut down" launchpad once we migrate to Storyboard. We will need to be conscious of that for quite some time. We'll all be learning to use Storyboard as we go, so there may be a few hiccups. It's okay. We'll deal. As Ed pointed out in the meeting "Be prepared for a "moving to git from svn" brain reshuffling. The workflow is not 1:1 to launchpad" Because of that I think that rather than hewing to a strict process we should be aware of our intentions and be flexible. With luck our general trend of "not that many bugs" (thus easy to manage) will continue. If you have questions, concerns, or suggestions for how to make it all delightful, please share them. /me knocks wood Thanks. [1] http://eavesdrop.openstack.org/meetings/nova_scheduler/2019/nova_scheduler.… -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

1 0