May 2019 - openstack-discuss - lists.openstack.org

Any ideas on fixing bug 1827083 so we can merge code?
by Matt Riedemann 15 May '19

15 May '19

I'm not sure what is causing the bug [1] but it's failing at a really high rate for about week now. Do we have ideas on the issue? Do we have thoughts on a workaround? Or should we disable the vexxhost-sjc1 provider until it's solved? [1] http://status.openstack.org/elastic-recheck/#1827083 -- Thanks, Matt

4 4

[oslo][all] Ending courtesy pings
by Ben Nemec 15 May '19

15 May '19

Hi, We discussed this some in the Oslo meeting yesterday[0], and I wanted to send a followup because there wasn't universal support for it. One of the outcomes of the PTL tips and tricks session in Denver was that courtesy pings like we use in the Oslo meeting are considered bad IRC etiquette. The recommendation was for interested parties to set up custom highlights on the "#startmeeting oslo" (or whichever meeting) command. Also, there is an ics file available on eavesdrop[1] that can be used to import the meeting to your calendaring app of choice. I should note that I don't seem to be able to configure notifications on the imported calendar entry in Google calendar though, so I'm not sure how useful this is as a reminder. A couple of concerns were raised yesterday. One was that people didn't know how to configure their IRC client to do this. Once you do configure it, there's a testing problem in that you don't get notified of your own messages, so you basically have to wait for the next meeting and hope you got it right. Or pull someone into a private channel and have them send a startmeeting command, which is a hassle. It isn't terribly complicated, but if it isn't tested then it's assumed broken. :-) The other concern was that this process would have to be done any time someone changes IRC clients, whereas the ping list was a central thing that always applies no matter where you're connecting from. Anyway, I said I would send an email out for further public discussion, and this is it. I'm interested to hear people's thoughts. Thanks. -Ben 0: http://eavesdrop.openstack.org/meetings/oslo/2019/oslo.2019-05-13-15.00.log… 1: http://eavesdrop.openstack.org/#Oslo_Team_Meeting

4 7

[First Contact] [SIG] Summit/Forum + PTG Summary
by Kendall Nelson 15 May '19

15 May '19

Hello All! Here's a super short summary of the going's on WRT the First Contact SIG. There were plenty of other relevant sessions, but this email summarizes the ones that we were directly responsible for. Summit Session + PTG -------------------------------- The Meet & Greet and the PTG session went similarly. We had a few new faces come in and introduce themselves. Most of the new faces were operators from various companies which was cool that they wanted to get engaged. Members of the SIG introduced themselves and answered any questions that we could. Forum Session (Welcoming New Contributors State of the Union and Deduplication of Efforts) ----------------------------------------------------------------------------------------------------------------------------- The biggest things that came out of this session were discussion about recording of onboarding sessions and a community goal of improving contributor documentation. Basically, we have never had the onboarding sessions recorded but if we could tt would really help new contributors even if they might get a little stale before we are able to record new ones. During that chat, we learned that Octavia does somewhat regular calls in whch they do onboarding for new contributors. I have asked for an outline to help encourage other projects to do similar. As for per project contributor documentation, some projects have it and some don't. Some projects have it and its incomplete. bauzas volunteered to do an audit of which projects have it and which don't and to propose a community goal for it. As a part of that, we should probably decide on a list of bare minimum things to include. Etherpad from those discussions: https://etherpad.openstack.org/p/new-contribs-state-and-deduplication -Kendall Nelson (diablo_rojo)

2 3

[cinder] ACTION REQUIRED: Updating weekly meeting ping list
by Jay Bryant 15 May '19

15 May '19

Team, There has been discussion recently about whether or not courtesy pings are good practice or not. The decision from the Cinder team is that we feel they are useful and we want to continue to use them. Our courtesy ping list, however, has become quite stale. So, we have decided to remove the existing list and allow people to add their names back in if they wish to continue to be pinged for the weekly meeting. I am going to keep the existing list one more week to give people time to see this e-mail and make changes. If, however, your name is not on the new list that I have started here [1] the assumption will be that you don't want to be pinged when the Cinder meetings start. Thanks for your attention to this change. In the future we will plan to refresh the ping list at the beginning of each release. Hope to see you in our weekly meetings! Thanks! Jay [1] https://etherpad.openstack.org/p/cinder-train-meetings

1 0

[placement][ptg] Summary of Summaries
by Chris Dent 15 May '19

15 May '19

I produced several summary messages at the end of the PTG [1] but then disappeared into a dark corporate belly, so I thought it might be useful to do a quick check in to assert and verify some status. I'll be doing a full pupdate on Friday. If any of the below doesn't align with your memories, please let me know. At the end the PTG there were some open questions about priorities so I produced an etherpad listing all the existing RFE stories along with the new ones produced by discussions at the PTG and asked people to register their preferences. Thus far only three people (including me) have voted. Please look at https://etherpad.openstack.org/p/placement-ptg-train-rfe-voter to register your input. As things currently stand our priorities are what you would expect based on who is available to work and the work that needs to be done to be able to satisfy other projects' dependencies on placement: * Consumer types, spec at: https://review.opendev.org/#/c/654799/ * Nested things: * request group mapping in allocation candidates: https://review.opendev.org/#/c/657582/ * the remaining "nested magic: https://review.opendev.org/#/c/658510/ The latter is a WIP and will almost certainly need someone besides Eric to finish it off. Multiple inter-related features will come out of that. See the related story [2]. "support any trait in allocation candidates" and "support mixing required traits with any traits" are still under review but have been de-emphasized. If we're able to get to them that's great, but they are not critical. Same is also true for managing a local-to-placement container. Since the PTG several of us have recognized/acknowledged a thing we already pretty much knew: Queries for nested providers in a highly populated but sparsely used cloud will be less performant than desired. There are things we can do to monitor and fix this. Tetsuro has already started some changes [3] and in the gaps I'm working on making changes to placeload [4] to build nested providers to be used in the perfload job. "Support resource provider partitioning" is also being de-emphasized, but based on several recent conversations I suspect we will want it soon, and any performance improvements we get before then will be important. So, to summarize this summary of summaries: Look at https://etherpad.openstack.org/p/placement-ptg-train-rfe-voter and review the three specs above. If you think other priorities are necessary please speak up and explain why. Thanks to everyone for their participation in the PTG and especially the virtual pre-PTG. We got a lot figured out and were still able to attend other sessions. [1] * Nested magic: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005823.html * Shared disk: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005829.html * Consumer Types: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005878.html * Placement, Ironic, Blazar: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005880.html [2] https://storyboard.openstack.org/#!/story/2005575 [3] https://review.opendev.org/#/c/658977/ [4] https://pypi.org/project/placeload/ -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

1 1

[nova] PTG aligning of nova spec: show-server-numa-topology
by yonglihe 15 May '19

15 May '19

Hi, Everyone I synced up with Alex about comments we got at PTG. It's a long discussion, I might lost something. What i got lists below, fix me: * Remove sockets * Remove thread_policy Not sure about following comments: * Remove the cpu topology from the proposal? * Using the cpu pinning info instead of cpu set? By apply the suggestion, the API ``GET /servers/{server_id}/topology`` response gonna to be like this, and let us align what it should be: { # overall policy: TOPOLOGY % 'index "nodes":[ { # Host Numa Node # control by policy TOPOLOGY % 'index:host_info' "host_numa_node": 3, # 0:5 means vcpu 0 pinning to pcpu 5 # control by policy TOPOLOGY % 'index:host_info' "cpu_pinning": {0:5, 1:6}, "vcpu_set": [0,1,2,3], "siblings": [[0,1],[2,3]], "memory_mb": 1024, "pagesize_kb": 4096, "cores": 2, # one core has at least one thread "threads": 2 } ... ], # nodes } links: ptg: https://etherpad.openstack.org/p/nova-ptg-train L334 spec review: https://review.opendev.org/#/c/612256/25/specs/stein/approved/show-server-n… code review: https://review.openstack.org/#/c/621476/ bp: https://blueprints.launchpad.net/nova/+spec/show-server-numa-topology Regards Yongli He

3 2

On reporting CPU flags that provide mitiation (to CVE flaws) as Nova 'traits'
by Kashyap Chamarthy 15 May '19

15 May '19

[When replying, please keep us Cced, I'm not subscribed to the list.] Grab a cup of tea, slightly long e-mail. Some of us in the upstream Nova channel are splitting hairs over this seemingly small issue about potential "security concern" in representing some CPU flags as "traits"[0]. Context ------- The 'os-traits' project lets you report CPU flags as "traits", e.g. to configure a flavor to require the "AVX2" CPU flag: $> openstack flavor set 1 --property trait:HW_CPU_X86_AVX2=required And here's the list of current CPU traits: https://github.com/openstack/os-traits/blob/master/os_traits/hw/cpu/x86.py The other day I casually noticed that the above file is missing some important CPU flags, and proposed a couple of drive-by small changes. And that set off this massive bike-shedding exercise sized Jupier. Some of the following flags provide mitigation from Meltdown/Spectre, others reduce performance degradation, yet others are 'benign' features or are in weird zone like the just-off-the-press (for "ZombieLoad") CPU flag: 'md-clear' -- which doesn't fix the vulnerability, but is a way to report to the operating system that the kernel is opportunistically issuing instructions (which, thankfully, already _exist_ in the CPU) needed for the fix. (See what all these acronyms mean in the rendered QEMU documentation here[1]). - Intel : PCID, STIBP, SPEC-CTRL, SSBD, PDPE1GB, MD-CLEAR - AMD : IBPB, STIBP, VIRT-SSBD, AMD-SSBD, AMD-NO-SSB, PDPE1GB Two things to distinguish ------------------------- It's worth reminding that there are two things here: (a) allowing CPU flags via Nova's config attribute: `[libvirt]/cpu_model_extra_flags`; and (b) allow scheduling of instances based on CPU traits. We're talking about case (b) here. Possible "security issue" ------------------------- As mentioned above, Nova has this ability to say: "DON'T land this instance on a host if it has $TRAIT". So, hypothetically, you can say: "don't land on the host that has AMD-SSBD fix", which can be done (not yet, as the patch[2] is being discussed) in one of the two ways: - Via placement request: `required=!HW_CPU_X86_AMD_SSBD` - From a flavor Extra Spec: `trait:HW_CPU_X86_AMD_SSBD=forbidden` So, theoretically there is scope for "exploiting" (but non-trivial) the above — however, it can be possible only when Nova exposes them via Compute (which it doesn't yet). Contention / unsolved question ------------------------------ Whether we should expose CPU flags (e.g. "SSBD", or "STIBP") that provide mitigation from CPU flaws as traits or not? It is a "policy" decision, and the 'traits' are "forever" (well, you can soft-deprecate them with a comment) once they're added, hence all the belaboring. There's no consensus here. Some think that we should _not_ allow those CPU flags as traits which can 'allow' you to target vulnerable hosts. Some think it is okay to add these as granular CPU traits. (Have a gander at the discussion on this[2] change.) Does the Security Team has any strong opinions? On "generic" traits ------------------- We also discussed whether it makes sense to add "generic roll-up traits" such as 'HW_CPU_HAS_SPECTRE_CURE' and 'HW_CPU_HAS_MELTDOWN_CURE'. However, that seems intuitively appealinty, the messy real world doesn't quite allow that (as there are multiple different Spectre flaws) So, for now we won't do these "generic" triats; but it can be added later, if we change our minds. Next steps ---------- If there is consensus on dropping those CPU-flags-as-traits that let you target vulnerable hosts, drop them. And add only those CPU flags as traits that provide either 'features' (what's the definition?) or those that reduce performance degradation. Otherwise, add all the required CPU flags consistently to 'os-traits', and move on. Another idea ------------ For "Meltdown" (and for other vulnerabilities; it should be case-by-case, based on fix availability), we can potentially make Nova check the 'sysfs' directory for vulnerabilities. And if it reports "Vulnerable" (instead of "Mitigation", as shown below): $> cat /sys/devices/system/cpu/vulnerabilities/meltdown Mitigation: PTI Then we can print a log warning for the current release that the host is vulnerable and warn that future Nova will refuse to run VMs on it, and then next release, make it mandatory. Likewise for "Spectre": $> grep . /sys/devices/system/cpu/vulnerabilities/spectre_* /sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user pointer sanitization /sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling Some think this is not "Nova's business", because: "just like how you don't want to stop based on CPU fan speed or temperature or firmware patch levels ...". But that argument doesn't quite apply, as CPU fan/speed are very different, and are not seen by the guest. If you take security seriously, it _is_ be fair game, IMHO, to make Nova warn (then stop) launching instances on Compute hosts with vulnerable hypervisors. But my umbilical cord isn't tied to this idea, just wanted to mention it for completeness' sake. [0] https://github.com/openstack/os-traits/ [1] https://qemu.weilnetz.de/doc/qemu-doc.html#important_005fcpu_005ffeatures_0… [2] https://review.opendev.org/#/c/655193/ Add CPU traits for Meltdown/Spectre mitigatio -- /kashyap

2 1

[PTL][SIG][WG] PTG Team Photos
by Kendall Nelson 15 May '19

15 May '19

Hello! If your team is attending the PTG and is interested in having a team photo taken, here is the signup[1]! There are slots Thursday and Friday from 10:00 AM to 4:30 PM. The location is TBD but will likely be close to where registration will be. I'll send an email out the day before with a reminder of your time slot and an exact location. -Kendall (diablo_rojo) [1] https://docs.google.com/spreadsheets/d/1DgsRHVWW2YLv7ewfX0M21zWJRf4wUfPG4ff…

5 10

[mistral] Proposing to have Mistral office hours weekly on Wed 8.00 UTC
by Renat Akhmerov 14 May '19

14 May '19

Hi, All people who are interested in any kind of discussions (technical, user questions) around Mistral are invited to participate regular Mistral office hours sessions starting next week. The proposed time slot for now is Wed 8.00 UTC. If you have other suggestions, let’s discuss. The Mistral IRC channel is #openstack-mistral. That doesn’t mean though that you can find us in the channel only within this hour once a week. Usually some of the Mistral contributors are there and available to talk. However, I’d like to renew office hours just so people know that somebody will be there for sure at this time. Since we haven’t had regular meetings for a few months the most important topics I’d like to propose are building a further roadmap and helping new contributors getting up to speed. But anything else is also welcome. Thanks Renat Akhmerov @Nokia

1 0

Open Infrastructure Summit CFP Open - Deadline: July 2
by Ashlee Ferguson 14 May '19

14 May '19

Hi everyone, The Call for Presentations (CFP) [1] for the Open Infrastructure Summit in Shanghai (November 4 - 6, 2019) [2] is open! Review the list of Tracks [3], and submit your presentations, panels, and workshops before July 2, 2019. Sessions will be presented in both Mandarin and English, so you may submit your presentation in either language. The content submission process for the Forum and Project Teams Gathering will be managed separately in the upcoming months. SUBMIT YOUR PRESENTATION [1] - Deadline July 2, 2019 at 11:59pm PT (July 3 at 6:59 UTC) Want to help shape the content for the Summit? The Programming Committee helps select sessions from the CFP for the Summit schedule. Nominate yourself or someone else for the Programming Committee [4] before May 20, 2019. Registration and Sponsorship • Shanghai Summit + PTG registration is available in the following currencies: • Register in USD [5] • Register in RMB (includes fapiao) [6] • Sponsorship opportunities [7] Please email speakersupport(a)openstack.org with any questions or feedback. Thanks, Ashlee [1] https://cfp.openstack.org/ [2] https://www.openstack.org/summit/shanghai-2019 [3] https://www.openstack.org/summit/shanghai-2019/summit-categories [4] http://bit.ly/ShanghaiProgrammingCommittee [5] https://app.eventxtra.link/registrations/6640a923-98d7-44c7-a623-1e2c9132b4… [6] https://app.eventxtra.link/registrations/f564960c-74f6-452d-b0b2-484386d33e… [7] https://www.openstack.org/summit/shanghai-2019/sponsors/ Ashlee Ferguson OpenStack Foundation ashlee(a)openstack.org

1 0