March 2020 - openstack-discuss - lists.openstack.org

Re: [Release-job-failures] Release of openstack/python-openstackclient for ref refs/tags/5.1.0 failed
by Thierry Carrez 25 Mar '20

25 Mar '20

zuul(a)openstack.org wrote: > Build failed. > > - release-openstack-python https://zuul.opendev.org/t/openstack/build/4519cd03309c4b5e834a9089183f6b3b : SUCCESS in 4m 00s > - announce-release https://zuul.opendev.org/t/openstack/build/7a30f87bded640f0a1f6c418d1dcce66 : FAILURE in 4m 34s > - propose-update-constraints https://zuul.opendev.org/t/openstack/build/eb428894cf24407a9991d22f40de98d7 : SUCCESS in 4m 25s During python-openstackclient 5.1.0 release: Error while sending announcement email: 'Temporary local problem - please try later' It's not the first time this happens, so maybe we should have a retry in there. -- Thierry Carrez (ttx)

1 0

[largescale-sig] Next meeting: Mar 25, 9utc
by Thierry Carrez 25 Mar '20

25 Mar '20

Hi everyone, The Large Scale SIG will have a meeting this week on Wednesday, Mar 25 at 9 UTC[1] in #openstack-meeting on IRC: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200325T09 Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting A reminder of the TODOs we had from last meeting, in case you have time to make progress on them: - amorin to create a wiki page for large scale documentation - amorin to propose patch against Nova doc - all to check/comment on https://etherpad.openstack.org/p/LargeScaleOps_OpenDev - all to review new patchset of oslo.metrics spec https://review.opendev.org/#/c/704733/ - oneswig to contribute a scaling story on bare metal cluster scaling Talk to you all on Wednesday, -- Thierry Carrez

1 1

[ops] need help configuring glance policies
by rajiv mucheli 25 Mar '20

25 Mar '20

Hi there, In regards to https://answers.launchpad.net/glance/+question/689450, i need help configuring glance policies. I did review the excellent videos but had no response from the authors over IRC. The 403 errors occur with admin_role = '' Please suggest how to proceed. Regards, Rajiv

1 0

10 Years of OpenStack Survey
by Sunny Cai 24 Mar '20

24 Mar '20

Hello Stackers! To celebrate 10 years of OpenStack, we are launching a campaign that not only celebrates the community’s achievement in the past 10 years, but also showcases the top 10 favorite things about OpenStack from our community members. We want to hear from you about what your top 10 favorite things related to OpenStack are! Go to this survey <https://openstackfoundation.formstack.com/forms/10_years_of_openstack> and choose one or more topics to answer. The topics range from your top 10 most memorable moments of OpenStack, your top 10 most used features in OpenStack to your top 10 favorite cities you visited for OpenStack. Top 10 favorites of OpenStack survey link: https://openstackfoundation.formstack.com/forms/10_years_of_openstack <https://openstackfoundation.formstack.com/forms/10_years_of_openstack> Please also share it to your networks including folks who were around at the beginning of OpenStack. We are looking forward to hearing your favorites, and we invite you all to join us and celebrate 10 awesome years of OpenStack! Thanks, Sunny Cai OpenStack Foundation sunny(a)openstack.org

1 0

[baremetal-sig][ironic] Lets finish the whitepaper doodle
by Julia Kreger 24 Mar '20

24 Mar '20

Greetings fellow humans! During today's ironic meeting, the idea of just getting on a video call and hammering out the remainder of the baremetal SIG whitepaper was raised. After some quick discussion in the community, I'm sending out a doodle[0] for us to find a few time windows that works, where we can try to gather together and hammer out words, cut/paste/massage content, and overall just try and get it done. In the next 24-48 hours, I'll contact the respondents to and schedule a call, where I'll also send out call details. Please respond to the doodle quickly. -Julia [0] https://doodle.com/poll/k4wnmuay34mvh94v

1 2

nova-api missing heartbeats to rabbitmq
by Satish Patel 24 Mar '20

24 Mar '20

Folks, Recently i am seeing lots of error message in rabbitmq logs saying missing heartbeats from nova-api nodes, I am not seeing any issue at functionality level as everything working fine but just noticed those error and trying to find root cause of it. 172.28.15.125 nova-api server 172.28.15.192 rabbitmq server on rabbit.log 2020-03-24 12:21:41.389 [error] <0.29772.4418> closing AMQP connection <0.29772.4418> (172.28.15.125:42656 -> 172.28.15.192:5671 - uwsgi:32419:9b8a323b-653d-4585-9916-d52b3fd81d59): missed heartbeats from client, timeout: 60s on nova-api.log 2020-03-24 12:19:06.554 32435 ERROR oslo.messaging._drivers.impl_rabbit [-] [4b8adff0-ff9f-4863-a939-537d391e5d9e] AMQP server on 172.28.15.192:5671 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer

4 3

[tc] April 2nd meeting agenda
by Jean-Philippe Evrard 24 Mar '20

24 Mar '20

Hello folks, If you have to update the agenda of our next monthly meeting, it's still time! I will send the definitive agenda on the ML Thursday. Regards, JP [1]: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[neutron] OVS inactivity probes
by James Denton 24 Mar '20

24 Mar '20

Hello all, Like others, we have seen an increase in the amount of messages like those below, followed up by disconnects: 2020-03-19T07:19:47.414Z|01614|rconn|ERR|br-int<->tcp:127.0.0.1:6633: no response to inactivity probe after 10 seconds, disconnecting 2020-03-19T07:19:47.414Z|01615|rconn|ERR|br-ex<->tcp:127.0.0.1:6633: no response to inactivity probe after 10 seconds, disconnecting 2020-03-19T07:19:47.414Z|01616|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 10 seconds, disconnecting We have since increased the value of of_inactivity_probe (https://bugs.launchpad.net/neutron/+bug/1817022) and confirmed the Controller connection reflects this new value: --- # ovs-vsctl list Controller ... _uuid : b4814677-c6f3-4afc-9c9e-999d5a5ac78f connection_mode : out-of-band controller_burst_limit: [] controller_rate_limit: [] enable_async_messages: [] external_ids : {} inactivity_probe : 60000 is_connected : true local_gateway : [] local_ip : [] local_netmask : [] max_backoff : [] other_config : {} role : other status : {last_error="Connection refused", sec_since_connect="1420", sec_since_disconnect="1423", state=ACTIVE} target : "tcp:127.0.0.1:6633" --- However, we also see disconnects on the manager side, which the config option does not address: 2020-03-23T11:01:02.871Z|00443|reconnect|ERR|tcp:127.0.0.1:50098: no response to inactivity probe after 5 seconds, disconnecting This bug (https://bugs.launchpad.net/neutron/+bug/1627106) and related commit (https://opendev.org/openstack/neutron/commit/1698bee770b84a2663ba940a6ded5d…) appear to leverage the ovs_vsctl_timeout value (since renamed to ovsdb_timeout), but the inactivity_probe for the Manager connection does not appear to be implemented. Honestly, I'm not sure if that code path is used. --- # ovs-vsctl list Manager _uuid : d61519ba-93fc-4fe5-b05c-b630778a44b0 connection_mode : [] external_ids : {} inactivity_probe : [] is_connected : true max_backoff : [] other_config : {} status : {bound_port="6640", n_connections="2", sec_since_connect="0", sec_since_disconnect="0"} target : "ptcp:6640:127.0.0.1" --- Running this by hand sets the inactivity_probe timeout on manager connection, but we'd prefer to use a built-in method, if possible: # ovs-vsctl set manager d61519ba-93fc-4fe5-b05c-b630778a44b0 inactivity_probe=30000 Any suggestions? Thanks, James

2 2

[glance] weekly meeting IRC channel change
by Abhishek Kekane 24 Mar '20

24 Mar '20

Hi All, Beginning with this Thursday (26 March), the Glance team weekly meeting will be held in: #openstack-meeting The time continues to be 1400 UTC on Thursdays. Thanks & Best Regards, Abhishek Kekane

1 0

[nova][ptg] PTG participation
by Balázs Gibizer 24 Mar '20

24 Mar '20

Hi, I've just got the news from my employer that due to COVID19 I cannot travel to Vancouver in June. cheers, gibi

3 4

[nova] feasibility to keep the two-company rule
by Balázs Gibizer 24 Mar '20

24 Mar '20

Hi, Nova has an unwritten rule that requires to have at least two companies involved in any new feature development (or even bugfix?). In the current Nova core diversity situation this rule puts extra burden to the remaining non Red Hat cores and I guess it also makes any Red Hat driven feature development harder. In parallel to working on increasing the size of the core team I suggest to reconsider dropping this rule. Some discussion happened already on the today's meeting[1]. Cheers, gibi [1] http://eavesdrop.openstack.org/meetings/nova/2020/nova.2020-03-19-16.00.log…

10 10

[nova] FYI for out-of-tree virt drivers : new argument for finish_migration() and finish_revert_migration()
by Sylvain Bauza 24 Mar '20

24 Mar '20

I wrote two changes [1] for passing down target allocations to virt drivers given we need to use them for closing some bugfix [2] Given the Nova contributor policy is saying that the public API is not related to virt drivers, we won't support virt drivers that won't accept this new argument. If you're a contributor that works on any out-of-tree Nova virt driver, please make sure that you modify your own two above methods. Thanks, -Sylvain [1] https://review.opendev.org/#/c/589085/7 and https://review.opendev.org/#/c/712118/2 [2] https://bugs.launchpad.net/nova/+bug/1778563

2 1

Summary: Ironic Mid-cycle at CERN
by Julia Kreger 23 Mar '20

23 Mar '20

A couple weeks ago (February 25-26), the Ironic community convened its first mid-cycle in quite a long time at the invitation of and encouragement of CERN. A special thanks goes to Arne Wiebalck for organizing the gathering. We spent two days discussing, learning, sharing, and working together to form a path forward. As long time contributors, some of us were able to bring context of not just how, but why. Other community members brought questions and requirements, meanwhile one of the hardware vendors brought their context and needs. And CERN was kind enough to show us how our work matters and makes a difference, which was the most inspiring part of all! Special thanks goes to Dmitry Tantsur, Riccardo Pittau, and Iury Gregory for helping me keep momentum moving forward on this summary. --------------------------------------- Deploy Steps ========== Discussed issues related to the actual workflow, concerns out process efficiency and path forward. The issue in question was the advance validation of deploy steps when some of the steps come from the ironic-python-agent ramdisk and are not reflected in the server code. Creating the whole list of steps for validation and execution requires information from the ramdisk, but it’s only available when the ramdisk is booted. We have discussed the following alternatives: * Start ramdisk before deploy step execution. This was ruled out for the following reasons: ** Some steps need to be executed out-of-band before the ramdisk is running. This is already an issue with iDRAC clean steps. ** The first deploy steps validation happens in the node validation API when the ramdisk is clearly not running. * Using cached deploy steps from the previous cleaning run. It was ruled out because: ** Some deployments disable automated cleaning. ** The deploy step list can change in between, e.g. because of hardware changes or any external input. * Accept that we cannot provide early validation of deploy steps and validate them as we go. This involves booting the ramdisk as one of the deploy steps (no special handling), with only out-of-band steps executed before that (ignoring any in-band steps with higher priority). We decided to go with the third option. In a side discussion we decided to file an RFE for a driver_info flag preventing booting the ramdisk during manual cleaning. Solving the cleaning issues completely probably requires making booting the ramdisk a separate clean step, similarly to deploy steps above. No final plan has been made for it, but we have more clarity than we did before. Security ====== Firmware Management -------------------------------- We entered a discussion about creating a possible “meta step”. After some back and forth discussions, we reached a consensus that it is likely not possible given different vendor parameters and requirements. During this topic, we also reached the point of discussing changes to “Active node” configuration as it is related in large part to firmware updates, and is necessary for larger fleet management, and eventually even for attestation process integration. The consensus kind of revolved around that the way to help enable some of this process was to leverage rescue, this is only theory. Hopefully we’ll have operator feedback in the next year on this subject and can make more informed decisions. By then, we should have deploy steps in a form that one doesn't need to be a python developer to leverage, and the team should be bandwidth to explore this further with operators. Attestation --------------- This is a topic that Julia has been raising for a while because there is a logical and legitimate reason to go ahead and implement some sort of integration with an attestation platform to perform system measurement and attestation during cleaning and deployment processes in order to help identify if machines were tampered with. In our case, remote attestation is likely the way forward, and inspiration can come looking at Keylime (TPM based highly boot attestation and run-time integrity measurement solution, and most important, opensource). We’ll need an implementation to cover at least Clean/Deploy steps, to be able to run and validate TPM measurement, and fail the deployment if attestation fails. We still need to figure out what the actual impact to firmware upgrade is, and how to safely ensure that a re-measurement is valid, or not, and when to trust the measurement is actually valid. Ironic’s next step is to begin to talk to the Keylime folks in more depth. Also one of our contributors, Kaifeng, who read our notes etherpad indicated that he is working towards the same area as well, so we may see some interesting and fruitful collaboration because ultimately we all have some of the same needs. Agent Tokens ------------------ Agent tokens was possibly the quickest topic that we visited with Dmitry suggesting we just needed to add a unit test and merge the code. To be further secure, we need the agent ramdisk to begin using TLS. TODO: Julia is to send out an email to the mailing list to serve as notice to operators that ironic intends to break backwards IPA compatibility next cycle by removing support for agents that do not support agent tokens. NOTE: As we type/refine this summary for distribution, the agent token code has largely merged, and should be completely merged before the end of the current development cycle. TLS in virtual media --------------------------- In order to secure Agent token use, we need to secure their transmission to the ironic-python-agent when commands are issued to the agent from the conductor. Ultimately we’re hoping to begin work on this soon in order to better secure interactions and communications with machines in remote “edge” environments. An RFE has been filed to automatically generate certificates and exchange them with the ramdisk: https://storyboard.openstack.org/#!/story/2007214. Implementing it may require downstream consumers to update their USA export control certification. FIPS 140-2 --------------- This was a late addition topic that was added while other discussion was coming up, and largely more for the purposes of community visibility. In short, we know based on some recent bugs that were fixed, that operators are starting to try and deploy Ironic in environments and on hosts that are configured for a FIPS 140-2 operating mode, in short a much more strict cryptography configuration. We ought to make sure that we don’t have any other surprises waiting for us, so the call went out for someone to at some point review the standard, and sanity check Ironic and components. Post-IPMI universe =============== The decline of IPMI is one that we, as a community, need to plan ahead for as some things become a little more difficult. Discovery ------------- Node discovery, as a feature, is anticipated to become a little more complicated. While we should still be able to identify a BMC address,that address may be the in-band communications channel address once vendors are supporting the Redfish host interface specification. This spurred discussion of alternatives, and one of the items that was raised was possibly supporting the discovery of BMCs using the SSDP and UPnP. This raises an interesting possibility in that the UUID of the BMC is retrievable through these means. It seems logical for us to one day consider the detection and enrollment of machines using an operator tool of some sort. This functionality is defined by the Redfish standard and, as everything in Redfish, is optional. The DMTF provided Redfish library contains an example implementation: https://github.com/DMTF/python-redfish-library/blob/master/src/redfish/disc…. Using System ID/GUID ------------------------------- The discovery topic spurred another topic of if we should be using the System UUID or GUID identifier in addition to or instead of a MAC address on the chassis. Ironic Inspector folks have been considering having additional or even plug-able matches for a long time. The system UUID can be discovered in-band via SMBios before launching inspection/discovery. Supporting this would largely be more of a feature matching a physical machine, but some of our code requires network information anyway, so it may not bring a huge benefit upfront beyond trying to match BMC<->Host. IPMI cipher changes ---------------------------- CERN folks were kind enough to raise an issue that has brought themselves some headaches as of recent which is that some BMC vendors have changed cipher logic and keying so they’ve had to put in some workarounds and modified ipmitool builds on their systems. As far as we’re aware as a community, there is really nothing we can directly do to help them remove this work around, but ultimately this headache may cause them to begin looking at Redfish and cause some development on serial console support for Redfish. Making inspector data a time series =========================== One of the challenges in data centers is identifying when the underlying hardware changes. When a disk is replaced, the serial number changes, and if that disk is in a running system, it would traditionally have needed to be re-inspected in order for information about the machine to be updated. But we added the ability to manually execute and submit this data in the last development cycle, so if there was any time series nature to inspection data, then it allows for changes to be identified, new serial numbers recorded, etc. The overwhelming response during the discussion was “Yes Please!”, in that such a feature would help a number of cases. Of course, what we quickly reached was disagreement over meaning. Turns out, the purpose is more about auditing and identifying changes, so even if there are only two copies, the latest and the previous inspection data, then differences could be identified by external tooling. A spec document or some sort of written MVP will ultimately be required, but the overall concept was submitted to Outreachy. DHCP-less deploy ============== In regard to the DHCP-less deploy specification (https://review.opendev.org/#/c/672780/) we touched upon several areas of the specification. We settled on Nova's network metadata format (as implemented by Glean) as the API format for this feature. Ilya has voiced concerns that it will tie us to Glean closer than we may want. Scalability of rebuilding ISO images per node. The CERN folks rightfully expressed concern that a parallel deployment of several hundred nodes can put a significant load on conductors, especially in terms of disk space. * In case of hardware that has more than one usable virtual media slot, we can keep the base ISO intact and use a 2nd slot (e.g. virtual USB) to provide configuration. * The only other option is documenting it as a limitation of our virtual media implementation. To get rid of the MAC address requirements for DHCP-less virtual media deployments, we determined that it will be necessary to return to sending node UUID and other configuration to the ramdisk via boot parameters. This way we can avoid the requirement for MAC addresses, although this has to be navigated carefully and with Operator feedback. An additional concern, beyond parallel deployment load, was “Rescue” support. The consensus seemed to be that we put giant security warnings in the documentation to signal the security risk of the ramdisk being exposed to a potentially un-trusted network. Agent token work _does_ significantly help improve operational security in these cases, but operators must be cognizant of the risks and potentially consider that rescue may be something they might not want to use under normal circumstances with network edge deployments. OOB DHCP-less deploy =================== We briefly touched on OOB dhcp-less deployment. This is HTTPBoot asserted through to the BMC with sufficient configuration details, ultimately looking a lot like DHCP-less deployments. Interest does seem to exist on this topic, but we can revisit once the base DHCP-less deployment work is done and hopefully an ironic contributor has access hardware where this is an explicit feature of the BMC. CI Improvements and Changes ======================== The upstream CI, and ways to make it more stable and just improve its efficiency, is a recurrent main discussion argument not only as part of the meetup. The impact of the CI in the day-to-day work is very high, and that’s why we took our time to talk about different aspects and do a proper analysis of the different jobs involved. The discussion started with the proposal of reducing the usage of the ironic-python-agent images based on TinyCoreLinux (the so-called tinyipa images), and rely more on the images built using diskimage-builder (DIB) and specifically CentOS 8 as base. This proposal is based on the fact that DIB-built images are what we recommend for production usage, while tinyIPA images have known issues on real hardware. Their only real benefit is a much smaller memory footprint (roughly 400MiB vs 2GiB of a CentOS 8 image). We have agreed to switch all jobs that use 1 testing VM to pre-built CentOS 8 images. This covers all jobs, except for the standalone, multi-node and grenade (upgrade testing) ones. While reviewing the current list of jobs, we realized that there is a lot of duplication between them. Essentially, most of the image type - deploy interface combinations are already tested in the standalone job. As these combinations are orthogonal to the management technology, we can use Redfish instead of IPMI for some of the tests. We decided to split the ironic-standalone job since it covers a lot of scenarios from tempest and it has a high failure rate. The idea to have one job testing software raid, manual cleaning and rescue, the other tests that consists in the combination of image type - deploy interface will be split in two jobs (one using IPMI and the other using Redfish). One other point that we reached was some consensus that more exotic, non-openstack focused CI jobs, were likely best to be implemented using Bifrost as opposed to Tempest. Third Party CI/Driver Requirements ----------------------------------------------- The question has been raised with-in the community if we reconsider 3rd Party CI requirements. For those that are unaware, it has been a requirement for drivers to merge into ironic to have Third-Party operated CI. Operating Third Party CI helps exercise drivers to ensure that the driver code is functional, and provides the community information in the event that there is a breaking changer or enhancement made. The community recognizes third party CI is difficult, and can be hard at times to keep working as the entire code base and dependencies evolve as time moves on. We discussed why some of these things are difficult, and what can we, and the larger community do to try and make it easier. As one would expect, a few questions arose: Q: Do we consider “supported = False” and keeping drivers in-tree until we know they no longer work? A: The consensus was that this is acceptable. That the community can keep unit tests working and code looking sane. Q: Do we consider such drivers as essentially frozen? A: The consensus is that drivers without third party CI will be functionally frozen unless changes are required to the driver for the project to move forward. Q: How do we provide visibility into the state of the driver? A: The current thought is to return a field in the /v1/drivers list to signal if the driver has upstream testing. The thought is to use a field named “Tested” or something similar as opposed to the internal name in the driver interface which is “supported” Q: Will we make it easier to merge a driver? A: Consensus was that we basically want to see it work at least once before we merge drivers. It was pointed out that this helped provide visibility with some of the recently proposed driver code which was developed originally against a much older version of ironic. Q: Do third party CI systems need to run on every patch? A: Consensus is No! A number of paths in the repository can be ignored from changes. In other words, there is no reason to trigger an integration test of Third Party CI for a documentation change or an update to a release note, or even a change to other vendor’s drivers. In summary Drivers without Third Party CI are “use at own risk” and removal is moving towards a model of “don’t be brutal”. This leaves us with a number of tasks in the coming months: * Update the contributor documentation with the Questions and Answers above. * Author an explicit exception path for the process of bringing CI back up as it pertains to drivers, essentially focusing on communication between the ironic community and driver maintainers. * Author a policy stating that unsupported drivers shall be removed immediately upon being made aware that the driver is no longer functional and without a clear/easy fix or path to resolution. * Solicit pain points from driver maintainers who have recently setup or do presently maintain Third Party CI and try to aggregate the data and maybe find some ways of improving the situation. “Fishy Politics”: Adapting sushy for Redfish spec versus implementation reality ============================================================ Everyone’s favorite topic is how implementations differ from specification documents. In part, the community is increasingly seeing cases where different vendors have behavior oddities in their Redfish implementations. We discussed various examples, such as https://storyboard.openstack.org/#!/story/2007071, and the current issues we’re observing on two different vendors around setting the machine boot mode and next boot device at the same time. For some of these issues, the idea of having some sort of Redfish flavor indicator suggested so the appropriate plugin could be loaded which might be able to handle larger differences like major field name differences, or possibly endpoint behavior differences like “PUT” instead of “PATCH”. This has not yet been explored but will likely need to be explored moving forward. Another item for the ironic team to be mindful moving forward, is that newer UEFI specific boot setting fields have been created, which we may want to explore using. This could give us a finer level of granularity of control, and at the same time may not be really usable in other vendor’s hardware due to the data in the field and how or what to correspond it back to. Kexec (or "Faster booting, yes?") ========================= This topic concerns using the kexec mechanism instead of rebooting from the ramdisk to the final instance. Additionally, if it is acceptable to run the agent on the user instance, it can be used for rebuilding and tearing down an instance. Potentially saving numerous reboots and "Power On Self Tests" in the process. We have one potential issue: with multi-tenant networking there is a possibility of a race between kexec and switching from the provisioning to the tenant network(s). In a normal deploy we avoid it by powering the node off first, then flipping the networks, then booting the final instance (on success). There is no such opportunity for kexec, meaning that this feature will be restricted to flat network cases. The group has expressed lots of interest in providing this feature as option for advanced operators, in other words “those who run large scale computing farms”. Julia proposed making a demo of super-fast deployment using fast-track and kexec as a goal moving forward and this received lots of positive feedback. Partitioning, What is next? ==================== Dmitry expressed that there seemed to be lots of interest from EU operators in supporting disk partitioning. This has been long sought, but with minimal consensus. We discussed some possible cases how this could be supported and we reached conclusion that the basis is largely just to support Linux Logical Volume Manager in the simplest possible configuration. At the same time the point was raised that parity basically means some mix of software RAID through LVM and UEFI boot. We soon realized we needed more information! So the decision was reached to start by creating a poll, with questions in three different languages to try and identify community requirements using some simple and feasible scenarios. Such as LVM on a single disk, LVM on multiple disks, LVM + image extraction on top of the LVM. The partitioning topic was actually very power in that we covered a number of different topics that we were not otherwise planning to explicitly cover. Network booting ---------------------- One of them being why not use network booting, and the point was made that Network is our legacy and fundamental for iSCSI based boot and ramdisk booting (such as the deployment ramdisk). During this dive into ironic’s history, we did reach an important point of consensus which is that Ironic should switch the default boot mode, as previously planned, and still keep at least one scenario test running in CI which uses network booting. Stated operator wants in terms of Partitioning ------------------------------------------------------------- Dmitry was able to provide us some insight into what the Russian operator community was seeking from Ironic, and Julia confirmed she had heard similar wants from public cloud operators wanting to offer Bare Metal as a Service. Largely these wants revolve around wanting LVM capability of the most basic possible scenario such as a single disk or a partition image with a LVM, or even Software RAID with partition images. Likely what has stalled some of these discussions in the past is the immediate focus on the more complex partitioning scenarios sought by some operators in the past, which has resulted in these discussions stalling due to complexity of requirements. Traits/Scheduling/Flavor Explosion =========================== Arne with CERN raised this topic to bring greater awareness. CERN presently has greater than 100 flavors representing their hardware fleet as each physical machine type gets its own flavor. This has resulted in pain from the lack of flavor specific quotas. What may help in this area is for Resource Class based quotas, but presently the state of that work is unknown. The bottom line: A user does not have clarity into their resource usage. The question then shifted to being able to report utilization since the current quota model is based on cores/RAM/instances but not resource_class consumption as a whole. The question largely being “How many am I allowed to create? [before I eat someone else’s capacity]”. https://github.com/stackhpc/os-capacity was raised as a reporting tool that may help with these sorts of situations with bare metal cloud operators. Another point raised in this discussion was the lack of being able to tie consumer and project ID to consumed resources, but it turns out the allocation list functionality in Placement now has this functionality. In the end of this discussion, there was consensus that this should be brought back to the Nova community radar. Machine Burn-in ============= Burning in machines is a regular topic that comes up, and it looks like we’re getting much closer to being able to support such functionality. Part of the reason behind discussing this was to determine how to move forward what organizations like CERN can offer the community. There are two use cases. The most important thing is to ensure that hardware does not fail. That doesn’t seem like a “cloudy” thing to have to worry about, but when you're building your cloud, you kind of want to make sure that suddenly half your hardware is not going to fail as soon as you put a workload on it. The second use case is nearly just as important, which is to be able to ensure that you are obtaining the performance you expect from the hardware. This brought a bit of discussion because there are fundamentally two different paths that could be taken. The first is to leverage the inspector, whereas the second is to use clean steps. Both have very useful possible configurations, largely there is no sophisticated data collection in terms of performance data. That being said, the consensus seemed to be that actual data collection was less of a problem than flexibility to invoke as part of cleaning and preparation of a machine into the deployment. In other words, the consensus seemed to be clean-steps would be ideal for community adoption and code acceptance. IPv6/Dual Stack ============ TL;DR, we need to remove the ip_version setting field. This is mostly a matter of time to sift through the PXE code, and determine the code paths that need to be taken. I.E. for IPv6, we would likely only want to signal flags for it if the machine is in UEFI mode. The dhcp-less work should provide some of the API side capabilities this will really need in terms of interaction with Neutron. Graphical Console ============== The question was raised to “what would it take to finally get this moving forward again?” This is because there is initial interface code, two proofs of concept, and it should be relatively straightforward to implement redfish support, OEM capability dependent. The answer was functionally “Someone needs to focus on this for a couple of months, keep it rebased, and engage the community”. The community expressed an absolute willingness to mentor. Software RAID - Specifying devices =========================== We have briefly discussed RFE https://storyboard.openstack.org/#!/story/2006369 that proposes a way to define which physical devices participate in software RAID. A similar functionality already exists in the RAID configuration format for hardware RAID, but software RAID always spans all hard drives, no matter how many. The RFE proposes re-using the same dictionary format as used for root device hints in the “physical_disks” field of the RAID configuration. This idea has been accepted by the audience, with Arne proposing to extend supported hints with a new “type” hint with values like “rotational”, “nvme” or “ssd”. Stickers ====== Yes, we really did discuss next steps for stickers. We have ideas. Lots of ideas… and we are all very busy. So we shall see if we’re able to make some awesome and fun stickers appear for the Berlin time frame.

2 1

[tempest][gate] tempest-tox-plugin-sanity-check broken by whitebox-tempest-plugin
by Artom Lifshitz 23 Mar '20

23 Mar '20

Hey all, A recent change in the whitebox-tempest-plugin [1] has broken the tempest-tox-plugin-sanity-check job (ex: [2]). The cause is a skipCheck that uses a whitebox-specific config option that had no default value and is obviously not set in any jobs outside of whitebox itself. When attempting to compare a None and an int, a TypeError is thrown. This has been fixed in [3]. The disruption should be minimal, but please don't: 1. recheck tempest until [3] merges 2. blacklist whitebox from the tempest-tox-plugin-sanity-check job :) Cheers! [1] https://review.opendev.org/#/c/699864/ [2] https://review.opendev.org/#/c/711049/8 [3] https://review.opendev.org/#/c/714539/

2 2

[cinder] reminder: weekly meeting location change
by Brian Rosmaita 23 Mar '20

23 Mar '20

(If you missed last week's meeting, this email may be the first you've heard about this change.) Beginning with this Wednesday (25 March), the Cinder team weekly meeting will be held in: #openstack-meeting-alt The time continues to be 1400 UTC on Wednesdays. cheers, brian

1 0

[neutron] Week 13 - bug deputy report
by Slawek Kaplonski 23 Mar '20

23 Mar '20

Hi, Last week I was on bug deputy. Below is my summary of it. *Critical* None *High* https://bugs.launchpad.net/neutron/+bug/1867936 - test_update_delete_extra_route failing due to timeout when creating subnets - HIGH as it’s gate failure This has no assignment yet. For me it looks like generall issue which we have from time to time with some very slow responses from API but it has to be verified. https://bugs.launchpad.net/neutron/+bug/1868237 - [tempest] Error in "test_multicast_between_vms_on_same_network" test case - HIGH as this is gate-failure No assignment yet. https://bugs.launchpad.net/neutron/+bug/1868110 - [OVN] neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log randomly fails - HIGH as it’s gate failure maciejjozefczyk is working on it *Medium* https://bugs.launchpad.net/neutron/+bug/1868100 - ncat process isn't always spawned on guest vm in scenario tests - MEDIUM Already fixed https://bugs.launchpad.net/neutron/+bug/1868125 - [ovn] Metadata agent spawns haproxy quickly twice on a single new port binding - MEDIUM Already fixed https://bugs.launchpad.net/neutron/+bug/1868137 - default geneve overhead is set not enough - MEDIUM Someone from ovn subteam can take a look *Low* https://bugs.launchpad.net/neutron/+bug/1867869 - [OVN] Remove dependency on fip_object - LOW Patch proposed https://review.opendev.org/#/c/713585/ *Incomplete* https://bugs.launchpad.net/neutron/+bug/1868098 - Neutron openvswitch agent error - INCOMPLETE But would be good if someone can take a look *Stadium projects* https://bugs.launchpad.net/neutron/+bug/1868515 - vpnaas ike policy is not allowed to be updated No assignment yet. Needs someone from vpnaas team to take a look. -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][privsep] Migrate from oslo.rootwrap to oslo.privsep
by Rodolfo Alonso 22 Mar '20

22 Mar '20

Hello all: With this mail I would like to propose the goal to move away from oslo.rootwrap and migrate to oslo.privsep. The last one offers a superior security model, faster and more secure. During the last cycles and since privsep was released, the Community has encouraged the usage of privsep and the deprecation of any existing code still using rootwrap. For any developer willing to collaborate, there are plenty of code examples, as I’ll provide later, implementing and using privsep for new methods and migrations. If this goal is approved, I'll open a Story (https://storyboard.openstack.org/) and any developer will be able to add a task for each patch or set of them related. This would be the tracker for this common effort. PROJECTS TO MIGRATE. -------------------- Projects that are still using rootwrap: http://codesearch.openstack.org/?q=rootwrap&i=nope&files=.*.py&repos= neutron os-brick designate cinder ironic-inspector neutron-vpnaas nova solum glance_store ironic zun magnum manila networking-bagpipe sahara ceilometer cinderlib freezer ironic-lib monasca-agent tacker tripleo-common USAGE DOCUMENTATION ABOUT PRIVSEP. ---------------------------------- How to create a privsep context, assign privileges and use it as a decorator: https://docs.openstack.org/oslo.privsep/latest/user/index.html HOW TO MIGRATE FROM ROOTWRAP TO PRIVSEP. ---------------------------------------- >From the same link provided previously, in the section “Converting from rootwrap to privsep”: https://docs.openstack.org/oslo.privsep/latest/user/index.html#converting-f… oslo.privsep provides a class, PrivContext, that can be used to create a decorator function. The instance created is a context of execution and has defined a list of capabilities, matching the Linux capabilities. The privsep context decorator should contain the minimum needed capabilities to execute the decorated function. For example: default = priv_context.PrivContext( __name__, cfg_section='privsep', pypath=__name__ + '.default', capabilities=[caps.CAP_SYS_ADMIN, caps.CAP_NET_ADMIN, caps.CAP_DAC_OVERRIDE, caps.CAP_DAC_READ_SEARCH, caps.CAP_SYS_PTRACE], ) The function “entrypoint” of this instance can be used as a decorator for another function: @privileged.default.entrypoint def delete_interface(ifname, namespace, **kwargs): _run_iproute_link("del", ifname, namespace, **kwargs) As commented in the given link, a straight 1:1 filter:function replacement generally results in functions that are still too broad for good security. It is better to replace each chmod rootwrap call with a narrow privsep function that will limit it to specific files. MIGRATION EXAMPLES. ------------------- Nova: https://review.opendev.org/#/q/project:openstack/nova+branch:master+topic:m… Neutron: https://review.opendev.org/#/q/status:merged+project:openstack/neutron+bran… os-vif: https://review.opendev.org/#/c/287725/ Thank you and regards.

6 7

[qeeens][neutron] migrating from iptables_hybrid to openvswitch
by Ignazio Cassano 22 Mar '20

22 Mar '20

Hello All, I am facing some problems migrating from iptables_hybrid frirewall to openvswitch firewall on centos 7 queens, I am doing this because I want enable security groups logs which require openvswitch firewall. I would like to migrate without restarting my instances. I startded moving all instances from compute node 1. Then I configured openvswitch firewall on compute node 1, Instances migrated from compute node 2 to compute node 1 without problems. Once the compute node 2 was empty, I migrated it to openvswitch. But now instances does not migrate from node 1 to node 2 because it requires the presence of qbr bridge on node 2 This happened because migrating instances from node2 with iptables_hybrid to compute node 1 with openvswitch, does not put the tap under br-int as requested by openvswich firewall, but qbr is still present on compute node 1. Once I enabled openvswitch on compute node 2, migration from compute node 1 fails because it exprects qbr on compute node 2 . So I think I should moving on the fly tap interfaces from qbr to br-int on compute node 1 before migrating to compute node 2 but it is a huge work on a lot of instances. Any workaround, please ? Ignazio

5 31

Diskimage-builder complex disk setup
by Stephen Nemeth 21 Mar '20

21 Mar '20

Hi all, We’re attempting to create user images to deploy with metal^3/ironic. I’m making almost no headway starting from any of the examples provided for the DIB_BLOCK_DEVICE_CONFIG setup. Is there a repository with practical values located anywhere? I’m finding the documentation insufficient to do anything more complicated than a single partition setup. Here’s what I’ve gotten so far. Help greatly appreciated: ``` [ { "local_loop": { "name": "image0", "size": "20GiB" } }, { "partitioning": { "base": "image0", "label": "gpt", "partitions": [ { "name": "ESP", "type": "EF00", "size": "16MiB" }, { "name": "boot", "type": "EF02", "size": "6GiB" }, { "name": "lvm", "type": 8, "size": "100%" } ] } }, { "lvm": { "name": "lvm2", "pvs": [ { "name": "pv", "options": [ "--force" ], "device": "lvm", "base": "image0" } ], "vgs": [ { "name": "vg", "base": [ "pv" ], "options": [ "--force" ] } ], "lvs": [ { "name": "root", "base": "vg", "extents": "100%FREE", "options": [ "--zero=n" ] } ] } }, { "mkfs": { { "name": "mkfs_root", "base": "root", "type": "btrfs", "label": "root", "opts": "-f", "mount": { "mount_point": "/" } }, { "name": "mkfs_efi", "base": "ESP", "type": "vfat", "label": "efi", "opts": "-f", "mount": { "mount_point": "/boot/efi/" } }, { "name": "mkfs_boot", "base": "boot", "type": "btrfs", "label": "boot", "opts": "-f", "mount": { "mount_point": "/boot/" } } } } ] ```

2 1

[tripleo] launchpad bug policy
by Wesley Hayutin 20 Mar '20

20 Mar '20

6 8

[neutron] How to fix break of connectivity in case of L3 HA after reboot of backup node
by Slawek Kaplonski 20 Mar '20

20 Mar '20

Hi, We have bug [1] to solve. Basically, when node which is backup node for some router, connectivity to external gateway may be broken for some time. It's like that because when host is up and L3 agent is configuring qrouter namespace, it flush all IPv6 addresses from the qg- interface. And due to that some MLDv2 packets are sent from this interface which may cause that ToR switch will learn qg port's mac address on wrong (backup) node. This don't happens every time and for every router because it is a race between L3 agent and OVS agent. When L3 agent creates qg interface in br-int, it sets tag 4095 for it and traffic sent with such vlan tag is always dropped in br-int. So if L3 agent will flush IPv6 addresses before OVS agent wires the port and sets correct tag for it, then all is fine. But if OVS agent is first, then MLDv2 packets are sent to the wire and there is this connectivity break. There are proposed 2 ways of fixing this: - [2] which propsoes to add some kind of "communication" between L3 agent and OVS agent and tell OVS agent that tag can be changed only after IPv6 config is finished by L3 agent. Downside of this solution is that it works for OVS agent only, Linuxbridge agent may still hit the same issue. But plus is that after initial configuration of the router, everything else regarding to failover is handled by keepalived only - in same way like it is now. - [3] which sets qg NIC to be DOWN always on backup nodes. So when keepalived failover active router to new node, L3 agent needs to come and switch interfaces to be UP before it will work. The plus of this solution is that it works for all OVS and Linuxbridge L2 agents (and probably for others too) but downside is that failover process is a bit longer and there may be potentially another race condition between L3 agent and keepalived. Keepalived tries to sent gARP packets after switch node to be active, first attempt will always fail as interface is still DOWN. But keepalived will retry those gARPs after some time and this should be fine if L3 agent will already bring interface to be UP. Both patches are waiting for pretty long time in gerrit and I want to bring more visibility for both of them. Please check them and maybe You will have some opinions about which solution would be better and which we should go with. [1] https://bugs.launchpad.net/neutron/+bug/1859832 [2] https://review.opendev.org/#/c/702856/ [3] https://review.opendev.org/#/c/707406/ -- Slawek Kaplonski Senior software engineer Red Hat

5 4

Neutron RabbitMQ issues
by Grant Morley 20 Mar '20

20 Mar '20

Hi all, We are currently experiencing some fairly major issues with our OpenStack cluster. It all appears to be with Neutron and RabbitMQ. We are seeing a lot of time out messages in responses to replies and because of this instance creation or anything to do with instances and networking is broken. We are running OpenStack Queens. We have already tuned Rabbit for Neutron by doing the following on neutron: heartbeat_timeout_threshold = 0 rpc_conn_pool_size = 300 rpc_thread_pool_size = 2048 rpc_response_timeout = 3600 rpc_poll_timeout = 60 ## Rpc all executor_thread_pool_size = 64 rpc_response_timeout = 3600 What we are seeing in the error logs for neutron for all services (l3-agent, dhcp, linux-bridge etc ) are these timeouts: https://pastebin.com/Fjh23A5a We have manually tried to get everything in sync by forcing fail-over of the networking which seems to get routers in sync. We are also seeing that there are a lot of "unacknowledged" messages in RabbitMQ for 'q-plugin' in the neutron queues. Some times restarting of the services on neutron gets these back acknowledged again, however the timeouts come back. The RabbitMQ servers themselves are not loaded at all. All memory, file descriptors and errlang processes have plenty of resources available. We are also seeing a lot of rpc issues: Timeout in RPC method release_dhcp_port. Waiting for 1523 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 965fa44ab4f6462fa378a1cf7259aad4 2020-03-10 19:02:33.548 16242 ERROR neutron.common.rpc [req-a858afbb-5083-4e21-a309-6ee53582c4d9 - - - - -] Timeout in RPC method release_dhcp_port. Waiting for 3347 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 7937465f15634fbfa443fe1758a12a9c Does anyone know if there is anymore tuning to be done at all? Upgrading for us at the moment to a newer version isn't really an option unfortunately. Because of our setup, we also have roughly 800 routers enabled and I know that will be putting a load on the system. However these problems have only started to happen roughly 1 week ago and have steadily got worse. If anyone has any use cases for this or any more recommendations that would be great. Many thanks,

3 25

[release] Release countdown for week R-7, March 23-27
by Sean McGinnis 20 Mar '20

20 Mar '20

Development Focus ----------------- We are entering the last weeks of the Ussuri development cycle. From now until the final release, we'll send a countdown email like this every week. It's probably a good time for teams to take stock of their library and client work that needs to be completed yet. The non-client library freeze is coming up, followed closely by the client lib freeze. Please plan accordingly so avoid any last minute rushes to get key functionality in. General Information ------------------- Next week is the Extra-ATC freeze, in preparation for elections. All contributions to OpenStack are valuable, but some are not expressed as Gerrit code changes. Please list active contributors to your project team who do not have a code contribution this cycle, and therefore won't automatically be considered an Active Technical Contributor and allowed to vote. This is done by adding extra-atcs to https://opendev.org/openstack/governance/src/branch/master/reference/projec… before the Extra-ATC freeze on March 26. A quick reminder of the upcoming freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (April 2). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (April 9) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, April 9. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (April 23) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (May 7) Finally, now is also a good time to start planning what highlights you want for your deliverables in the cycle highlights. The deadline to submit an initial version for those is set to Feature freeze (April 9). Background on cycle-highlights: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125613.html Project Team Guide, Cycle-Highlights: https://docs.openstack.org/project-team-guide/release-management.html#cycle… knelson [at] openstack.org/diablo_rojo on IRC is available if you need help selecting or writing your highlights Note that cycle highlights are per-team, not per-deliverable. Upcoming Deadlines & Dates -------------------------- Extra-ATC freeze: March 26 (R-7 week) Non-client library freeze: April 2 (R-6 week) Client library freeze: April 9 (R-5 week) Ussuri-3 milestone: April 9 (R-5 week) Cycle Highlights Due: April 9 (R-5 week)

1 0

[queens][cinder][unity] iscsi driver
by Ignazio Cassano 20 Mar '20

20 Mar '20

Hello All, I would like to suggest to modify the unity iscsi cinder driver documentation. After 1 month of intensive test, we discovered the mutipath.conf needs the following option in multipath.conf kvm nodes: skip_kpartx yes Without the above option instances with more than 2 volumes attached could fail live migration because the os_brick code fails in "multipath -f" command reporting map in use error. Ignazio

1 0

[openstack.org] SEO Improvements
by Jimmy McArthur 19 Mar '20

19 Mar '20

Hi all - We've contracted a professional SEO firm to help improve search placement for all OSF projects. I'd like to crowd source this on each of the project mailing lists, so the community is able to be involved. Could you all help out in providing the below: “Wish List” of keywords: 8-12 big terms you think fit your domain (if you only have 4-5 to share, fewer is okay) - open source ci - ? At least 3 competitors - Jenkins - ? Any other input on positioning, offerings, etc. that you think will help best filter for relevance - ? Cheers, Jimmy

6 12

Re: [Release-job-failures] Tag of openstack/glance_store for ref refs/tags/rocky-em failed
by Sean McGinnis 19 Mar '20

19 Mar '20

On 3/19/20 4:56 PM, zuul(a)openstack.org wrote: > Build failed. > > - publish-openstack-releasenotes-python3 https://zuul.opendev.org/t/openstack/build/66b1173250724747bc9b7630277a5b1a : POST_FAILURE in 4m 24s > > _______________________________________________ > Release-job-failures mailing list > Release-job-failures(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures This failure appears to be from a temporary glitch that we've seen time to time in the gate: "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" This was on publishing release notes. Since no new release notes have been added as part of this (just tagging rocky-em) it is safe to ignore this failure. Sean

1 0

[nova] bug triage
by Balázs Gibizer 19 Mar '20

19 Mar '20

Hi, We surpassed the somewhat magical line of having more than 100 untriaged nova bugs [1]. For me it seems that how we, as a team, are handling the incoming bugs is not sustainable. I'm guilty as well of not doing bug triage in a last two months. So I'm personally trying to change now and dedicate a weekly times lot to look at the bug list. But I also want to open a discussion about bug triage in genera. How can we handle the incoming bugs? I see that neutron does a weekly rotation of bug deputy and for them it works nicely. What do you think? Do we want to try that? Do we have enough volunteers to create a nice rotation period? Cheers, gibi [1] https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New

2 1

[python-openstackclient] --force bug
by jayaditya gupta 19 Mar '20

19 Mar '20

Hello all, new contributor/developer here. I have just submitted a bug for python-openstackclient project : https://storyboard.openstack.org/#!/story/2007440 I am not sure how to provide --force option to openstackclient .. I am comparing quota.py files in nova-client and in openstackclient.but can't seem to understand how to provide --force option. Any suggestions? Best Regards

3 3

[neutron]Drivers meeting cancelled
by Slawek Kaplonski 19 Mar '20

19 Mar '20

Hi, Due to lack of agenda, lets cancel tomorrows (20.03.2020) drivers meeting and lets focus on reviewing patches related to the already accepted RFEs [1]. [1] https://tinyurl.com/vezk6n6 — Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][tc] Moving PTL role to "Maintainers"
by Mohammed Naser 19 Mar '20

19 Mar '20

Hi everyone: We're now in a spot where we have an increasing amount of projects that don't end up with a volunteer as PTL, even if the project has contributors .. no one wants to hold that responsibility alone for many reasons. With time, the PTL role has become far more overloaded with many extra responsibilities than what we define in our charter: https://governance.openstack.org/tc/reference/charter.html#project-team-lea… I think it's time to re-evaluate the project leadership model that we have. I am thinking that perhaps it would make a lot of sense to move from a single PTL model to multiple maintainers. This would leave it up to the maintainers to decide how they want to sort the different requirements/liaisons/contact persons between them. The above is just a very basic idea, I don't intend to diving much more in depth for now as I'd like to hear about what the rest of the community thinks. Thanks, Mohammed

24 34

[ops][cinder] deprecation notice: os-reset_status notifications
by Brian Rosmaita 19 Mar '20

19 Mar '20

tl;dr: the notifications themselves aren't being deprecated, there's a change in where they're being sent While working on https://launchpad.net/bugs/1863806, Walt discovered that notifications associated with the Block Storage API os-reset_status calls were being sent to nonstandard publisher_ids, namely: * 'volumeStatusUpdate' for volume status resets * 'volumeStatusUpdate' for snapshot status resets * 'backupStatusUpdate' for backup status resets Walt has a patch up to direct these notifications to the same publisher_id as other Cinder resource actions: * 'volume' for volume status resets * 'snapshot' for snapshot status resets * 'backup' for backup status resets Walt's patch continues to send the notifications to the nonstandard publisher_ids, but this behavior is DEPRECATED. In Ussuri, notifications will be sent to *both* publisher_ids. We propose to remove sending notifications to the nonstandard publisher_ids during the Victoria development cycle. This deprecation will be announced in the cinder release notes, but we wanted to give operators advance notice. No one currently working on cinder knows why these notifications were being sent to nonstandard publisher_ids, which makes it difficult to assess what, if any, impact this change will have on operators. If you have concerns about either the nonstandard publisher_ids being deprecated and removed, or about these notifications being sent to the standard publisher_ids, please express them on the following review before Tuesday 24 March: https://review.opendev.org/#/c/708563/ Thank you.

1 0

upgrade qemu 4.2 on CentOS 7.x for openstack
by Satish Patel 18 Mar '20

18 Mar '20

Folk, I am running openstack stein on CentOS 7.5 so default version of qemu-kvm is 2.12.x. as we know current qemu version is 4.2. Question: If i compile/install latest version of qemu 4.2 using source tarball on compute node in that case do i need to compile/install kvm or Libvirt? ~S

1 0

[all][elections][ptl][tc] Combined PTL/TC Election Season
by Kendall Nelson 18 Mar '20

18 Mar '20

Hello Everyone! Election details: https://governance.openstack.org/election/ The nomination period officially begins Mar 24, 2020 23:45 UTC. Please read the stipulations and timelines for candidates and electorate contained in this governance documentation. Due to circumstances of timing, PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you which to discuss in private please email any of the election officials[1] so that we may address your concerns. Thank you! -Kendall Nelson (diablo_rojo) & the Election Officials [1] https://governance.openstack.org/election/#election-officials

1 0

FW: 2020 OSF Events & coronavirus
by Mark Collier 18 Mar '20

18 Mar '20

I wanted to make sure everyone saw this thread on the foundation mailing list, since I know not everyone is subscribed to both lists: Archive: http://lists.openstack.org/pipermail/foundation/2020-March/002852.html Please join that ML thread to share feedback on this topic, or you can reach out directly to myself or jonathan(a)openstack.org I saw first hand how we all pulled together during the Snowpenstack in Dublin, so I know we'll once again pull together as a community to get through this! Mark On Friday, March 6, 2020 12:55pm, "Mark Collier" <mark(a)openstack.org> said: > Stackers, > > Before I get into the current plans for the OSF events in Vancouver and Berlin, I > wanted to say a few words in general about the virus impacting so many people > right now. > > First, I wanted to acknowledge the very difficult situation many are facing > because of COVID-19 (Coronavirus), across open source communities and local > communities in general (tech or otherwise). I also want to say to everyone who is > on the front lines managing events, from the full time staffers to the volunteers, > to the contractors and production partners, that we have some idea of what you're > going through and we know this is a tough time. If there's anything we can do to > help, please reach out. In the best of times, event organization can be grueling > and thankless, and so I just want to say THANK YOU to everyone who does the > organizing work in the communities we all care so much about. > > OSF 2020 EVENTS > > When it comes to the 2020 events OSF is managing, namely the OpenDev + PTG in > Vancouver June 8-11 and the Open Infrastructure Summit in Berlin October 19-23, > please read and bookmark this status page which we will continue to update: > https://www.openstack.org/events/covid-19-coronavirus-disease-updates > > When it comes to our community, the health of every individual is of paramount > concern. We have always aimed to produce events "of the community, by the > community" and the upcoming event in Vancouver is no exception. The OpenDev tracks > each morning will be programmed by volunteers from the community, and the project > teams will be organizing their own conversations as well each afternoon M-W, and > all day Thursday. > > But the larger question is here: should the show go on? > > The short answer is that as of now, the Vancouver and Berlin events are still > scheduled to happen in June (8-11) and October (19-23), respectively. > > However, we are willing to cancel or approach the events in a different way (i.e. > virtual) if the facts indicate that is the best path, and we know the facts are > changing rapidly. One of the most critical inputs we need is to hear from each of > you. We know that many of you rely on the twice-annual events to get together and > make rapid progress on the software, which is one reason we are not making any > decisions in haste. We also know that many of you may be unable or unwilling to > travel in June, and that is critical information to hear as we get closer to the > event so that we can make the most informed decision. > > I also wanted to answer a FAQ by letting everyone know that if either event is > cancelled, event tickets and sponsorships will be fully refunded. Please note that > if you're making travel arrangements (e.g. flights, hotels) those are outside of > our control. > > So as we continue to monitor the news and listen to health experts to make an > informed decision on any changes to our event plans, we'd like to hear from > everyone in the community who has a stake in these events. Our most pressing topic > is of course Vanvouver, but if you have questions or concerns about the Berlin > plan feel free to share those as well. > > If you'd like to connect directly, you can always contact Executive Director > Jonathan Bryce (jonathan(a)openstack.org) or myself (mark(a)openstack.org) > > Key Links: > - STATUS PAGE: > https://www.openstack.org/events/covid-19-coronavirus-disease-updates > - Vancouver OpenDev + PTG https://www.openstack.org/events/opendev-ptg-2020/ > - Berlin Open Infrastructure Summit: https://www.openstack.org/summit/berlin-2020/ > > Key Dates for OpenDev + PTG in Vancouver: > - Schedule will be published in early April > - Early bird deadline is April 4 > - Final day to sponsor will be May 4 > - Final registration price increase will be in early May > > Mark Collier > COO, OpenStack Foundation > @sparkycollier > >

11 11

Not running for TC this round
by Thierry Carrez 18 Mar '20

18 Mar '20

Hi everyone, I've been elected on the TC (or its previous incarnations) since we had community-elected seats in that body (2011). However, as announced during the previous election round (and in the discussion about reducing the number of TC seats) I do not plan to run for the TC in the upcoming election round. There are multiple reasons for this. The main one is that with name recognition and a Condorcet voting system, it's relatively easy for incumbents to be reelected. We only elect 5 seats this time, so I think it's important to make room for new names to step up. The second reason is that I'd like more people with an operational background to be present on the TC. I think we can simplify our governance by having a single elected body handling the open source project, rather than maintain separate representation for users and developers. As we get more people bringing operational perspective to the TC, that simplification will be even more obvious. The final reason is that I've always been saying that you can work on governance issues and voice your opinion on them, without being elected on the TC and formally voting on things. It's time for me to walk that talk. I do not intend to go anywhere, and I'll be as present as always in proposing the OpenStack governance changes that I deem needed in this day and age. I'll finish this email by encouraging anyone interested in open source project governance, or wanting to help shape the next chapter of the OpenStack story, to step up and run for TC election. As for all governance bodies, the experience can be frustrating, as you'd like to fix more things but there is only so much you can do. But it's a great experience, one that I recommend everyone working on open source project should have. The only prerequisite is an interest and care for "OpenStack" as a whole. No specific experience or diploma needed. If you wonder if you can do it, the answer is yes. Nominations[1] open up March 24. Please consider running, especially if you have operational experience with some cycles to spare. If you have any questions on the job, feel free to reach out to me -- you know where to find me. [1] https://governance.openstack.org/election/ -- Thierry Carrez (ttx)

3 2

[all][PTL][tc] U Community Goal: Project PTL & Contrib Docs Update #5
by Kendall Nelson 17 Mar '20

17 Mar '20

Hello Everyone! First off, I want to say thank you so much to Watcher, Trove, Searchlight, Qinling, Nova, Neutron, Kolla, Ironic, i18n, Glance, Cinder, and Blazar for getting started/completing the goal! For those projects that have not started yet, you can take a look at what the others have done for some inspiration[1], or simply fill in the template[2]. We did our best to make the process as low effort as possible. If you have any questions, please let me know or refer to the goal[3]. Once you get started, please make sure to keep the story up to date[4]. I look forward to reviewing your patches!! If there is anything I can do to help, please let me know. -Kendall (diablo_rojo) [1] Progress: https://review.opendev.org/#/q/topic:project-ptl-and-contrib-docs+(status:o… [2] Cookiecutter Template: https://opendev.org/openstack/cookiecutter/src/branch/master/%7b%7bcookiecu… [3] Accepted Goal: https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-c… [4] Task Tracking: https://storyboard.openstack.org/#!/story/2007236

1 0

Using #opendev on Freenode for OpenDev Infrastructure Discussions
by Clark Boylan 17 Mar '20

17 Mar '20

Hello all, We have reached a point in time where it now makes sense for us to use #opendev on Freenode for OpenDev discussions, rather than #openstack-infra. We are updating our various bots to reflect this change and are asking you to find us in the new channel. We expect this to be a time of transition so don't be afraid of bringing things up in either the old or new channel if you are unsure of where you should go; we'll get you pointed in the right direction. The old channel, #openstack-infra, won't be going away. We expect it to become more OpenStack specific. Topics like how to use Zuul for OpenStack jobs could go here. This likely will have some overlap with the #openstack-qa channel and we may end up collapsing them, but that can happen if we notice the conversations do overlap significantly. We hope this gives non OpenStack users a friendly neutral location to engage us. Sorry, for any disruptions this may cause. Clark

1 0

[all] Guides for newbies to OpenStack
by Mike Carden 17 Mar '20

17 Mar '20

Our small team at ${DAYJOB} has built a handful of OpenStack clusters based on Red Hat OpenStack 13 (aka Queens) over the last couple of years. We now find ourselves in the position of being 'gifted' human resources in the shape of mid-level 'IT people' who are sent to join our team for a short time to 'Learn OpenStack'. These tend to be people for whom, "Here's a Horizon URL and some creds - go log in and launch a VM"[1]... is a bit much. I've done a wee bit of web searching (enough to find the dead links) trying to find some newbie friendly tutorials on OpenStack basics. Before I attempt to re-invent the wheel, can anyone suggest some public resources I might point people to? Deity help us if we have to explain Tripleo's Undercloud, Overcloud, partially containered, partially pacemakered, fully flaky... underpinnings. Thanks, MC [1] Even with a step by step guide

5 4

[openstack-helm] Virtual Midcycle
by Gage Hugo 17 Mar '20

17 Mar '20

Good afternoon, The openstack-helm team will be hosting their virtual midcycle next Tuesday, March 24th 2020 at 1400 to 1700 UTC. Please check out the etherpad link for more details about the agenda and conferencing tool. Since this meeting is overlapping our normal meeting time, the actual IRC meeting will be canceled for next week. https://etherpad.openstack.org/p/osh-virtual-ptg-2020-03 Hope to see you all there.

1 0

OpenStack support for msgpack 1.0.0
by Thomas Goirand 17 Mar '20

17 Mar '20

Hi, msgpack 1.0.0 was released last Sept. I'm being pressed by the rest of Debian people to upload 1.0.0 to Sid. I'm concern by it: will this break any of the OpenStack package, like oslo.messaging? Is there any plan to have support for 1.0.0 for Ussuri? Cheers, Thomas Goirand (zigo)

5 5

[tc][requirements][horizon] Broken gates due to the new setuptools and pyScss package
by Ivan Kolodyazhny 17 Mar '20

17 Mar '20

Hi team, I'm sorry for being too noisy, but I decided to tag TC to get more attention to the current Horizon situation. We've got a bug reported by Kolla team two days ago [1]. We merged some workaround [2] and [3] yesterday. Thanks a lot to the Requirements team for the really quick merge! I appreciate it. For now, we've got horizon gates broken because of hose patches fix only devstack but not unit tests jobs. The root cause of this situation is pyScss package which is not maintained for the last two years. It's not a surprise to me that it doesn't work with the new setuptools. I'm really surprised that we've found this kind of bugs only now. Since I don't believe we can block new setuptools forever, I decided to fork pyScss [4] and django-pyscss [5] projects. I'm still not sure that I've done everything right with licensing and versioning, but it works now on my environment. Any help on these areas would be much appreciated. I proposed patches to requirements and horizon repos [6] to use new libraries. The reason I've tagged TC in the mail thread is described below. Horizon has a lot of too old and unmaintained libraries. I'm pretty sure that this only one of the first issues with outdated dependencies which blocks horizon and other gates. I do understand why we've got this situation. Unfortunately, we don't have any full-time horizon developers in the community. Horizon is mostly in maintenance phrase but not in active development. I would like to get more attention on this issue because we have to update all dependencies not because they are new, have new features and/or security fixes. We have to take care of our dependencies asap to avoid usage of unmaintained libraries to have the whole OpenStack and Horizon healthy. P.S. I'm sorry if this message is too rude or emotional, I really don't want to make it such one. [1] https://bugs.launchpad.net/kolla/+bug/1866961 [2] https://review.opendev.org/#/c/711930/ [3] https://review.opendev.org/#/c/712777/ [4] https://github.com/e0ne/pyScss/ [5] https://github.com/e0ne/django-pyscss [6] https://review.opendev.org/#/q/status:open+topic:fix-pyscss Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

5 5

[cinder] virtual mid-cycle session 2 summary available
by Brian Rosmaita 17 Mar '20

17 Mar '20

Hello Cinderinos, Thanks for a productive discussion yesterday. I've posted a summary on the wiki so we can remember what happened and for the edification of those who didn't attend: https://wiki.openstack.org/wiki/CinderUssuriMidCycleSummary A link to the recording will be posted on that page when it's available. cheers, brian

1 0

PyOS
by Martin Uhrin 16 Mar '20

16 Mar '20

Dear OpenStack devs, Just a quick question: I came across your project 'pyos' on pypi after using this name for internally one of my projects. I see that you haven't updated the package for some time and was wondering if it was dead and if so if you would consider transfering the name to me for hosting my project? Thanks for your time. All the best, -Martin

2 5

[ironic] proposing Iury Gregory for bifrost-core, ironic-inspector-core, sushy-core
by Julia Kreger 16 Mar '20

16 Mar '20

Iury has been working hard across the ironic community and has been quite active in changing and improving our CI, as well as reviewing code contributions and helpfully pointing out issues or items that need to be fixed. I feel that he is on track to join ironic-core in the next few months, but first I propose we add him to bifrost-core, ironic-inspector-core, and sushy-core. Any objections?

3 2

[ops] how to setup bgp-to-the-host and Neutron Dynamic Routing ?
by Thomas Goirand 16 Mar '20

16 Mar '20

Hi, In my setup, the bgp-to-the-host part is already operational. We're using IPv6 local link to 2 switches on each host, each of them advertising for itself to both routers. Now, I'm trying to add Neutron stuff to it. In my setup, I don't have any IPv4 on my nics, both have ipv6 local link only. So, where do I connect br-ex? I don't have a bond0 interface to connect to anymore... Then, we do have an FRR instance to do the bgp-to-the-host IPv4 routing on each host. How does this work on the hosts where the dr-agent is running? Can it re-use the FRR that I've setup to do the route advertisements too? Or do I need another instance of a routing daemon? Cheers, Thomas Goirand (zigo)

1 0

[Tempest] OpenSack Powered * vs OpeStack Compatible
by prakash RAMCHANDRAN 16 Mar '20

16 Mar '20

Hi all, I am curious to find out if any Distribution or Products based on Openstack Train or Usuri are seeking the latest certifications based on 2019.02. Similarly does any Hardware Driver of Software application seeking OpenStack compatibility Logo? Finally does anyone think that Open Infra Distro like Airship or StarlingX should promote Open Infra Airship or Open Infra StarlingX powered as a new way to promote eco system surrounding them similar to OpenStack compatible drivers and software. Will then Argo, customize, Metal3.io or Ironic be qualified as Open Infra Airship compatible? If so how tempest can help in testing the above comments? Refer to this market place below as how Distos and Products leverage OpenStack logos and branding programs. https://www.openstack.org/marketplace/distros/ Discussions and feedback are welcome. A healthy debate as how k8s modules used in Open Infra can be certified will be a good start. ThanksPrakash Sent from Yahoo Mail on Android

6 11

[tripleo][RDO] Version of Ansible in RDO CentOS8 repository
by Alfredo Moralejo Alonso 16 Mar '20

16 Mar '20

Hi all, During CentOS 8 dependencies preparation we've built ansible 2.9 in RDO dependencies repo which was released on Oct 2019, While testing TripleO with CentOS8 it has been discovered that the latest release of ceph-ansible does not support ansible 2.9 but only 2.8, so I'm opening discussion about the best way to move on in CentOS 8: - Make ceph-ansible 4.0 to work with ansible 2.8 *and* 2.9 so that the same releases can be used in CentOS7 with Stein and Train and CentOS8 Train and Ussuri. - Maintain separated ceph-ansible releases and builds for centos7/ansible 2.8 and centos8/ansible 2.9 able to deploy Nautilus. - Move ansible back to 2.8 in CentOS 8 Ussuri repository. I wonder if TripleO or other projects using ansible from RDO repositories has any requirement or need to move to ansible 2.9 in Ussuri cycle or can stay in 2.8 until next release, any thoughts? Best regards, Alfredo

1 2

[neutron] Bug deputy report (week starting on 2020-03-09)
by Bernard Cafarelli 16 Mar '20

16 Mar '20

Hello neutrinos, last week we started a new bug deputy rotation, opening this round here are the bugs reported in week 11. This was relatively quiet (for new bugs count), and most bugs have active discussion or suggested fix Critical * [security] Add allowed-address-pair 0.0.0.0/0 to one port will open all others' protocol under same security group - https://bugs.launchpad.net/neutron/+bug/1867119 A follow-up to security bug https://bugs.launchpad.net/neutron/+bug/1793029 (which was fixed in documentation) Potential code fix at https://review.opendev.org/712632 - reviews and opinions most welcome Medium * Restart neutron-linuxbridge-agent service led to all ports status changed - https://bugs.launchpad.net/neutron/+bug/1866743 Reported on Pike/Queens, pretty standard configuration, may be l2pop Related gerrit question: https://review.opendev.org/713156 * MTU too large error presented on create but not update - https://bugs.launchpad.net/neutron/+bug/1867214 Suggested fix: https://review.opendev.org/712801 Low * Packets incorrectly marked as martian - https://bugs.launchpad.net/neutron/+bug/1866615 Martian packets logged with some specific setup, VMs are working fine though, switching to ovs firewall workarounds the issue * Deployment has security group with empty tenant id - https://bugs.launchpad.net/neutron/+bug/1867101 Some master devstacks deployments like networking-odl get empty project ID for default security group * Unnecessary network flapping while update floatingip without port or fixed ip changed - https://bugs.launchpad.net/neutron/+bug/1867122 OVN mech driver only, some discussion about relevant use-case for FIP update in LP and patch https://review.opendev.org/712641 Incomplete * router-update for internal networking not correct when restarting ovs-agent - https://bugs.launchpad.net/neutron/+bug/1866635 Missing flows on restart, I asked for more logs - may be missing tunnel during restart Update from previous week * br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes - https://bugs.launchpad.net/neutron/+bug/1866445 Was closed as duplicate of bug #1732067 but they do not use OVS firewall Patch for iptables_hybrid proposed: https://review.opendev.org/712640 Last bug I triaged is 1867214, handing over the deputy baton to slaweq -- Bernard Cafarelli

1 0

[goals][Drop Python 2.7 Support] Week R-9 Update
by Ghanshyam Mann 15 Mar '20

15 Mar '20

Hello Everyone, Below is the progress on "Drop Python 2.7 Support" at end of R-9 week. Schedule: https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html#sc… Highlights: ======== * We already passed the deadline but still work is not completed on this. * Few pythonclient and tempest plugins are not merged yet. * I request projects again to merge the passing patches asap. Project wise status and need reviews: ============================ Phase-1 status: All the OpenStack services have dropped the python2.7. Phase-2 status: * Pending Tempest plugins: ** barbican-tempest-plugin: https://review.opendev.org/#/c/704083/ ** cyborg-tempest-plugin: https://review.opendev.org/#/c/704076/ ** kuryr-tempest-plugin: https://review.opendev.org/#/c/704072/ * Pending pythonclient: ** python-barbicanclient: https://review.opendev.org/#/c/699096/2 ** python-zaqarclient: https://review.opendev.org/#/c/692011/4 ** python-tripleoclient: https://review.opendev.org/#/c/703344 * Few more repo patches need to merge: ** masakari-specs: https://review.opendev.org/#/c/698982/ ** cyborg-specs: https://review.opendev.org/#/c/698824/ ** masakari-monitors: https://review.opendev.org/#/c/694552/ ** nova-powervm: https://review.opendev.org/#/c/700683/ ** paunch: https://review.opendev.org/#/c/703344/ ** cloudkitty: https://review.opendev.org/#/c/709585/3 * Started pushing the required updates on deployment projects. ** Completed or no updates required: *** Openstack-Chef - not required *** Packaging-Rpm - Done *** Puppet Openstack- Done ** In progress: *** Openstack Charms - Patches are ready to merge. Need review. *** Openstackansible - In-progress. centos7 jobs are failing on few projects. ** Waiting from projects team to know the status: *** Openstack-Helm (Helm charts for OpenStack services) *** Tripleo (Deployment service) * Open review: https://review.opendev.org/#/q/topic:drop-py27-support+status:open Phase-3 status: This is audit and requirement repo work which is not started yet. I will start this once all the phase-2 work mentioned above is completed. How you can help: ============== - Review the patches. Push the patches if I missed any repo. -gmann

1 0

[requirements][all] migration OFF of mock to the unittest.mock built-in library
by Matthew Thode 13 Mar '20

13 Mar '20

I'd like to suggest, now that we are on modern python, that we stop using the mock library and instead use the built in mock library. unittest.mock has been around since python-3.3 so we should not have a problem with python support. This would allow us to drop a library and also help prevent issues with stuff like https://review.opendev.org/712713 is going to expose (nova/neutron fail tests with the external mock-4 library). -- Matthew Thode

5 6

[requirements] add sqlalchemy-filters
by Prashant Bhole 13 Mar '20

13 Mar '20

Hello requirements people, I wanted to discuss about addition of this new requirement: sqlalchemy-filters https://pypi.org/project/sqlalchemy-filters/ Why it is needed: This library is used in Tacker project for the implementation of Attribute Filtering of VNF Packages defined in ETSI NFV_SOL005 section 9.4.2.3.2. It is used for adding filter criterion to SQL alchemy queries. These criterion are based on attribute values of VNF Packages passed in the GET request to list VNF Packages. This library can be reused for similar cases in future. Matthew Thode had raised a concern about this project being stale as there are pending PRs. On contacting with the developer of the library the developer said that he will will work on pending PRs. The project is being actively maintained. Please review it. https://review.opendev.org/#/c/710555/ Thanks, Prashant

2 1

[cinder] final reminder: ussuri virtual mid-cycle 16 march at 12:00 UTC
by Brian Rosmaita 13 Mar '20

13 Mar '20

For people in the New York timezone, this event starts at 8:00 am on Monday. (Feedback from the last virtual meet-up was that it needed more promotion, so if you know anyone who might be (or should be) interested, please tell them, in addition to marking your own calendar.) Session Two of the Cinder Ussuri virtual mid-cycle will be held: DATE: Monday, 16 March 2020 TIME: 1200-1400 UTC LOCATION: https://bluejeans.com/3228528973 The meeting will be recorded. Please add topics to the planning etherpad: https://etherpad.openstack.org/p/cinder-ussuri-mid-cycle-planning cheers, brian

1 0

[neutron] Propose Lajos Katona for Neutron core team
by Slawek Kaplonski 13 Mar '20

13 Mar '20

Hi neutrinos, I would like to propose Lajos Katona (irc: lajoskatona) as a member of the Neutron core team. Lajos is Neutron contributor Neutron since around Queens cycle and now he is one of the most active reviewers in the Neutron group projects. He was one of the key contributors in cooperation with Nova and Placement teams to deliver guaranteed minimum bandwidth feature in OpenStack. He is very active and helpful with triaging and fixing Neutron bugs and issues in our CI. During last few cycles he proved that he has wide knowledge about Neutron code base. He is currently also a maintainer of some neutron stadium projects which shows that he has knowledge about code base not only about neutron but also Neutron stadium. The quality and number of his reviews are comparable to other members of the Neutron core team: https://www.stackalytics.com/?release=ussuri&module=neutron-group and are higher every cycle :) I think he will be great addition to our core team. I will keep this nomination open for a week or until all current cores will respond. — Slawek Kaplonski Senior software engineer Red Hat

8 8

[API][SDK][SIG] distilling our community wisdom
by Michael McCune 13 Mar '20

13 Mar '20

hello all, given the recent trend of discussing how we can condense and streamline various parts of the openstack community, i would like to start socializing the idea of merging the API-SIG, and its artifacts/processes/members, with the SDK group. over the last few years we have seen the activity of the API SIG greatly reduced. although there are a few outstanding guidelines that should be merged, we are having fewer and fewer contributors. additionally, although we migrated from weekly meetings down to weekly office hours, we have seen that aside from facilitating discussions for other groups these office hours are largely unnecessary. i don't have a proposal at the moment, beyond this email, but i would appreciate any and all feedback to help us reach a place where the SIG can still react as needed to the community but also reduce its surface area. the heart of these goals is to sustain and preserve the hard work that has been done over the years, and also to recognize the diminishing availability of our membership. peace o/

2 1

[Multi-Arch-SIG][devstack][nova] Implement Arm64 CI now for Nova and DevStack
by kevinz 13 Mar '20

13 Mar '20

Hi OpenStacker, Linaro has donated a cluster for OpenStack CI on Arm64. Now the cluster is ready and some jobs(Kolla and disk builder, thanks hrw's hard-working) are running here, https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl0… Thanks for the help from Infra team and nova team, I am now working on Devstack tempest jobs running on arm64: https://review.opendev.org/708317 Some changes from Nova also need to be covered in order to make devstack jobs successfully. https://review.opendev.org/712607 https://review.opendev.org/709494 Yet, there are still a lot of tasks need to fix for arm64 CI work. If you are interested in this, any forms of help will be really welcomed! let's get together to make arm64 CI happens.

1 0

[cyborg] Proposing core reviewers
by Nadathur, Sundar 12 Mar '20

12 Mar '20

Hello all, Brin Zhang has been actively contributing to Cyborg in various areas, adding new features, improving quality, reviewing patches, and generally helping others in the community. Despite the relatively short time, he has been one of the most prolific contributors, and brings an enthusiastic and active mindset. I would like to thank him and acknowledge his significant contributions by proposing him as a core reviewer for Cyborg. Shogo Saito has been active in Cyborg since Train release. He has been driving the Cyborg client improvements, including its revamp to use OpenStackSDK. Previously he was instrumental in the transition to Python 3, testing and fixing issues in the process. As he has access to real FPGA hardware, he brings a users’ perspective and also tests Cyborg with real hardware. I would like to thank and acknowledge him for his steady valuable contributions, and propose him as a core reviewer for Cyborg. Some of the currently listed core reviewers have not been participating for a lengthy period of time. It is proposed that those who have had no contributions for the past 18 months – i.e. no participation in meetings, no code contributions and no reviews – be removed from the list of core reviewers. If no objections are made known by March 20, I will make the changes proposed above. Thanks. Regards, Sundar

3 9

[tripleo]
by Wesley Hayutin 12 Mar '20

12 Mar '20

Greetings, So, an update on centos-8 jobs for master / ussuri. The tooling that tags containers w/ "current-tripleo" is almost available. Until that time one can derive the hash used to tag containers by navigating to [1] For example [2] you'll see the centos-binary-base is tagged w/ [1] I'm retagging the current containers in current-tripleo w/ the tag current-tripleo to resolve this issue for now. The tooling will be back up shortly :) I appreciate your patience. Thanks [1] https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.m… 3621159be13b41f8ead2e873b357f4a5 [2] https://hub.docker.com/layers/tripleomaster/centos-binary-base/3621159be13b…

1 1

[OpenDev] We want your discussion ideas!
by Ashlee Ferguson 12 Mar '20

12 Mar '20

Hi everyone, We still need your help shaping the discussion content for OpenDev + PTG, June 8-11, 2020 <https://www.openstack.org/events/opendev-ptg-2020/>! Our vision is for the content to be programmed by you-- the community. Programming Committees for each Track have begun meeting to discuss potential topics for the event, but they need your ideas and input to make sure all of the content is relevant and interesting to the community. The Programming Committee will also select moderators who will lead interactive discussions on a particular topic within the Track, so make sure to volunteer through the form below if you’re interested in that as well. Check out this etherpad <https://etherpad.openstack.org/p/r.961497ec02b63da123fbda235a7ba02e> to see what topics the Committees are currently discussing. If you’re interested in volunteering as an OpenDev discussion moderator, or would like to suggest topics for moderated discussions within a particular Track, please read the details below, and then fill out this form <https://openstackfoundation.formstack.com/forms/opendev_vancouver2020_volun…>. We’re looking for discussion topics and Moderators for the following OpenDev Tracks: - Hardware Automation - Large-scale Usage of Open Source Infrastructure Software - Containers in Production - Key Challenges for Open Source in 2020 OpenDev Discussion Moderators will: - Be appointed by the Programming Committees - Facilitate discussions within a particular Track - Have adequate knowledge and experience to lead and moderate discussion around certain topics during the event - Work with Programming Committee to decide focal point of discussion Moderators need to be available to attend OpenDev, June 8 - 10, 2020. Volunteer to be a moderator or suggest discussion topics here before March 20: https://openstackfoundation.formstack.com/forms/opendev_vancouver2020_volun… <https://openstackfoundation.formstack.com/forms/opendev_vancouver2020_volun…> Cheers, Ashlee Ashlee Ferguson Community & Events Coordinator OpenStack Foundation

1 0

[neutron][largescale-sig] Debugging and tracking missing flows with l2pop
by Krzysztof Klimonda 12 Mar '20

12 Mar '20

Hi, (This is stein deployment with 14.0.2 neutron release) I’ve just spent some time debugging a missing connection between two VMs running on OS stein with ovs+l2pop enabled and the direct cause was missing flows in table 20 and a very incomplete flood flow in table 22. Restarting neutron-openvswitch-agent on that host has fixed the issue. Last time we’ve encountered missing flood flows (in another pike-based deployment), we tracked it down to https://review.opendev.org/#/c/600151/ and since then it was stable. My initial thought was that we were hitting the same bug - a couple of VMs are scheduled on the same compute, 3 ports are activated at the same time, and the flood entry is not broadcasted to other computes. However that issue was only affecting one of the computes, and it was the only one missing both MAC entries in table 20 and VXLAN tunnels in table 22. The only other idea I have is that the compute with missing flows have not received them from rabbitmq, but there I see nothing in logs that would suggest that agent was disconnected from rabbitmq. So at this point I have three questions: - what would be a good place to look next to track down those missing flows - for other operators, how stable do you find l2pop in general? and if you have problems with missing flows in your environment, do you try to monitor your deployment for that? -Chris

2 3

[tripleo]
by Wesley Hayutin 12 Mar '20

12 Mar '20

Greetings, Everyone you are going to find your jobs going red atm. My apologies some containers were not pushed manually properly and I'm having to repull and and repush to docker.io atm. We should have this fixed in about 2 hours. Sorry again, and hopefully you'll be back up and running shortly. Thanks

4 6

Missing OVA in GitHub
by Mats Karlsson 12 Mar '20

12 Mar '20

Hi, I’m new in OpenStack and found out that there is a an VM with OpenStack at https://github.com/openstack/upstream-institute-virtual-environment But it look like that the VM (http://bit.ly/vm-2019-shanghai-v1) is missing and I can’t file a support issue in that repo so that’s why I’m asking here. Is this a known issue ? Regards Mats Karlsson Trainer [cid:image002.jpg@01D4C2DE.FE359640] Rosenlundsgatan 54 SE-118 63 Stockholm, Sweden M: +46 766 967 835 T: +46 8 555 105 15 E:mats.karlsson@apistraining.com W: www.apistraining.com<http://www.apistraining.com/> Connect with us: LinkedIn<http://bit.ly/LINKEDINAPIS> Youtube<http://bit.ly/YoutubeChannelApis> Facebook<https://www.facebook.com/apistraining/> Twitter <https://twitter.com/ApisIPsolutions> Instagram<https://www.instagram.com/apisipsolutions/>

1 0

Add pytest, pytest-django and pytest-html to global requirements
by Oleksii Petrenko 11 Mar '20

11 Mar '20

Adding pytest, pytest-django and pytest-html allows import of tests results in xml, html formats for openstack-horizon. What do you think about this?

5 6

[tripleo] Missing tag in cron container image - no recheck please
by Emilien Macchi 11 Mar '20

11 Mar '20

Hi folks, We seem to have an issue with container images, where one (at least) has a missing tag: https://bugs.launchpad.net/tripleo/+bug/1866927 It is causing most of our jobs to go red and fail on: tripleo_common.image.exception.ImageNotFoundException: Not found image: docker:// docker.io/tripleomaster/centos-binary-cron:3621159be13b41f8ead2e873b357f4a5 Please refrain from approving or rechecking patches until we have sorted this out. Thanks and stay tuned. -- Emilien Macchi

2 1

[OSSA-2020-002] Manila: Unprivileged users can retrieve, use and manipulate share networks (CVE-2020-9543)
by Goutham Pacha Ravi 11 Mar '20

11 Mar '20

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 ================================================================================= OSSA-2020-002: Unprivileged users can retrieve, use and manipulate share networks ================================================================================= :Date: March 10, 2020 :CVE: CVE-2020-9543 Affects ~~~~~~~ - - Manila: <7.4.1, >=8.0.0 <8.1.1, >=9.0.0 <9.1.1 Description ~~~~~~~~~~~ Tobias Rydberg from City Network Hosting AB reported a vulnerability with the manila's share network APIs. An attacker can retrieve and manipulate share networks that do not belong to them if they possess the share network ID. By exploiting this vulnerability, they can view and manipulate share network subnets and use the share network to create resources such as shares and share groups. Patches ~~~~~~~ - - https://review.opendev.org/712167 (Pike) - - https://review.opendev.org/712166 (Queens) - - https://review.opendev.org/712165 (Rocky) - - https://review.opendev.org/712164 (Stein) - - https://review.opendev.org/712163 (Train) - - https://review.opendev.org/712158 (Ussuri) Credits ~~~~~~~ - - Tobias Rydberg from City Network Hosting AB (CVE-2020-9543) References ~~~~~~~~~~ - - https://launchpad.net/bugs/1861485 - - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-9543 Notes ~~~~~ - - The stable/queens and stable/pike branches are under extended maintenance and will receive no new point releases, but patches for them are provided as a courtesy. - -- Goutham Pacha Ravi PTL, OpenStack Manila -----BEGIN PGP SIGNATURE----- wsFcBAEBCAAGBQJeaVVWAAoJEDEySBmyuw9i8c0P/Rjkr4mxbDi7GzDCLdvC 4SK31LaF92uop/t2XXnm/p2Lui/4nG6ss46ajnmsplN2D//f+/NhBC+Oa/+R 3rwEl1YFFO8NoNcpjWS+6oE66HNPEPTxSMheyfWJTjl8bmH4wL0ZGnQ+cNWM q1XhO5Qjwv58epa0IK5vRA6lfWEmZQ69/+7nf6Tyha8vuLFOpStWXj7sV0SZ j/AxvTeCu/30EH9U4E10VQ/GpHz00WuueEYUCJgOZw4jGk32238yXmuF1fBU il4PR53ZPFqb20It56t/rrr0sGB8lLui7KiBhaHFmjRK8YqwD1pqz9XAaxNq CsgbkMnR8+WsheAgMr49NeYsQ1PD6SCLBXPQGVNus/pl5bzctIaqmswPN1ey p23tREpTEjOxg9mQJLkTCKICvi0alx3Nlk9EsrSapovJk/v8BJGrjkIj8iH0 a1pAMzjcHfGpCTGO2dHBOfJs7BXL9B6Jdba9bdRTt5BRI4NHKwvM9SP9yBb6 F7UNoo8cd+pQp0EV6i8CPUTF/qWU5rqOyIr9tGTAOPm0lg8+uIOot7oZzJcu QBaKyEZu9X4OV1o5mZ68KokiVP7RWYGMGz94NV4ZMNNfmgpsxP/h2+MZCUQJ +lmMPInx5abdwMtqiyhrSQxdgLCOKlWMYXgrs7w225sjv2+LpuVltIPXGPEJ tJq+ =tXeN -----END PGP SIGNATURE-----

1 0

[nova] Breaking up and migrating the nova-live-migration job to Zuulv3
by Lee Yarwood 11 Mar '20

11 Mar '20

Hello all, I've started PoC'ing some ideas around $subject in the topic below and wanted to ask the wider team for feedback on the approach I'm taking: https://review.opendev.org/#/q/topic:nova-live-migration-zuulv3 My initial idea is to break the job up into the following smaller multinode jobs that are hopefully easier to understand and maintain. * nova-multinode-live-migration-py3 A simple LM job using the qcow2 imagebackend and LVM/iSCSI c-vol. * nova-multinode-live-migration-ceph-py3 A ceph based LM job using rbd for both imagebackend and c-vol. * nova-multinode-evacuate-py3 A separate evacuation job using qcow2 imagebackend and LVM/iSCSI c-vol. The existing script *could* then be ported to an Ansible role as part of the migration to Zuulv3. Hopefully this is pretty straight forward but I'd appreciate any feedback on this all the same. Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76

4 5

Not running for TC next election
by Jim Rollenhagen 11 Mar '20

11 Mar '20

Hi all, I won't be running for TC next election. As you probably noticed, I don't really have enough time these days to meaningfully contribute, so leaving it open for someone new. It's been fun and a great learning experience, so I highly encourage others in the community to run! I'll still be around to heckle in the background, don't worry. :) // jim

2 1

[neutron] Why network performance is extremely bad and linearly related with number of VMs?
by Yi Yang (杨�D)-云服务集团 11 Mar '20

11 Mar '20

Hi, All Anybody has noticed network performance between VMs is extremely bad, it is basically linearly related with numbers of VMs in same compute node. In my case, if I launch one VM per compute node and run iperf3 tcp and udp, performance is good, it is about 4Gbps and 1.7Gbps, for 16 bytes small UDP packets, it can reach 180000 pps (packets per second), but if I launch two VMs per compute node (note: they are in the same subnet) and only run pps test case, that will be decrease to about 90000 pps, if I launch 3 VMs per compute node, that will be about 50000 pps, I tried to find out the root cause, other VMs in this subnet (they are in the same compute node as iperf3 client) can receive all the packets iperf3 client VM sent out although destination MAC isn’t broadcast MAC or multicast MAC, actually it is MAC of iperf3 server VM in another compute node, by further check, I did find qemu instances of these VMs have higher CPU utilization and corresponding vhost kernel threads also also higher CPU utilization, to be importantly, I did find ovs was broadcasting these packets because all the ovs bridges didn’t learn this destination MAC. I tried this in Queens and Rocky, the same issue is there. By the way, we’re using linux bridge for security group, so VM tap interface is attached into linux bridge which is connected to br-int by veth pair. Here is output of “ovs-appctl dpif/dump-flows br-int” after I launched many VMs: recirc_id(0),in_port(12),eth(src=fa:16:3e:49:26:51,dst=fa:16:3e:a7:0a:3a),et h_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:11012944, bytes:726983412, used:0.000s, flags:SP., actions:push_vlan(vid=1,pcp=0),2,set(tunnel(tun_id=0x49,src=10.3.2.17,dst=10 .3.2.16,ttl=64,tp_dst=4789,flags(df|key))),pop_vlan,9,8,11,13,14,15,16,17,18 ,19 $ sudo ovs-appctl fdb/show br-floating | grep fa:16:3e:49:26:51 $ sudo ovs-appctl fdb/show br-tun | grep fa:16:3e:49:26:51 $ sudo ovs-appctl fdb/show br-bond1 | grep fa:16:3e:49:26:51 $ sudo ovs-appctl fdb/show br-int | grep fa:16:3e:49:26:51 All the bridges can’t learn this MAC. My question is why ovs bridges can’t learn MACs of other compute nodes, is this common issue of all the Openstack versions? Is there any known existing way to fix it? Look forward to hearing your insights and solutions, thank you in advance and have a good day.

4 5

[largescale-sig] Meeting summary and next actions
by Thierry Carrez 11 Mar '20

11 Mar '20

Hi everyone, The Large Scale SIG held a meeting today. You can catch up with the summary and logs of the meeting at: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_si… No progress on "Documenting large scale operations" this week. masahito just posted a new revision for the oslo.metrics spec: https://review.opendev.org/#/c/704733/ belmoreira asks for input/comment on the proposed "Large Scale operations" Opendev track content: https://etherpad.openstack.org/p/LargeScaleOps_OpenDev Standing TODOs: - amorin to create a wiki page for large scale documentation - amorin to propose patch against Nova doc - all to check/comment on https://etherpad.openstack.org/p/LargeScaleOps_OpenDev - all to review new patchset of oslo.metrics spec https://review.opendev.org/#/c/704733/ - oneswig to contribute a scaling story on bare metal cluster scaling The next meeting will happen on March 25, at 9:00 UTC on #openstack-meeting. Cheers, -- Thierry Carrez (ttx)

1 0

[manila] share group replication spike/questions
by Dong.Ding＠dell.com 11 Mar '20

11 Mar '20

Hi, guys, As we talked about the topic in a virtual PTG few months ago. https://etherpad.openstack.org/p/shanghai-ptg-manila-virtual (Support promoting several shares in group (DELL EMC: dingdong) I'm trying to write a manila-spec for it. It's my first experience to implement such feature in framework. I need to double check with you something, and hope you can give me some guides like: 1. Where is the extra-spec defined for group/group type, it's in Manila repo, right? (like manila.db.sqlalchemy.models....) 2. The command cli should be implemented for 'python-manilaclinet' repo, right? (I have never touched this repo before) 3. Where is the rest-api should be implemented? 4. And more tips you have? like any other related project should be changed? Just list what I know, and more details questions will be raised when implementing, I think. FYI Thanks, Ding Dong

2 7

[horizon][kolla] pyscss failure on newest setuptools
by Marcin Juszkiewicz 11 Mar '20

11 Mar '20

One of Horizon's requirements is pyscss package. Which had last release over 4 years ago... Two days ago setuptools v46 got released. One of changes was drop of Features feature. Today Kolla builds started to fail: INFO:kolla.common.utils.horizon:Collecting pyScss===1.3.4 INFO:kolla.common.utils.horizon: Downloading http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8… (120 kB) INFO:kolla.common.utils.horizon:[91m ERROR: Command errored out with exit status 1: INFO:kolla.common.utils.horizon: command: /var/lib/kolla/venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rr0db3qs/pyScss/pip-egg-info INFO:kolla.common.utils.horizon: cwd: /tmp/pip-install-rr0db3qs/pyScss/ INFO:kolla.common.utils.horizon: Complete output (5 lines): INFO:kolla.common.utils.horizon: Traceback (most recent call last): INFO:kolla.common.utils.horizon: File "<string>", line 1, in <module> INFO:kolla.common.utils.horizon: File "/tmp/pip-install-rr0db3qs/pyScss/setup.py", line 9, in <module> INFO:kolla.common.utils.horizon: from setuptools import setup, Extension, Feature INFO:kolla.common.utils.horizon: ImportError: cannot import name 'Feature' Are there any plans to fix it? pyscss project got issue: https://github.com/Kronuz/pyScss/issues/385 In Kolla I made ugly workaround: https://paste.centos.org/view/2e29d284 What are plans of Horizon team?

1 0

[Watcher]no topics and cancelling the irc meeting today
by licanwei 11 Mar '20

11 Mar '20

1 0

[nova][ptl] Temporary Nova PTL until election
by Balázs Gibizer 10 Mar '20

10 Mar '20

Hi, Since Eric announced that he has to leave us [1] I have been working internally with my employee to be able to take over the Nova PTL position. Now I've got the necessary approvals. The official PTL election is close [2] and I'm ready to fill the PTL gap until the proper PTL election in April. Is this a workable solution for the community? Cheers, gibi [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012663… [2] https://governance.openstack.org/election/

9 8

[cyborg] Updating the list of core reviewers
by Nadathur, Sundar 10 Mar '20

10 Mar '20

Hello all, Brin Zhang has been actively contributing to Cyborg in various areas, adding new features, improving quality, reviewing patches, and generally helping others in the community. Despite the relatively short time, he has been one of the most prolific contributors, and brings an enthusiastic and active mindset. I would like to thank him and acknowledge his significant contributions by proposing him as a core reviewer for Cyborg. Shogo Saito has been active in Cyborg since Train release. He has been driving the Cyborg client improvements, including its revamp to use OpenStackSDK. Previously he was instrumental in the transition to Python 3, testing and fixing issues in the process. As he has access to real FPGA hardware, he brings a users' perspective and also tests Cyborg with real hardware. I would like to thank and acknowledge him for his steady valuable contributions, and propose him as a core reviewer for Cyborg. Some of the currently listed core reviewers have not been participating for a lengthy period of time. It is proposed that those who have had no contributions for the past 18 months - i.e. no participation in meetings, no code contributions and no reviews - be removed from the list of core reviewers. If no objections are made known by March 20, I will make the changes proposed above. Thanks. Regards, Sundar

1 0

[congress][ptl] new PTL for V-cycle
by Eric Kao 10 Mar '20

10 Mar '20

Hello all, Due to a change in my role, I do not intend to seek another term as Congress project PTL. I will continue through the end of the U-cycle. After that, I look forward to new voices and new perspectives to help us take the next step in the Congress journey. Cheers, Eric

1 0

OSC version
by Radosław Piliszek 10 Mar '20

10 Mar '20

Hiya Folks, while doing something else entirely (as usual!), I noticed something related to OSC version. OSC metapackage README [1] states that: "The major version of openstackclient will always correspond to the major version of python-openstackclient that will be installed." But OSC metapackage is 4.0.0 atm and installs latest python-osc which is 5.0.0. I don't follow that "correspondence" unless it was meant to mean "always not less than". :-) Still confusing. [1] https://pypi.org/project/openstackclient/ -yoctozepto

1 0

[horizon][kolla] pyscss failure on newest setuptools
by Marcin Juszkiewicz 10 Mar '20

10 Mar '20

One of Horizon's requirements is pyscss package. Which had last release over 4 years ago... Two days ago setuptools v46 got released. One of changes was drop of Features feature. Today Kolla builds started to fail: INFO:kolla.common.utils.horizon:Collecting pyScss===1.3.4 INFO:kolla.common.utils.horizon: Downloading http://mirror.ord.rax.opendev.org:8080/pypifiles/packages/1d/4a/221ae7561c8… (120 kB) INFO:kolla.common.utils.horizon:[91m ERROR: Command errored out with exit status 1: INFO:kolla.common.utils.horizon: command: /var/lib/kolla/venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rr0db3qs/pyScss/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rr0db3qs/pyScss/pip-egg-info INFO:kolla.common.utils.horizon: cwd: /tmp/pip-install-rr0db3qs/pyScss/ INFO:kolla.common.utils.horizon: Complete output (5 lines): INFO:kolla.common.utils.horizon: Traceback (most recent call last): INFO:kolla.common.utils.horizon: File "<string>", line 1, in <module> INFO:kolla.common.utils.horizon: File "/tmp/pip-install-rr0db3qs/pyScss/setup.py", line 9, in <module> INFO:kolla.common.utils.horizon: from setuptools import setup, Extension, Feature INFO:kolla.common.utils.horizon: ImportError: cannot import name 'Feature' Are there any plans to fix it? pyscss project got issue: https://github.com/Kronuz/pyScss/issues/385 In Kolla I made ugly workaround: https://paste.centos.org/view/2e29d284 What are plans of Horizon team?

1 0

[openstack-helm] VIrtual Midcycle - March 2020
by Gage Hugo 10 Mar '20

10 Mar '20

Hello everyone, just a reminder that we are looking to host a virtual midcycle later this month. If you wish to attend, please fill out the poll[0] and etherpad[1] if you have any topics to discuss. We are looking to host a roughly 4 hour session virtually either next week or the following week. [0] https://doodle.com/poll/g6uvdb4rbad9s8gb [1] https://etherpad.openstack.org/p/osh-virtual-ptg-2020-03

1 0

[all] Victoria schedule
by Sean McGinnis 10 Mar '20

10 Mar '20

Hello everyone, The proposed schedule for the Victoria release has now been approved and the schedule published: https://releases.openstack.org/victoria/schedule.html Some of the key dates for Victoria: * Milestone 1 - June 18 * Milestone 2 - July 30 * Milestone 3 - Sept 10 * RC1 deadline - Sept 24 * Final RC deadline - Oct 8 * Victoria coordinated release - Oct 14 Thanks! Sean

1 0

[nova] US meeting slot
by Balázs Gibizer 10 Mar '20

10 Mar '20

Hi, Nova has alternate meeting slots on Thursdays to try to cover contributors from different time zones. * 14:00 UTC * 21:00 UTC As I'm taking over the PTL role form Eric I need to figure out how to run the nova meetings. I cannot really run the UTC 21:00 as it is pretty late for me. (I will run the 14:00 UTC slot). I see different options: a) Somebody from the US side of the globe volunteers to run the 21:00 UTC slot. Please speak up if you would like to run it. I can help you with agenda refresh and technicalities if needed. b) Have only one meeting time, and move that to 16:00 UTC. In this case I will be able to run it most of the weeks. c) Do not have a dedicated meeting slot but switch to office hours. Here we also need to find a time slot. I think 16:00 UTC could work there as well. Please share your view! Any other proposal is very welcome. Cheers, gibi

7 10

[nova][stable] Nova stable branch liaison
by Balázs Gibizer 10 Mar '20

10 Mar '20

Hi Team, According to the wiki [1] we still have Matt as a stable branch liaison for nova. Obviously we need to find somebody else as Matt is gone. I'm not part of the nova-stable-core team so I cannot assume default ownership of that. Also I see that we have a small but active stable team so I hope somebody from that team can step forward and can take this role. Cheers, gibi [1] https://wiki.openstack.org/wiki/CrossProjectLiaisons#Stable_Branch

2 2

[Watcher]about meeting on March 11
by licanwei 09 Mar '20

09 Mar '20

Hi, We will have the team meeting tomorrow at 08:00 UTC on #openstack-meeting-alt. pls update the meeting agenda if you have something want to be discussed. if nothing need to be discussed, maybe i'll cancel the meeting. thanks, licanwei | | licanwei_cn | | 邮箱：licanwei_cn(a)163.com | 签名由网易邮箱大师定制

1 0

[all][ideas] Introducing Project Teapot: a baremetal cloud for the 2020s
by Zane Bitter 09 Mar '20

09 Mar '20

I'm pleased to announce the first entry on the new Ideas site (I wanted to call it Crazy Ideas, but JP overruled me and he did all the work to set it up), which also happens to be the reason the Ideas site exists: Project Teapot. (Before we go any further; we all wear a lot of hats in this community, so I'd like to make clear that I'm writing this email wearing my cap-and-bells in my capacity as self-appointed spokes-jester for Project Teapot, and no other.) What is Project Teapot? It's a new kind of cloud that has only one type of workload: Kubernetes on bare metal. Intrigued? Read on: https://governance.openstack.org/ideas/ideas/teapot/index.html What does this have to do with OpenStack? Well, it turns out that the OpenStack community has already built a lot (but not all) of the implementations of things that a cloud like this would need, in the form of projects like Ironic, Manila, Cinder, Keystone, Designate, Barbican. Plus we think it could probably be run as an OpenStack service alongside an existing OpenStack cloud when desired as well. Thanks are due to all of the folks who helped develop this idea, and the domain experts who reviewed it to hopefully eliminate the most egregious errors. Now it's over to you. If you need this, or want to help implement it, we'd like to hear from you on this thread. If you think this is a terrible idea and we should all be run out of town on a rail, we want to hear that too! (Pro tip: use the hashtag #ProjectCrackpot when you complain about it on Twitter.) If you have corrections or additional implementation ideas to add, feel free to submit a patch to the openstack/ideas repo. You can also add questions as inline comments on the original review (https://review.opendev.org/710173) if you want. It might pay to flag anything you post to Gerrit in this thread as well to make sure it's not missed. Finally, if you have ideas of your own, please submit them to the Ideas repo. Remember, they can be as crazy as you want. Let's not let the wisdom of the OpenStack community remain locked up in individual heads. cheers, Zane.

1 0

[qa][cinder][devstack] proposed governance changes for some devstack plugins
by Brian Rosmaita 09 Mar '20

09 Mar '20

Hello QA team and devstack-plugin-ceph-core people, The Cinder team has some proposals we'd like to float. 1. The Cinder team is interested in becoming more active in the maintenance of openstack/devstack-plugin-ceph [0]. Currently, the devstack-plugin-ceph-core is https://review.opendev.org/#/admin/groups/1196,members The cinder-core is already represented by Eric and Sean; we'd like to replace them by including the cinder-core group. 2. The Cinder team is interested in becoming more active in the maintenance of x/devstack-plugin-nfs [1]. Currently, the devstack-plugin-nfs-core is https://review.opendev.org/#/admin/groups/1330,members It's already 75% cinder-core members; we'd like to replace the individual members with the cinder-core group. We also propose that devstack-core be added as an included group. 3. The Cinder team is interested in implementing a new devstack plugin: openstack/devstack-plugin-open-cas This will enable thorough testing of a new feature [2] being introduced as experimental in Ussuri and expected to be finalized in Victoria. Our plan would be to make both cinder-core and devstack-core included groups for the gerrit group governing the new plugin. 4. This is a minor point, but can the devstack-plugin-nfs repo be moved back into the 'openstack' namespace? Let us know which of these proposals you find acceptable. [0] https://opendev.org/openstack/devstack-plugin-ceph [1] https://opendev.org/x/devstack-plugin-nfs [2] https://blueprints.launchpad.net/cinder/+spec/support-volume-local-cache

3 8

[oslo][infra] Oslo core security team on Launchpad
by Ben Nemec 09 Mar '20

09 Mar '20

Hi, I just noticed that the Oslo core security team includes a number of people no longer active in Oslo and also only me for current cores. We should really clean that up so random people aren't getting notified of private security bugs and ideally add some current cores so we have more eyes on said security bugs. How do we go about doing that? I see it's owned by the OpenStack Administrators team, so do I put in a request with the changes or can they just make me an administrator for that group? Thanks. -Ben

2 2

[cinder] reminder: ussuri virtual mid-cycle 16 march at 12:00 UTC
by Brian Rosmaita 09 Mar '20

09 Mar '20

(Feedback from the last virtual meet-up was that it needed more promotion, so if you know anyone who might be (or should be) interested, please tell them, in addition to marking your own calendar.) Session Two of the Cinder Ussuri virtual mid-cycle will be held: DATE: Monday, 16 March 2020 TIME: 1200-1400 UTC LOCATION: https://bluejeans.com/3228528973 The meeting will be recorded. Please add topics to the planning etherpad: https://etherpad.openstack.org/p/cinder-ussuri-mid-cycle-planning cheers, brian

1 0

[all][tc] Stepping down from TC
by Alexandra Settle 09 Mar '20

09 Mar '20

Hi all, This should come as no shock as I have been relatively quite for some time now, but I will not standing for the Technical Committee for a second term. I have thoroughly enjoyed my tenure, learning so much about open source governance than I ever thought I needed 😉 My work takes me elsewhere, as it did several years ago, and I simply do not have the time to manage both. I encourage anyone who is interested in governance, or is passionate about OpenStack and wants to learn more, to stand for the TC elections. As was proven by my own nomination and subsequent successful election, you do not have to be "purely technical" to stand and be a part of something great. Diversity of skill is so important to our survival. Thanks to all those that have supported me to get to the point, I appreciate you all and will miss working intimately with the community. Please do not hesitate to reach out and ask any questions if you are interested in the positions available, happy to help encourage and answer any questions you may have. All the best, Alex ________________________________ Alexandra Settle Senior Technical Writer London, United Kingdom (GMT)

4 3

[neutron] bug deputy report for week of 2020-03-02
by Bence Romsics 09 Mar '20

09 Mar '20

Hi All, Here's the deputy report for the week of 2020-03-02. Please note we have a new rotation schedule at the usual place: https://wiki.openstack.org/wiki/Network/Meetings High * https://bugs.launchpad.net/neutron/+bug/1865453 neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestVirtualPorts.test_virtual_port_created_before fails randomly Random error in the gate. Maciej is working on it. * https://bugs.launchpad.net/neutron/+bug/1866039 [OVN] QoS gives different bandwidth limit measures than ml2/ovs Clearing prerequisites of testing qos in tempest against ovn backend. Work in progress by Maciej: https://review.opendev.org/711048 * https://bugs.launchpad.net/neutron/+bug/1866336 Binding of floating ip agent gateway port and agent_id isn't removed Work in progress by Slawek: https://review.opendev.org/711623 * https://bugs.launchpad.net/neutron/+bug/1866560 FIP Port forwarding description API extension don't work Work in progress by Slawek: https://review.opendev.org/711888 Medium * https://bugs.launchpad.net/neutron/+bug/1865891 Race condition during removal of subnet from the router and removal of subnet Downstream bug reproduced on master too. Slawek is working on it. * https://bugs.launchpad.net/neutron/+bug/1866068 [OVN] neutron_pg_drop port group table creation race condition Work in progress by Jakub: https://review.opendev.org/711404 * https://bugs.launchpad.net/neutron/+bug/1866160 Update security group failed with the same stateful data Work in progress by Lina: https://review.opendev.org/711385 RFE * https://bugs.launchpad.net/neutron/+bug/1865889 Routed provider networks support in OVN To be scheduled and discussed on the drivers meeting. Could turn into an RFE * https://bugs.launchpad.net/neutron/+bug/1866077 [L3][IPv6] IPv6 traffic with DVR in compute host Incomplete * https://bugs.launchpad.net/neutron/+bug/1866139 GARP not sent on provider network after live migration * https://bugs.launchpad.net/neutron/+bug/1866445 br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes Invalid * https://bugs.launchpad.net/neutron/+bug/1866353 Neutron API returning HTTP 201 for SG rule create when not fully created yet octavia-ovn-provider * https://review.opendev.org/711244 [OVN Octavia Provider] Deleting of listener fails Work in progress by Maciej: https://review.opendev.org/711244 Cheers, Bence (rubasov)

1 0

Re: [Release-job-failures] Release of openstack/monasca-agent for ref refs/tags/2.8.1 failed
by Thierry Carrez 09 Mar '20

09 Mar '20

zuul(a)openstack.org wrote: > Build failed. > > - release-openstack-python https://zuul.opendev.org/t/openstack/build/25e1809c970044708c503cda05ac84f9 : SUCCESS in 7m 24s > - announce-release https://zuul.opendev.org/t/openstack/build/646a0bbaa0054504a863b3900b476006 : FAILURE in 7m 55s > - propose-update-constraints https://zuul.opendev.org/t/openstack/build/5d85de0f42c740299fb5c6c7f24f3aa1 : SUCCESS in 4m 42s Analysis: Release announcement for monasca-agent failed due to the following transient email error: 451 Temporary local problem - please try later Release went out OK, only the announce was missed. -- Thierry Carrez (ttx)

1 0

[largescale-sig] Next meeting: Mar 11, 9utc
by Thierry Carrez 09 Mar '20

09 Mar '20

Hi everyone, The Large Scale SIG will have a meeting this week on Wednesday, Mar 11 at 9 UTC[1] in #openstack-meeting on IRC. As we evolve in DST hell (US having moved to summer time while Europe hasn't), please doublecheck when that falls for you: [1] https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200311T09 As always, the agenda for our meeting is available at: https://etherpad.openstack.org/p/large-scale-sig-meeting Feel free to add topics to it! Talk to you all on Wednesday, -- Thierry Carrez

1 0

Goodbye for now
by Albert Braden 09 Mar '20

09 Mar '20

My contract at Synopsys ends today, so I will have to continue my efforts to sign up as an Openstack developer at my next role. Thanks to everyone for all of the help and advice. Best wishes! Albert

2 1

[Horizon] Unable to access the dashboard page
by Rahul Sharma 09 Mar '20

09 Mar '20

Hi, I am trying to get OpenStack (Train) up and running on CentOS 7 by following the "OpenStack Installation Guide" provided on OpenStack's website and have completed installation of below components - Keystone - Glance - Placement - Nova - Networking After installing each component, I have also verified its operation and it seems to be working successfully. However, there is a problem I am facing after installing "Horizon" for dashboard services. In order to verify its operation, one is supposed to browse to the URL "http://controller/horizon/" where "controller" could be the hostname or IP address of the Node which is running the controller Browsing to the above URL throws an error "The requested URL /horizon was not found on this server." In the apache access logs, I see below error: " 103.44.50.92 - - [07/Mar/2020:13:14:52 +0000] "GET /horizon/ HTTP/1.1" 404 206 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" 103.44.50.92 - - [07/Mar/2020:13:14:52 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "http://3.21.90.63/horizon/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" " If I browse to the URL "http://controller/", the default "Testing123" page of apache gets loaded. Please assist. Thanks, Rahul

4 7

[CINDER] Snapshots export
by Alfredo De Luca 09 Mar '20

09 Mar '20

Hi all. We have our env with Openstack (Train) and cinder with CEPH (nautilus) backend. We are creating automatic volumes snapshots and now we'd like to export them as a backup/restore plan. After exporting the snapshots we will use Acronis as backup tool. I couldn't find the right steps/commands to exports the snapshots. Any info? Cheers -- *Alfredo*

5 6

[all][tc] What happened in OpenStack Governance recently
by Jean-Philippe Evrard 09 Mar '20

09 Mar '20

Hello, It's been a while there was no update on "what happened in governance recently?" (February). So here is a summary for you! We had two meetings since the last community update, February [1] and March [2]. We have established our first (business-focused) upstream investment opportunities for 2020, which currently consists of: - Goal champions - Consistent and secure policy defaults - QA developers You can see the latest version in [3]. The timing for the next PTL and TC elections is now determined [4]. We've introduced the ideas framework, allowing you to propose, follow, and track large scale changes of OpenStack, with their conversations history [5]. We are starting to split OpenDev out of OpenStack-infra. Our next release names are Victoria and Wallaby. In terms of community goals [6], we've changed the U goal for documentation, so please have a look if you haven't already [7][8], and/or talk to the goal champion Kendall (diablo_rojo). The 'Switch legacy Zuul jobs to native' goal has been selected for Victoria. Please talk to its champion, Luigi Toscano (tosky)! On the projects' side, we've clarified the guidelines to drop official project teams [9]. We want to retire teams more actively if necessary, to allow us to increase our focus. Some projects have changed: - networking-ovn won't get releases anymore, as it's merged back into neutron in U [10] - neutron-lbaas was retired in Ussuri [11] Have a good day to you all! Regards, Jean-Philippe Evrard (evrardjp) [1]: http://eavesdrop.openstack.org/meetings/tc/2020/tc.2020-02-06-14.00.html [2]: http://eavesdrop.openstack.org/meetings/tc/2020/tc.2020-03-05-14.00.html [3]: https://governance.openstack.org/tc/reference/upstream-investment-opportuni… [4]: https://review.opendev.org/#/c/708470/3/configuration.yaml [5]: http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012847… [6]: https://governance.openstack.org/tc/goals/selected/index.html [7]: https://review.opendev.org/#/c/708672/ [8]: https://review.opendev.org/#/c/709617/ [9]: https://review.opendev.org/#/c/707421/ [10]: https://review.opendev.org/#/c/705781/ [11]: https://review.opendev.org/#/c/705780/

1 0

[tc] April meeting
by Jean-Philippe Evrard 09 Mar '20

09 Mar '20

Hello everyone, It would be nice if you could update the agenda on the wiki [1] for the April meeting, happening on April, 2nd. Thank you in advance, Jean-Philippe Evrard (evrardjp). [1]: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[tc] March meeting
by Jean-Philippe Evrard 09 Mar '20

09 Mar '20

Hello, Here are the minutes for the march meeting [1]. There are a few action points for you, please have a look! Regards, Jean-Philippe Evrard (evrardjp) [1]: http://eavesdrop.openstack.org/meetings/tc/2020/tc.2020-03-05-14.00.html

1 0

[queens] [neutron]security_groups_log]
by Ignazio Cassano 08 Mar '20

08 Mar '20

Hello, I have queens installation based on centos7. Before implementing security groups logs, I had the following configuration in /etc/neutron/plugins/ml2/openvswitch_agent.ini: firewall_driver = iptables_hybrid Enabling security groups log I had to change it in : firewall_driver = openvswitch It seems to work end security logs are logged . After restarting kvm nodes and controllers, virtual machines do not live migrate. The firewall driver change could be the cause of my problem ? firewall_driver = openvswitch is mandatory for security groups log ? Please, any help ? I cannot reproduce the problem rebooting all my nodes. I rebooted them because I hat to transfer from a rack to another. Ignazio

2 5

[queens] [neutron]security_groups_log] issues
by Ignazio Cassano 07 Mar '20

07 Mar '20

Hello, I have queens installation based on centos7. Before implementing security groups logs, I had the following configuration in /etc/neutron/plugins/ml2/openvswitch_agent.ini: firewall_driver = iptables_hybrid Enabling security groups log I had to change it in : firewall_driver = openvswitch It seems to work end security logs are logged . After restarting kvm nodes and controllers, virtual machines do not live migrate. I rolled back my configuration without security groups logs feature and now all works fine. The firewall driver change could be the cause of my problem ? firewall_driver = openvswitch is mandatory for security groups log ? Please, any help ? I cannot reproduce the problem rebooting all my nodes. I rebooted them because I hat to transfer from a rack to another. Ignazio

1 0

[nova][qa] When creating a instance using a flavor with "hw:cpu_policy=dedicated" "hw:cpu_realtime=yes" and "hw:cpu_realtime_mask=^0-1" failed.
by Juntingqiu Qiujunting (邱军婷) 06 Mar '20

06 Mar '20

When I create a instance using a flavor with "hw:cpu_policy=dedicated" "hw:cpu_realtime=yes" and "hw:cpu_realtime_mask=^0-1". The error is "libvirtError:Cannot set scheduler parameters for pid 3666:operation not permitted". https://bugs.launchpad.net/starlingx/+bug/1866311 The instance xml as following: <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='3'/> <vcpupin vcpu='1' cpuset='11'/> <vcpupin vcpu='2' cpuset='0'/> <vcpupin vcpu='3' cpuset='8'/> <emulatorpin cpuset='3,11'/> <vcpusched vcpus='2' scheduler='fifo' priority='1'/> <vcpusched vcpus='3' scheduler='fifo' priority='1'/> </cputune> The error log as following: ERROR nova.compute.manager [instance: 74dcc0a1-9ed4-4d13-bef7-ac9623bca2d0] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags ERROR nova.compute.manager [instance: 74dcc0a1-9ed4-4d13-bef7-ac9623bca2d0] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) ERROR nova.compute.manager [instance: 74dcc0a1-9ed4-4d13-bef7-ac9623bca2d0] libvirtError: Cannot set scheduler parameters for pid 6504: Operation not permitted ERROR nova.compute.manager [instance: 74dcc0a1-9ed4-4d13-bef7-ac9623bca2d0]

2 1

Re: [sdk][congress][octavia][designate][magnum][senlin][swift] Adding project-specific cores to SDK
by Eric Fried 06 Mar '20

06 Mar '20

Any chance of reusing the “PTL ack” bot that has recently appeared in the releases repo? But as a “SME ack” that would recognize anyone from $project’s core team? (How does the releases bot know which project the patch is for? Might have to be a bit fuzzy on that logic for SDK/OSC.) Then the team could adopt a policy of single core approval if the patch has this SME +1, and no real danger of “abuse”. Eric Fried

4 4

[sdk][congress][octavia][designate][magnum][senlin][swift] Adding project-specific cores to SDK
by Monty Taylor 06 Mar '20

06 Mar '20

Heya, I’d like to try something. I’d like to try adding some project-specific people to the core team so that they can more directly help maintain the support for their service in SDK. In some of these cases the person I’m suggestion has next to no review experience in SDK. I think let’s be fine with that for now - we’re still a 2x +2 in general thing - but I know currently when reviewing neutron or ironic changes I always want to see a +2 from slaweq or dtantsur … so in the spirit of trying new things and trying to move the project forward in a healthy and welcoming way - how about we give this a try? The idea here is that we’re trusting people to use their good judgement and to only use their new +2 powers for good in their project. Over time, if they feel like they’ve gotten a handle on things more widely, there’s nothing stopping them from reviewing other patches - but I think that most of us aren’t looking for additional review work anyway. Specifically this would be: Shogo Saito - congress Adam Harwell - octavia Graham Hayes - designate Bharat Kumar - magnum Erik Olof Gunnar Andersson - senlin Tim Burke - swift I think we should also add a file in the repo that lists “subject matter experts” for each service we’ve got support for, where we have them. My list of current cores who I’d ping for specific service suitability are: Sean McGinnis - cinder Slawek Kaplonski - neutron Dmitry Tantsur - ironic Eric Fried - nova (at least until tomorrow my friend) How does that sound to folks? Monty

4 4