- openstack-discuss - lists.openstack.org

[tc] Add non-voting py38 for ussuri
by Corey Bryant 14 Nov '19

14 Nov '19

Hello TC members, Python 3.8 is available in Ubuntu Bionic now and while I understand it's too late to enable voting py38 unit tests for ussuri, I'd like to at least enable non-voting py38 unit tests. This email is seeking approval and direction from the TC to move forward with enabling non-voting py38 tests. For some further background: The next release of Ubuntu, Focal (20.04) LTS, is scheduled to release in April 2020. Python 3.8 will be the default in the Focal release, so I'm hopeful that non-voting unit tests will help close some of the gap. I have a review here for the zuul project template enablement for ussuri: https://review.opendev.org/#/c/693401 Also should this be updated considering py38 would be non-voting? https://governance.openstack.org/tc/reference/runtimes/ussuri.html Thanks, Corey

9 17

[neutron][docs] networking-onos EOL?
by Mark Goddard 14 Nov '19

14 Nov '19

Hi, We (kolla) had a bug report [1] from someone trying to use the neutron onos_ml2 ML2 driver for the ONOS SDN controller. As far as I can tell [2], this project hasn't been released since 2015. However, the 'latest' documentation is still accessible [3], and does not mention that the project is dead. What can we do to help steer people away from projects like this? Cheers, Mark [1] https://bugs.launchpad.net/bugs/1850763 [2] https://pypi.org/project/networking-onos/#history [3] https://docs.openstack.org/networking-onos/latest/

3 3

[nova][ptg] pinned and unpinned CPUs in one instance
by Balázs Gibizer 14 Nov '19

14 Nov '19

spec: https://review.opendev.org/668656 Agreements from the PTG: How we will test it: * do functional test with libvirt driver, like the pinned cpu tests we have today * donyd's CI supports nested virt so we can do pinned cpu testing but not realtime. As this CI is still work in progress we should not block on this. * coverage inhttps://opendev.org/x/whitebox-tempest-pluginis a nice to have Naming: use the 'shared' and 'dedicated' terminology Support both the hw:pinvcpus=3 and the resources:PCPU=2 flavor extra specs syntaxtbut not in the same flavor. The resources:PCPU=2 syntax will have less expression power until nova models NUMA in placement. So nova will try to evenly distribute PCPUs between numa nodes. If it not possible we reject the request and ask the user to use the hw:pinvcpus=3 syntax. Realtime mask is an exclusion mask, any vcpus not listed there has to be in the dedicated set of the instance. TODOInvestigate whether we want to enable NUMA by default * Pros: Simpler, everything is NUMA by default * Cons: We'll either have to break/make configurablethe 1:1 guest:host NUMA mapping else we won't be able to boot e.g. a 40 core shared instance on a 40 core, 2 NUMA node host Cheers, gibi

4 6

[nova][ptg] Expose auto converge and post copy
by Balázs Gibizer 14 Nov '19

14 Nov '19

spec: https://review.opendev.org/#/c/687199 There was multiple discussion during the PTG around this. I think at the end mdbooth and yawang found a possible solution that they liked and sounded OK to me too. Unfortunately the etherpad was not updated and my memory is bad enough that I cannot gather what was the exact proposal. yawang: could you please post a short summary here and / or simply update the spec with the discussed solution? Cheers, gibi

3 4

Re: [nova][ptg] Flavor explosion
by Brin Zhang(张百林) 14 Nov '19

14 Nov '19

Thanks Gibizer, I will update this specs to the ussuri directory, and update & reply your comments. brinzhang > 发件人: Balázs Gibizer [mailto:balazs.gibizer@est.tech] > > On Sun, Nov 10, 2019 at 16:09, Brin Zhang(张百林) > <zhangbailin(a)inspur.com> wrote: > > Hi all, > > Based on the discussion on the Train PTG, and reference to the > > records on the etherpad and ML, I was updated that SPEC, and I think > > there are some details need to be discussed, and I have listed some > > details, if there are any other things that I have not considered, or > > if some place that I thoughtless, please post a discussion. > > > > List some details as follows, and you can review that spec in > > https://review.opendev.org/#/c/663563. > > > > Listed details: > > - Don't change the model of the flavor in nova code and in the db. > > > > - No change for operators who choose not to request the flavor extra > > specs group. > > > > - Requested more than one flavor extra specs groups, if there are > > different values for the same spec will be raised a 409. > > > > - Flavor in request body of server create that has the same spec in > > the request ``flavor_extra_specs_group``, it will be raised a 409. > > > > - When resize an instance, you need to compare the > > ``flavor_extra_specs_group`` with the spec request spec, otherwise > > raise a 400. > > > > Thanks Brin for updating the spec, I did a review round on it and left comments. > > gibi >

1 0

[horizon] next weekly meeting cancelled
by Akihiro Motoki 14 Nov '19

14 Nov '19

Hi, The weekly team meeting next week (Nov 20) is cancelled. I will be on a business trip to join a conference in US and cannot run it. We agreed to cancel it in the team meeting yesterday. Akihiro Motoki (amotoki)

1 0

Keystone user ID case sensitivity issue
by PARSONS, CLIFF 14 Nov '19

14 Nov '19

Hello everyone! My organization has a need to make the user name/ID retrieval from Heat template to be case insensitive. For example: suppose we already have a user in keystone, "xyz123". Then we have a client that creates a heat stack containing a UserRoleAssignment resource, in which the user was specified as "XYZ123". The user would not be found in the Keystone database (due to Keystone user IDs being case sensitive) and the role assignment would not occur. Either Keystone could be changed so that its users are treated case insensitive, or we could make the change to heat (Heat KeystoneClientPlugin class) like in https://review.opendev.org/#/c/694117/ so that it converts to lower case before querying keystone. Can I get some thoughts on this? Would something like this be acceptable at all? Would we need to make it configurable, and if we did, would that be acceptable? Thanks in advance for your thoughts/concerns/suggestions. Thank you, Cliff Parsons

3 2

[nova][ptg] Allow compute nodes to use DISK_GB from shared storage RP by using aggregate relationship
by Balázs Gibizer 14 Nov '19

14 Nov '19

spec: https://review.opendev.org/#/c/650188/ Agreements from the room[1]: * new config option 'using_shared_disk_provider' (just an example name) on the compute level to ignore the DISK_GB reporting from the driver * new config option 'sharing_disk_aggregate' (just an example name) on the compute level to tell nova compute what is the UUID of the placement aggregate that contains the sharing DISK_GB providers in placement. * the "using_shared_disk_provider" flag necessarly has to be explicit since if not, it would be a chicken-and-egg problem on a greenfields install as the shared RP wouldn't be created * deployer needs to create the sharing disk RP and report inventory / traits on it * deployer needs to define the placement aggregate and add the sharing disk RP into it * when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape * os-hypervisor API response (in a new microversion) will have a link to the sharing disk RP if the compute is so configured. TODO: * tpatil to update the spec [1] https://etherpad.openstack.org/p/nova-shanghai-ptg

4 8

[nova] track error migrations and orphans in Resource Tracker
by Zhong, Luyao 14 Nov '19

14 Nov '19

Hi Nova experts, "Not tracking error migrations and orphans in RT." is probably a bug. This may trigger some problems in update_available_resources in RT at the moment. That is some orphans or error migrations are using cpus/memory/disk etc, but we don't take these usage into consideration. And instance.resources is introduced from Train used to contain specific resources, we also track assigned specific resources in RT based on tracked migrations and instances. So this bug will also affect the specific resources tracking. I draft an doc to clarify this bug and possible solutions: https://etherpad.openstack.org/p/track-err-migr-and-orphans-in-RT Looking forward to suggestions from you. Thanks in advance. Best Regards, Luyao

6 7

Filter costs / filter order
by Albert Braden 13 Nov '19

13 Nov '19

I'm running Rocky and trying to figure out filter order. I'm reading this doc: https://docs.openstack.org/nova/rocky/user/filter-scheduler.html It says: Each filter selects hosts in a different way and has different costs. The order of filter_scheduler.enabled_filters affects scheduling performance. The general suggestion is to filter out invalid hosts as soon as possible to avoid unnecessary costs. We can sort filter_scheduler.enabled_filters items by their costs in reverse order. For example, ComputeFilter is better before any resource calculating filters like RamFilter, CoreFilter. Is there a document that specifies filter costs, or ranks filters by cost? Is there a well-known process for determining the optimal filter order?

2 2

[ironic][ptg] Summary of discussions/happenings related to ironic
by Julia Kreger 13 Nov '19

13 Nov '19

Overall, There was quite a bit of interest in Ironic. We had great attendance for the Project Update, Rico Lin’s Heat/Ironic integration presentation, demonstration of dhcp-less virtual media boot, and the forum discussion on snapshot support for bare metal machines, and more! We also learned there are some very large bare metal clouds in China, even larger than the clouds we typically talk about when we discuss scale issues. As such, I think it would behoove the ironic community and OpenStack in general to be mindful of hyper-scale. These are not clouds with 100s of compute nodes, but with baremetal clouds containing thousands to tens of thousands of physical bare metal machines. So in no particular order, below is an overview of the sessions, discussions, and commentary with additional status where applicable. My apologies now since this is over 4,000 words in length. Project Update =========== The project update was fairly quick. I’ll try and record a video of it sometime this week or next and post it online. Essentially Ironic’s code addition/deletion levels are relatively stable cycle to cycle. Our developer and Ironic operator commit contribution levels have increased in Train over Stein, while the overall pool of contributors has continued to decline cycle after cycle, although not dramatically. I think the takeaway from this is that as ironic has become more and more stable, and that the problems being solved in many cases are operator specific needs or wants, or bug fixes in cases that are only raised in particular environment configurations. The only real question that came out of the project update was, if my memory is correct, was “What does Metal^3 mean for Ironic”, and “Who is driving forward Metal^3?” The answers are fairly straight forward, more ironic users and more use cases from Metal^3 driving ironic to deploy machines. As for who is driving it forward, it is largely being driven forward by Red Hat along with interested communities and hardware vendors. Quick, Solid, and Automatic OpenStack Bare-Metal Orchestration ================================================== Rico Lin, the Heat PTL, proposed this talk promoting the possibility of using ironic naively to deploy bare metal nodes. Specifically where configuration pass-through can’t be made generic or somehow articulated through the compute API. Cases where they may be is where someone wishes to utilize something like our “ramdisk” deploy_interface which does not deploy an image to the actual physical disk. The only real question that I seem to remember coming up was the question why might someone want or need to do this, which again becomes more of a question of doing things that are not quite “compute” API-ish. The patches are available in gerrit[10]. Operator Feedback Session ===================== The operator feedback[0] session was not as well populated with maybe ~20-25 people present. Overall the feeling of the room was that “everything works”, however there is a need and desire for information and additional capabilities. * Detailed driver support matrix * Reduce the deployment times further * Disk key rotation is an ask from operators for drives that claim smart erase support but end up doing a drive wipe instead. In essence, to reduce the overall time spent cleaning * Software RAID is needed at deploy time. * IPA needs improved error handling. - This may be a case where something of the communication flow changes that had been previously discussed could help in that we could actively try and keep track of the agent a little more. Additional discussion will definitely be required. * There does still seem to be some interest in graphical console support. A contributor has been revising patches, but I think it would really help for a vendor to become involved here and support accessing their graphical interface through such a method. * Information and an information sharing location is needed. I’ve reached out to the Foundation staff regarding the Bare Metal Logo Program to see if we can find a common place that we can build/foster moving forward. In this topic, the one major pain point began being stressed, issues with the resource tracker at 3,500 bare metal nodes. Privately another operator reached out with the same issue in the scale of tens of thousands of bare metal nodes. As such, this became a topic during the PTG which gained further discussion. I’ll cover that later. Ironic – Snapshots? =============== As a result of some public discussion of adding snapshot capability, I proposed a forum session to discuss the topic[1] such that requirements can be identified and the discussion can continue over the next cycle. I didn't expect the number of attendees present to swell from the operator's feedback session. The discussion of requirements went back and forth to ultimately define "what is a snapshot" in this case, and "what should Ironic do?" There was quite a bit of interaction in this session and the consensus seemed to be the following: * Don’t make it reliant on nova, for standalone users may want/need to use it. * This could be a very powerful feature as an operator could ``adopt`` a machine into ironic and then ``snapshot`` it to capture the disk contents. * Block level only and we can’t forget about capturing/storing content checksum * Capture the machine’s contents with the same expectation as we would have for a VM, and upload this to someplace. In order to make this happen in a fashion which will scale, the ironic team will likely need to leverage the application credentials. Ironically reeling in large bare metal deployment without PXE ============================================== This was a talk submitted by Ilya Etingof, who unfortunately was unable to make it to the summit. Special thanks goes to Both Ilya and Richard Pioso for working together to make this demonstration happen. The idea was to demonstrate where the ironic team sees the future of deployment of machines on the edge using virtual media and how vendors would likely interact with that in some cases as slightly different mechanics may be required even if the BMCs all speak Redfish, which is the case for a Dell iDRAC BMC. The idea[2] ultimately being is that the conductor would inject the configuration information into the virtual media ISO image that is attached via virtual media negating the need for DHCP. We have videos posted that allow those interested to see what this functionality looks like with neutron[3] and without neutron[4]. While the large audience was impressed, it seemed to be a general surprise that Ironic had virtual media support in some of the drivers previously. This talk spurred quite a bit of conversation and hallway track style discussion after the presentation concluded which is always an excellent sign. Project Teams Gathering =================== The ironic community PTG attendance was nothing short of excellent. Thank you everyone who attended! At one point we had fifteen people and a chair had to be pulled up to our table for a 16th person to join us. At which point, we may have captured another table and created confusion. We did things a little differently this time around. Given some of the unknowns, we did not create a strict schedule around the topics. We simply went through and prioritized topics and tried to discuss them each as thoroughly as possible until we had reached the conclusion or a consensus on the topic. Topics and a few words on each topic we discussed in the notes section on the PTG etherpad[5]. On-boarding ----------------- We had three contributors that attended a fairly brief on-boarding overview of Ironic. Two of them were more developer focused where as the third was more of an operator focus looking to leverage ironic and see how they can contribute back to the community. BareMetal SIG - Next Steps ------------------------------------- Arne Wiebalck and I both provided an update including current conversations where we saw the SIG, the Logo Program, the white paper, and what should the SIG do beyond the whitepaper. To start with the Logo program, it largely seems there that somewhere along the way a message or document got lost and that largely impacted the Logo Program -> SIG feedback mechanism. I’m working with the OpenStack Foundation to fix that and get communication going again. Largely what spurred that was that some vendors expressed interest in joining, and wanted additional information. As for the white paper, contributions are welcome and progress is being made again. >From a next steps standpoint, the question was raised how do we build up an improved Operator point of contact. There was some consensus that we as a community should try to encourage at least one contributor to attend the operations mid-cycles. This allows for a somewhat shorter feedback look with a different audience. We also discussed knowledge sharing, or how to improve it. Included with this is how do we share best practices. I’ve got the question out to folks at the foundation if there is a better way as part of the Logo program, or if we should just use the Wiki. I think this will be an open discussion topic in the coming weeks. The final question that came up as part of the SIG is how to show activity. I reached out to Amy on the UC regarding this, and it seems the process is largely just reach out to the current leaders of the SIG, so it is critical that we keep that up to date moving forward. Sensor Data/Metrics --------------------------- The barrier between Tenant level information and Operator level information is difficult with this topic. The consensus among the group was that the capability to collect some level of OOB sensor data should be present on all drivers, but there is also a recognition that this comes at a cost and possible performance impact. Mainly this performance impact question was raised with Redfish because this data is scattered around the API where multiple API calls are required, and may even cause some interruption to actively inquire upon some data points. The middle ground in the discussion came to adding a capability of somehow saying “collect power status, temp every minute, fan speeds every five minutes, drive/cpu health data maybe every 30 minutes”. I would be remiss if I didn't note that there was joking about how this would in essence be re-implementation of Cron. What this would end up looking like, we don’t know, but it would provide operators the data resolution necessary for the failure risk/impact. The analogy used was that “If the temperature sensor has risen to an alarm level, either a AC failure or a thermal hot spot forming based upon load in the data center, checking the sensor too often is just not going to result in a human investigating that on the data center floor any faster.” Mainly I believe this discussion largely stresses that the information is for the operator of the bare metal and not to provide insight into a tenant monitoring system, that those activities should largely be done with-in the operating system. One question among the group was if anyone was using the metrics framework built into ironic already for metrics of ironic itself, to see if we can re-use it. Well, it uses a plugin interface! In any event, I've sent a post to the openstack-discuss mailing list seeking usage information. Node Retirement ----------------------- This is a returning discussion from the last PTG, and in discussing the topic we figured out where the discussion became derailed at previously. In essence, the desire was to mix this with the concept of being able to take a node “out of service”. Except, taking a node out of service is an immediate state related flag, where as retiring might be as soon as the current tenant vacates the machine… possibly in three to six months. In other words, one is “do something or nothing now”, and the other is “do something later when a particular state boundary is crossed”. Trying to make one solution for both, doesn’t exactly work. Unanimous consensus among those present was that in order to provide node retirement functionality, that the logic should be similar to maintenance/maintenance reason. A top level field in the node object that would allow API queries for nodes slated for retirement, which helps solve an operator workflow conundrum “How do I know what is slated for retirement but not yet vacated?” Going back to the “out of service” discussion, we reached consensus that this was in essence a “user declarable failed state”, and as such that it should be done only in the state machine as it is in the present, not a future action. Should we implement out of service, we’ll need to check the nova.virt.ironic code and related virt code to properly handle nodes dropping from `ACTIVE` state, which could also be problematic and need to be API version guarded to prevent machines from accidentally entering `ERROR` state if they are not automatically recovered in nova. Multi-tenancy ------------------ Lots of interest existed around making the API somewhat of a mutli-tenant aware interaction, and the exact interactions and uses involved there are not exactly clear. What IS clear is that providing functionality as such will allow operators to remove complication in their resource classes and tenant specific flavors which is presently being used to enable tenant specific hardware pools. The added benefit of providing some level for normally non-admin users to access the ironic API is that it would allow those tenants to have a clear understanding of their used resources and available resources by directly asking ironic, where as presently, they don’t have a good way to collect nor understand that short of asking the cloud operator when it comes to bare metal. Initial work has been posted for this to gerrit[6]. In terms of how tenants resources would be shared, there was consensus that the community should stress that new special use tenants should be created for collaborative efforts. There was some discussion regarding explicitly dropping fields for non-privileged users that can see the nodes, such as driver_info and possibly even driver_internal_info. Definitely a topic that requires more discussion, but that would solve operator reporting and use headaches. Manual Cleaning Out-Of-Band ---------------------------------------- The point was raised that we unconditionally start the agent ramdisk to perform manual cleaning. Except, we should support a method of out of band cleaning operators to only be executed so the bare metal node doesn’t need to be booted to a ramdisk. The consensus seemed to be that we should consider a decorator or existing decorator change that allows the conductor to hold off actually powering the node on for ramdisk boot unless or until a step is reached that is not purely out of band. In essence, fixing this allows a “fix_bmc” out of band clean step to be executed first without trying to modify BMC settings, which would presently fail. Scale issues ----------------- A number of scaling issues between how nova and ironic interact, specifically with the resource tracker and how inventory is updated from ironic and loaded into nova. Largely this issue revolves around the concept in nova that each ``nova-compute`` is a hypervisor. And while one can run multiple ``nova-compute`` processes to serve as the connection to ironic, the underlying lock in Nova is at the level of the compute node, not the node level. This means as thousands of records are downloaded, synced, copied into the resource tracker, the compute process is essentially blocked from other actions while this serialized job runs. In a typical VM case, you may only have at most a couple hundred VMs on a hypervisor, where as with bare metal, we’re potentially servicing thousands of physical machines. It should be noted that there are several large scale operators that indicated during the PTG that this was their pain point. Some of the contributors from CERN sat down with us and the nova team to try and hammer out a solution to this issue. A summary of that cross project session can be found at line 212 in the PTG etherpad[0]. But there is another pain point that contributes to this performance issue and that is the speed at which records are returned by our API. We’ve had some operators voice some frustration with this before, and we should at least be mindful of this and hopefully see if we can improve record retrieval performance. In addition to this, if we supported some form of bulk “GET” of nodes, it might be able to be leveraged as opposed to a get on each node one at a time which is presently what occurs in the nova-compute process. Boot Mode Config ------------------------ Previously, when scheduling occurred with flavors and filters were appropriately set, if a machine was declared as supporting only one boot mode, requests would only ever land on that node. Now with Traits, this is a bit different and unfortunately optional without logic to really guard the setting application for an instance. So in this case, if filters are such that a request for a Legacy boot instance lands on a UEFI only machine, we’ll still try to deploy it. In reality, we really should try and fail fast. Ideally the solution here is we consult with the BMC through some sort of get_supported_boot_modes method, and if we determine a mismatch between what the settings are or what the requested instance is from the data we have, we fail the deploy. This ultimately may require work in the nova.virt.ironic driver code to identify the cause of the failure being an invalid configuration and reporting that back, however it may not be fatal on another machine. Security of /heartbeat and /lookup endpoints ----------------------------------------------------------- We had a discussion of adding some additional layers of security mechanics around the /heartbeat and /lookup endpoints in ironic’s REST API. These limited endpoints are documented as being unauthenticated, so naturally some issues can arise from these and we want to minimize the vectors in which an attacker that has gained access to a cleaning/provisioning/rescue network could possibly impersonate a running ironic-python-agent. Conversely, the ironic-python-agent runs in a similar fashion, intended to run on secure trusted networks which is only accessible to the ironic-conductor. As such, we also want to add some validation to the API request is from the same Ironic deployment that IPA is heart-beating to. The solution to this introduce a limited lifetime token that is unique per node per deployment. It would be stored in RAM on the agent, and in the node.driver_internal_info so it is available to the conductor. It would be provided only once via out of band OR via the first “lookup” of a node, and then only become accessible again during known reboot steps. Conceptually the introduction of tokens was well supported in the discussions and there were zero objections to doing so. Some initial patches[7][8] are under development to move this forward. An additional item is to add IP address filtering capabilities to both endpoints such that we only process the heartbeat/lookup address if we know it came from the correct IP address. An operator has written this feature downstream and consensus was unanimous at the PTG that we should accept this feature upstream. We should expect a patch for this functionality to be posted soon. Persistent Agents ------------------------ The use case behind persistent agents is “I want to kexec my way to the agent ramdisk, or the next operating system.” and “I want to have up to date inspection data.” We’ve already somewhat solved the latter, but the former is a harder problem requiring the previously mentioned endpoint security enhancements to be in-place first. There is some interest from CERN and some other large scale operators. In other words, we should expect more of this from an bare metal fleet operations point of view for some environments as we move forward. “Managing hardware the Ironic way” ------------------------------------------------- The question that spurred this discussion was “How do I provide a way for my hardware manager to know what it might need to do by default.” Except, those defaults may differ between racks that serve different purposes. “Rack 1, node0” may need a port set to FiberChannel mode, where as “Rack2, node1” may require it to be Ethernet. This quickly also reaches the discussion of “What if I need different firmware versions by default?” This topic quickly evolved from there and the idea that surfaced was that we introduce a new field on the node object for the storage of such data. Something like ``node.default_config``, where it would be a dictionary sort of like what a user provides for cleaning steps or deploy steps, that provides argument values which is iterated through when in automated cleaning mode to allow operators to fill in configuration requirement gaps for hardware managers. Interestingly enough, even today we just had someone ask a similar question in IRC. This should ultimately be usable to assert desired/default firmware from an administrative point of view. Adrianc (Mellanox) is going to reach out to bdobb (DMTF) regarding the redfish PLDM firmware update interface to see where this may go from here. Edge computing working group session ---------------------------------------------------- The edge working group largely became a session to update everyone on where Ironic was going and where we see things going in terms of managing bare metal at the edge/far-edge. This included some in-depth questions about dhcp-less deployment and related mechanics as well as HTTPBoot’ing machines. Supporting HTTPBoot does definitely seem to be of interest to a number of people, although at least after sharing my context only five or six people in attendance really seemed interested in ironic prioritizing such functionality. The primary blocker, for those that are unaware, is pre-built UEFI images for us to do integration testing for IPv4 HTTPBoot. Functionally ironic already supports IPv6 HTTPBoot via DHCPv6 as part of our IPv6 support with PXE/iPXE, however we also don’t have an integration test job for this code path for the same reason, pre-built UEFI firmware images lack the built-in support. More minor PTG topics ------------------------------- * Smartnics - A desire to attach virtual ports in ironic baremetal nodes with smartnics was raised. Seems that we don’t need to try and create a port entry in ironic. It seems we only need to track/signal and remove the “vif” attachment” to the node in general as there is no physical mac required for that virtual port in ironic. The constraint that at least one MAC address would be required to identify the machine is understood. If anyone sees an issue with this, please raise this to adrianc. * Metal^3 - Within the group attending the PTG, there was not much interest in Metal^3 or using CRDs to manage bare metal resources with ironic hidden behind the CRD. One factor related to this is the desire to define more data to be passed through to ironic which is not presently supported in the CRD definition.. Stable Backports with Ironic's release model ================================== I was pulled into a discussion with the TC and the Stable team regarding frustrations that have been expressed with-in the ironic team regarding stable back-porting of fixes, mainly drivers. There is consensus that it is okay for us as the ironic team to backport drivery things when needed to support vendors as long as they are not breaking or overall behavior contracts. This quickly leads us to needing to also modify constraints for drivery things as well. Constraints changes will continue to be evaluated on a case by case basis, but the general consensus is there is full support to "do the right thing" for ironic's users, vendors, and community. The key is making sure we are on the same page and agreeing to what that right thing is. This is where asynchronous communication can get us into trouble, and I would highly encourage trying to start higher bandwidth discussion when these cases arise in the future. The key takeaway that we should likely keep in mind is policy is there for good reasons, but policy is not and can not be a crutch to prevent the right thing from being done. Additional items worth noting - Q1 Gatherings =================================== There will be an operations mid-cycle at Bloomberg in London, January 7th-8th, 2020. It would be good if at least one ironic contributor could attend as the operators group tends to be closer to the physical baremetal, and it is a good chance to build mutual context between developers and operations people actually using our software. Additionally, we want to gauge the interest of having an ironic mid-cycle in central Europe in Q1 of 2020. We need to identify the number of contributors that would be interested in and able to attend since the next PTG will be in June. Please email me off-list if your interested in attending and I'll make a note of it as we're still having initial discussions. And now I've reached a buffer under-run on words. If there are any questions, just reply to the list. -Julia Links: [0]: https://etherpad.openstack.org/p/PVG-ironic-operator-feedback [1]: https://etherpad.openstack.org/p/PVG-ironic-snapshot-support [2]: https://review.opendev.org/#/c/672780/ [3]: https://tinyurl.com/vwts36l [4]: https://tinyurl.com/st6azrw [5]: https://etherpad.openstack.org/p/PVG-Ironic-Planning [6]: https://review.opendev.org/#/c/689551/ [7]: https://review.opendev.org/692609 [8]: https://review.opendev.org/692614 [9]: https://etherpad.openstack.org/p/ops-meetup-1st-2020 [10]: https://review.opendev.org/#/q/topic:story/2006403+(status:open+OR+status:m…

1 1

[cinder] virtual PTG datetime selection poll
by Brian Rosmaita 13 Nov '19

13 Nov '19

At the Shanghai PTG last week, the Cinder team decided to hold a Virtual PTG the last week of November. The format will be 2 two-hour sessions, ideally held on consecutive days. Since most people have the 16:00 UTC Cinder meeting already on their calendars, I suggest that we skip the meeting on 27 November and instead use that time for Virtual PTG. So, basically, I'd like to meet: Wednesday for sure, and either Tuesday or Thursday (with Monday or Friday as possibilities if Tuesday or Thursday are impossible for too many people). Let me know what your preferences are on this poll: https://forms.gle/rKDJpSZvAxbnBESp7 The poll closes at 23:39 on *Tuesday* 19 November 2019. thanks, brian

1 1

[cinder] meeting time change poll
by Brian Rosmaita 13 Nov '19

13 Nov '19

The poll to help decide whether we move the time of the weekly Cinder meeting is now available: https://forms.gle/kA2JGzoxegy2KRDB6 The poll closes on 20 November at 23:59 UTC. If the time is changed, it will be effective for the first meeting in December (4 December 2019). cheers, brian

1 1

[ironic] Anyone using the process metrics collection functionality in ironic?
by Julia Kreger 13 Nov '19

13 Nov '19

A question from the PTG was raised if anyone was using the existing statsd metrics publishing support in ironic/ironic-python-agent to publish internal performance times? There seems to be some interest in expanding this metric data publishing capability so Prometheus can also be used, but before we really even think of heading down that path, we wanted to understand if there were present users of that feature. Thanks, -Julia

1 0

[blazar] Why/how does Blazar use Keystone trusts?
by Jason Anderson 13 Nov '19

13 Nov '19

Hi Blazar contributors, We hit an issue today involving trusts in Blazar, where a host couldn't be deleted due to some issue authenticating against the trust associated with the host. We still haven't resolved this issue, but it felt odd to me: why is a trust even involved here? I have often wondered what the reason is for using trusts in Blazar, as I can't think of anything Blazar is doing that could not be done by the Blazar system user (and in fact, many operations are done via this user... via another trust.) There are also issues where a user leaves a project before their leases have ended; in this case Blazar has difficulty cleaning up because it tries to resurrect a trust that is not tied to a valid user/project relationship. Does anybody have context on to how trusts are used in Blazar and if they are still necessary? Does it make sense to remove this functionality? Thank you, -- Jason Anderson Chameleon DevOps Lead Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

3 3

[cinder] meeting time reminder
by Brian Rosmaita 13 Nov '19

13 Nov '19

For people who were on Daylight Savings Time but now are not, just a reminder that this week's Cinder meeting at 16:00 UTC may be an hour earlier for you. The meeting-time-change-poll link will be in a separate email. If the time is changed, it will be effective for the first meeting in December (4 December 2019). That's because we'll be having the Virtual PTG during the last week of November (that poll will be out shortly). cheers, brian

1 0

[stable][glance] Proposal to remove Flavio Percoco from glance-stable-maint
by Brian Rosmaita 13 Nov '19

13 Nov '19

I just noticed that Flavio is still a member of glance-stable-maint. Nothing against him personally -- he's an excellent dude -- but he hasn't been working on Glance (or OpenStack) for quite a while now and is no longer a member of glance-core, so he probably shouldn't be on the stable-maint team. (Not that he'd do anything bad, it just makes the glance-stable-maint team look larger than it actually is.) On the off chance he'll see this message, I'd like to thank Flavio for all his hard work in the past keeping Glance stable! cheers, brian

3 3

[ptg] Auto-generated etherpad links !
by Thierry Carrez 13 Nov '19

13 Nov '19

Hi everyone, The PTGbot grew a new feature over the summer. It now dynamically generates the list of PTG track etherpads. You can find that list at: http://ptg.openstack.org/etherpads.html If you haven't created your etherpad already, just follow the link there to create your etherpad. If you have created your track etherpad already under a different name, you can overload the automatically-generated name using the PTGbot. Just join the #openstack-ptg channel and (as a Freenode authenticated user) send the following command: #TRACKNAME etherpad <URL for your etherpad> Example: #keystone etherpad https://etherpad.openstack.org/p/awesome-keystone-pad That will update the link on that page automatically. Hoping to see you in Shanghai! -- Thierry Carrez (ttx)

2 1

[kuryr] [tc] kuryr project mission
by Michał Dulko 13 Nov '19

13 Nov '19

Hi, It's been more than a year after fuxi [1] and fuxi-kubernetes [2] were declared as retired. First one was supposed to integrate Docker with Cinder and Manila. The second was aiming to do the same with Kubernetes. I'm not really sure how to serve the first use case, but it seems like there are some alternatives [3], [4]. fuxi-kubernetes use case is now served by Cloud Provider OpenStack in Cinder [5] and Manila [6] CSI plugins, which seem like a much better fit. Currently the only maintained deliverables in kuryr project are kuryr, kuryr-libnetwork and kuryr-kubernetes. All of them are related to networking. Given that I'd like to propose rephrasing Kuryr mission statement from: > Bridge between container framework networking and storage models > to OpenStack networking and storage abstractions. to > Bridge between container framework networking models > to OpenStack networking abstractions. effectively getting storage out of project scope. Are there any thoughts or objections? Maybe someone sees a better phrasing? Thanks, Michał [1] https://opendev.org/openstack/fuxi [2] https://opendev.org/openstack/fuxi-kubernetes [3] https://rexray.readthedocs.io/en/stable/user-guide/schedulers/docker/plug-i… [4] https://github.com/j-griffith/cinder-docker-driver [5] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/usi… [6] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/usi…

5 4

[tc][horizon][all] Horizon plugins maintenance
by Ivan Kolodyazhny 13 Nov '19

13 Nov '19

Hi team, As you may know, we've got a pretty big list of Horizon Plugins [1]. Unfortunately, not all of them are in active development due to the lack of resources in projects teams. As a Horizon team, we understand all the reasons, and we're doing our best to help other teams to maintain plugins. That's why we're proposing our help to maintain horizon plugins. We raised this topic during the last Horizon weekly meeting [2] and we'll have some discussion during the PTG [3] too. There are a lot of Horizon changes which affect plugins and horizon team is ready to help: - new Django versions - dependencies updates - Horizon API changes - etc. To get faster fixes in, it would be good to have +2 permissions for the horizon-core team for each plugin. We helped Heat team during the last cycle adding horizon-core to the heat-dashboard-core team. Also, we've got +2 on other plugins via global project config [4] and via Gerrit configuration for (neutron-*aas-dashboard, tuskar-ui). Vitrage PTL agreed to do the same for vitrage-dashboard during the last meeting [5]. Of course, it's up to each project to maintain horizon plugins and it's responsibilities but I would like to raise this topic to the TC too. I really sure, that it will speed up some critical fixes for Horizon plugins and makes users and operators experience better. [1] https://docs.openstack.org/horizon/latest/install/plugin-registry.html [2] http://eavesdrop.openstack.org/meetings/horizon/2019/horizon.2019-10-16-15.… [3] https://etherpad.openstack.org/p/horizon-u-ptg [4] http://codesearch.openstack.org/?q=horizon-core&i=nope&files=&repos=opensta… [5] http://eavesdrop.openstack.org/meetings/vitrage/2019/vitrage.2019-10-23-08.… Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

9 9

[ptg][glance] PTG summary
by Abhishek Kekane 13 Nov '19

13 Nov '19

Hi All, I attended OpenInfra summit and PTG at Shanghai last week. It was an interesting event with lots of discussion happening around different OpenStack projects. I was mostly associated with Glance and cross-projects work related to Glance. There were other topics around Edge, UI and QA. During summit Me and Erno gave a Glance project update where we discussed what we achieved in Train cycle and what we are going to do in Ussuri cycle. As multiple stores feature is stabilized during Train, In Ussuri main focus of glance is on enhancing /v2/import API to import single image into multiple stores and copying existing images to multiple stores to avoid the manual efforts required by operator to copy the image across the stores. New delete API will also be added to delete the image from single store, also cinder driver of glance_store needs refactoring so that it can use multiple backends configured by cinder. Efforts will be continued for cluster awareness of glance API during this cycle as well. Apart from these edge related work, Glance team will also work on removing deprecated registry and related functional tests, removing of sheepdog driver from glance_store, adding s3 driver with multiple stores support in glance_store and some urgent bug fixes. Cross-Project work: In this PTG we had discussion with Nova and Cinder regarding the adoption of multiple store feature of Glance. As per discussion we have finalized the design and Glance team will work together with Nova and Cinder towards adding multiple store support feature in Train cycle. Support for Glance multiple stores in Cinder: As per discussion, volume-type will be used to add which store the image will be uploaded on upload-to-image operation, also cinder will send base image id to glance as a header using which glance will upload the image created from volume to all those stores in which base image is present. Nova snapshots to dedicated store: Agreement is, Nova will send us a base image id to glance as a header using which glance will upload the instance snapshot to all those stores in which base image is present. Talk with QA team: Glance has also talked with QA team for adding new tempest coverage for newly added features in the last couple of cycles, Glance team will work with tempest to add below new tempest tests. 1. New import workflow (image conversion, inject metadata etc.) - Depends on https://review.opendev.org/#/c/545483/ devstack patch 2. Hide old images 3. Multiple stores: https://review.opendev.org/#/c/689104/ in devstack 3.1 Devstack patch + zuul job to setup multiple stores and the job will run on glance and run glance api and scenario tests 4. Delete barbican secrets from glance images 4.1 add the tests the in barbican-tempest-plugin 4.2 run as part of barbican gate using their job 4.3 run that tests with new job (multi stores) on glance gate. do not run barbican job on glance. Below is the Ussuri cycle planning and deadline for Glance. Ussuri milestone planning: Ussuri U1 - December 09-13: 1. Import image in multiple stores (Specs + Implementation) 2. Copy existing image in multiple stores (Specs + Implementation) 3. S3 driver for glance 4. remove sheepdog driver from glance_store 5. Fix subunit parser error 6. Modify existing nova and cinder specs Ussuri U2 - February 10-14 1. Cluster awareness of glance API nodes 2. remove registry code 3. Delete image from single store 4. Nova and Cinder upload snapshot/volume to glance 5. image-import.conf parsing issue with uwsgi Ussuri U3 - April 06-10 1. Multiple cinder store support in glance_store (specs + implementation) 2. Creating image from volume using ceph (slow uploading issue) 3. Image encryption 4. Tempest work Glance PTG planning etherpad: https://etherpad.openstack.org/p/Glance-Ussuri-PTG-planning Let me know if you guys need more details on this. Thanks & Best Regards, Abhishek Kekane

1 0

[nova][ops][stable] Any interest in backporting --dry-run and/or --instance options for heal_allocations?
by Matt Riedemann 13 Nov '19

13 Nov '19

I was helping someone recover from a stuck live migration today where the migration record was stuck in pre-migrating status and somehow the request never hit the compute or was lost. The guest was stopped on the guest and basically the live migration either never started or never completed properly (maybe rabbit dropped the request or the compute service was restarted, I don't know). I instructed them to update the database to set the migration record status to 'error' and hard reboot the instance to get it running again. Then they pointed out they were seeing this in the compute logs: "There are allocations remaining against the source host that might need to be removed" That's because the source node allocations are still tracked in placement by the migration record and the dest node allocations are tracked by the instance. Cleaning that up is non-trivial. I have a troubleshooting doc started for manually cleaning up that kind of stuff here [1] but ultimately just told them to delete the allocations in placement for both the migration and the instance and then run the heal_allocations command to recreate the allocations for the instance. Since this person's nova deployment was running Stein, they don't have the --dry-run [2] or --instance [3] options for the heal_allocations command. This isn't a huge problem but it does mean they could be healing allocations for instances they didn't expect. They could work around this by installing nova from train or master in a VM/container/virtual environment and running it against the stein setup, but that's maybe more work than they want to do. The question I'm posing is if people would like to see those options backported to stein and if so, would the stable team be OK with it? I'd say this falls into a gray area where these are things that are optional, not used by default, and are operational tooling so less risk to backport, but it's not zero risk. It's also worth noting that when I wrote those patches I did so with the intent that people could backport them at least internally. [1] https://review.opendev.org/#/c/691427/ [2] https://review.opendev.org/#/c/651932/ [3] https://review.opendev.org/#/c/651945/ -- Thanks, Matt

5 5

[nova][ptg] PCI refactoring needs and a strawman proposal inside
by Sylvain Bauza 13 Nov '19

13 Nov '19

Based on some Forum discussions, we had a de facto conversation at the Nova PTG about any potential things we could do for helping our operators, but also potentially Cyborg since they use the PCI passthrough capabilities. The feedback from ops (thanks mnaser) was that PCI passthrough works pretty smoothly but there are some ugly issues where we could improve the UX. The agreement in the room was, at least for Ussuri, to start collecting some ideas on how we could model Placement usage for PCI devices and also write motivations for such things in a spec. We also agreed on a smooth upgrade plan where PCITracker would still be present for a couple of releases until we're able to close the feature gap. Bauzas volunteered for drafting the spec and gibi, stephenfin and bauzas started to sketch up the Placement modeling on a whiteboard, picture to be shared in a follow-up.

3 2

[nova] spec is ready to review and for feature liasion needed
by Brin Zhang(张百林) 13 Nov '19

13 Nov '19

Hi everyone: This spec comment by melwitt, and already updated, wait for review and need a feature liaison. Proposal for a safer noVNC console with password authentication https://review.opendev.org/#/c/623120 Brin Zhang

2 2

[stable][cinder] Proposal to remove John Griffith from cinder-stable-maint
by Brian Rosmaita 12 Nov '19

12 Nov '19

John Griffith has taken on other commitments and stepped down as a cinder-core recently, so it doesn't make sense for him to continue on the cinder-stable-maint list. I'd like to acknowledge his role as "The Father of Cinder", though, and express my thanks on behalf of the Cinder team for all his past work on the project. cheers, brian

2 1