November 2020 - openstack-discuss

freezer
by Ali Ebrahimpour 06 Nov '20

06 Nov '20

hi i recently install the freezer api and i have problem with using it if run this command: freezer-agent --debug --action backup --nova-inst-id a143d11d-fa27-4716-9795-e03b4e601df4 --storage local --container /tmp/ali --backup-name test --mode nova --engine nova --no-incremental true --log-file a.log --consistency-check and freezer api doesn't work current and error with: Engine error: a bytes-like object is required, not 'str' Engine error: Forced stop Adn attach the log please help me ! thank you for attention

3 3

[nova] Wallaby spec review day
by Balázs Gibizer 06 Nov '20

06 Nov '20

Hi, Wallaby Milestone 1 is four weeks from now. As Sylvain raised on the weekly meeting it is a good time to have a spec review day somewhere before M1. So I'm proposing 17th of November as spec review day. Let me know if it what you think. Cheers, gibi

1 0

[cinder][ceilometer][heat][instances] Installation Problems
by WALSH CHANG 06 Nov '20

06 Nov '20

To whom it may concern, I am new to openstack, I got several errors when I installed the service followed by the installation guide. My openstack version : stein Ubuntu 18.0.4 1. Cinder-api is in /usr/bin/cinder-api, but when I type service cinder-api status, it shows Unit cinder-api.service could not be found. 2. Ceilometer and Heat tab didn't show in the Dashboard. 3. I was trying to launch an instance, but I got the status error and I tried to delete, but the instances can not be deleted. And I used nova force-delete instance-id, the error message is ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.AllocationDeleteFailed'> (HTTP 500) (Request-ID: req-e6c0d966-c7ec-4f20-8e9f-c788018a8d81) I explored the openstack installation myself, just wondering if there is any online hands-on training or course that I can enroll in? Thanks. Kind regards, Walsh

1 0

[neutron] Multi-AZ without L2 spanning between datacenter for external network
by Steve Vézina 06 Nov '20

06 Nov '20

Hi everyone, I am wondering if anyone got a multi-az deployment using OVS that work with seperate network fabrics which mean no L2 spanning between the AZs for external network? Thanks! Steve.

1 0

[tripleo][ci] ubi-8:3 has broken all centos-8 tripleo jobs
by Wesley Hayutin 05 Nov '20

05 Nov '20

Greetings, Bug: https://bugs.launchpad.net/tripleo/+bug/1902846 Patch: https://review.opendev.org/#/c/761463 Status: RED Thanks to Sagi and Sandeep ( ruck / rovers ) the issue was caught yesterday. Few iterations on where and how to pin this back to a working ubi-8 image for containers builds. We are confident the current patch will set things back to working. Context: https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image CentOS / RHEL ecosystem are not quite in sync yet.

1 1

ovs-dpdk bonding question
by Satish Patel 05 Nov '20

05 Nov '20

Folks, I have a compute node with only 2x10G port in that case, how do I configure ovs-dpdk with bonding support? I can understand if i have more than 2 nic ports then i can make one of nic port to management but in my case i have only two. Does that mean I can't utilize bonding features, is that true? How do other folks deal with this kind of scenario? (my deployment tool is openstack-ansible) ~S

1 0

[glance] Single store support removal
by Abhishek Kekane 05 Nov '20

05 Nov '20

Hi All The Single store glance support(default) was deprecated in Rocky release by Change-Id: I50189862454ada978eb401ec24a46988517ea73b The glance team will start working on removing single store support from the wallaby cycle and intended it to finish the same in wallaby or during milestone 1 of the next cycle. Below is the roadmap for removing single store configuration support from glance and glance_store repository. 1. Default Glance to configure multiple stores - Wallaby milestone 1 - https://review.opendev.org/#/c/741802/ 2. Convert single store unit and functional tests to use multiple stores - Wallaby milestone 1 3. Remove use of single store from glance - Wallaby milestone 2 4. Remove single stores support from glance_store - Wallaby milestone 3 to X milestone 1 We would like to encourage our users to start moving to multiple stores of glance if you are still using traditional single stores of glance in your deployments. Please let us know if you have any queries or suggestions about the same. Thank you, Abhishek

1 0

Openstack - File system changing to read-only mode
by ANUSUYA MANI 05 Nov '20

05 Nov '20

Openstack - File system changing to read-only mode after 1 or 2 days of creating the instance. Not able to access any files after that. I am using Ubuntu 18.04 instances. I have installed openstack on my server which is also ubuntu 18.04. I have enough disk space. After this issues, when i try to remount , i am not able to do. what is the cause of this kind of issue and how to resolve this. Kindly help me in resolving this. Attaching logs here.

2 1

[manila][ptg] Manila Wallaby PTG Summary
by Goutham Pacha Ravi 05 Nov '20

05 Nov '20

Hello Zorillas and all other interested Stackers! The OpenStack Manila Project Technical Gathering for the Wallaby development cycle was a successful meeting of project developers, users, operators and contributors to the rest of OpenStack, Kubernetes, Ceph and other communities - simply all zorillas and friends! I would like to thank the PTG organizing committee, (the Kendalls! :D) the infrastructure team, and everyone that contributed to the discussions. The following is my summary of the proceedings. I've linked associated etherpads, and resources to dig in further, and chat with us here, or on freenode's #openstack-manila! == Day 1 - Oct 26, 2020, Mon == === Manila Interop testing === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-interop Discussion Items: - OSF Foundation Board and Interop Working Group members were apprised about Manila, its capabilities and problem space - Manila contributors answered questions regarding user workflows, tempest integration, vendor coverage and project maturity - Manila's addition to the Interop refstack suite will allow: * OpenStack clouds and distributions to certify their manila APIs for the "OpenStack Powered" trademark program, as well as, * OpenStack Manila third party integrations can claim the "OpenStack Compatible" label - A plan was charted to target an upcoming interop advisory with manila as an "Add-On" component - Interop team presented their idea for E2E testing for running container workloads on top of openstack Actions: - Vida Haririan (vhari), Liron Kuchlani (lkuchlan) and Goutham Pacha Ravi (gouthamr) will create the interop test plan for Manila - Test plan target is the upcoming 2020.10 advisory === Manila Retrospective === Topic Etherpad: https://etherpad.opendev.org/p/manila-victoria-retrospective Discussion Items: - Victoria Cycle Bug/Documentation Squash days were a big hit. 45% of the bug backlog was addressed with these squashes - thanks vhari/vkmc! - Team's doing a great job of farming low-hanging-fruit; but resolution is taking longer, we should timebox fixes for Medium prio low-hanging-fruit; high prio bugs cannot be tagged "low-hanging-fruit" - Team recognized the mentorship eagerness, and the opportunity providers in Victoria/Wallaby - Grace Hopper Celebration, Outreachy, Boston University Project and other venues - We couldn't complete the "migrate to focal fossa goal" - team was short handed at the end of the release, thanks to the goal champion, Ghanshyam Mann for ensuring we can make decent progress and target completion in wallaby - New TC tags (kubernetes-in-virt, assert:supports-accessible-upgrade, erstwhile tc:approved-release) were great, more to come in Wallaby (vulnerability-managed, supports-api-interoperability) - A GREAT DEAL of work was done on manila-tempest-plugin, and the team is thankful, and is sure users are appreciative of the focus we have on CI. - Great PTL leadership - but there's room to grow :D - Collaborative review sessions are a hit. Let's keep doing them! Actions: - We'll have a bug squash event closer to Wallaby-1 for bigger bugs that require API/DB changes as identified in the last Cycle (vkmc/vhari) - We'll have Bug/Documentation Squash after Wallaby-3 as well - Work on TC tags by Wallaby-1 - Contributors can actively seek Collaborative Review sessions for the work they're doing - CI flakiness needs to be audited and recurring test failures will be identified (dviroel) and discussed at an upcoming IRC meeting - Backend Feature Support Matrix will be improved and doc bugs will be filed for other issues identified (dviroel) === Cephadm, Triple-O and NFS-Ganesha === Topic Etherpad: https://etherpad.opendev.org/p/tripleo-ceph Discussion Items: - fultonj/gfidente/fpantano presented three specs to chart the future of ceph deployment with tripleo. The ceph community will no longer support ceph-ansible as a deployment tool. - particularly interesting to the manila team was the fate of nfs-ganesha ("ceph-nfs") which doesn't have adequate support in ceph's new deployment tooling (cephadm and ceph orch) to cover tripleo's requirements of HA, or allow deployment flexibility like ceph-ansible used to (ceph-ansible would allow deploying nfs-ganesha as a standalone service on tripleo hosts) - the proposal was to retain the portions of ceph-ansible that were used for nfs-ganesha in tripleo while replacing the use of ceph-ansible with cephadm/ceph orch. - tripleo contributors were concerned about the future supportability of taking over the portion of the setup that ceph ansible does for tripleo today; they would be more comfortable if cephadm can be fixed in time for wallaby, or the portions of the code taken from ceph-ansible be in an independent repo Actions: - pursue cephadm/orch support for nfs-ganesha with ceph-orchestration team and understand timeline and upgrade implications when substituting the work - advance the work via specs == Day 2 - Oct 27, 2020, Tue == === Sub teams === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-subteams Discussion Items: - gouthamr/vkmc proposed that client repos (python-manilaclient, manila-ui) and tempest (manila-tempest-plugin) have their own core teams - manila-core can be part of the initial core teams of those repos - the idea is to create focussed subteams that can to some degree of autonomy, govern these projects with the domain knowledge they have, have independent meetings and bug triages if they wish and dictate a roadmap - this way, review bandwidth can be adequately shared - the sub teams will allow manila core team mentoring efforts and provide an easier on ramp for new maintainers - team agreed that this is a good way forward, and acknowledged that they would not want to create more maintenance burden or create silos. Actions: - get https://review.opendev.org/758868/ merged - to avoid siloing, weekly IRC meeting will call upon subteams to share updates - focussing on a sub project doesn't mean we don't pay attention to other repos - attempt to record mentoring sessions and post it publicly for posterity === Extending shares via the Manila scheduler === Topic Etherpad: https://etherpad.opendev.org/p/share-stuck-in-extend-discussion Discussion Items: - manila share extension currently doesn't go through the scheduler - they directly hit the share service/host responsible for the share ( https://bugs.launchpad.net/manila/+bug/1855391) - this needs to be fixed for two reasons - a) capacity based calculations are off if scheduler is ignored, b) service health checks are not accounted for - there was a concern this change would break administrators that rely on this wrong behavior as storage pools fill up - this was called out as a corner case, and instead of having a global way to opt-in/out of using the scheduler for share extensions, we can build a "force" flag for privileged users that will preserve existing behavior Actions: - haixin will propose changes to the patch based on this feedback === Virtio-FS === Topic Etherpad: https://etherpad.opendev.org/p/nova-manila-virtio-fs-support-vptg-october-2… Discussion Items: - virtiofs support has debuted in newer linux kernels and the libvirt/qemu/kvm communities have adopted it - in OpenStack, Nova can be made to use virtio-fs and expose Manila shares to tenant VMs - this provides a user experience improvement for manila users where mount automation has been a concern for a while, along with the desire to have strong multi-tenant separation on the network path which only a subset of manila backends are able to provide directly - virtiofs support does not replace DHSS=True or use of gateways like NFS-Ganesha (e.g.: CephFS, GPFS, etc) since those are still very much valuable when the consumer isn't nova VMs but baremetals, containers or non-OpenStack VMs. - nova team helped understand the feature submission process and provided guidance on how to integrate this feature in stages - the user experience, data models and cross service RPCs for this can be modeled around block device mapping, however we don't need to share or copy the implementation - this is surely a large multi-cycle work - nova team suggested that we begin with only attach/detach workflows in Wallaby Actions: - tbarron will propose a nova spec to allow "nova attach fs" (and detach) workflows - we'll continue to design "boot with share" or "boot from share" APIs, including scheduler changes through this cycle === Share Server Updates === Topic Etherpad: https://etherpad.opendev.org/p/share-server-update Discussion Items: - objective is to be able to update security services and share network subnets after share servers have been provisioned - users that want to make these updates have no access to share servers - a security service update can impact multiple share networks (and hence multiple share servers) and in turn multiple share resources, so there was a concern with granularity of the update - operators don't want to be involved with the updates, want users to be able to make the updates on resources they can see (security services or share networks) - if we allow end users to make these changes, we'll still need to break down the operation into more granular bits, and provide APIs to perform validation and multi-step updates Actions: - dviroel will propose spec updates based on the feedback gathered here: https://review.opendev.org/729292 - need other DHSS=True driver maintainers to pay attention to these changes === Share Affinity Hints === Topic Etherpad: https://etherpad.opendev.org/p/manila-anti-affinity Discussion Items: - need a way to specify affinity or anti-affinity placement hints wrt existing resources (e.g: shares, share groups) - share groups provide a way to indicate affinity, but they may also encase shares into a construct and make individual operations on shares harder (you cannot add existing shares to a group or remove shares from a share group without deleting them, you can only create new members in a share group) - need a vehicle other than groups or share type extra specs to carry scheduling decisions that affect single shares - this mechanism shouldn't break cloud storage abstraction - cinder has had the ability for end users to add placement hints for affinity without breaking abstractions Actions: - carthaca will propose a specification for this work - we'll seek reviews from cinder contributors on the spec == Day 3 - Oct 28, 2020, Wed == === Responsibility for Code Reviews === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - a bulk of the day-2 operations that are being enhanced have significant impact on vendor driver implementations - we do have first party "reference" driver support for testing and designing manila APIs in a generic fashion but, not having vendor driver maintainer feedback is unfortunate - operator feedback is also very much appreciated, and we've made significant decisions because of it Actions: - proposers for new features that affect drivers must actively seek out vendor driver maintainers and operators and add them to review. The team will help identify these individuals - we'll track vendor/operator feedback and reviews as part of the review focus tracking we do through the cycle === Acting on Deprecation removals === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - tbarron proposed https://review.opendev.org/745206/ to remove old deprecated config options from manila - we should be doing this more actively each cycle, however, we held off from merging the code change above considering the impact to configuration and deployment tooling - driver module maintainers must adjust the deployment tooling/other usages accordingly asap Actions: - dviroel, carloss and gouthamr will review and merge https://review.opendev.org/745206/ === rootwrap-to-privsep migration === Topic Etherpad: https://etherpad.opendev.org/p/manila-migration-from-rootwrap-to-privsep Discussion Items: - this has been selected as a community goal for Wallaby - privsep has not been used in manila so far - most first party and reference drivers (lvm, zfsonlinux, generic, container, ceph) use rootwrap heavily - some third party drivers also use rootwrap - team was concerned with the completion criteria wrt third party drivers - a stretch goal would be to investigate if dropping privileges for operations by substituting mechanisms is possible Actions: - carloss will work on this goal, no specification is necessary - adequate testing, admin/operator documentation will be provided - carloss will reach out to the third party driver maintainers to act on rootwrap removals - carloss/team will attack de-escalation/removing use of root privs to perform an operation, opportunistically === Secure RBAC improvements === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-policy-changes Discussion Items: - manila will need to support splitting of the existing admin, member roles to system, domain and project admins and members - operators can define personas with these default roles using keystone's "role scope" - an accompanying feature is to provide "reader" role permissions as well - manila accepts the use of "policy.yaml" to drive RBAC, over "policy.json", but, the default is still "policy.json" and this needs to change due to several challenges, including the ability to have comments in the files Actions: - gouthamr will propose a spec for policy refresh across manila APIs/resources using these new role personas - making policy.yaml the default for policy overrides is going to be a community goal. We're targeting this transition to wallaby-1 in manila - we'll attempt to complete this effort in wallaby. there are several parts, collaboration is welcome! === CephFS Updates === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-cephfs-driver-update Discussion Items: - vkmc provided an update on the cephfs feature roadmap - we need changes in cephfs (preferably in ceph-nautilus) to perform authorization, subvolume cloning, capacity reporting etc. This work is being tracked through tracker tickets in Ceph's tracking system - we stopped using shaman builds for ceph and nfs-ganesha in victoria cycle - consuming nfs-ganesha from ppa's has introduced new scenario test failures, vkmc is currently triaging them Actions: - start testing ceph octopus in the wallaby cycle - keep momentum on the ceph trackers and switch over from ceph-volume-client to ceph-mgr and implement create share from snapshot in the cephfs drivers - engage with ceph and nfs-ganesha communities to create a support matrix across nfs-ganesha versions, manila and cephfs === Divisive Language Correction === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - spotz joined us, and recapped the diversity and inclusion wg's work so far - manila team is determined to help resolve any divisive language and references in code and documentation - the team realizes that unconscious use of negative language is affecting the community's inherent desire to be welcoming and inclusive - no manila architecture or code references currently *need* the use of the divisive words identified by the working group and the wider openstack community - making changes is important to us, but we recognize it's just part of the story Actions: - we'll begin with a documentation change to remove the words identified in https://etherpad.opendev.org/p/divisivelanguage - we'll wait to coordinate a change of name for the development/trunk branch, and other "upstream" changes to databases, git, opendev - team actively welcomes any feedback == Day 5 - Oct 30, 2020, Fri == === Share and Server Migration graduation === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - we've graduated one major feature through the last couple of cycles, share migration is next - team agreed to keep share server migration as experimental APIs since they haven't had sufficient soak time - no significant changes have been made to share migration migration APIs since Ocata, leading to believe the APIs are stable and can be improved via microversions where necessary - not a lot of vendor driver implementations for driver optimized migration, NetApp driver only allows moving within a backend at the moment - generic migration improvements have been deprioritized, and need help in terms of High Availability of the Data service as well as providing a jobs table to track, and coordinate ongoing migrations - tests are being skipped at the gate because of flakiness Actions: - team will get in touch with third party vendors and guage interest in implementing driver optimized migration (or testing/supporting generic migration with their drivers) (carloss). Feedback from the vendors will drive our decision to graduate the share migration APIs in Wallaby. === Metadata for all share resources === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-metadata-controller Discussion Items: - the proposal is to introduce a metadata API controller that can be inherited by the resource controllers - all user facing resources will have metadata capabilities, with consistent API methods - this is expected to benefit automation use cases, manila-csi and the upcoming virtio-fs effort Actions: - gouthamr will propose the spec for team's review, including identifying the resources that will be targeted in the Wallaby release - share metadata APIs will be improved to adhere to the API SIG spec: https://specs.openstack.org/openstack/api-sig/guidelines/metadata.html === Client Project updates === Topic Etherpads: https://etherpad.opendev.org/p/manila-osc-wallaby-ptg and https://etherpad.opendev.org/p/manila-ui-wallaby-ptg Discussion Items: - the team met with Nicole Chen, Ashley Rodriguez, Mark Tony from Boston University who will be working on the OpenStackSDK integration for Manila APIs - maaritamm provided an update on the OSC plugin interface implementation - vkmc provided an update on manila-ui feature gaps and the upcoming Outreachy internship Actions: - for some OSC plugin command implementations, we'll benefit from integrating with OpenStackSDK rather than manilaclient SDK, so we'll coordinate that work - maaritamm/vkmc will send an email to openstack-discuss to prioritize the commands left to implement in the OSC plugin - vkmc will drive consistency across cinder/manila for the user messages dashboards that were added in Victoria - we'll continue to work on closing the feature gap in manila-ui === Manila CSI Wallaby roadmap === Topic Document: https://docs.google.com/document/u/1/d/1YJcv0EROWzYbV_8DM3DHVvDqdhlPhYRV8Np… Discussion Items: - users of manila-csi today deploy one driver per share protocol - future work on monitoring, common node plugin interactions requires us to rework the architecture to have one driver multiplex with multiple share protocol node plugins - upgrade impact as well as UI/UX impact was discussed and gman0 has documented them - share metadata is necessary to tag and identify shares Actions: - gman0 has started working on the code patches, and will benefit from reviews - mfedosin will take a look at the upgrade impact and share details for automation that can be done in the csi-operator - mfedosin will test gman0's latest patch to add share metadata via the storage class (as will folks from CERN) and provide feedback Thanks for reading thus far, there were a number of items that we couldn't cover, that you can see on the PTG Planning etherpad [1], if you own those topics, please add them to the weekly IRC meeting agenda [2] and we can go over them. The master meeting minutes document is available as well [3]. As usual, the whole PTG was recorded and posted on the OpenStack Manila Youtube channel [4] We had a great turn out and we heard from a diverse set of contributors, operators, interns and users from several affiliations and time zones. On behalf of the OpenStack Manila team, I deeply appreciate your time, and help in keeping the momentum on the project! [1] https://etherpad.opendev.org/p/wallaby-ptg-manila-planning [2] https://wiki.openstack.org/wiki/Manila/Meetings [3] https://etherpad.opendev.org/p/wallaby-ptg-manila [4] https://www.youtube.com/playlist?list=PLnpzT0InFrqATBbJ60FesKGYRejnTIoCW

1 0

[gate][all] assorted tempest jobs fail when IPv4 default route is not available
by melanie witt 05 Nov '20

05 Nov '20

Hi all, We have a gate bug [1] where if a CI VM does not have an IPv4 default route configured, the job will fail with one of the following errors: "[ERROR] /opt/stack/devstack/functions-common:230 Failure retrieving default route device" [2] or "[ERROR] /opt/stack/devstack/functions-common:104 Failure retrieving default IPv4 route devices" [3] because the die_if_not_set is being triggered. Most of the time, CI VMs come configured with an IPv4 default route but something changed recently (around Oct 23 or 24) to where we often get VMs that do not have an IPv4 default route configured and presumably only have an IPv6 default route. According to this logstash query [4] we have: 58 hits in the last 7 days, check and gate, all failures, all on the limestone-regionone node provider, various projects, various jobs Nate Johnson has proposed a fix here: https://review.opendev.org/761178 Reviews appreciated. Cheers, -melanie [1] https://bugs.launchpad.net/devstack/+bug/1902002 [2] example: https://zuul.openstack.org/build/7543021732cc4886b608553da58e81f9/log/contr… [3] example: https://zuul.opendev.org/t/openstack/build/7fff839dd0bc4214ba6470ef40cf78fd… [4] http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:…

1 0

Editing libvirt xml file
by vladimir franciz blando 04 Nov '20

04 Nov '20

Hi, I edited the libvirt xml (virsh edit...) file of a running Windows server instance because I changed the video model type from cirrus to vmvga so it can support a higher resolution on a console (cirrus supports up to 1280 only). After editing, i "soft-rebooted" the instance and the new configuration sticks. That's awesome. But when you do a "hard-reboot" it will revert back to the original config. I also tried to do a "shut-down" then "start" again, but it still reads the original config. Not sure where it is reading the config from... Regards Vlad

3 5

[neutron][ovn] Neutron <-> OVN DB synchronization code
by Piotr Misiak 04 Nov '20

04 Nov '20

Hi, Do you know why Neutron <-> OVN DB sync code synchronizes "DEVICE_OWNER_ROUTER_HA_INTF" router ports here in this line: https://github.com/openstack/neutron/blob/164f12349fd8be09b9fd4f23b8cf8d2f3… ? As far as know there is no need for HA network ports for L3 virtual routers implementation in OVN, because OVN uses the internal BFD mechanism for Gateway Chassis reachability tests. Thanks, Piotr

1 0

[kolla] Kolla klub meetings paused
by Mark Goddard 04 Nov '20

04 Nov '20

Hi, We agreed at the PTG to try rebooting the Kolla klub meetings at a different time to involve some new members. Let's pause the meetings until then. Regards, Mark

1 0

[tripleo] next meeting Tuesday Nov 10 @ 1400 UTC in #tripleo
by Marios Andreou 04 Nov '20

04 Nov '20

Hello tripleo as discussed at PTG - we voted [1] to have IRC meetings every 2 weeks and next week will be 2 weeks since PTG :) Keeping the same day and time our next meeting will be ** Tuesday 10th November at 1400 UTC in #tripleo. ** Our regular agenda is at https://wiki.openstack.org/wiki/Meetings/TripleO Please add items you want to raise at https://etherpad.opendev.org/p/tripleo-meeting-items . There were concerns voiced at PTG about low attendance in these meetings but at the same time quite a few folks voted to have it. So, let's try again and as with all the things we will re-evaluate based on all our feedback, Can I get a o/ if you plan on attending these meetings? Or even if you can't but plan to read the meetbot logs ;) Obviously now would be a good time to speak up if you hate tuesdays at 1400 UTC and want to propose some alternative for us to consider, thanks, marios [1] https://etherpad.opendev.org/p/tripleo-ptg-wallaby-community-process

1 0

[ptl] Victoria Release Community Meeting
by helena＠openstack.org 04 Nov '20

04 Nov '20

Hi Everyone, We are looking to do a community meeting following The Open Infrastructure Summit and PTG to discuss the Victoria Release. If you’re a PTL, please let me know if you’re interested in doing a prerecorded explanation of your project’s key features for the release. We will show a compilation of these recordings at the community meeting and follow it with a live Q&A session. Post community meeting we will have this recording live in the project navigator. Cheers, Helena

8 10

[cinder] Block Storage API v2 removal
by Brian Rosmaita 04 Nov '20

04 Nov '20

The Block Storage API v2 was deprecated in the Pike release by Change-Id: I913c44799cddc37c3342729ec0ef34068db5b2d4 The Cinder team will be removing it during the Wallaby development cycle. At the Wallaby PTG, the question came up about Block Storage API v2 support in the python-cinderclient. We looked at some proposals [0] at today's Cinder meeting and plan to make a decision at next week's meeting [1]. If you have strong feelings about this issue, please make them known on the etherpad [0], the ML, or by attending the next Cinder meeting (1400 UTC in #openstack-meeting-alt on Wednesday 11 November). [0] https://etherpad.opendev.org/p/python-cinderclient-v2-support-removal [1] https://etherpad.opendev.org/p/cinder-wallaby-meetings

1 0

[tc] weeklu update
by Mohammed Naser 04 Nov '20

04 Nov '20

Hi everyone, Here's an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. # Patches ## Open Reviews - Adding reno and warnings example for JSON->YAML goal doc https://review.opendev.org/760956 - Select JSON to YAML goal for Wallaby cycle https://review.opendev.org/760957 - Add goal for migrate RBAC policy format from JSON to YAML https://review.opendev.org/759881 - Add diablo_rojo liaison preferences https://review.opendev.org/759718 - Add danms liaison preferences https://review.opendev.org/758822 - Add ricolin liaison preferences https://review.opendev.org/759704 - Add gmann liaison preference https://review.opendev.org/758602 - Resetting projects' TC liaisons empty https://review.opendev.org/758600 - Move Placement under Nova's governance https://review.opendev.org/760917 - Clarify the requirements for supports-api-interoperability https://review.opendev.org/760562 - Add Resolution of TC stance on the OpenStackClient https://review.opendev.org/759904 ## General Changes - Implement distributed leadership in tools and schema https://review.opendev.org/757966 - Adding missing IRC nick https://review.opendev.org/759724 - Update dansmith IRC nick https://review.opendev.org/760561 - Update projects with Telemetry PTL election result https://review.opendev.org/760206 - Appoint XueFeng Liu as Senlin PTL https://review.opendev.org/758122 - Begin the 'X' release naming process https://review.opendev.org/759888 - Appoint Adam Harwell as Octavia PTL https://review.opendev.org/758121 - Add nomination for TC chair https://review.opendev.org/758181 - Appoint Rafael Weingartner as CloudKitty PTL https://review.opendev.org/758120 - Appoint Frode Nordahl as OpenStack Charms PTL https://review.opendev.org/758119 - Select Privsep as the Wallaby Goal https://review.opendev.org/755590 ## Project Updates - Adopting the DPL governance model for oslo https://review.opendev.org/757906 Thanks for reading! Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

Random 503 error while running using api/command line
by Deepa KR 04 Nov '20

04 Nov '20

Hi All Good Day We have a Train Openstack setup using juju maas configured in HA with 3 controllers. When using the openstack api getting a timeout error randomly. Also sometimes getting "Unknown Error (HTTP 503)" or Remote end closed connection without response while using the command line . openstack server list --all-projects *Unknown Error (HTTP 503)* Kindly suggest to look in the right direction. Regards, Deepa K R

1 0

[tripleo][ci] Monday Nov 2
by Wesley Hayutin 04 Nov '20

04 Nov '20

Greetings, The tripleo ci team has identified a handful of patches that we'd like to land prior to Nov 2, the day docker.io goes away. We've hit some new bugs and have also tuned a few things to try and make sure we can get patches to merge. Our current focus is across master, victoria, ussuri and centos-8 train, and queens while reducing coverage in rocky and stein. A list of the prioritized gerrit reviews can be found here: https://hackmd.io/MlbZ_izSTEuZsCWTJvu_Kg?view The entire topic can be found here: https://review.opendev.org/#/q/topic:new-ci-job Thanks all.

4 4

[nova] Why nova needs password-less SSH to do live migraiton?
by Zhi CZ Chang 04 Nov '20

04 Nov '20

3 5

[ironic] Configuration mold follow-up
by Jaunteva, Aija 04 Nov '20

04 Nov '20

Hi, to follow up on Ironic PTG session about Configuration molds [1] I'm scheduling a call to discuss remaining items (mainly storage of the molds). Anyone interested please add your availability in Doodle [2]. When time slot decided, will share call details. Regards, Aija [1] https://etherpad.opendev.org/p/ironic-wallaby-ptg line 216 [2] https://doodle.com/poll/dry4x5tbmhi6x6p3

2 3

[Neutron] PTG summary
by Slawek Kaplonski 04 Nov '20

04 Nov '20

Hi, Below is my summary of the Neutron team sessions which we had during the virtual PTG last week. Etherpad with notes from the discussions can be found at [1]. ## Retrospective of the Victoria cycle >From the good things during _Victoria_ cycle team pointed: * Complete 8 blueprints including the _Metadata over IPv6_ [2], * Improved feature parity in the OVN driver, * Good review velocity >From the not so good things we mentioned: * CI instability - average number of rechecks needed to merge patch in last year can be found at [3], * Too much "Red Hat" in the Neutron team - and percentage of the reviews (and patches) done by people from Red Hat is constantly increasing over last few cycles. As a main reason of that we pointed that it is hard to change companies way of thinking that dedicating developer to the upstream project means that they lose developer in downstream. * Migration of our CI and Zuul v3 took us a lot of time and effort - but good thing is that we accomplished that in the Victoria cycle :) During that session we also agreed on some action items for the Wallaby cycle: * keep talking about OVN details - Miguel Lavalle and me will work on some way to deliver talks about Neutron OVN backend and OVN internals to the community during next months. The idea is to maybe propose some talks ever 6 weeks or so. This may make ovn driver development more diverse and let operators to thing about migration to the ovn backend. ## Review of the existing meetings We reviewed list of our existing upstream meetings and discussed about ideas on how to increase number of attendees on the meetings. We decided to: * drop neutron-qos meeting as it's not needed anymore * advertise more meetings and meetings' agenda on the OpenStack mailing list - I will send reminders with links to the agenda before every meeting * Together with Lajos Katona we will give some introduction to the debugging of CI issues in Neutron* ## Support for old versions Bernard started discussion about support for the old releases in Neutron and Neutron stadium projects. For Neutron we decided to mark __Ocata__ branch as unmaintained already as its gate is already broken. For the __Pike__ and never branches we will keep them in the __EM__ phase as there is still some community interest to keep those branches open. For the stadium projects we decided to do it similary to what we did while looking for new maintainers for the projects. We will send email "call for maintainers" for such stable branches. If there will be no voluneers to step in, fix gate issues and keep those branches healthy, we will mark them as __unmaintained__ and later as __End of Life__ (EOL). Currently broken CI is in projects: * networking-sfc, * networking-bgpvpn/bagpipe, * neutron-fwaas And those are candidates to be marked as unmaintained if there will be no volunteers to fix them. Bernard Cafarelli volunteered to work on that in next days/weeks. ## Healtcheck API endpoint We discussed as our healtcheck API should works. During the discussion we decided that: * healtcheck result should __NOT__ rely on the agents status, it should rely on worker's ability to connect to the DB and MQ (rabbitmq) * Lajos will ask community (API experts) about some guidance how it should works on the whole OpenStack level, * As for reference implementation we can check e.g. Octavia [4] and Keystone [5] which already implemented it. ## Squash of the DB migration script Rodolfo explained us what are benefits of doing such squash of the db migration scripts from the old versions: * Deployment is faster: we don't need to create/delete tables or create+update other ones - the win is small possibly in the magnitude of 5s per job, * DB definition is centralized in one place, not in original definition plus further migrations - that is most important reason why we really should do that, * UTs faster: removal of some older checks. The problem with this may be that we need to do that carefully and be really verbose about with such changes we may break stadium projects or 3rd party projects which are doing db migration too. To minimalize potential breakage, we will announce such changes on the OpenStack discuss mailing list. Rodolfo volunteered to take propose squash up to Liberty release in W cycle. Together with this squash we will also document that process so in next cycles we should be able to do squashes for newer releases in easier way. Lajos volunteered to help with fixing Neutron stadium projects if that will be needed. ## Switch to the new engine facade We were discussing how to move on and finally finish old Blueprint [6]. We decided that together with Rodolfo we will try how this new engine facade will work without using transaction guards in the code. Hopefully that will let us move on with this. If not, we will try to reach out to some DB experts for some help with this. ## Change from rootwrap to the privsep This is now community goal during the Wallaby cycle so we need to focus on it to accomplish that transition finally. This transition may speed up and make our code a bit more secure. Rodolfo explained us multiple possible strategies of migration: * move to native, e.g. * replace ps with python psutils, not using rootwrap or privsep * replace ip commands with pyroute2, under a privsep context (elevated permissions needed) * directly translate rootwrap to privsep, executing the same shell command but under a privsep context To move on with this I will create list of the pieces of code which needs to be transitioned in the Neutron repo and in the stadium projects. Current todo items can be found on the storyboard [7]. ## Migration to the NFtables During this session we were discussing potential strategies on how to migrate from the old iptables to the new nftables. We need to start planning that work as it major Linux distributions (e.g. RHEL) are planning to deprecate iptables in next releases. It seems that currently all major distros (Ubuntu, Centos, OpenSuSE) supports nftables already. We decided that in Wallaby cycle we will propose new _Manager_ class and we will add some config option which will allow people to test new solution. In next cycles we will continue work on it to make it stable and to make upgrade and migration path for users as easy as possible. There is already created blueprint to track progress on that topic [8]. We need to migrate: * Linuxbridge firewall, iptables OVS hybrid firewall, * L3 code (legacy router, DVR router, conntrack, port forwarding), * iptables metering, * metadata proxy, * dhcp agent for when it does metadata for isolated networks and namespace creation, * neutron-vpnaas - ipsec code, * and maybe something else what we didn't found yet. ## Nova-Neutron cross project session We had very interesting discussion with Nova team. We were discussing topics like: * NUMA affinity in the neutron port * vhost-vdpa support * default __vnic_type__/__port flavour__ Notes from that discussion are available in the nova's etherpad [9]. ## Neutron scalling issues At this session we were discussing issues mentioned by operators on the Forum sessions a week before the PTG. There was couple of issues mentioned there: * problems with retries of the DB operations - we should migrate all our code to the oslo.db retries mechanism - new blueprint [10] is created to track progress on that one. * problems with maintenance of the agents, like e.g. DHCP or L3 agents - many of those issues are caused by how our agents are designed and to really fix that we would need very deep and huge changes. But also many of those issues can be solved by the __ovn__ backend - **and that is strategic direction in which neutron wants to go in the next cycles**, * Miguel Lavalle and I volunteered to do some profiling of the agents to see where we are loosing most of the time - maybe we will be able to find some _low hanging fruits_ which can be fixed and improve the situation at least a bit, * Similar problem with neutron-ovs-agent and especially security groups which are using _remove group id_ as a reference - here we also need some volunteers who will try to optimize that. ## CI (in)stablility On Thursday we were discussing how to improve our very poor CI. Finally we decided to: * not recheck patches without giving reason of recheck in the comment - there should be already reported bug which should be linked in the _recheck_ comment, or user should open new one and link to it also. IN case if the problem was e.g. related to infra some simple comment like _infra issue_ will also be enough there, * To lower number of existing jobs we will do some changes like: * move *-neutron-lib-master and *-ovs-master jobs to the experimental and periodic queues to not run them on every patch, * I will switch _neutron-tempest-plugin-api_ job to be deployed with uwsgi so we can drop _neutron-tempest-with-uwsgi_ job, * Consolidate _neutron-tempest-plugin-scenario-linuxbridge_ and _neutron- tempest-linuxbridge_ jobs, * Consolidate _neutron-tempest-plugin-scenario-iptables_hybrid and _neutron- tempest-iptables_hybrid jobs, Later we also discussed about the way how to run or skip tests which can be only run when some specific feature is available in the cloud (e.g. _Metadata over IPv6_). After some discussion we decided to add new config option with list of enabled features. It will be very similar to the existing option _api_extensions_. Lajos volunteered to work on that. As last CI related topic we discussed about testing DVR in our CI. Oleg Bondarev volunteered to check and try to fix broken _neutron-tempest-plugin- dvr-multinode-scenario_ job. ## Flow based DHCP Liu Yulong raised topic about new way of doing fully distributed DHCP service, instead of using _DHCP agent_ on the nodes - RFE is proposed at [11]. His proposal of doing Open Flow based DHCP (similar to what e.g. ovn-controller is doing) is described in [12]. It could be implemented as an L2 agent extension and enabled by operators in the config when they would need it. As a next step Liu will now propose spec with details about this solution and we will continue discussion about it in the spec's review. ## Routed provider networks limited to one host As a lost topic on Thursday we briefly talked about old RFE [13]. Miguel Lavalle told us that his company, Verizon Media, is interested in working on this RFE in next cycles. This also involves some work on Nova's side which was started by Sylvain Bauza already. Miguel will sync with Sylvain on that RFE. ## L3 feature improvements On Friday we were discussing some potential improvements in the L3 area. Lajos and Bence shown us some features which their company is interested in and on which they plan to work. Those are things like: * support for Bidirectional Forwarding Detection * some additional API to set additional router parameters like: * ECMP max path, * ECMP hash algorith * --provider-allocation-pool parameter in the subnets - in some specific cases it may help to use IPs from such _special_ pool for some infrastructure needs, more details about that will come in the RFE in future, For now all those described above improvements are in very early planning phase but Bence will sync with Liu and Liu will dedicate some time to discuss progress on them during the __L3 subteam meetings__. ## Leveraging routing-on-the-host in Neutron in our next-gen clusters As a last topic on Friday we were discussing potential solutions of the _L3 on the host_ in the Neutron. The idea here is very similar to what e.g. __Calico plugin__ is doing currently. More details about potential solutions are described in the etherpad [14]. During the discussion Dawid Deja from OVH told us that OVH is also using very similar, downstream only solution. Conclusion of that discussion was that we may have most of the needed code already in Neutron and some stadium projects so as a first step people who are interested in that topic, like Jan Gutter, Miguel and Dawid will work on some deployment guide for such use case. ## Team photo During the PTG we also made team photos which You can find at [15]. [1] https://etherpad.opendev.org/p/neutron-wallaby-ptg [2] https://blueprints.launchpad.net/neutron/+spec/metadata-over-ipv6 [3] https://ibb.co/12sB9N9 [4] https://opendev.org/openstack/octavia/src/branch/master/octavia/api/ healthcheck/healthcheck_plugins.py [5] https://docs.openstack.org/keystone/victoria/admin/health-check-middleware.… [6] https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch [7] https://storyboard.openstack.org/#!/story/2007686 [8] https://blueprints.launchpad.net/neutron/+spec/nftables-migration [9] https://etherpad.opendev.org/p/nova-wallaby-ptg [10] https://blueprints.launchpad.net/neutron/+spec/oslo-db-retries [11] https://bugs.launchpad.net/neutron/+bug/1900934 [12] https://github.com/gotostack/shanghai_ptg/blob/master/ shanghai_neutron_ptg_topics_liuyulong.pdf [13] https://bugs.launchpad.net/neutron/+bug/1764738 [14] https://etherpad.opendev.org/p/neutron-routing-on-the-host [15] http://kaplonski.pl/files/Neutron_virtual_PTG_October_2020.tar.gz -- Slawek Kaplonski Principal Software Engineer Red Hat

3 4

Intel 82599 dpdk support with centos8
by Satish Patel 04 Nov '20

04 Nov '20

Folks, I am trying to get dpdk working with my OVS on CentOS-8 and I am stuck here so I need advice from folks who can help me here. I have an HP c7000 chassis with ProLiant BL460c Gen9 blade. I am running CentOS-8 with following version of ovs-dpdk [root@compute-2 usertools]# ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.12.0 DPDK 18.11.2 I am using openstack-ansible to set up openvswitch with dpdk, and so far everything went well but I am stuck in the end. [root@compute-2 usertools]# ovs-vsctl add-port br-provider dpdk-0 -- set Interface dpdk-0 type=dpdk options:dpdk-devargs=0000:06:00.0 ovs-vsctl: Error detected while setting up 'dpdk-0': Error attaching device '0000:06:00.0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". Here is the error message. [root@compute-2 usertools]# tail -f /var/log/openvswitch/ovs-vswitchd.log 2020-11-04T02:46:31.751Z|00303|dpdk|INFO|EAL: PCI device 0000:06:00.0 on NUMA socket 0 2020-11-04T02:46:31.751Z|00304|dpdk|INFO|EAL: probe driver: 8086:10f8 net_ixgbe 2020-11-04T02:46:31.751Z|00305|dpdk|ERR|EAL: Driver cannot attach the device (0000:06:00.0) 2020-11-04T02:46:31.751Z|00306|dpdk|ERR|EAL: Failed to attach device on primary process 2020-11-04T02:46:31.751Z|00307|netdev_dpdk|WARN|Error attaching device '0000:06:00.0' to DPDK 2020-11-04T02:46:31.751Z|00308|netdev|WARN|dpdk-0: could not set configuration (Invalid argument) 2020-11-04T02:46:31.751Z|00309|dpdk|ERR|Invalid port_id=32 This is what my NIC firmware/driver [root@compute-2 usertools]# ethtool -i eno50 driver: ixgbe version: 5.1.0-k-rh8.2.0 firmware-version: 0x800008fb, 1.2028.0 expansion-rom-version: bus-info: 0000:06:00.1 [root@compute-2 usertools]# python3 ./dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:06:00.0 '82599 10 Gigabit Dual Port Backplane Connection 10f8' drv=vfio-pci unused=ixgbe Network devices using kernel driver =================================== 0000:06:00.1 '82599 10 Gigabit Dual Port Backplane Connection 10f8' if=eno50 drv=ixgbe unused=vfio-pci I did enabled IOMMU & huge mem in grub iommu=pt intel_iommu=on hugepagesz=2M hugepages=30000 transparent_hugepage=never Not sure what is wrong here and the big question is does this NIC support DPDK or not?

1 0

In Openstack (devstack) i am able to create maximum of 3 volumes only
by ANUSUYA MANI 04 Nov '20

04 Nov '20

I have 3 volumes of 80GB each and when i try to create a volume snapshot or a new volume i am not able to do that. i get the error 'schedule allocate volume:Could not find any available weighted backend'. The volume quotas are set properly yet i am not able to create more than 3 volumes. i have attached screenshots of the volume quotas and error message. Kindly <https://i.stack.imgur.com/ThvMo.jpg> requesting to help me solve this issue.

1 0

[cinder] Wallaby PTG summary available
by Brian Rosmaita 04 Nov '20

04 Nov '20

I've posted a summary of the discussions we had last week: https://wiki.openstack.org/wiki/CinderWallabyPTGSummary Feel free to make any corrections or clarifications. Also, look through to remind yourself what actions you volunteered to perform (you can search for your IRC nick or the text ACTION). cheers, brian

1 0