November 2020 - openstack-discuss

ovs-dpdk bonding question
by Satish Patel 05 Nov '20

05 Nov '20

Folks, I have a compute node with only 2x10G port in that case, how do I configure ovs-dpdk with bonding support? I can understand if i have more than 2 nic ports then i can make one of nic port to management but in my case i have only two. Does that mean I can't utilize bonding features, is that true? How do other folks deal with this kind of scenario? (my deployment tool is openstack-ansible) ~S

1 0

[glance] Single store support removal
by Abhishek Kekane 05 Nov '20

05 Nov '20

Hi All The Single store glance support(default) was deprecated in Rocky release by Change-Id: I50189862454ada978eb401ec24a46988517ea73b The glance team will start working on removing single store support from the wallaby cycle and intended it to finish the same in wallaby or during milestone 1 of the next cycle. Below is the roadmap for removing single store configuration support from glance and glance_store repository. 1. Default Glance to configure multiple stores - Wallaby milestone 1 - https://review.opendev.org/#/c/741802/ 2. Convert single store unit and functional tests to use multiple stores - Wallaby milestone 1 3. Remove use of single store from glance - Wallaby milestone 2 4. Remove single stores support from glance_store - Wallaby milestone 3 to X milestone 1 We would like to encourage our users to start moving to multiple stores of glance if you are still using traditional single stores of glance in your deployments. Please let us know if you have any queries or suggestions about the same. Thank you, Abhishek

1 0

Openstack - File system changing to read-only mode
by ANUSUYA MANI 04 Nov '20

04 Nov '20

Openstack - File system changing to read-only mode after 1 or 2 days of creating the instance. Not able to access any files after that. I am using Ubuntu 18.04 instances. I have installed openstack on my server which is also ubuntu 18.04. I have enough disk space. After this issues, when i try to remount , i am not able to do. what is the cause of this kind of issue and how to resolve this. Kindly help me in resolving this. Attaching logs here.

2 1

[manila][ptg] Manila Wallaby PTG Summary
by Goutham Pacha Ravi 04 Nov '20

04 Nov '20

Hello Zorillas and all other interested Stackers! The OpenStack Manila Project Technical Gathering for the Wallaby development cycle was a successful meeting of project developers, users, operators and contributors to the rest of OpenStack, Kubernetes, Ceph and other communities - simply all zorillas and friends! I would like to thank the PTG organizing committee, (the Kendalls! :D) the infrastructure team, and everyone that contributed to the discussions. The following is my summary of the proceedings. I've linked associated etherpads, and resources to dig in further, and chat with us here, or on freenode's #openstack-manila! == Day 1 - Oct 26, 2020, Mon == === Manila Interop testing === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-interop Discussion Items: - OSF Foundation Board and Interop Working Group members were apprised about Manila, its capabilities and problem space - Manila contributors answered questions regarding user workflows, tempest integration, vendor coverage and project maturity - Manila's addition to the Interop refstack suite will allow: * OpenStack clouds and distributions to certify their manila APIs for the "OpenStack Powered" trademark program, as well as, * OpenStack Manila third party integrations can claim the "OpenStack Compatible" label - A plan was charted to target an upcoming interop advisory with manila as an "Add-On" component - Interop team presented their idea for E2E testing for running container workloads on top of openstack Actions: - Vida Haririan (vhari), Liron Kuchlani (lkuchlan) and Goutham Pacha Ravi (gouthamr) will create the interop test plan for Manila - Test plan target is the upcoming 2020.10 advisory === Manila Retrospective === Topic Etherpad: https://etherpad.opendev.org/p/manila-victoria-retrospective Discussion Items: - Victoria Cycle Bug/Documentation Squash days were a big hit. 45% of the bug backlog was addressed with these squashes - thanks vhari/vkmc! - Team's doing a great job of farming low-hanging-fruit; but resolution is taking longer, we should timebox fixes for Medium prio low-hanging-fruit; high prio bugs cannot be tagged "low-hanging-fruit" - Team recognized the mentorship eagerness, and the opportunity providers in Victoria/Wallaby - Grace Hopper Celebration, Outreachy, Boston University Project and other venues - We couldn't complete the "migrate to focal fossa goal" - team was short handed at the end of the release, thanks to the goal champion, Ghanshyam Mann for ensuring we can make decent progress and target completion in wallaby - New TC tags (kubernetes-in-virt, assert:supports-accessible-upgrade, erstwhile tc:approved-release) were great, more to come in Wallaby (vulnerability-managed, supports-api-interoperability) - A GREAT DEAL of work was done on manila-tempest-plugin, and the team is thankful, and is sure users are appreciative of the focus we have on CI. - Great PTL leadership - but there's room to grow :D - Collaborative review sessions are a hit. Let's keep doing them! Actions: - We'll have a bug squash event closer to Wallaby-1 for bigger bugs that require API/DB changes as identified in the last Cycle (vkmc/vhari) - We'll have Bug/Documentation Squash after Wallaby-3 as well - Work on TC tags by Wallaby-1 - Contributors can actively seek Collaborative Review sessions for the work they're doing - CI flakiness needs to be audited and recurring test failures will be identified (dviroel) and discussed at an upcoming IRC meeting - Backend Feature Support Matrix will be improved and doc bugs will be filed for other issues identified (dviroel) === Cephadm, Triple-O and NFS-Ganesha === Topic Etherpad: https://etherpad.opendev.org/p/tripleo-ceph Discussion Items: - fultonj/gfidente/fpantano presented three specs to chart the future of ceph deployment with tripleo. The ceph community will no longer support ceph-ansible as a deployment tool. - particularly interesting to the manila team was the fate of nfs-ganesha ("ceph-nfs") which doesn't have adequate support in ceph's new deployment tooling (cephadm and ceph orch) to cover tripleo's requirements of HA, or allow deployment flexibility like ceph-ansible used to (ceph-ansible would allow deploying nfs-ganesha as a standalone service on tripleo hosts) - the proposal was to retain the portions of ceph-ansible that were used for nfs-ganesha in tripleo while replacing the use of ceph-ansible with cephadm/ceph orch. - tripleo contributors were concerned about the future supportability of taking over the portion of the setup that ceph ansible does for tripleo today; they would be more comfortable if cephadm can be fixed in time for wallaby, or the portions of the code taken from ceph-ansible be in an independent repo Actions: - pursue cephadm/orch support for nfs-ganesha with ceph-orchestration team and understand timeline and upgrade implications when substituting the work - advance the work via specs == Day 2 - Oct 27, 2020, Tue == === Sub teams === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-subteams Discussion Items: - gouthamr/vkmc proposed that client repos (python-manilaclient, manila-ui) and tempest (manila-tempest-plugin) have their own core teams - manila-core can be part of the initial core teams of those repos - the idea is to create focussed subteams that can to some degree of autonomy, govern these projects with the domain knowledge they have, have independent meetings and bug triages if they wish and dictate a roadmap - this way, review bandwidth can be adequately shared - the sub teams will allow manila core team mentoring efforts and provide an easier on ramp for new maintainers - team agreed that this is a good way forward, and acknowledged that they would not want to create more maintenance burden or create silos. Actions: - get https://review.opendev.org/758868/ merged - to avoid siloing, weekly IRC meeting will call upon subteams to share updates - focussing on a sub project doesn't mean we don't pay attention to other repos - attempt to record mentoring sessions and post it publicly for posterity === Extending shares via the Manila scheduler === Topic Etherpad: https://etherpad.opendev.org/p/share-stuck-in-extend-discussion Discussion Items: - manila share extension currently doesn't go through the scheduler - they directly hit the share service/host responsible for the share ( https://bugs.launchpad.net/manila/+bug/1855391) - this needs to be fixed for two reasons - a) capacity based calculations are off if scheduler is ignored, b) service health checks are not accounted for - there was a concern this change would break administrators that rely on this wrong behavior as storage pools fill up - this was called out as a corner case, and instead of having a global way to opt-in/out of using the scheduler for share extensions, we can build a "force" flag for privileged users that will preserve existing behavior Actions: - haixin will propose changes to the patch based on this feedback === Virtio-FS === Topic Etherpad: https://etherpad.opendev.org/p/nova-manila-virtio-fs-support-vptg-october-2… Discussion Items: - virtiofs support has debuted in newer linux kernels and the libvirt/qemu/kvm communities have adopted it - in OpenStack, Nova can be made to use virtio-fs and expose Manila shares to tenant VMs - this provides a user experience improvement for manila users where mount automation has been a concern for a while, along with the desire to have strong multi-tenant separation on the network path which only a subset of manila backends are able to provide directly - virtiofs support does not replace DHSS=True or use of gateways like NFS-Ganesha (e.g.: CephFS, GPFS, etc) since those are still very much valuable when the consumer isn't nova VMs but baremetals, containers or non-OpenStack VMs. - nova team helped understand the feature submission process and provided guidance on how to integrate this feature in stages - the user experience, data models and cross service RPCs for this can be modeled around block device mapping, however we don't need to share or copy the implementation - this is surely a large multi-cycle work - nova team suggested that we begin with only attach/detach workflows in Wallaby Actions: - tbarron will propose a nova spec to allow "nova attach fs" (and detach) workflows - we'll continue to design "boot with share" or "boot from share" APIs, including scheduler changes through this cycle === Share Server Updates === Topic Etherpad: https://etherpad.opendev.org/p/share-server-update Discussion Items: - objective is to be able to update security services and share network subnets after share servers have been provisioned - users that want to make these updates have no access to share servers - a security service update can impact multiple share networks (and hence multiple share servers) and in turn multiple share resources, so there was a concern with granularity of the update - operators don't want to be involved with the updates, want users to be able to make the updates on resources they can see (security services or share networks) - if we allow end users to make these changes, we'll still need to break down the operation into more granular bits, and provide APIs to perform validation and multi-step updates Actions: - dviroel will propose spec updates based on the feedback gathered here: https://review.opendev.org/729292 - need other DHSS=True driver maintainers to pay attention to these changes === Share Affinity Hints === Topic Etherpad: https://etherpad.opendev.org/p/manila-anti-affinity Discussion Items: - need a way to specify affinity or anti-affinity placement hints wrt existing resources (e.g: shares, share groups) - share groups provide a way to indicate affinity, but they may also encase shares into a construct and make individual operations on shares harder (you cannot add existing shares to a group or remove shares from a share group without deleting them, you can only create new members in a share group) - need a vehicle other than groups or share type extra specs to carry scheduling decisions that affect single shares - this mechanism shouldn't break cloud storage abstraction - cinder has had the ability for end users to add placement hints for affinity without breaking abstractions Actions: - carthaca will propose a specification for this work - we'll seek reviews from cinder contributors on the spec == Day 3 - Oct 28, 2020, Wed == === Responsibility for Code Reviews === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - a bulk of the day-2 operations that are being enhanced have significant impact on vendor driver implementations - we do have first party "reference" driver support for testing and designing manila APIs in a generic fashion but, not having vendor driver maintainer feedback is unfortunate - operator feedback is also very much appreciated, and we've made significant decisions because of it Actions: - proposers for new features that affect drivers must actively seek out vendor driver maintainers and operators and add them to review. The team will help identify these individuals - we'll track vendor/operator feedback and reviews as part of the review focus tracking we do through the cycle === Acting on Deprecation removals === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - tbarron proposed https://review.opendev.org/745206/ to remove old deprecated config options from manila - we should be doing this more actively each cycle, however, we held off from merging the code change above considering the impact to configuration and deployment tooling - driver module maintainers must adjust the deployment tooling/other usages accordingly asap Actions: - dviroel, carloss and gouthamr will review and merge https://review.opendev.org/745206/ === rootwrap-to-privsep migration === Topic Etherpad: https://etherpad.opendev.org/p/manila-migration-from-rootwrap-to-privsep Discussion Items: - this has been selected as a community goal for Wallaby - privsep has not been used in manila so far - most first party and reference drivers (lvm, zfsonlinux, generic, container, ceph) use rootwrap heavily - some third party drivers also use rootwrap - team was concerned with the completion criteria wrt third party drivers - a stretch goal would be to investigate if dropping privileges for operations by substituting mechanisms is possible Actions: - carloss will work on this goal, no specification is necessary - adequate testing, admin/operator documentation will be provided - carloss will reach out to the third party driver maintainers to act on rootwrap removals - carloss/team will attack de-escalation/removing use of root privs to perform an operation, opportunistically === Secure RBAC improvements === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-policy-changes Discussion Items: - manila will need to support splitting of the existing admin, member roles to system, domain and project admins and members - operators can define personas with these default roles using keystone's "role scope" - an accompanying feature is to provide "reader" role permissions as well - manila accepts the use of "policy.yaml" to drive RBAC, over "policy.json", but, the default is still "policy.json" and this needs to change due to several challenges, including the ability to have comments in the files Actions: - gouthamr will propose a spec for policy refresh across manila APIs/resources using these new role personas - making policy.yaml the default for policy overrides is going to be a community goal. We're targeting this transition to wallaby-1 in manila - we'll attempt to complete this effort in wallaby. there are several parts, collaboration is welcome! === CephFS Updates === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-cephfs-driver-update Discussion Items: - vkmc provided an update on the cephfs feature roadmap - we need changes in cephfs (preferably in ceph-nautilus) to perform authorization, subvolume cloning, capacity reporting etc. This work is being tracked through tracker tickets in Ceph's tracking system - we stopped using shaman builds for ceph and nfs-ganesha in victoria cycle - consuming nfs-ganesha from ppa's has introduced new scenario test failures, vkmc is currently triaging them Actions: - start testing ceph octopus in the wallaby cycle - keep momentum on the ceph trackers and switch over from ceph-volume-client to ceph-mgr and implement create share from snapshot in the cephfs drivers - engage with ceph and nfs-ganesha communities to create a support matrix across nfs-ganesha versions, manila and cephfs === Divisive Language Correction === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - spotz joined us, and recapped the diversity and inclusion wg's work so far - manila team is determined to help resolve any divisive language and references in code and documentation - the team realizes that unconscious use of negative language is affecting the community's inherent desire to be welcoming and inclusive - no manila architecture or code references currently *need* the use of the divisive words identified by the working group and the wider openstack community - making changes is important to us, but we recognize it's just part of the story Actions: - we'll begin with a documentation change to remove the words identified in https://etherpad.opendev.org/p/divisivelanguage - we'll wait to coordinate a change of name for the development/trunk branch, and other "upstream" changes to databases, git, opendev - team actively welcomes any feedback == Day 5 - Oct 30, 2020, Fri == === Share and Server Migration graduation === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila Discussion Items: - we've graduated one major feature through the last couple of cycles, share migration is next - team agreed to keep share server migration as experimental APIs since they haven't had sufficient soak time - no significant changes have been made to share migration migration APIs since Ocata, leading to believe the APIs are stable and can be improved via microversions where necessary - not a lot of vendor driver implementations for driver optimized migration, NetApp driver only allows moving within a backend at the moment - generic migration improvements have been deprioritized, and need help in terms of High Availability of the Data service as well as providing a jobs table to track, and coordinate ongoing migrations - tests are being skipped at the gate because of flakiness Actions: - team will get in touch with third party vendors and guage interest in implementing driver optimized migration (or testing/supporting generic migration with their drivers) (carloss). Feedback from the vendors will drive our decision to graduate the share migration APIs in Wallaby. === Metadata for all share resources === Topic Etherpad: https://etherpad.opendev.org/p/wallaby-ptg-manila-metadata-controller Discussion Items: - the proposal is to introduce a metadata API controller that can be inherited by the resource controllers - all user facing resources will have metadata capabilities, with consistent API methods - this is expected to benefit automation use cases, manila-csi and the upcoming virtio-fs effort Actions: - gouthamr will propose the spec for team's review, including identifying the resources that will be targeted in the Wallaby release - share metadata APIs will be improved to adhere to the API SIG spec: https://specs.openstack.org/openstack/api-sig/guidelines/metadata.html === Client Project updates === Topic Etherpads: https://etherpad.opendev.org/p/manila-osc-wallaby-ptg and https://etherpad.opendev.org/p/manila-ui-wallaby-ptg Discussion Items: - the team met with Nicole Chen, Ashley Rodriguez, Mark Tony from Boston University who will be working on the OpenStackSDK integration for Manila APIs - maaritamm provided an update on the OSC plugin interface implementation - vkmc provided an update on manila-ui feature gaps and the upcoming Outreachy internship Actions: - for some OSC plugin command implementations, we'll benefit from integrating with OpenStackSDK rather than manilaclient SDK, so we'll coordinate that work - maaritamm/vkmc will send an email to openstack-discuss to prioritize the commands left to implement in the OSC plugin - vkmc will drive consistency across cinder/manila for the user messages dashboards that were added in Victoria - we'll continue to work on closing the feature gap in manila-ui === Manila CSI Wallaby roadmap === Topic Document: https://docs.google.com/document/u/1/d/1YJcv0EROWzYbV_8DM3DHVvDqdhlPhYRV8Np… Discussion Items: - users of manila-csi today deploy one driver per share protocol - future work on monitoring, common node plugin interactions requires us to rework the architecture to have one driver multiplex with multiple share protocol node plugins - upgrade impact as well as UI/UX impact was discussed and gman0 has documented them - share metadata is necessary to tag and identify shares Actions: - gman0 has started working on the code patches, and will benefit from reviews - mfedosin will take a look at the upgrade impact and share details for automation that can be done in the csi-operator - mfedosin will test gman0's latest patch to add share metadata via the storage class (as will folks from CERN) and provide feedback Thanks for reading thus far, there were a number of items that we couldn't cover, that you can see on the PTG Planning etherpad [1], if you own those topics, please add them to the weekly IRC meeting agenda [2] and we can go over them. The master meeting minutes document is available as well [3]. As usual, the whole PTG was recorded and posted on the OpenStack Manila Youtube channel [4] We had a great turn out and we heard from a diverse set of contributors, operators, interns and users from several affiliations and time zones. On behalf of the OpenStack Manila team, I deeply appreciate your time, and help in keeping the momentum on the project! [1] https://etherpad.opendev.org/p/wallaby-ptg-manila-planning [2] https://wiki.openstack.org/wiki/Manila/Meetings [3] https://etherpad.opendev.org/p/wallaby-ptg-manila [4] https://www.youtube.com/playlist?list=PLnpzT0InFrqATBbJ60FesKGYRejnTIoCW

1 0

[gate][all] assorted tempest jobs fail when IPv4 default route is not available
by melanie witt 04 Nov '20

04 Nov '20

Hi all, We have a gate bug [1] where if a CI VM does not have an IPv4 default route configured, the job will fail with one of the following errors: "[ERROR] /opt/stack/devstack/functions-common:230 Failure retrieving default route device" [2] or "[ERROR] /opt/stack/devstack/functions-common:104 Failure retrieving default IPv4 route devices" [3] because the die_if_not_set is being triggered. Most of the time, CI VMs come configured with an IPv4 default route but something changed recently (around Oct 23 or 24) to where we often get VMs that do not have an IPv4 default route configured and presumably only have an IPv6 default route. According to this logstash query [4] we have: 58 hits in the last 7 days, check and gate, all failures, all on the limestone-regionone node provider, various projects, various jobs Nate Johnson has proposed a fix here: https://review.opendev.org/761178 Reviews appreciated. Cheers, -melanie [1] https://bugs.launchpad.net/devstack/+bug/1902002 [2] example: https://zuul.openstack.org/build/7543021732cc4886b608553da58e81f9/log/contr… [3] example: https://zuul.opendev.org/t/openstack/build/7fff839dd0bc4214ba6470ef40cf78fd… [4] http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:…

1 0

Editing libvirt xml file
by vladimir franciz blando 04 Nov '20

04 Nov '20

Hi, I edited the libvirt xml (virsh edit...) file of a running Windows server instance because I changed the video model type from cirrus to vmvga so it can support a higher resolution on a console (cirrus supports up to 1280 only). After editing, i "soft-rebooted" the instance and the new configuration sticks. That's awesome. But when you do a "hard-reboot" it will revert back to the original config. I also tried to do a "shut-down" then "start" again, but it still reads the original config. Not sure where it is reading the config from... Regards Vlad

3 5

[neutron][ovn] Neutron <-> OVN DB synchronization code
by Piotr Misiak 04 Nov '20

04 Nov '20

Hi, Do you know why Neutron <-> OVN DB sync code synchronizes "DEVICE_OWNER_ROUTER_HA_INTF" router ports here in this line: https://github.com/openstack/neutron/blob/164f12349fd8be09b9fd4f23b8cf8d2f3… ? As far as know there is no need for HA network ports for L3 virtual routers implementation in OVN, because OVN uses the internal BFD mechanism for Gateway Chassis reachability tests. Thanks, Piotr

1 0

[kolla] Kolla klub meetings paused
by Mark Goddard 04 Nov '20

04 Nov '20

Hi, We agreed at the PTG to try rebooting the Kolla klub meetings at a different time to involve some new members. Let's pause the meetings until then. Regards, Mark

1 0

[tripleo] next meeting Tuesday Nov 10 @ 1400 UTC in #tripleo
by Marios Andreou 04 Nov '20

04 Nov '20

Hello tripleo as discussed at PTG - we voted [1] to have IRC meetings every 2 weeks and next week will be 2 weeks since PTG :) Keeping the same day and time our next meeting will be ** Tuesday 10th November at 1400 UTC in #tripleo. ** Our regular agenda is at https://wiki.openstack.org/wiki/Meetings/TripleO Please add items you want to raise at https://etherpad.opendev.org/p/tripleo-meeting-items . There were concerns voiced at PTG about low attendance in these meetings but at the same time quite a few folks voted to have it. So, let's try again and as with all the things we will re-evaluate based on all our feedback, Can I get a o/ if you plan on attending these meetings? Or even if you can't but plan to read the meetbot logs ;) Obviously now would be a good time to speak up if you hate tuesdays at 1400 UTC and want to propose some alternative for us to consider, thanks, marios [1] https://etherpad.opendev.org/p/tripleo-ptg-wallaby-community-process

1 0

[ptl] Victoria Release Community Meeting
by helena＠openstack.org 04 Nov '20

04 Nov '20

Hi Everyone, We are looking to do a community meeting following The Open Infrastructure Summit and PTG to discuss the Victoria Release. If you’re a PTL, please let me know if you’re interested in doing a prerecorded explanation of your project’s key features for the release. We will show a compilation of these recordings at the community meeting and follow it with a live Q&A session. Post community meeting we will have this recording live in the project navigator. Cheers, Helena

8 10