November 2020 - openstack-discuss

[cinder] Block Storage API v2 removal
by Brian Rosmaita 04 Nov '20

04 Nov '20

The Block Storage API v2 was deprecated in the Pike release by Change-Id: I913c44799cddc37c3342729ec0ef34068db5b2d4 The Cinder team will be removing it during the Wallaby development cycle. At the Wallaby PTG, the question came up about Block Storage API v2 support in the python-cinderclient. We looked at some proposals [0] at today's Cinder meeting and plan to make a decision at next week's meeting [1]. If you have strong feelings about this issue, please make them known on the etherpad [0], the ML, or by attending the next Cinder meeting (1400 UTC in #openstack-meeting-alt on Wednesday 11 November). [0] https://etherpad.opendev.org/p/python-cinderclient-v2-support-removal [1] https://etherpad.opendev.org/p/cinder-wallaby-meetings

1 0

[tc] weeklu update
by Mohammed Naser 04 Nov '20

04 Nov '20

Hi everyone, Here's an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. # Patches ## Open Reviews - Adding reno and warnings example for JSON->YAML goal doc https://review.opendev.org/760956 - Select JSON to YAML goal for Wallaby cycle https://review.opendev.org/760957 - Add goal for migrate RBAC policy format from JSON to YAML https://review.opendev.org/759881 - Add diablo_rojo liaison preferences https://review.opendev.org/759718 - Add danms liaison preferences https://review.opendev.org/758822 - Add ricolin liaison preferences https://review.opendev.org/759704 - Add gmann liaison preference https://review.opendev.org/758602 - Resetting projects' TC liaisons empty https://review.opendev.org/758600 - Move Placement under Nova's governance https://review.opendev.org/760917 - Clarify the requirements for supports-api-interoperability https://review.opendev.org/760562 - Add Resolution of TC stance on the OpenStackClient https://review.opendev.org/759904 ## General Changes - Implement distributed leadership in tools and schema https://review.opendev.org/757966 - Adding missing IRC nick https://review.opendev.org/759724 - Update dansmith IRC nick https://review.opendev.org/760561 - Update projects with Telemetry PTL election result https://review.opendev.org/760206 - Appoint XueFeng Liu as Senlin PTL https://review.opendev.org/758122 - Begin the 'X' release naming process https://review.opendev.org/759888 - Appoint Adam Harwell as Octavia PTL https://review.opendev.org/758121 - Add nomination for TC chair https://review.opendev.org/758181 - Appoint Rafael Weingartner as CloudKitty PTL https://review.opendev.org/758120 - Appoint Frode Nordahl as OpenStack Charms PTL https://review.opendev.org/758119 - Select Privsep as the Wallaby Goal https://review.opendev.org/755590 ## Project Updates - Adopting the DPL governance model for oslo https://review.opendev.org/757906 Thanks for reading! Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

Random 503 error while running using api/command line
by Deepa KR 04 Nov '20

04 Nov '20

Hi All Good Day We have a Train Openstack setup using juju maas configured in HA with 3 controllers. When using the openstack api getting a timeout error randomly. Also sometimes getting "Unknown Error (HTTP 503)" or Remote end closed connection without response while using the command line . openstack server list --all-projects *Unknown Error (HTTP 503)* Kindly suggest to look in the right direction. Regards, Deepa K R

1 0

[tripleo][ci] Monday Nov 2
by Wesley Hayutin 04 Nov '20

04 Nov '20

Greetings, The tripleo ci team has identified a handful of patches that we'd like to land prior to Nov 2, the day docker.io goes away. We've hit some new bugs and have also tuned a few things to try and make sure we can get patches to merge. Our current focus is across master, victoria, ussuri and centos-8 train, and queens while reducing coverage in rocky and stein. A list of the prioritized gerrit reviews can be found here: https://hackmd.io/MlbZ_izSTEuZsCWTJvu_Kg?view The entire topic can be found here: https://review.opendev.org/#/q/topic:new-ci-job Thanks all.

4 4

[nova] Why nova needs password-less SSH to do live migraiton?
by Zhi CZ Chang 04 Nov '20

04 Nov '20

3 5

[ironic] Configuration mold follow-up
by Jaunteva, Aija 04 Nov '20

04 Nov '20

Hi, to follow up on Ironic PTG session about Configuration molds [1] I'm scheduling a call to discuss remaining items (mainly storage of the molds). Anyone interested please add your availability in Doodle [2]. When time slot decided, will share call details. Regards, Aija [1] https://etherpad.opendev.org/p/ironic-wallaby-ptg line 216 [2] https://doodle.com/poll/dry4x5tbmhi6x6p3

2 3

[Neutron] PTG summary
by Slawek Kaplonski 04 Nov '20

04 Nov '20

Hi, Below is my summary of the Neutron team sessions which we had during the virtual PTG last week. Etherpad with notes from the discussions can be found at [1]. ## Retrospective of the Victoria cycle >From the good things during _Victoria_ cycle team pointed: * Complete 8 blueprints including the _Metadata over IPv6_ [2], * Improved feature parity in the OVN driver, * Good review velocity >From the not so good things we mentioned: * CI instability - average number of rechecks needed to merge patch in last year can be found at [3], * Too much "Red Hat" in the Neutron team - and percentage of the reviews (and patches) done by people from Red Hat is constantly increasing over last few cycles. As a main reason of that we pointed that it is hard to change companies way of thinking that dedicating developer to the upstream project means that they lose developer in downstream. * Migration of our CI and Zuul v3 took us a lot of time and effort - but good thing is that we accomplished that in the Victoria cycle :) During that session we also agreed on some action items for the Wallaby cycle: * keep talking about OVN details - Miguel Lavalle and me will work on some way to deliver talks about Neutron OVN backend and OVN internals to the community during next months. The idea is to maybe propose some talks ever 6 weeks or so. This may make ovn driver development more diverse and let operators to thing about migration to the ovn backend. ## Review of the existing meetings We reviewed list of our existing upstream meetings and discussed about ideas on how to increase number of attendees on the meetings. We decided to: * drop neutron-qos meeting as it's not needed anymore * advertise more meetings and meetings' agenda on the OpenStack mailing list - I will send reminders with links to the agenda before every meeting * Together with Lajos Katona we will give some introduction to the debugging of CI issues in Neutron* ## Support for old versions Bernard started discussion about support for the old releases in Neutron and Neutron stadium projects. For Neutron we decided to mark __Ocata__ branch as unmaintained already as its gate is already broken. For the __Pike__ and never branches we will keep them in the __EM__ phase as there is still some community interest to keep those branches open. For the stadium projects we decided to do it similary to what we did while looking for new maintainers for the projects. We will send email "call for maintainers" for such stable branches. If there will be no voluneers to step in, fix gate issues and keep those branches healthy, we will mark them as __unmaintained__ and later as __End of Life__ (EOL). Currently broken CI is in projects: * networking-sfc, * networking-bgpvpn/bagpipe, * neutron-fwaas And those are candidates to be marked as unmaintained if there will be no volunteers to fix them. Bernard Cafarelli volunteered to work on that in next days/weeks. ## Healtcheck API endpoint We discussed as our healtcheck API should works. During the discussion we decided that: * healtcheck result should __NOT__ rely on the agents status, it should rely on worker's ability to connect to the DB and MQ (rabbitmq) * Lajos will ask community (API experts) about some guidance how it should works on the whole OpenStack level, * As for reference implementation we can check e.g. Octavia [4] and Keystone [5] which already implemented it. ## Squash of the DB migration script Rodolfo explained us what are benefits of doing such squash of the db migration scripts from the old versions: * Deployment is faster: we don't need to create/delete tables or create+update other ones - the win is small possibly in the magnitude of 5s per job, * DB definition is centralized in one place, not in original definition plus further migrations - that is most important reason why we really should do that, * UTs faster: removal of some older checks. The problem with this may be that we need to do that carefully and be really verbose about with such changes we may break stadium projects or 3rd party projects which are doing db migration too. To minimalize potential breakage, we will announce such changes on the OpenStack discuss mailing list. Rodolfo volunteered to take propose squash up to Liberty release in W cycle. Together with this squash we will also document that process so in next cycles we should be able to do squashes for newer releases in easier way. Lajos volunteered to help with fixing Neutron stadium projects if that will be needed. ## Switch to the new engine facade We were discussing how to move on and finally finish old Blueprint [6]. We decided that together with Rodolfo we will try how this new engine facade will work without using transaction guards in the code. Hopefully that will let us move on with this. If not, we will try to reach out to some DB experts for some help with this. ## Change from rootwrap to the privsep This is now community goal during the Wallaby cycle so we need to focus on it to accomplish that transition finally. This transition may speed up and make our code a bit more secure. Rodolfo explained us multiple possible strategies of migration: * move to native, e.g. * replace ps with python psutils, not using rootwrap or privsep * replace ip commands with pyroute2, under a privsep context (elevated permissions needed) * directly translate rootwrap to privsep, executing the same shell command but under a privsep context To move on with this I will create list of the pieces of code which needs to be transitioned in the Neutron repo and in the stadium projects. Current todo items can be found on the storyboard [7]. ## Migration to the NFtables During this session we were discussing potential strategies on how to migrate from the old iptables to the new nftables. We need to start planning that work as it major Linux distributions (e.g. RHEL) are planning to deprecate iptables in next releases. It seems that currently all major distros (Ubuntu, Centos, OpenSuSE) supports nftables already. We decided that in Wallaby cycle we will propose new _Manager_ class and we will add some config option which will allow people to test new solution. In next cycles we will continue work on it to make it stable and to make upgrade and migration path for users as easy as possible. There is already created blueprint to track progress on that topic [8]. We need to migrate: * Linuxbridge firewall, iptables OVS hybrid firewall, * L3 code (legacy router, DVR router, conntrack, port forwarding), * iptables metering, * metadata proxy, * dhcp agent for when it does metadata for isolated networks and namespace creation, * neutron-vpnaas - ipsec code, * and maybe something else what we didn't found yet. ## Nova-Neutron cross project session We had very interesting discussion with Nova team. We were discussing topics like: * NUMA affinity in the neutron port * vhost-vdpa support * default __vnic_type__/__port flavour__ Notes from that discussion are available in the nova's etherpad [9]. ## Neutron scalling issues At this session we were discussing issues mentioned by operators on the Forum sessions a week before the PTG. There was couple of issues mentioned there: * problems with retries of the DB operations - we should migrate all our code to the oslo.db retries mechanism - new blueprint [10] is created to track progress on that one. * problems with maintenance of the agents, like e.g. DHCP or L3 agents - many of those issues are caused by how our agents are designed and to really fix that we would need very deep and huge changes. But also many of those issues can be solved by the __ovn__ backend - **and that is strategic direction in which neutron wants to go in the next cycles**, * Miguel Lavalle and I volunteered to do some profiling of the agents to see where we are loosing most of the time - maybe we will be able to find some _low hanging fruits_ which can be fixed and improve the situation at least a bit, * Similar problem with neutron-ovs-agent and especially security groups which are using _remove group id_ as a reference - here we also need some volunteers who will try to optimize that. ## CI (in)stablility On Thursday we were discussing how to improve our very poor CI. Finally we decided to: * not recheck patches without giving reason of recheck in the comment - there should be already reported bug which should be linked in the _recheck_ comment, or user should open new one and link to it also. IN case if the problem was e.g. related to infra some simple comment like _infra issue_ will also be enough there, * To lower number of existing jobs we will do some changes like: * move *-neutron-lib-master and *-ovs-master jobs to the experimental and periodic queues to not run them on every patch, * I will switch _neutron-tempest-plugin-api_ job to be deployed with uwsgi so we can drop _neutron-tempest-with-uwsgi_ job, * Consolidate _neutron-tempest-plugin-scenario-linuxbridge_ and _neutron- tempest-linuxbridge_ jobs, * Consolidate _neutron-tempest-plugin-scenario-iptables_hybrid and _neutron- tempest-iptables_hybrid jobs, Later we also discussed about the way how to run or skip tests which can be only run when some specific feature is available in the cloud (e.g. _Metadata over IPv6_). After some discussion we decided to add new config option with list of enabled features. It will be very similar to the existing option _api_extensions_. Lajos volunteered to work on that. As last CI related topic we discussed about testing DVR in our CI. Oleg Bondarev volunteered to check and try to fix broken _neutron-tempest-plugin- dvr-multinode-scenario_ job. ## Flow based DHCP Liu Yulong raised topic about new way of doing fully distributed DHCP service, instead of using _DHCP agent_ on the nodes - RFE is proposed at [11]. His proposal of doing Open Flow based DHCP (similar to what e.g. ovn-controller is doing) is described in [12]. It could be implemented as an L2 agent extension and enabled by operators in the config when they would need it. As a next step Liu will now propose spec with details about this solution and we will continue discussion about it in the spec's review. ## Routed provider networks limited to one host As a lost topic on Thursday we briefly talked about old RFE [13]. Miguel Lavalle told us that his company, Verizon Media, is interested in working on this RFE in next cycles. This also involves some work on Nova's side which was started by Sylvain Bauza already. Miguel will sync with Sylvain on that RFE. ## L3 feature improvements On Friday we were discussing some potential improvements in the L3 area. Lajos and Bence shown us some features which their company is interested in and on which they plan to work. Those are things like: * support for Bidirectional Forwarding Detection * some additional API to set additional router parameters like: * ECMP max path, * ECMP hash algorith * --provider-allocation-pool parameter in the subnets - in some specific cases it may help to use IPs from such _special_ pool for some infrastructure needs, more details about that will come in the RFE in future, For now all those described above improvements are in very early planning phase but Bence will sync with Liu and Liu will dedicate some time to discuss progress on them during the __L3 subteam meetings__. ## Leveraging routing-on-the-host in Neutron in our next-gen clusters As a last topic on Friday we were discussing potential solutions of the _L3 on the host_ in the Neutron. The idea here is very similar to what e.g. __Calico plugin__ is doing currently. More details about potential solutions are described in the etherpad [14]. During the discussion Dawid Deja from OVH told us that OVH is also using very similar, downstream only solution. Conclusion of that discussion was that we may have most of the needed code already in Neutron and some stadium projects so as a first step people who are interested in that topic, like Jan Gutter, Miguel and Dawid will work on some deployment guide for such use case. ## Team photo During the PTG we also made team photos which You can find at [15]. [1] https://etherpad.opendev.org/p/neutron-wallaby-ptg [2] https://blueprints.launchpad.net/neutron/+spec/metadata-over-ipv6 [3] https://ibb.co/12sB9N9 [4] https://opendev.org/openstack/octavia/src/branch/master/octavia/api/ healthcheck/healthcheck_plugins.py [5] https://docs.openstack.org/keystone/victoria/admin/health-check-middleware.… [6] https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch [7] https://storyboard.openstack.org/#!/story/2007686 [8] https://blueprints.launchpad.net/neutron/+spec/nftables-migration [9] https://etherpad.opendev.org/p/nova-wallaby-ptg [10] https://blueprints.launchpad.net/neutron/+spec/oslo-db-retries [11] https://bugs.launchpad.net/neutron/+bug/1900934 [12] https://github.com/gotostack/shanghai_ptg/blob/master/ shanghai_neutron_ptg_topics_liuyulong.pdf [13] https://bugs.launchpad.net/neutron/+bug/1764738 [14] https://etherpad.opendev.org/p/neutron-routing-on-the-host [15] http://kaplonski.pl/files/Neutron_virtual_PTG_October_2020.tar.gz -- Slawek Kaplonski Principal Software Engineer Red Hat

3 4

Intel 82599 dpdk support with centos8
by Satish Patel 03 Nov '20

03 Nov '20

Folks, I am trying to get dpdk working with my OVS on CentOS-8 and I am stuck here so I need advice from folks who can help me here. I have an HP c7000 chassis with ProLiant BL460c Gen9 blade. I am running CentOS-8 with following version of ovs-dpdk [root@compute-2 usertools]# ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.12.0 DPDK 18.11.2 I am using openstack-ansible to set up openvswitch with dpdk, and so far everything went well but I am stuck in the end. [root@compute-2 usertools]# ovs-vsctl add-port br-provider dpdk-0 -- set Interface dpdk-0 type=dpdk options:dpdk-devargs=0000:06:00.0 ovs-vsctl: Error detected while setting up 'dpdk-0': Error attaching device '0000:06:00.0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". Here is the error message. [root@compute-2 usertools]# tail -f /var/log/openvswitch/ovs-vswitchd.log 2020-11-04T02:46:31.751Z|00303|dpdk|INFO|EAL: PCI device 0000:06:00.0 on NUMA socket 0 2020-11-04T02:46:31.751Z|00304|dpdk|INFO|EAL: probe driver: 8086:10f8 net_ixgbe 2020-11-04T02:46:31.751Z|00305|dpdk|ERR|EAL: Driver cannot attach the device (0000:06:00.0) 2020-11-04T02:46:31.751Z|00306|dpdk|ERR|EAL: Failed to attach device on primary process 2020-11-04T02:46:31.751Z|00307|netdev_dpdk|WARN|Error attaching device '0000:06:00.0' to DPDK 2020-11-04T02:46:31.751Z|00308|netdev|WARN|dpdk-0: could not set configuration (Invalid argument) 2020-11-04T02:46:31.751Z|00309|dpdk|ERR|Invalid port_id=32 This is what my NIC firmware/driver [root@compute-2 usertools]# ethtool -i eno50 driver: ixgbe version: 5.1.0-k-rh8.2.0 firmware-version: 0x800008fb, 1.2028.0 expansion-rom-version: bus-info: 0000:06:00.1 [root@compute-2 usertools]# python3 ./dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:06:00.0 '82599 10 Gigabit Dual Port Backplane Connection 10f8' drv=vfio-pci unused=ixgbe Network devices using kernel driver =================================== 0000:06:00.1 '82599 10 Gigabit Dual Port Backplane Connection 10f8' if=eno50 drv=ixgbe unused=vfio-pci I did enabled IOMMU & huge mem in grub iommu=pt intel_iommu=on hugepagesz=2M hugepages=30000 transparent_hugepage=never Not sure what is wrong here and the big question is does this NIC support DPDK or not?

1 0

In Openstack (devstack) i am able to create maximum of 3 volumes only
by ANUSUYA MANI 03 Nov '20

03 Nov '20

I have 3 volumes of 80GB each and when i try to create a volume snapshot or a new volume i am not able to do that. i get the error 'schedule allocate volume:Could not find any available weighted backend'. The volume quotas are set properly yet i am not able to create more than 3 volumes. i have attached screenshots of the volume quotas and error message. Kindly <https://i.stack.imgur.com/ThvMo.jpg> requesting to help me solve this issue.

1 0

[cinder] Wallaby PTG summary available
by Brian Rosmaita 03 Nov '20

03 Nov '20

I've posted a summary of the discussions we had last week: https://wiki.openstack.org/wiki/CinderWallabyPTGSummary Feel free to make any corrections or clarifications. Also, look through to remind yourself what actions you volunteered to perform (you can search for your IRC nick or the text ACTION). cheers, brian

1 0