- openstack-discuss - lists.openstack.org

[tripleo] next meeting Tuesday Nov 10 @ 1400 UTC in #tripleo
by Marios Andreou 04 Nov '20

04 Nov '20

Hello tripleo as discussed at PTG - we voted [1] to have IRC meetings every 2 weeks and next week will be 2 weeks since PTG :) Keeping the same day and time our next meeting will be ** Tuesday 10th November at 1400 UTC in #tripleo. ** Our regular agenda is at https://wiki.openstack.org/wiki/Meetings/TripleO Please add items you want to raise at https://etherpad.opendev.org/p/tripleo-meeting-items . There were concerns voiced at PTG about low attendance in these meetings but at the same time quite a few folks voted to have it. So, let's try again and as with all the things we will re-evaluate based on all our feedback, Can I get a o/ if you plan on attending these meetings? Or even if you can't but plan to read the meetbot logs ;) Obviously now would be a good time to speak up if you hate tuesdays at 1400 UTC and want to propose some alternative for us to consider, thanks, marios [1] https://etherpad.opendev.org/p/tripleo-ptg-wallaby-community-process

1 0

[ptl] Victoria Release Community Meeting
by helena＠openstack.org 04 Nov '20

04 Nov '20

Hi Everyone, We are looking to do a community meeting following The Open Infrastructure Summit and PTG to discuss the Victoria Release. If you’re a PTL, please let me know if you’re interested in doing a prerecorded explanation of your project’s key features for the release. We will show a compilation of these recordings at the community meeting and follow it with a live Q&A session. Post community meeting we will have this recording live in the project navigator. Cheers, Helena

8 10

[cinder] Block Storage API v2 removal
by Brian Rosmaita 04 Nov '20

04 Nov '20

The Block Storage API v2 was deprecated in the Pike release by Change-Id: I913c44799cddc37c3342729ec0ef34068db5b2d4 The Cinder team will be removing it during the Wallaby development cycle. At the Wallaby PTG, the question came up about Block Storage API v2 support in the python-cinderclient. We looked at some proposals [0] at today's Cinder meeting and plan to make a decision at next week's meeting [1]. If you have strong feelings about this issue, please make them known on the etherpad [0], the ML, or by attending the next Cinder meeting (1400 UTC in #openstack-meeting-alt on Wednesday 11 November). [0] https://etherpad.opendev.org/p/python-cinderclient-v2-support-removal [1] https://etherpad.opendev.org/p/cinder-wallaby-meetings

1 0

[tc] weeklu update
by Mohammed Naser 04 Nov '20

04 Nov '20

Hi everyone, Here's an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. # Patches ## Open Reviews - Adding reno and warnings example for JSON->YAML goal doc https://review.opendev.org/760956 - Select JSON to YAML goal for Wallaby cycle https://review.opendev.org/760957 - Add goal for migrate RBAC policy format from JSON to YAML https://review.opendev.org/759881 - Add diablo_rojo liaison preferences https://review.opendev.org/759718 - Add danms liaison preferences https://review.opendev.org/758822 - Add ricolin liaison preferences https://review.opendev.org/759704 - Add gmann liaison preference https://review.opendev.org/758602 - Resetting projects' TC liaisons empty https://review.opendev.org/758600 - Move Placement under Nova's governance https://review.opendev.org/760917 - Clarify the requirements for supports-api-interoperability https://review.opendev.org/760562 - Add Resolution of TC stance on the OpenStackClient https://review.opendev.org/759904 ## General Changes - Implement distributed leadership in tools and schema https://review.opendev.org/757966 - Adding missing IRC nick https://review.opendev.org/759724 - Update dansmith IRC nick https://review.opendev.org/760561 - Update projects with Telemetry PTL election result https://review.opendev.org/760206 - Appoint XueFeng Liu as Senlin PTL https://review.opendev.org/758122 - Begin the 'X' release naming process https://review.opendev.org/759888 - Appoint Adam Harwell as Octavia PTL https://review.opendev.org/758121 - Add nomination for TC chair https://review.opendev.org/758181 - Appoint Rafael Weingartner as CloudKitty PTL https://review.opendev.org/758120 - Appoint Frode Nordahl as OpenStack Charms PTL https://review.opendev.org/758119 - Select Privsep as the Wallaby Goal https://review.opendev.org/755590 ## Project Updates - Adopting the DPL governance model for oslo https://review.opendev.org/757906 Thanks for reading! Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

Random 503 error while running using api/command line
by Deepa KR 04 Nov '20

04 Nov '20

Hi All Good Day We have a Train Openstack setup using juju maas configured in HA with 3 controllers. When using the openstack api getting a timeout error randomly. Also sometimes getting "Unknown Error (HTTP 503)" or Remote end closed connection without response while using the command line . openstack server list --all-projects *Unknown Error (HTTP 503)* Kindly suggest to look in the right direction. Regards, Deepa K R

1 0

[tripleo][ci] Monday Nov 2
by Wesley Hayutin 04 Nov '20

04 Nov '20

Greetings, The tripleo ci team has identified a handful of patches that we'd like to land prior to Nov 2, the day docker.io goes away. We've hit some new bugs and have also tuned a few things to try and make sure we can get patches to merge. Our current focus is across master, victoria, ussuri and centos-8 train, and queens while reducing coverage in rocky and stein. A list of the prioritized gerrit reviews can be found here: https://hackmd.io/MlbZ_izSTEuZsCWTJvu_Kg?view The entire topic can be found here: https://review.opendev.org/#/q/topic:new-ci-job Thanks all.

4 4

[nova] Why nova needs password-less SSH to do live migraiton?
by Zhi CZ Chang 04 Nov '20

04 Nov '20

3 5

[ironic] Configuration mold follow-up
by Jaunteva, Aija 04 Nov '20

04 Nov '20

Hi, to follow up on Ironic PTG session about Configuration molds [1] I'm scheduling a call to discuss remaining items (mainly storage of the molds). Anyone interested please add your availability in Doodle [2]. When time slot decided, will share call details. Regards, Aija [1] https://etherpad.opendev.org/p/ironic-wallaby-ptg line 216 [2] https://doodle.com/poll/dry4x5tbmhi6x6p3

2 3

[Neutron] PTG summary
by Slawek Kaplonski 04 Nov '20

04 Nov '20

Hi, Below is my summary of the Neutron team sessions which we had during the virtual PTG last week. Etherpad with notes from the discussions can be found at [1]. ## Retrospective of the Victoria cycle >From the good things during _Victoria_ cycle team pointed: * Complete 8 blueprints including the _Metadata over IPv6_ [2], * Improved feature parity in the OVN driver, * Good review velocity >From the not so good things we mentioned: * CI instability - average number of rechecks needed to merge patch in last year can be found at [3], * Too much "Red Hat" in the Neutron team - and percentage of the reviews (and patches) done by people from Red Hat is constantly increasing over last few cycles. As a main reason of that we pointed that it is hard to change companies way of thinking that dedicating developer to the upstream project means that they lose developer in downstream. * Migration of our CI and Zuul v3 took us a lot of time and effort - but good thing is that we accomplished that in the Victoria cycle :) During that session we also agreed on some action items for the Wallaby cycle: * keep talking about OVN details - Miguel Lavalle and me will work on some way to deliver talks about Neutron OVN backend and OVN internals to the community during next months. The idea is to maybe propose some talks ever 6 weeks or so. This may make ovn driver development more diverse and let operators to thing about migration to the ovn backend. ## Review of the existing meetings We reviewed list of our existing upstream meetings and discussed about ideas on how to increase number of attendees on the meetings. We decided to: * drop neutron-qos meeting as it's not needed anymore * advertise more meetings and meetings' agenda on the OpenStack mailing list - I will send reminders with links to the agenda before every meeting * Together with Lajos Katona we will give some introduction to the debugging of CI issues in Neutron* ## Support for old versions Bernard started discussion about support for the old releases in Neutron and Neutron stadium projects. For Neutron we decided to mark __Ocata__ branch as unmaintained already as its gate is already broken. For the __Pike__ and never branches we will keep them in the __EM__ phase as there is still some community interest to keep those branches open. For the stadium projects we decided to do it similary to what we did while looking for new maintainers for the projects. We will send email "call for maintainers" for such stable branches. If there will be no voluneers to step in, fix gate issues and keep those branches healthy, we will mark them as __unmaintained__ and later as __End of Life__ (EOL). Currently broken CI is in projects: * networking-sfc, * networking-bgpvpn/bagpipe, * neutron-fwaas And those are candidates to be marked as unmaintained if there will be no volunteers to fix them. Bernard Cafarelli volunteered to work on that in next days/weeks. ## Healtcheck API endpoint We discussed as our healtcheck API should works. During the discussion we decided that: * healtcheck result should __NOT__ rely on the agents status, it should rely on worker's ability to connect to the DB and MQ (rabbitmq) * Lajos will ask community (API experts) about some guidance how it should works on the whole OpenStack level, * As for reference implementation we can check e.g. Octavia [4] and Keystone [5] which already implemented it. ## Squash of the DB migration script Rodolfo explained us what are benefits of doing such squash of the db migration scripts from the old versions: * Deployment is faster: we don't need to create/delete tables or create+update other ones - the win is small possibly in the magnitude of 5s per job, * DB definition is centralized in one place, not in original definition plus further migrations - that is most important reason why we really should do that, * UTs faster: removal of some older checks. The problem with this may be that we need to do that carefully and be really verbose about with such changes we may break stadium projects or 3rd party projects which are doing db migration too. To minimalize potential breakage, we will announce such changes on the OpenStack discuss mailing list. Rodolfo volunteered to take propose squash up to Liberty release in W cycle. Together with this squash we will also document that process so in next cycles we should be able to do squashes for newer releases in easier way. Lajos volunteered to help with fixing Neutron stadium projects if that will be needed. ## Switch to the new engine facade We were discussing how to move on and finally finish old Blueprint [6]. We decided that together with Rodolfo we will try how this new engine facade will work without using transaction guards in the code. Hopefully that will let us move on with this. If not, we will try to reach out to some DB experts for some help with this. ## Change from rootwrap to the privsep This is now community goal during the Wallaby cycle so we need to focus on it to accomplish that transition finally. This transition may speed up and make our code a bit more secure. Rodolfo explained us multiple possible strategies of migration: * move to native, e.g. * replace ps with python psutils, not using rootwrap or privsep * replace ip commands with pyroute2, under a privsep context (elevated permissions needed) * directly translate rootwrap to privsep, executing the same shell command but under a privsep context To move on with this I will create list of the pieces of code which needs to be transitioned in the Neutron repo and in the stadium projects. Current todo items can be found on the storyboard [7]. ## Migration to the NFtables During this session we were discussing potential strategies on how to migrate from the old iptables to the new nftables. We need to start planning that work as it major Linux distributions (e.g. RHEL) are planning to deprecate iptables in next releases. It seems that currently all major distros (Ubuntu, Centos, OpenSuSE) supports nftables already. We decided that in Wallaby cycle we will propose new _Manager_ class and we will add some config option which will allow people to test new solution. In next cycles we will continue work on it to make it stable and to make upgrade and migration path for users as easy as possible. There is already created blueprint to track progress on that topic [8]. We need to migrate: * Linuxbridge firewall, iptables OVS hybrid firewall, * L3 code (legacy router, DVR router, conntrack, port forwarding), * iptables metering, * metadata proxy, * dhcp agent for when it does metadata for isolated networks and namespace creation, * neutron-vpnaas - ipsec code, * and maybe something else what we didn't found yet. ## Nova-Neutron cross project session We had very interesting discussion with Nova team. We were discussing topics like: * NUMA affinity in the neutron port * vhost-vdpa support * default __vnic_type__/__port flavour__ Notes from that discussion are available in the nova's etherpad [9]. ## Neutron scalling issues At this session we were discussing issues mentioned by operators on the Forum sessions a week before the PTG. There was couple of issues mentioned there: * problems with retries of the DB operations - we should migrate all our code to the oslo.db retries mechanism - new blueprint [10] is created to track progress on that one. * problems with maintenance of the agents, like e.g. DHCP or L3 agents - many of those issues are caused by how our agents are designed and to really fix that we would need very deep and huge changes. But also many of those issues can be solved by the __ovn__ backend - **and that is strategic direction in which neutron wants to go in the next cycles**, * Miguel Lavalle and I volunteered to do some profiling of the agents to see where we are loosing most of the time - maybe we will be able to find some _low hanging fruits_ which can be fixed and improve the situation at least a bit, * Similar problem with neutron-ovs-agent and especially security groups which are using _remove group id_ as a reference - here we also need some volunteers who will try to optimize that. ## CI (in)stablility On Thursday we were discussing how to improve our very poor CI. Finally we decided to: * not recheck patches without giving reason of recheck in the comment - there should be already reported bug which should be linked in the _recheck_ comment, or user should open new one and link to it also. IN case if the problem was e.g. related to infra some simple comment like _infra issue_ will also be enough there, * To lower number of existing jobs we will do some changes like: * move *-neutron-lib-master and *-ovs-master jobs to the experimental and periodic queues to not run them on every patch, * I will switch _neutron-tempest-plugin-api_ job to be deployed with uwsgi so we can drop _neutron-tempest-with-uwsgi_ job, * Consolidate _neutron-tempest-plugin-scenario-linuxbridge_ and _neutron- tempest-linuxbridge_ jobs, * Consolidate _neutron-tempest-plugin-scenario-iptables_hybrid and _neutron- tempest-iptables_hybrid jobs, Later we also discussed about the way how to run or skip tests which can be only run when some specific feature is available in the cloud (e.g. _Metadata over IPv6_). After some discussion we decided to add new config option with list of enabled features. It will be very similar to the existing option _api_extensions_. Lajos volunteered to work on that. As last CI related topic we discussed about testing DVR in our CI. Oleg Bondarev volunteered to check and try to fix broken _neutron-tempest-plugin- dvr-multinode-scenario_ job. ## Flow based DHCP Liu Yulong raised topic about new way of doing fully distributed DHCP service, instead of using _DHCP agent_ on the nodes - RFE is proposed at [11]. His proposal of doing Open Flow based DHCP (similar to what e.g. ovn-controller is doing) is described in [12]. It could be implemented as an L2 agent extension and enabled by operators in the config when they would need it. As a next step Liu will now propose spec with details about this solution and we will continue discussion about it in the spec's review. ## Routed provider networks limited to one host As a lost topic on Thursday we briefly talked about old RFE [13]. Miguel Lavalle told us that his company, Verizon Media, is interested in working on this RFE in next cycles. This also involves some work on Nova's side which was started by Sylvain Bauza already. Miguel will sync with Sylvain on that RFE. ## L3 feature improvements On Friday we were discussing some potential improvements in the L3 area. Lajos and Bence shown us some features which their company is interested in and on which they plan to work. Those are things like: * support for Bidirectional Forwarding Detection * some additional API to set additional router parameters like: * ECMP max path, * ECMP hash algorith * --provider-allocation-pool parameter in the subnets - in some specific cases it may help to use IPs from such _special_ pool for some infrastructure needs, more details about that will come in the RFE in future, For now all those described above improvements are in very early planning phase but Bence will sync with Liu and Liu will dedicate some time to discuss progress on them during the __L3 subteam meetings__. ## Leveraging routing-on-the-host in Neutron in our next-gen clusters As a last topic on Friday we were discussing potential solutions of the _L3 on the host_ in the Neutron. The idea here is very similar to what e.g. __Calico plugin__ is doing currently. More details about potential solutions are described in the etherpad [14]. During the discussion Dawid Deja from OVH told us that OVH is also using very similar, downstream only solution. Conclusion of that discussion was that we may have most of the needed code already in Neutron and some stadium projects so as a first step people who are interested in that topic, like Jan Gutter, Miguel and Dawid will work on some deployment guide for such use case. ## Team photo During the PTG we also made team photos which You can find at [15]. [1] https://etherpad.opendev.org/p/neutron-wallaby-ptg [2] https://blueprints.launchpad.net/neutron/+spec/metadata-over-ipv6 [3] https://ibb.co/12sB9N9 [4] https://opendev.org/openstack/octavia/src/branch/master/octavia/api/ healthcheck/healthcheck_plugins.py [5] https://docs.openstack.org/keystone/victoria/admin/health-check-middleware.… [6] https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch [7] https://storyboard.openstack.org/#!/story/2007686 [8] https://blueprints.launchpad.net/neutron/+spec/nftables-migration [9] https://etherpad.opendev.org/p/nova-wallaby-ptg [10] https://blueprints.launchpad.net/neutron/+spec/oslo-db-retries [11] https://bugs.launchpad.net/neutron/+bug/1900934 [12] https://github.com/gotostack/shanghai_ptg/blob/master/ shanghai_neutron_ptg_topics_liuyulong.pdf [13] https://bugs.launchpad.net/neutron/+bug/1764738 [14] https://etherpad.opendev.org/p/neutron-routing-on-the-host [15] http://kaplonski.pl/files/Neutron_virtual_PTG_October_2020.tar.gz -- Slawek Kaplonski Principal Software Engineer Red Hat

3 4

Intel 82599 dpdk support with centos8
by Satish Patel 03 Nov '20

03 Nov '20

Folks, I am trying to get dpdk working with my OVS on CentOS-8 and I am stuck here so I need advice from folks who can help me here. I have an HP c7000 chassis with ProLiant BL460c Gen9 blade. I am running CentOS-8 with following version of ovs-dpdk [root@compute-2 usertools]# ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.12.0 DPDK 18.11.2 I am using openstack-ansible to set up openvswitch with dpdk, and so far everything went well but I am stuck in the end. [root@compute-2 usertools]# ovs-vsctl add-port br-provider dpdk-0 -- set Interface dpdk-0 type=dpdk options:dpdk-devargs=0000:06:00.0 ovs-vsctl: Error detected while setting up 'dpdk-0': Error attaching device '0000:06:00.0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". Here is the error message. [root@compute-2 usertools]# tail -f /var/log/openvswitch/ovs-vswitchd.log 2020-11-04T02:46:31.751Z|00303|dpdk|INFO|EAL: PCI device 0000:06:00.0 on NUMA socket 0 2020-11-04T02:46:31.751Z|00304|dpdk|INFO|EAL: probe driver: 8086:10f8 net_ixgbe 2020-11-04T02:46:31.751Z|00305|dpdk|ERR|EAL: Driver cannot attach the device (0000:06:00.0) 2020-11-04T02:46:31.751Z|00306|dpdk|ERR|EAL: Failed to attach device on primary process 2020-11-04T02:46:31.751Z|00307|netdev_dpdk|WARN|Error attaching device '0000:06:00.0' to DPDK 2020-11-04T02:46:31.751Z|00308|netdev|WARN|dpdk-0: could not set configuration (Invalid argument) 2020-11-04T02:46:31.751Z|00309|dpdk|ERR|Invalid port_id=32 This is what my NIC firmware/driver [root@compute-2 usertools]# ethtool -i eno50 driver: ixgbe version: 5.1.0-k-rh8.2.0 firmware-version: 0x800008fb, 1.2028.0 expansion-rom-version: bus-info: 0000:06:00.1 [root@compute-2 usertools]# python3 ./dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:06:00.0 '82599 10 Gigabit Dual Port Backplane Connection 10f8' drv=vfio-pci unused=ixgbe Network devices using kernel driver =================================== 0000:06:00.1 '82599 10 Gigabit Dual Port Backplane Connection 10f8' if=eno50 drv=ixgbe unused=vfio-pci I did enabled IOMMU & huge mem in grub iommu=pt intel_iommu=on hugepagesz=2M hugepages=30000 transparent_hugepage=never Not sure what is wrong here and the big question is does this NIC support DPDK or not?

1 0

In Openstack (devstack) i am able to create maximum of 3 volumes only
by ANUSUYA MANI 03 Nov '20

03 Nov '20

I have 3 volumes of 80GB each and when i try to create a volume snapshot or a new volume i am not able to do that. i get the error 'schedule allocate volume:Could not find any available weighted backend'. The volume quotas are set properly yet i am not able to create more than 3 volumes. i have attached screenshots of the volume quotas and error message. Kindly <https://i.stack.imgur.com/ThvMo.jpg> requesting to help me solve this issue.

1 0

[cinder] Wallaby PTG summary available
by Brian Rosmaita 03 Nov '20

03 Nov '20

I've posted a summary of the discussions we had last week: https://wiki.openstack.org/wiki/CinderWallabyPTGSummary Feel free to make any corrections or clarifications. Also, look through to remind yourself what actions you volunteered to perform (you can search for your IRC nick or the text ACTION). cheers, brian

1 0

[barbican] Stable Branch Liaison Update
by Douglas Mendizabal 03 Nov '20

03 Nov '20

Hi openstack-discuss, I've updated the Cross Project Liaison Wiki [1] to revert our Stable Branch liaison to myself as current Barbican PTL. I am wondering what the process is to get myself added to the barbican-stable-maint group in gerrit? [2] Unfortunately, our only team member on that team has not been very active the last few months and we have a growing list of patches to the stable branches that we have been unable to merge. Any pointers would be greatly appreciated. Thanks - Douglas Mendizábal [1] https://wiki.openstack.org/wiki/CrossProjectLiaisons#Stable_Branch [2] https://review.opendev.org/#/admin/groups/1097

2 1

[masakari] Wallaby PTG Summary
by Radosław Piliszek 03 Nov '20

03 Nov '20

Hello Everyone! Masakari had 3 2-hour sessions and we managed to discuss a broad range of proposals and assign tasks to interested parties. (And this was my first time chairing a PTG as PTL). Thank you, everyone who participated! Here is a summary of the PTG discussions. The whole discussion has been written down in the etherpad. [1] We acknowledged that the project has lost its momentum during the last cycle (Victoria) and we hope to restore it now. To this end, we are reviving interesting old blueprints and specs. Some of these are already at the implementation stage, ready for reviews. Some of the discussed proposals fall into the 'low code impact, high (positive) usability impact' category, the most prominent being ways to have more flexibility in disabling/enabling the HA scope as well as generally more friendly approach towards user errors (not evacuating 'pinned' instances). On the other hand, some proposals involve deeper architectural changes to Masakari, such as the shift from reliance on Pacemaker/Corosync stack to Consul and/or etcd (possibly also for coordination), as well as making Masakari inspect the host status via both inbound and outbound means (including power management for checks and fencing). To this end, we plan to investigate a possible integration with Ironic as our gate to the baremetal world. Another proposal worth noting is the Masakari 'restore previous state' workflow, meant to help operators move instances back to their original location once the local problem is solved. Yet another proposal is increasing the resilience under the current scope (Pacemaker/Corosync) by retrying host checks (several sampled observations rather than one) and implementing adaptive rules to consider large-scale failures. We also discussed general maintenance tasks. We decided to give some love to the documentation and make sure it's not self-contradictory and also in agreement with the actual code, as well as include general recommendation on how Masakari is supposed to be deployed to have it work optimally. Finally, we discussed a few miscellaneous proposals such as making Masakari a better container world citizen by removing the reliance on systemd. [1] https://etherpad.opendev.org/p/masakari-wallaby-vptg Kind regards, -yoctozepto

1 0

[openstack-helm] No OSH meeting today
by Gage Hugo 03 Nov '20

03 Nov '20

Hello, the meeting for today has been cancelled, we will meet next week.

1 0

[keystone] No keystone team meeting Nov 3
by Nikolla, Kristi 03 Nov '20

03 Nov '20

Hi all, There will be no keystone team meeting on IRC today, on November 3rd. Best, Kristi

1 0

[cinder] reminder: weekly meeting wednesday 4 nov at 1400utc
by Brian Rosmaita 03 Nov '20

03 Nov '20

After two weeks of not having meetings due to summit/forum/ptg, the cinder team will be holding its weekly meeting at the usual time (1400 UTC) and location (#openstack-meeting-alt) tomorrow (wednesday 4 november). People who had been following daylight saving (or summer) time until recently, keep in mind that although the UTC time of the meeting has not changed, it may be an hour earlier for you in your local time. cheers, brian

1 0

[neutron] Team meeting
by Slawek Kaplonski 03 Nov '20

03 Nov '20

Hi, Just a quick reminder. Today at 14:00 UTC we will have neutron team meeting in the #openstack-meeting-3 room @freenode. Meeting's agenda is available in [1]. If You have any topics which You want to discuss, please add them to the "On demand" section. See You all at the meeting :) [1] https://wiki.openstack.org/wiki/Network/Meetings -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[placement][nova][cinder][neutron][blazar] Placement governance switch(back)
by Stephen Finucane 03 Nov '20

03 Nov '20

Placement has been a separate project with its own governance since Stein [1]. Since then, the main drivers behind the separation have moved onto pastures new and with Tetsuro sadly declaring his non- candidacy for the PTL position for Wallaby [2], we're left in the unenviable position of potentially not having a PTL for the Wallaby cycle. As such, it's probably time to discuss the future of Placement governance. Assuming no one steps forward for the Placement PTL role, it would appear to me that we have two options. Either we look at transitioning Placement to a PTL-less project, or we move it back under nova governance. To be honest, given how important placement is to nova and other projects now, I'm uncomfortable with the idea of not having a point person who is ultimately responsible for things like cutting a release (yes, delegation is encouraged but someone needs to herd the cats). At the same time, I do realize that placement is used by more that nova now so nova cores and what's left of the separate placement core team shouldn't be the only ones making this decision. So, assuming the worst happens and placement is left without a PTL for Victoria, what do we want to do? Stephen PS: Apologies if I missed other projects with an interest in placement. I did try to catch them all /o\ [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002575… [2] http://lists.openstack.org/pipermail/openstack-discuss/2020-September/01749…

8 10

[neutron][core team] (Re)adding Oleg Bondarev to the Neutron core team
by Slawek Kaplonski 03 Nov '20

03 Nov '20

Hi, I'm glad to announce that I just fast-tracked back Oleg to the Neutron core team. Oleg was Neutron core in the past and now, since few months he is again one of the most active Neutron reviewers [1]. Welcome back in the core team Oleg :) [1] https://www.stackalytics.com/report/contribution/neutron-group/90 -- Slawek Kaplonski Principal Software Engineer Red Hat

3 2

[cyborg] PTG Summary and Wallaby Release Schedule
by yumeng bao 03 Nov '20

03 Nov '20

Hi all, Thank you all for the PTG participation! All our agreements and decisions are marked in the etherpad[1] with AGREED word. And I have proposed the Wallaby_Release_Schedule[2] on Cyborg wiki page to track features with milestone. [1]https://etherpad.opendev.org/p/cyborg-wallaby-goals [2]https://wiki.openstack.org/wiki/Cyborg/Wallaby_Release_Schedule Regards, Yumeng

1 0

[neutron] bug deputy report for week of 2020-10-26
by Bence Romsics 03 Nov '20

03 Nov '20

Hi Team, We had the following new bugs on the week of the PTG. Sorry for sending the report a bit late. Critical: * https://bugs.launchpad.net/neutron/+bug/1901534 Neutron fail to create networks because of MTU value changes proposed: https://review.opendev.org/#/q/topic:bug/1901534 Medium: * https://bugs.launchpad.net/neutron/+bug/1901527 Agent API gets broken if OVN DB is upgraded to the one introducing chassis_private change series proposed: https://review.opendev.org/#/q/topic:bug/1901527 * https://bugs.launchpad.net/neutron/+bug/1901992 Original default route in DVR SNAT namespace is not recovered after creating and deleting /0 extraroute The first fix proposed won't work since it would have been an API change. No fix proposed for the re-stated problem (bug comments #4 and #7) at the moment. * https://bugs.launchpad.net/neutron/+bug/1902512 neutron-ovn-tripleo-ci-centos-8-containers-multinode fails on private networ creation (mtu size) unassigned Low: * https://bugs.launchpad.net/neutron/+bug/1901936 [OVN Octavia Provider] OVN provider loadbalancer failover should fail as unsupported fix proposed: https://review.opendev.org/760209 Incomplete: * https://bugs.launchpad.net/neutron/+bug/1902211 Router State standby on all l3 agent when create Looks vpnaas related. The reporter is asking for help in debugging a lock that's never released. Not reproducible at will at the moment. * https://bugs.launchpad.net/neutron/+bug/1901707 race condition on port binding vs instance being resumed for live-migrations Broken out of https://bugs.launchpad.net/neutron/+bug/1815989. * https://bugs.launchpad.net/neutron/+bug/1463631 60_nova/resources.sh:106:ping_check_public fails intermittently Sporadic re-appearance of an ages old bug, with hardly enough information to reproduce at will. Cheers, Bence

1 0

[tripleo] please avoid creating ansible playbooks that call modules for trivial tasks
by Bogdan Dobrelya 02 Nov '20

02 Nov '20

Hi, In today TripleO client development we invoke ansible playbooks instead of Mistral workflows. I believe this may only be justified for non-trivial things. Otherwise please just import python modules into the client command actions without introducing additional moving parts in the form of ansible modules (that import the same python modules as well). Ansible brings declarative DSL that sometimes can better and simpler and faster describes things to do and shape inputs data, than if the same would be done in python, in the client. What should be the trivial thing and non-trivial then? Well, I think that example [0] illustrates such a trivial thing. A new playbook [1] would only install a few packages then "returns" back into python and calls a new ansible module [2]. And that new module only wraps [3] the same code being removed from the client: from tripleo_common.image import build ... manager = build.ImageBuildManager(<args>) I see no point of adding a new playbook and a new module for that trivial example. Those 4 packages could (and should) be as well installed from the client caller code without any ansible involved in the middle IMO. [0] https://review.opendev.org/#/c/759954/ [1] https://review.opendev.org/#/c/759953/3/tripleo_ansible/playbooks/cli-overc… [2] https://review.opendev.org/#/c/759953/3/tripleo_ansible/ansible_plugins/mod… -- Best regards, Bogdan Dobrelya, Irc #bogdando

6 9

[charms] OpenStack Charms 20.10 release is now available
by Peter Matulis 02 Nov '20

02 Nov '20

The 20.10 release of the OpenStack Charms is now available. This release brings several new features to the existing OpenStack Charms deployments for Queens, Rocky, Stein, Train, Ussuri, Victoria and many stable combinations of Ubuntu + OpenStack. Please see the Release Notes for full details: https://docs.openstack.org/charm-guide/latest/2010.html == Highlights == * OpenStack Victoria OpenStack Victoria is now supported on Ubuntu 20.10 (natively) and Ubuntu 20.04 LTS (via UCA). * Gnocchi and external root CA for S3 The gnocchi charm now supports the configuration of an external root CA in the gnocchi units. This is useful when an encrypted S3 (storage backend) endpoint uses certificates that are not managed by Vault. * Arista - Supported releases The neutron-api-plugin-arista charm now supports all OpenStack releases starting with Queens. * Neutron Gateway and additional data on ports and bridges The neutron-gateway charm now marks the openvswitch ports and bridges it creates as managed by the charm. This allows for less riskier cleanup mechanisms to be implemented. * cinder charm: Allow specifying the default volume type The cinder charm can now set the default volume type. This is to support scenarios where multiple storage backends are being used with Cinder. * Action based migration from Neutron ML2+OVS to ML2+OVN New actions have been introduced to enable migration of a Neutron ML2+OVS based deployment to ML2+OVN. * New charm: ceph-iscsi The ceph-iscsi charm has been promoted to supported status. This charm implements the Ceph iSCSI gateway, which provides iSCSI targets backed by a Ceph cluster. == OpenStack Charms team == The OpenStack Charms team can be contacted on the #openstack-charms IRC channel on Freenode. == Thank you == Lots of thanks to the below 30 charm contributors who squashed 45 bugs, enabled support for a new release of OpenStack, improved documentation, and added exciting new functionality! Alex Kavanagh Andrea Ieri Andreas Jaeger Aurelien Lourot Chris MacNaughton Corey Bryant Dan Ardelean David Ames Dmitrii Shcherbakov Edin Sarajlic Edward Hope-Morley Felipe Reyes Frode Nordahl Gabriel Adrian Hemanth Nakkina James Page Jorge Niedbalski Liam Young Martin Kalcok Martin Kopec Marton Kiss Michael Quiniola Nobuto Murata Pedro Guimaraes Peter Matulis Ponnuvel Palaniyappan Rodrigo Barbieri Xav Paice camille.rodriguez zhhuabj -- OpenStack Charms Team

1 0

Re: [keystone][policy] user read-only role not working
by melanie witt 02 Nov '20

02 Nov '20

Adding back the mailing list +openstack-discuss@ On 11/1/20 23:15, its-openstack(a)zohocorp.com wrote: > Dear Openstack, > > we are implementing this reader role through kolla-ansible. Need > help in understanding the policy file for adding custom role both in > nova and keystone. You can learn how to use the policy file directly by reading the docs I linked earlier: * https://docs.openstack.org/security-guide/identity/policies.html * https://docs.openstack.org/oslo.policy/train/admin/policy-json-file.html And then the APIs you can control access to in nova are shown in this sample file: * https://docs.openstack.org/nova/train/configuration/sample-policy.html APIs in keystone are shown in this sample file: * https://docs.openstack.org/keystone/train/configuration/samples/policy-yaml… I'm afraid I don't know anything about how to adjust the policy file through kolla-ansible though. Cheers, -melanie > ---- On Fri, 02 Oct 2020 02:12:39 +0530 *melanie witt > <melwittt(a)gmail.com>* wrote ---- > > On 9/25/20 07:25, Ben Nemec wrote: > > I don't believe that the reader role was respected by most > projects in > > Train. Moving every project to support it is still a work in > progress. > > This is true and for nova, we have added support for the reader role > beginning in the Ussuri release as part of this spec work: > > https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/p… > <https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/p…> > > > Documentation: > > https://docs.openstack.org/nova/latest/configuration/policy-concepts.html > <https://docs.openstack.org/nova/latest/configuration/policy-concepts.html> > > > To accomplish a read-only user in the Train release for nova, you can > DIY to a limited extent by creating custom roles and adjusting your > policy.json file [1][2] accordingly. There are separate policies for > GET/POST/PUT/DELETE in many cases so if you were to create a role > ReadWriteUser you could specify that for POST/PUT/DELETE APIs and > create > another role ReadOnlyUser and specify that for GET APIs. > > Hope this helps, > -melanie > > [1] > https://docs.openstack.org/nova/train/configuration/sample-policy.html > <https://docs.openstack.org/nova/train/configuration/sample-policy.html> > > [2] https://docs.openstack.org/security-guide/identity/policies.html > <https://docs.openstack.org/security-guide/identity/policies.html> > > > On 9/24/20 11:58 PM, its-openstack(a)zohocorp.com > <mailto:its-openstack@zohocorp.com> wrote: > >> Dear Openstack, > >> > >> We have deployed openstack train branch. > >> > >> This mail is in regards to the default role in openstack. we are > >> trying to create a read-only user i.e, the said user can only > view in > >> the web portal(horizon)/using cli commands. > >> the user cannot create an instance or delete an instance , the same > >> with any resource. > >> > >> we created a user in a project test with reader role, but in > >> horizon/cli able to create and delete instance and similar to other > >> access also > >> if you so kindly help us fix this issue would be grateful. > >> > >> the commands used for creation > >> > >> > >> > >> $ openstack user create --domain default --password-prompt > >> test-reader(a)test.com <mailto:test-reader@test.com> > <mailto:test-reader@test.com <mailto:test-reader@test.com>> > >> $ openstack role add --project test --user test-reader(a)test.com > <mailto:test-reader@test.com> > >> <mailto:gowtham.sankar@zohocorp.com > <mailto:gowtham.sankar@zohocorp.com>> reader > >> > >> > >> > >> Thanks and Regards > >> sysadmin > >> > >> > >> > >> > >> > > > > >

1 0

[tripleo] stable/victoria for tripleo repos
by Marios Andreou 02 Nov '20

02 Nov '20

Hello all, I got some pings about tripleo stable/victoria so here is an update. The tripleo-ci team is busy preparing and we are almost ready. Basically we've blocked branching until we have adequate CI coverage. This means check & gate jobs but also promotion jobs so that check & gate aren't running with stale content. tl;dr We are aiming to branch victoria tripleoclient & common today and everything else next week - with more details below but please note: PLEASE AVOID MERGING to the stable/victoria tripleoclient/common that will appear after [4] until we are confident that our ci coverage is working well. Status: Most of our 'core' (standalone/scenarios/undercloud etc) check/gate jobs match on branch like [1] so we expect those to run OK once there is a stable/victoria available. There is ongoing work for branchful, upgrade, 3rd party and ovb jobs but we will not block branching on those. The victoria integration pipeline [2] (which produces current-tripleo) is in good shape. It was blocked on a nit in the component pipeline but it is resolved with [3]. Once we have a victoria promotion then we should be all clear to branch stable/victoria. Actions: A). today: make stable/victoria for python-tripleoclient & tripleo-common. The change at [4] will do this once merged. We will use this to see our check/gate jobs run against victoria & fix issues. B). next week: once we are happy with A). AND we've had a promotion, we release and make stable/victoria for everything else. Hopefully B). will happen end of next week but that depends on how well the above goes (plus inevitable delays cos PTG), I'll repeat this cos it's important: PLEASE AVOID MERGING to stable/victoria tripleoclient/common for now until we are confident that our jobs are doing the right thing! As always grateful for a sanity check or any thoughts about any of the above - I'm pretty new to this (weshay has been a great help so far thank you weshay++), thanks, marios [1] https://opendev.org/openstack/tripleo-ci/src/commit/d5514028452f9d427949f5a… [2] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integ… [3] https://review.rdoproject.org/r/#/c/30654/ [4] https://review.opendev.org/#/c/759449/

1 2

[nova] python3.9 support
by Balázs Gibizer 02 Nov '20

02 Nov '20

Hi, I run unit and functional tests (where applicable) for nova, python-novaclient, os-vif, placement, osc-placement, os-traits, os-resource-classes with python 3.9 and everything is green. I've proposed some tox.ini targets where it was needed [1][2][3] Cheers, gibi [1]https://review.opendev.org/#/c/760884/ [2]https://review.opendev.org/#/c/760890/ [3]https://review.opendev.org/#/c/760912/

2 1

Python 3.9 is here in Debian Sid
by Thomas Goirand 02 Nov '20

02 Nov '20

Hi there! Matthias Klose decided that uploading Python 3.9 on the 14th of this month (yes, the same day of the OpenStack release) would be fun. I am now really having fun ... fixing bugs! :) What's not so funny for me, is that some key packages like Greenlet or lxml aren't built yet for Python 3.9, even though they should. As a consequence, I just need to wait to be able to upload to Unstable. So this will delay my usual announcement for the release GA in Debian (even though the unofficial Victoria repo for Buster is ready and I could spawn VMs on Buster + Victoria). Anyways, all of this will settle slowly. Though Python 3.9 is here to stay, and to soon reach Ubuntu as well (it's planned for the next release). It'd be nice if projects were starting to investigate gating on Python 3.9 as early as possible. As much as I can tell, there's not so many issues to fix. The first one I fixed is here: https://review.opendev.org/758237 but I don't expect much more, and hopefully, I'll be able to propose more patch as I find the issues. Must we wait until Python 3.9 reaches Ubuntu to enable gating in the OpenStack projects? FYI, I do expect Victoria to run over Python 3.9 in Bullseye. Please don't reply with the usual "Victoria doesn't support it", because that's irrelevant. I'm fine with backporting patches, as long as they also exist in Wallaby. Cheers, Thomas Goirand (zigo)

7 9

[magnum] hyperkube deprecation situation
by Ionut Biru 02 Nov '20

02 Nov '20

Hi guys, I was hoping that in Victoria there will be a decision regarding this topic. Currently, if I use vanilla magnum, I'm stuck on 1.18.6 being the latest version, even so 1.18.10 was released. We are kinda stuck on a version that maybe in the near future is pruned to security issues. How do other operators are using magnum (vanilla) and keeping their kubernetes up to date? Are operators using a forked version of https://review.opendev.org/#/c/752254/ ? -- Ionut Biru - https://fleio.com

2 2

[openstack-ansible] Wallaby PTG results
by Dmitriy Rabotyagov 02 Nov '20

02 Nov '20

Hi, Bellow you may find summarization of descisions that were taken during PTG discussion. Victoria retrospective ---------------------------- 1. We're pretty far from dropping resource creation code out of the tempest, as we should respect TripleO needs here. So leaving as is during this cycle but adding option to be possible to omit resource creation. 2. We decided to postpone prometheus integration (as well as libvirtd exporter) as we don't have enough resource to implement and moreover support it in the future. 3. Decided to leave ELK stack in ops repo as for now. 4. Adjusting bootstrap-ansible to pick up gerrit patches idea was also postponed as has super low prio and we have a lot of more useful things on our hands 5. We didn't have much progress in tempest coverage of existing roles, we should proceed with it during W. Same is applicable to the roles overall speedup topic. Descisions taken for Wallaby -------------------------------------- 1. Remove nspawn from docs, and set it to unmaintained state in V. Remove code in W. 2. Document how upgrade paths for distro installs are difficult, and may/may not align with OSA releases 3. Check through the logs what roles are we covering with tempest, and what not 4. extract deprecation messages from journal files to the separate log 5. Continue working on speeding up OSA runtime: - Fight with skipped tasks (ie by moving them to separate files that would be included) - most valid for systemd service, systemd_networkd and python_venv_build roles - Try to split up variables by group_vars - Try to use include instead of imports again 6. Set TRANSFORM_INVALID_GROUP_CHARS to ignore in V and replace invalid chars in groups later. For octavia we can temporary add 2 groups (old style and new style) for migration process. 7. Try to speedup zuul required projects clone process and talk to zuul folks 8. Finally do Integrated repo deploy for Zun and sync deployment process with service docs 9. Installation of ara for deployer needs - ARA deploy on localhost, configured by variables - Option for remote ARA - Option for UI server locally 10. Create Root CA on deploy host and overall SSL approach refactoring. Futher discussion in SPEC: https://review.opendev.org/#/c/758805/ Original etherpad for the reference https://etherpad.opendev.org/p/osa-certificates-refactor 11. For ansible 2.11 (which requires py3.8 on deploy host) we can use pyenv for ansible venv to workaround this, and leave hosts themselves on their native python 12. Move neutron-server to standalone group in env.d 13. Add Senlin tempest test - Add plugins (source only) to os_tempest https://review.opendev.org/754044 - Upstream senlin endpoints fix https://review.opendev.org/74987 - Enable senlin tempest plugin in integrated repo https://review.opendev.org/#/c/754105/ - Test patch for senlin/tempest https://review.opendev.org/754045 14 Document in os_nova defaults way to manage ratios via api (ie set cpu_allocation_ratio to 0.0) (regarding https://review.opendev.org/#/c/758029/) 15. Add support for zookeeper deployment for services coordination (low priority) 16. Check where we don't use uwsgi role and see if we can use it there now (like designate, neutron-api) (low priority) -- Kind Regards, Dmitriy Rabotyagov

1 0

[undercloud][train]Fatal error during train undercloud deployment
by dangerzone ar 02 Nov '20

02 Nov '20

Hi all, I keep getting the same fatal error below during undercloud deployment. My environment is VMware VM Centos7 undercloud (openstack train) and refers to the official openstack guideline. TASK [tripleo_container_manage : Print failing containers] ********************* Monday 02 November 2020 19:55:07 +0800 (0:00:00.182) 0:17:50.677 ******* fatal: [undercloud]: FAILED! => changed=false msg: 'Container(s) with bad ExitCode: [u''container-puppet-ironic'', u''container-puppet-zaqar''], check logs in /var/log/containers/stdouts/' I even tried with clean different VM and all give me same error. Appreciate some help please. Thank you.

2 2

[Cloudkitty][ptg] CloudKitty Wallaby PTG outcome
by Rafael Weingärtner 02 Nov '20

02 Nov '20

Hey people, Thanks to everyone who attended the CloudKitty PTG sessions, we covered a lot during the meeting. The Etherpad is available at [1]. In Etherpad [1], I added the notation "TODO(XXX):" in the topics we discussed, that require effort from someone. The work can be either writing a use case story, developing extensions/features, or creating test cases. Therefore, if you are interested in working with that item, do not hesitate and put your name there. Anyone in the community is welcome. [1] https://etherpad.opendev.org/p/cloudkitty-ptg-wallaby -- Rafael Weingärtner

1 0

[nova] PTG
by Balázs Gibizer 02 Nov '20

02 Nov '20

Hi, Thank you all for the PTG participation! All our agreements and decisions are marked in the etherpad[1] with AGREED word. To avoid loosing this information I made a dump of the etherpad on Friday at the end of the PTG and attached to this mail Cheers, gibi [1] https://etherpad.opendev.org/p/nova-wallaby-ptg

1 0

[Glance] Wallaby PTG summary
by Abhishek Kekane 01 Nov '20

01 Nov '20

Hi All, We had our second virtual PTG last week from 26th October to 30th October 2020. Thanks to everyone who joined the virtual PTG sessions. Using bluejeans app we had lots of discussion around different topics for glance, glance + cinder and glance + tempest. I have created etherpad [1] with Notes from session and which also includes the recordings of each discussion. The same etherpad includes milestone wise priorities at the bottom. Below topics we discussed during PTG: 1. Victoria retrospection 2. Image encryption (progress) 3. Remove single store configuration from glance_store 4. Cache-API 5. Duplicate downloads 6. Extend issue when using nfs cinder backend (with cinder team) 7. Multi-format images 8. Cluster awareness 9. Improve performance of ceph/rbd store of glance (multi-threaded rbd driver) 10. Task API's for general users 11. Tempest enhancement/bridge gap between glance and tempest 12. Code cleanup, enhancements 13. Wallaby priorities Kindly let me know if you have any questions about the same. [1] https://etherpad.opendev.org/p/glance-wallaby-ptg Thank you, Abhishek

1 0

Fwd: [MURANO] DevStack Murano "yaql_function" Error
by İzzettin Erdem 01 Nov '20

01 Nov '20

Thanks Andy. Regards. Forwarded Conversation Subject: [MURANO] DevStack Murano "yaql_function" Error ------------------------ Gönderen: Andy Botting <andy(a)andybotting.com> Date: 30 Nis 2020 Per, 14:13 To: İzzettin Erdem <root.mch(a)gmail.com> Cc: Rong Zhu <aaronzhu1121(a)gmail.com>, < openstack-discuss(a)lists.openstack.org> Hi İzzettin, Thanks for your interest. Complete log link is at below. It is happening > when i try to deploy or update an environment. > > http://paste.openstack.org/show/792444/ > I've encountered the error AttributeError: 'method' object has no attribute '__yaql_function__' in our environment recently. I believe it started after we upgraded to the Stein release. I haven't tracked it down yet, but calls to the YAQL function: std:Project.getEnvironmentOwner() in a Murano app seem to generate it. It's good to know that at least someone else has reproduced this. cheers, Andy ---------- Gönderen: İzzettin Erdem <root.mch(a)gmail.com> Date: 2 May 2020 Cmt, 16:08 To: Andy Botting <andy(a)andybotting.com> Hi Andy, I solved this problem, it is happening when you try to create or update an environment on existing network. If you change the network with "create new" parameter, it stuck on "creating -or updating- environment" state and after that throws connection error. So i searched Murano network errors and i found this: https://docs.oracle.com/cd/E73172_01/E73173/html/issue-21976631.html. I applied these steps and i created successfully an environment just only with "create new" network parameter. After all these, i realized my devstack environment has no internet connection and because of this murano-agent could not install. So i tried to install Murano on my OSA stable/train test environment and i came with success until last step. Murano-agent installed but this time agent could not connect to rabbitmq instance. I discussed this with OSA IRC channel and i think i have to struggle with this for a long time. Apologize for this long mail but i wanted to inform you. Thanks. Regards. ---------- Gönderen: Andy Botting <andy(a)andybotting.com> Date: 3 May 2020 Paz, 14:24 To: İzzettin Erdem <root.mch(a)gmail.com> Hi İzzettin, Glad you got it sorted. I think you've probably just worked around the issue. Using your new method, you're probably just avoiding the std:Project.getEnvironmentOwner YAQL call now. I've also worked around it in our environment, but I'd like to solve it properly. When I get some time to look into it properly, I'll let you know. cheers, Andy

3 3

Weird iptables problem
by Florian Rommel 01 Nov '20

01 Nov '20

Hi, so we deployed a test cluster with 8 total nodes. Openstack ussuri, Ubuntu 20.04 , opevswitch, octavia. Everything works more or less as it should , however when deploying a loadbalancer (which works perfectly), instances on the same node as the amphora deployment loose all access via public up after open switch logs a message : modified security group “long uuid”, which is a sec group belonging to octavia. What effect does that have with the rest? Questions: Should ufw be on and started? How can we troubleshoot this more in depth? Why does the lb work perfectly whole instance with floating up loos all access from the internet but the lb works fine passing thru the web services. One more thing. Does anyone have a recommended sysctl.conf Settings list for Ubuntu/ openstack ussuri? Thanks already and have a nice weekend, //f

1 0

[tc][all][ Wallaby Virtual PTG Summary
by Ghanshyam Mann 31 Oct '20

31 Oct '20

Hello Everyone, I am summarizing the TC PTG discussion and action items. Wallaby Leaderless assignment --------------------------------------- Four projects are still pending for leadership assignment. We discussed the next step. If you would like to help any of these or using in your cloud/distro, this is the right time to step up: 1. Karbor - Action item: diablo_rojo will start the retirement process. 2. Qinling - Action item: gmann to start the retirement process. 3. Searchlight -Action item: gmann to start the retirement process. 4. Placement - Action item: gibi will update TC about the final discussion and accordingly mnaser to propose the patch in governance. Victoria Retrospective: ----------------------------- There are a couple of things TC finished in last cycle, few of them are: * Reduced size of TC to speed up the things. * Updated policy to add projects faster. * Outlined Distributed Project Leadership Model as an alternate leadership solution of PTL. Oslo will be the first project to try this new model. * Normalized retirement processes. All the retired repos cleanup is finished. * Tag cleanups & additions This is not complete yet but started with tc-approved release tag in the Victoria cycle. * Merged UC with TC. This is one of the key things finished and the best strategy to make users-developers work more closely. TC Meeting time: ----------------- TC is going to try (re-try) the weekly meeting every Thursday 15:00 UTC. * Action Items: mnaser: propose patch. TC policy regarding OpenStack client: ----------------------------------------------- Forum Discussion: https://etherpad.opendev.org/p/vSummit2020_FeatureGap This is one of the important items and still no good progress on this. From TC perspective, we talked about how to ask or motivate projects to migrate to OSC. TC needs to be very clear on whether it's a strict policy or just guidelines. After long discussion on this we are going with the below strategy: These strategies will be documented as TC resolution. * All user docs should have `openstack` client commands. * Focus on user docs first and then ops docs later * All ci-tooling should have `openstack` client usage * Use the process to find what is missing so we can prioritize it. * Action items: - diablo_rojo to work on initial resolution proposal[1] Finalize the Wallaby cycle goal selection: --------------------------------------------------- There are two proposals for the Wallaby community-wide goals. First is 'oslo.rootwrap to oslo.privsep' which is already being discussed in the last cycle also and all set to be selected[2]. 2nd proposal came up during PTG itself. During policy-popup PTG[3], it came up that deprecating the JSON formatted of policy file will be a good advance step before the projects move to new RBAC policies. This will help operators to smoothly migrate to new policies. This does not involve much work and ok to select as 2nd goal for Wallaby cycle, TC agreed to have below goals for Wallaby cycle: * Migrate from oslo.rootwrap to oslo.privsep [2] * Migrate Default Policy Format from JSON to YAML [4] *Action: - TC to merge the patches which selects both goals for Wallaby cycle. TC tag cleanup: ------------------- In the Victoria cycle, we started the process to audit all the TC tags and start cleanup those. We removed the 'tc:approved-release' tag in the Victoria cycle[5]. In this PTG we discussed two more tags. 1. assert:supports-zero-downtime-upgrade: Currently, there is no project who has this tag and also no testing framework available. Testing for zero downtime is not so easy in upstream testing. We decided to remove this tag as it is advertising something we aren't doing. If anyone interested to spend time on this in the future then we can add it after projects start testing it and document it. 2. assert:supports-api-interoperability: This tag is whether project API is interoperable also or not and this is important from the interop trademark program also. We only have Nova having this tag. Our goal is to promote more projects to apply for this tag. During the discussion, we found that we need to clarify this tag more clearly. For example, this tag is not about implementing the microversion but any versioning schema which provides the feature (API changes) discoverability. And also it is about how we change the API not how our APIs are currently. As long as services have some versioning mechanism to discover the changes and follow the API SIG guidelines for interoperability, and test it via branchless testing way, that service is applicable to apply for this tag. TC will document this tag in a more clear way and encourage each project to start working on applying this tag. * Action: - Graham to put up a change to update the wording to allow for discovery mechanisms[6] - Should include the release in which they started to assert this tag. - also documents that it's about following the guidelines for future changes not existing one. - gmann - Should socialize this out to the projects after we get the documentation improvements landed. - gmann k8s Community Bridging : -------------------------------- This is one of the exciting discussion for everyone. We hosted this cross-community meeting with Kubernetes steering committee teams. Bob and dims from the Kubernetes steering committee joined us in this discussion. It was started with a quick introduction from both teams and then started the discussion on the below topics [11] * Sharing governance models: k8s governance hierarchy is LF->CNCF->k8s steering commitee->various SIG/working group. k8s Steering the committee consists of 7 elected members and doesn't actually influence the direction of the code instead leave it to SIG and arch committee. There is no chair in this committee and host biweekly private as well as public meetings. Each SIG (repos) team has approver and reviewer roles where reviewer review the code and approver are responsible for merging the code. Naser explained the OpenStack governance model. * What are your 3 biggest issues that are barriers to growth/sustainability as a community? as a project? - Going up the contribution ladder is hard. - Stale/emeritus membership in reviewer/approver level groups. - Mono repo makes it so that each SIG involved needs to review which can slow things down. * Area/Domain mapping vs service/repo centered teams - mono repo forces people to work together and it's a little more diverse who is reviewing - mono repo is bad because sometimes things need to be in multiple places like documentation - mono repo struggles with libraries especially on testing the external one. * Related Ecosystem projects comparisons The SIG lead has to sign off and accept that they are willing to take ownership + handle maintenance + releasing etc. It is general consensus that we try to distribute much work as possible to the subgroup and keep it out of k/k. There is general consensus across k8s leadership that work should be delegated out to subgroups. For API interoperability challenge in a distributed model, k8s has conformance tests which exercise the API performance and vendors try and upload conformance results every release or dot release. * Common FOSS topics release and CI part was discussed. and also on how COVID things are impacting community health. k8s community almost lost their independents or part-time contributors. also, the k8s community is doing 3 releases per year compared to 4. * Staying connected going forward To stay connected it will be a good idea if we extend an invite for K8s to join PTG. Upstream Investment Opportunities: ----------------------------------------------- We have three upstream opportunities defined for 2020[1] but there is no help on any of these, even in previous (2018, 2019) upstream opportunities also. We started the discussion if we need to continue this for 2021 years also or just stop defining it and decide the area to help when we have someone interested. Before deciding anything on this mnaser will discuss this with the board of directors and get their opinion. * Action Items: - mnaser to find ways to engage with members and talk with them. - mnaser to talk to board about infrastructure - mnaser to talk to board about platinum members contribution levels + what are they doing to drive engagement. Pop Up Team Check In: ----------------------------- Currently, we have two active popup teams 1. policy 2. Encryption. TC checked the progress on both teams. The policy team is very active and finished some work in Voctoria cycle (cyborg finished it and Barbican started) and also hosted forum[8], PTG sessions[9], and discussed Wallaby development plan. This team host biweekly meeting to discuss and review the progress. The encryption team is also active. Josephine explained the progress on this. Glance spec is merged. Both teams will continue in Wallaby cycle also. Completing the Gerrit breach audit : ---------------------------------------------- There are still 19 teams pending to finish this audit. Those are listed in etherpad - https://etherpad.opendev.org/p/code-audit-gerrit-breach-tracker We encourage all those pending teams to finish the audit, TC members will start following with those projects every week. * Action: - TC to follow up every week on progress Other topics: ----------------- We ran out of time and all the pending (below) topics will be discussed in TC regular meetings. TC will skip this month's meeting[10] but will have weekly meetings on 12th Nov onwards. *Better way to test defined testing runtime *Monitoring in OpenStack: Ceilometer + Telemetry + Gnocchi state *Stable core team maintainer and process to recruit the new members and adding core team members in stable core list. Detailed discussions are in this Etherpad: https://etherpad.opendev.org/p/tc-wallaby-ptg [1] https://review.opendev.org/759904 [2] https://review.opendev.org/#/c/755590/ [3] https://etherpad.opendev.org/p/policy-popup-wallaby-ptg [4] https://review.opendev.org/#/c/759881/ [5] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/017786.… [6] https://review.opendev.org/760562 [7] https://governance.openstack.org/tc/reference/upstream-investment-opportuni… [8] https://etherpad.opendev.org/p/consistent-and-secure-default-policies-walla… [9] https://etherpad.opendev.org/p/policy-popup-wallaby-ptg [10] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018439.… [11] https://etherpad.opendev.org/p/kubernetes-cross-community-topics -gmann

1 0

devstack installation error - [qa-octavia]
by Ratnopam Chakrabarti 31 Oct '20

31 Oct '20

I tried installing devstack in ubuntu 20.04 with no luck. Are the instructions > at https://docs.openstack.org/devstack/latest/ and > https://docs.openstack.org/devstack/latest/guides/devstack-with-lbaas-v2.ht… > still valid? Could anyone please confirm?

2 3

Weird iptables problem
by frommel＠gmail.com 31 Oct '20

31 Oct '20

Hi, so we deployed a test cluster with 8 total nodes. Openstack ussuri, Ubuntu 20.04 , opevswitch, octavia. Everything works more or less as it should , however when deploying a loadbalancer (which works perfectly), instances on the same node as the amphora deployment loose all access via public up after open switch logs a message : modified security group “long uuid”, which is a sec group belonging to octavia. What effect does that have with the rest? Questions: Should ufw be on and started? How can we troubleshoot this more in depth? Why does the lb work perfectly whole instance with floating up loos all access from the internet but the lb works fine passing thru the web services. One more thing. Does anyone have a recommended sysctl.conf Settings list for Ubuntu/ openstack ussuri? Thanks already and have a nice weekend, //f

1 0

[magnum] Keystone trust credentials failing authentication (after previously working)
by Paul Browne 30 Oct '20

30 Oct '20

Hello Magnum users and devs, I was wondering if any may be able to provide some clues on a mystifying problem we've been seeing with some Magnum-deployed k8s clusters, of older deployment date (v1.17, deployed ~6 months ago) and Keystone trusts used to manage OpenStack cloud resources. I've attached a copy of the Magnum template for the k8s cluster to give an idea of its initial environment. The issue we've been seeing is that the Keystone trust generated at Magnum cluster creation time seems no longer be usable to successfully authenticate to the OpenStack APIs, and so operations such as creation of Manila shares used as k8s Persistent Volume Claims then get stuck in Pending state, forever. The strange thing is that this doesn't happen for newly created Magnum clusters of the same template (in attached file, I use trust credentials for a newly deployed and old cluster; OpenStack API calls for new credentials succeed, whereas old cluster trust fails). Unfortunately following debug-level Keystone logging for rejected trust auth attempts hasn't led me much further to understanding the root issue; the Magnum generated trust exists and has no expiry, but it still seems to be rejected. Keystone debug logging doesn't seem to give much indication as to *why* it is being rejected or what policy violations may be involved, but perhaps there's details hidden in there I'm not seeing yet. Does anyone have any clues as to why this may be happening, or any advice on how we may be able to replace or refresh the trust that the Magnum-created k8s cluster is expecting to use in calling OpenStack APIs? Many thanks, Paul -- ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk Tel: 0044-1223-746548 *******************

1 0

[kolla] Wallaby PTG summary
by Mark Goddard 30 Oct '20

30 Oct '20

Hi, Thanks to everyone who joined the virtual PTG sessions this week. We had some great discussions covering many different topics. I will try to provide a summary here. # Victoria cycle retrospective The Ubuntu Focal migration went smoothly. Core reviewer bandwidth was hurt by the pandemic, but we did pick up towards the end of the cycle and landed a good number of features. As usual, lots of CI fires to extinguish. # Review Victoria PTG Actions We looked back at the actions from the last PTG. Quite a few are still incomplete - we should do a better job of reviewing these. We agreed that the docs-related actions are still the most important. Any help with improving our documentation is appreciated. # General topics ## Ability to run CI jobs locally (without Zuul, but possibly with Ansible) This depends on a Zuul feature called Zuul Runner [1], which should be ready soon. yoctozepto signed up to test it out and document how to use it with Kolla. ## Future of Kolla Klub and Kall The Kolla Klub started well, with lots of involvement, and some good sessions. Over time, attendance has fallen, and the agenda still seems to be largely driven by me. Keeping up productive sessions every two weeks is starting to feel like a bit of a chore, and I think we need to change things up or stop. We agreed to try rebooting at a time that better suits contributors in APAC, possibly recycling some earlier material. There was a proposal at the summit to have monthly ops meetups, and maybe we would do well to link these related communities. We had a few ideas for how to improve the productivity of the more development oriented Kolla Kall, including multiple smaller groups, pair/mob programming, and fewer longer sessions. ## Keep or drop lower-constraints jobs? We didn't find many reasons to keep the lower-constraints jobs, and they do seem to need a bit of maintenance. We voted 2 in favour to remove them, with everyone else not having strong feelings. ## How can we integrate better with our user & contributor base outside of EU/US timezones? Kolla has a large user base outside of EU/US timezones, but the core team and surrounding community is not always that effective at involving these users. We agreed to try to improve the situation by reaching out to known community members in other regions, and offering some limited mentorship. # Kolla ## Infra images This feature has been in progress for a few cycles, but is proving difficult to land. The main blocker now is in CI. We need to split our image build CI jobs into several pipelines. They need to share a local docker registry. This pattern should open up a few other interesting features. ## New Docker Hub limits Docker hub is changing its policy for anonymous and free users. This affects retention of unused images, and limits the number of pulls in a 6 hour period (100 for anonymous users, limited by IP, 200 for logged in free users). This is likely to affect users running without a local registry. Fl1nt agreed to add some information to our documentation to help users. This is likely also to impact our CI jobs, which need to pull images. There is an application process to be listed as an open source project, which we need to ask the opendev infra team to apply for. # Kolla Ansible ## More fine grained skipping of tasks, e.g. allow to skip service registration priteau proposed making it easier to skip certain tasks during deployment, such as bootstrapping or service registration. There are two obvious ways to do this - a new 'action' (e.g. deploy-without-register), or add tags to the relevant tasks, which may be skipped. A third approach is to combine two commands - genconfig & deploy-containers. This is related to work currently going to allow configuration to be generated separately from container deployment. In the end, we considered adding tags, as well as the two command approach. ## Deprecate reconfigure command? The reconfigure command is now essentially the same as deploy. This is a source of confusion for users, so we agreed to deprecate reconfigure. We will keep reconfigure as an alias for deploy at the CLI level. ## Modernise the old skool Swift role The Swift role is the last (except Bifrost, which is generally weird), to use the old format, prior to the better-reconfigure blueprint. This has some limitations, so we should bring it into line. Nothing controversial here, just needs someone to pick it up. ## DNS-based endpoint naming rafaelweingartne has proposed a mini spec [2] for DNS-based endpoint naming. I think we now have a reasonable idea of his solution (which is to use Consul for service discovery, and avoid HAProxy for internal API communication), however the hard part is defining the problem that is being solved. ## Support to identity federation (OpenID Connect) configurations in Keystone and Horizon via Kolla-ansible There is a patch [3] to support this which has been open for some time. It requires some fairly specific knowledge to review, and the challenge will be in making core reviewers confident that it works. We agreed that CI testing and/or documentation of a simple development environment would help. ## podman Podman keeps coming up as a replacement for Docker. There are several issues here, including support on Debian/Ubuntu - we do not want to maintain support for two container engines. We agreed that this is likely to be a large feature, and that we should get started with a proof of concept in Wallaby, possibly with a spec to tease out the details. Fl1nt may have some time early 2021 for this, but anyone is welcome to pick it up. ## Finish Masakari integration We don't have support for the host monitor, which is arguably the most useful part of Masakari. With yoctozepto as Masakari PTL, we expect to see good progress in Wallaby. ## Performance improvements We made some good progress with performance improvement in Victoria. There is always more that can be done, and the next phase will probably involve splitting roles that target both control and compute hosts. Mitogen [4] is also an interesting option. As always, changes should be based on benchmarks [5]. ## Let's Encrypt integration - container running certbot, triggers certificate distribution yoctozepto added initial support for the ACME protocol in Ussuri, and the next step is to support deployment of a certbot container, and unattended certificate rotation. There are lots of fiddly details here, including how to bootstrap it. headphoneJames offered to write up a high level description of how the pieces fit together. ## Support for HAProxy reload (e.g. for TLS) Currently any configuration changes to HAProxy require a restart to be applied. This causes connections to be dropped. We discussed adding support for hitless reloads of HAProxy [6]. ## Native fluentd logging We didn't discuss this as kevko was busy, but I thought I'd add it here anyway. The idea is to use use logging.conf to configure services for native fluentd logging. This avoids all the regexes in fluentd, but may change our document schema in Elasticsearch. Patch [7] is up for review. # Kayobe ## Support multiple environments from a single kayobe configuration This feature has been on the cards for some time, and is an important step for Kayobe. priteau has proposed a patch [8] which is a good starting point. We spent a good hour going over some of the outstanding issues, and made some good progress. We agreed on the need to keep the meaning of KAYOBE_CONFIG_PATH intact, and to allow a branch to define the default environment to use. We agreed to write a high level design document to pull out the remaining uncertainties, and aim to get it over the line in Wallaby. ## Switching Docker storage from devicemapper to overlay2 We switched the default Docker storage driver from devicemapper to overlay2 in Victoria. For existing users of devicemapper, the plan is to try to ensure they are migrated to overlay2 during CentOS 8 migration. Further, we will deprecate devicemapper on stable branches, and require users to take some action (set a variable) to allow its use. This should deter new users from using devicemapper, without completely breaking existing users. # Switch to NetworkManager? jovial proposed this discussion, as he likes the checkpoint/restore feature of NetworkManager. There are various ways to interface with NM, including nm-cli, an ansible module, templated config files + nmcli con reload. ## Strip out Grafana post configure functionality and move it to Kolla-Ansible dougszu proposed this topic, to improve the separation of concerns between Kayobe and Kolla Ansible. It may also apply to setup we do for ironic. ## Create a reference custom playbook repo We have a stackhpc repo [9] with various useful playbooks and scripts. dougszu proposed making something more official. I suggested that Ansible collections could be used to improve the user experience with custom playbooks in external repos. ## Config gen on ansible control host This is something we've talked about for a while - offline generation of configuration, and potentially validating it in a CI pipeline. jovial noted the importance of being able to redact passwords. This is mostly dependent on ongoing changes to Kolla Ansible, but could be wrapped up nicely by Kayobe. ## Hashivault integration While we have integrated with Hashicorp vault for secret retrieval, the user experience could be better. We also lack support for pushing passwords generated by kolla up to Vault. This is likely to be a Kolla Ansible feature, plus some Kayobe integration. [1] https://review.opendev.org/681277 [2] https://review.opendev.org/759706 [3] https://review.opendev.org/695432 [4] https://github.com/dw/mitogen [5] https://github.com/stackhpc/ansible-scaling [6] https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/ [7] https://review.opendev.org/755775 [8] https://review.opendev.org/734867 [9] https://github.com/stackhpc/kayobe-ops Cheers, Mark

1 0

Devstack installation failure
by Ratnopam Chakrabarti 30 Oct '20

30 Oct '20

Hi, Has anyone successfully installed devstack in last 2 weeks with stack.sh script?It was working well for me before that. I am seeing failures when I try to install on Ubuntu 18.04. Here's the trace - [Call Trace] ./stack.sh:1304:start_glance /home/stack/devstack/lib/glance:480:die [ERROR] /home/stack/devstack/lib/glance:480 g-api did not start Error on exit nova-compute: no process found neutron-dhcp-agent: no process found neutron-l3-agent: no process found neutron-metadata-agent: no process found neutron-openvswitch-agent: no process found git repo i am using to download devstack is https://git.openstack.org/openstack-dev/devstackI tried with with stable/victoria branch also, same issue. Any idea on how to solve this? Thanks

2 1

[tc] monthly meeting skip
by Mohammed Naser 30 Oct '20

30 Oct '20

Hello everyone, Our monthly meeting was scheduled for next Thursday, Nov 5th. Since the PTG took place this week, we will be skipping this month’s meeting and consider the PTG as our monthly meeting instead. Thanks, Regards, Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

[ansible][tripleo][openstack-ansible-modules] Openstack Ansible modules PTG wrap up
by Sagi Shnaidman 30 Oct '20

30 Oct '20

Openstack Ansible modules PTG wrap up 1. We continue to align new features in SDK, OSC and modules a) need to complete switch of cloud layer to use ONLY proxy layer and not reimplement the APIs b) in Ansible modules we'd better to use cloud layer when it's available and not proxy layer 2. For exposing SDK logs when running modules, we need to find a right place to print them. Either it would be one of module returns or special plugin printing logs with more verbosity. The discussion will be in the maillist. 3. We will start developing unit tests for modules and run them with ansible-test. 4. Also need to rework our current test system and make it more atomic and add coverage. 5. We decided to align all *_info modules to one standard - <resource>_info, some of them named as <resource>s_info. 6. We'll need to generate documentation for modules and place it on docs.openstack.org if it's possible. Current documentation in Ansible covers the release that is built-in in Ansible and doesn't reflect last changes. 7. Effort of moving modules to use a single OpenstackModule class will continue. 8. Transition and adding new baremetal modules will continue as well. Thanks -- Best regards Sagi Shnaidman

1 0

Re: [Openstack][cinder] dell unity iscsi faulty devices
by Gorka Eguileor 30 Oct '20

30 Oct '20

On 30/10, Ignazio Cassano wrote: > Please, se the last email where I upgraded to the last openstack nova on > queens. > [root@compute-0 nova]# rpm -qa|grep queens > centos-release-openstack-queens-1-2.el7.centos.noarch > [root@compute-0 nova]# rpm -qa|grep nova > openstack-nova-compute-17.0.13-1.el7.noarch > openstack-nova-common-17.0.13-1.el7.noarch > python-nova-17.0.13-1.el7.noarch > python2-novaclient-10.1.0-1.el7.noarch > > I sent you the logs on the update release. > I am not so skilled for reading fine logs output about this issue. > Sorry > Ignazio > Hi, I missed that email and the attachment, sorry. The logs you sent me were missing most of the connect_volume call, and only the end of the call was present, but I think it doesn't matter as I see what the problem is. The problem is that some of the nodes and sessions are duplicated. An example of a duplicated node: tcp: [3] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) tcp: [4] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) An example of that node's duplicated session: tcp: [3] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) tcp: [4] 10.102.189.156:3260,15 iqn.1992-04.com.emc:cx.ckm00200502005.b11 (non-flash) And os-brick is not prepared to handle that, because it is programmed to reuse the nodes and sesssions. So on disconnect it get's the first of each to look for the volumes provided by it. In the example of the duplicated node-session aboveit sees that it provides /dev/sdd, but that is not one of the disks that belong to the multipath that we are disconnecting, so it gets ignored. The volume we are looking for it's probably on the second session. So from this point forward (where we have duplicated node-sessions) it will not work again. I recommend you clean up that system so that you don't have duplicated nodes and sessions before trying to do a VM migration with a single volume attached. If that works, then try to attach 2 volumes on instances on the same host and see if the nodes and sessions are duplicated. Cheers, Gorka. > Il giorno ven 30 ott 2020 alle ore 09:27 Gorka Eguileor <geguileo(a)redhat.com> > ha scritto: > > > On 30/10, Ignazio Cassano wrote: > > > Hello, these are versions we are using: > > > [root@podto2-kvm02 ansible]# rpm -qa|grep queens > > > centos-release-openstack-queens-1-2.el7.centos.noarch > > > [root@podto2-kvm02 ansible]# rpm -qa|grep nova > > > openstack-nova-common-17.0.11-1.el7.noarch > > > python2-novaclient-10.1.0-1.el7.noarch > > > openstack-nova-compute-17.0.11-1.el7.noarch > > > python-nova-17.0.11-1.el7.noarch > > > > > > Cheers,Gorka. > > > > Hi, > > > > That release has the Nova bug fix, so Nova should not be calling Cinder > > to do an initialize connection on the source on the post-migration step > > anymore. > > > > I recommend comparing the connection info passed to connect_volume when > > the volume is attached on the source host and when it's disconnected on > > the post-migration step on the source host. > > > > Cheers, > > Gorka. > > > > > > > > > > > > > [1]: https://bugs.launchpad.net/nova/+bug/1754716 > > > > [2]: > > > > > > https://github.com/openstack/nova/commit/013f421bca4067bd430a9fac1e3b290cf1… > > > > > > > > > > > > > > > > > > > Il giorno gio 29 ott 2020 alle ore 09:12 Gorka Eguileor < > > > > geguileo(a)redhat.com> > > > > > ha scritto: > > > > > > > > > > > On 28/10, Ignazio Cassano wrote: > > > > > > > Hello Gorka, I would like to know if with unity iscsi driver, I > > must > > > > > > > configure iscsi initiator on both compute and controller nodes. > > > > > > > At this time I installed and cinfigured iscsi initiator only on > > > > compute > > > > > > > nodes and I got a lot of faulty devices when volumes ate > > detached. > > > > > > > Thanks > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > Both compute and controller nodes are in the data path. Computes > > when > > > > > > instances use the volumes, and controllers when we create volume > > from > > > > > > images, do generic volume migrations, create or restore backups, > > etc. > > > > > > > > > > > > Unless your deployment isn't doing any of the Cinder operations > > that > > > > > > involve the data plane, you'll have to configure iSCSI on the > > > > controller > > > > > > as well. > > > > > > > > > > > > Having said that, whether you configure the iSCSI initiator or not > > on > > > > > > the controller will have no effect on the paths used by the > > compute. > > > > > > > > > > > > I've seen the iSCSI initiator going crazy when the iscsid and the > > > > > > iscsiadm are from different versions. I've seen this in > > containerized > > > > > > environments. > > > > > > > > > > > > Faulty paths on multipathing is a tricky business, because there > > are > > > > > > different checkers, some generic (readsector0, tur, directio) and > > some > > > > > > vendor specific (emc_clarrion, hp_wd, rdac), and each one behaves > > in a > > > > > > different way. > > > > > > > > > > > > If you have a multipath device with faulty paths, that you think > > should > > > > > > not be faulty, you should look into what's going on with those > > paths: > > > > > > > > > > > > - Confirm that the device is still in the system under /dev/XYZ > > > > > > - Confirm in your storage array's console/webconsole that the > > volume is > > > > > > still mapped on that target-portal to that host's iscsi initiator > > > > > > name. > > > > > > - Confirm you can read the faulty devices with dd on the host > > > > > > - Confirm that the WWN of the device is the same in all the paths > > > > (using > > > > > > /lib/udev/scsi_id) > > > > > > - Finally look into what checker is multipath using for your device > > > > > > (sometimes checkers have bugs). > > > > > > > > > > > > Cheers, > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > Il Mar 20 Ott 2020, 19:58 Gorka Eguileor <geguileo(a)redhat.com> > > ha > > > > > > scritto: > > > > > > > > > > > > > > > On 20/10, Ignazio Cassano wrote: > > > > > > > > > This is the entre log from when the migration started: > > > > > > > > > > > > > > > > > > http://paste.openstack.org/show/799199/ > > > > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > There are no os-brick calls in there. :-( > > > > > > > > > > > > > > > > You should look for the call to connect_volume that should have > > > > > > > > something like: > > > > > > > > > > > > > > > > ==> disconnect_volume: call "{'args': > > > > > > > > (<os_brick.initiator.connectors.iscsi > > > > > > > > > > > > > > > > And the second parameter to that call is a dictionary where you > > > > can see > > > > > > > > the target_lun, target_luns, target_portals, target_portal, > > > > target_iqn, > > > > > > > > target_iqns... This will allow us to check if we are actually > > > > > > connected > > > > > > > > to those targets-portals > > > > > > > > > > > > > > > > The third parameter should contain two things that are > > relevant, > > > > the > > > > > > > > scsi_wwn and the path. You can check if the path exists and if > > > > that > > > > > > > > path actually has that wwn using /lib/udev/scsi_id --page 0x83 > > > > > > > > --whitelisted $path > > > > > > > > > > > > > > > > Those are the things I would check, because the only reason I > > can > > > > think > > > > > > > > that os-brick is not disconnecting any volumes are that the > > > > connection > > > > > > > > info is not right, or that the volume is no longer connected. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno mar 20 ott 2020 alle ore 11:23 Gorka Eguileor < > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > On 20/10, Ignazio Cassano wrote: > > > > > > > > > > > Hello Gorka,this is what happens on nova compute with > > debug > > > > > > enabled, > > > > > > > > > > when I > > > > > > > > > > > migrate an instance with iscsi volumes ( note > > Disconnecting > > > > > > from[] > > > > > > > > should > > > > > > > > > > > be the issue): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > The disconnect from [] is the right clue, not necessarily > > the > > > > > > issue. > > > > > > > > > > > > > > > > > > > > OS-Brick is saying that for the connection information > > that has > > > > > > been > > > > > > > > > > passed in the "disconnect_volume" call (which is not > > present > > > > in the > > > > > > > > > > emailed logs) there are no volumes present in the system. > > > > > > > > > > > > > > > > > > > > You should check the connection info that Nova is passing > > to > > > > > > > > > > disconnect_volume and confirm if that data is correct. For > > > > example > > > > > > > > > > checking if the path present in the connection info > > dictionary > > > > is > > > > > > the > > > > > > > > > > same as the one in the instance's XML dump, or if the LUN > > from > > > > the > > > > > > > > > > connection info dict is actually present in the system. > > > > > > > > > > > > > > > > > > > > There are multiple reasons why Nova could be passing the > > wrong > > > > > > > > > > connection info to os-brick. The ones that come to mind > > are: > > > > > > > > > > > > > > > > > > > > - There was a failed migration at some point, and Nova > > didn't > > > > > > rollback > > > > > > > > > > the connection info on the BDM table. > > > > > > > > > > - Nova is calling multiple times initialize_connection on > > > > Cinder > > > > > > for > > > > > > > > the > > > > > > > > > > same host and the driver being used is not idempotent. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > stderr= _run_iscsiadm_bare > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1122 > > > > > > > > > > > 2020-10-20 09:52:33.066 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] iscsi session list stdout=tcp: [10] > > 10.138.209.48:3260,9 > > > > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a3 (non-flash) > > > > > > > > > > > tcp: [11] 10.138.215.17:3260,8 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [12] 10.138.215.17:3260,8 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [13] 10.138.215.18:3260,7 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [14] 10.138.215.18:3260,7 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b2 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [15] 10.138.209.47:3260,6 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [16] 10.138.209.47:3260,6 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.b3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > tcp: [9] 10.138.209.48:3260,9 > > > > > > > > iqn.1992-04.com.emc:cx.ckm00184400687.a3 > > > > > > > > > > > (non-flash) > > > > > > > > > > > stderr= _run_iscsi_session > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1111 > > > > > > > > > > > 2020-10-20 09:52:33.078 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] Resulting device map defaultdict(<function <lambda> > > at > > > > > > > > > > 0x7f4f1b1f7cf8>, > > > > > > > > > > > {(u'10.138.215.17:3260', > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.a2'): > > > > > > > > > > > (set([]), set([u'sdg', u'sdi'])), (u'10.138.209.47:3260 > > ', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.b3'): (set([]), > > > > > > set([u'sdo', > > > > > > > > > > > u'sdq'])), (u'10.138.209.48:3260', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.a3'): (set([]), > > > > > > set([u'sdd', > > > > > > > > > > > u'sdb'])), (u'10.138.215.18:3260', > > > > > > > > > > > u'iqn.1992-04.com.emc:cx.ckm00184400687.b2'): (set([]), > > > > > > set([u'sdm', > > > > > > > > > > > u'sdk']))}) _get_connection_devices > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:844 > > > > > > > > > > > 2020-10-20 09:52:33.078 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] Disconnecting from: [] _disconnect_connection > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1099 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > oslo_concurrency.lockutils > > > > > > [-] > > > > > > > > Lock > > > > > > > > > > > "connect_volume" released by > > > > > > > > > > > "os_brick.initiator.connectors.iscsi.disconnect_volume" > > :: > > > > held > > > > > > > > 1.058s > > > > > > > > > > > inner > > > > > > > > > > /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > [-] <== disconnect_volume: return (1057ms) None > > > > > > trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:170 > > > > > > > > > > > 2020-10-20 09:52:33.079 132171 DEBUG > > > > > > nova.virt.libvirt.volume.iscsi > > > > > > > > [-] > > > > > > > > > > > [instance: 0c846f66-f194-40de-b31e-d53652570fa7] > > Disconnected > > > > > > iSCSI > > > > > > > > > > Volume > > > > > > > > > > > disconnect_volume > > > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py:78 > > > > > > > > > > > 2020-10-20 09:52:33.080 132171 DEBUG os_brick.utils [-] > > ==> > > > > > > > > > > > get_connector_properties: call u"{'execute': None, > > 'my_ip': > > > > > > > > > > > '10.138.208.178', 'enforce_multipath': True, 'host': > > > > > > > > 'podiscsivc-kvm02', > > > > > > > > > > > 'root_helper': 'sudo nova-rootwrap > > /etc/nova/rootwrap.conf', > > > > > > > > 'multipath': > > > > > > > > > > > True}" trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:146 > > > > > > > > > > > 2020-10-20 09:52:33.125 132171 DEBUG > > > > os_brick.initiator.linuxfc > > > > > > [-] > > > > > > > > No > > > > > > > > > > > Fibre Channel support detected on system. get_fc_hbas > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157 > > > > > > > > > > > 2020-10-20 09:52:33.126 132171 DEBUG > > > > os_brick.initiator.linuxfc > > > > > > [-] > > > > > > > > No > > > > > > > > > > > Fibre Channel support detected on system. get_fc_hbas > > > > > > > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py:157 > > > > > > > > > > > 2020-10-20 09:52:33.145 132171 DEBUG os_brick.utils [-] > > <== > > > > > > > > > > > get_connector_properties: return (61ms) {'initiator': > > > > > > > > > > > u'iqn.1994-05.com.redhat:fbfdc37eed4c', 'ip': > > > > u'10.138.208.178', > > > > > > > > 'system > > > > > > > > > > > uuid': u'4C4C4544-0051-4E10-8057-B6C04F425932', > > 'platform': > > > > > > > > u'x86_64', > > > > > > > > > > > 'host': u'podiscsivc-kvm02', 'do_local_attach': False, > > > > 'os_type': > > > > > > > > > > > u'linux2', 'multipath': True} trace_logging_wrapper > > > > > > > > > > > /usr/lib/python2.7/site-packages/os_brick/utils.py:170 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > Il giorno gio 15 ott 2020 alle ore 10:57 Gorka Eguileor < > > > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > > > > > On 14/10, Ignazio Cassano wrote: > > > > > > > > > > > > > Hello, thank you for the answer. > > > > > > > > > > > > > I am using os-brick 2.3.8 but I got same issues on > > stein > > > > with > > > > > > > > > > os.brick > > > > > > > > > > > > 2.8 > > > > > > > > > > > > > For explain better the situation I send you the > > output of > > > > > > > > multipath > > > > > > > > > > -ll > > > > > > > > > > > > on > > > > > > > > > > > > > a compute node: > > > > > > > > > > > > > root@podvc-kvm01 ansible]# multipath -ll > > > > > > > > > > > > > Oct 14 18:50:01 | sdbg: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbe: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbd: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbf: alua not supported > > > > > > > > > > > > > 360060160f0d049007ab7275f743d0286 dm-11 DGC > > ,VRAID > > > > > > > > > > > > > size=30G features='1 retain_attached_hw_handler' > > > > hwhandler='1 > > > > > > > > alua' > > > > > > > > > > wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > | |- 15:0:0:71 sdbg 67:160 failed faulty running > > > > > > > > > > > > > | `- 12:0:0:71 sdbe 67:128 failed faulty running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > |- 11:0:0:71 sdbd 67:112 failed faulty running > > > > > > > > > > > > > `- 13:0:0:71 sdbf 67:144 failed faulty running > > > > > > > > > > > > > 360060160f0d049004cdb615f52343fdb dm-8 DGC ,VRAID > > > > > > > > > > > > > size=80G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 15:0:0:210 sdau 66:224 active ready running > > > > > > > > > > > > > | `- 12:0:0:210 sdas 66:192 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:210 sdar 66:176 active ready running > > > > > > > > > > > > > `- 13:0:0:210 sdat 66:208 active ready running > > > > > > > > > > > > > 360060160f0d0490034aa645fe52265eb dm-12 DGC > > ,VRAID > > > > > > > > > > > > > size=100G features='2 queue_if_no_path > > > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 12:0:0:177 sdbi 67:192 active ready running > > > > > > > > > > > > > | `- 15:0:0:177 sdbk 67:224 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:177 sdbh 67:176 active ready running > > > > > > > > > > > > > `- 13:0:0:177 sdbj 67:208 active ready running > > > > > > > > > > > > > 360060160f0d04900159f225fd6126db9 dm-6 DGC ,VRAID > > > > > > > > > > > > > size=40G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:26 sdaf 65:240 active ready running > > > > > > > > > > > > > | `- 13:0:0:26 sdah 66:16 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 12:0:0:26 sdag 66:0 active ready running > > > > > > > > > > > > > `- 15:0:0:26 sdai 66:32 active ready running > > > > > > > > > > > > > Oct 14 18:50:01 | sdba: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbc: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdaz: alua not supported > > > > > > > > > > > > > Oct 14 18:50:01 | sdbb: alua not supported > > > > > > > > > > > > > 360060160f0d049007eb7275f93937511 dm-10 DGC > > ,VRAID > > > > > > > > > > > > > size=40G features='1 retain_attached_hw_handler' > > > > hwhandler='1 > > > > > > > > alua' > > > > > > > > > > wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > | |- 12:0:0:242 sdba 67:64 failed faulty running > > > > > > > > > > > > > | `- 15:0:0:242 sdbc 67:96 failed faulty running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=0 status=enabled > > > > > > > > > > > > > |- 11:0:0:242 sdaz 67:48 failed faulty running > > > > > > > > > > > > > `- 13:0:0:242 sdbb 67:80 failed faulty running > > > > > > > > > > > > > 360060160f0d049003a567c5fb72201e8 dm-7 DGC ,VRAID > > > > > > > > > > > > > size=40G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 12:0:0:57 sdbq 68:64 active ready running > > > > > > > > > > > > > | `- 15:0:0:57 sdbs 68:96 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 11:0:0:57 sdbp 68:48 active ready running > > > > > > > > > > > > > `- 13:0:0:57 sdbr 68:80 active ready running > > > > > > > > > > > > > 360060160f0d04900c120625f802ea1fa dm-9 DGC ,VRAID > > > > > > > > > > > > > size=25G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:234 sdav 66:240 active ready running > > > > > > > > > > > > > | `- 13:0:0:234 sdax 67:16 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 15:0:0:234 sday 67:32 active ready running > > > > > > > > > > > > > `- 12:0:0:234 sdaw 67:0 active ready running > > > > > > > > > > > > > 360060160f0d04900b8b0615fb14ef1bd dm-3 DGC ,VRAID > > > > > > > > > > > > > size=50G features='2 queue_if_no_path > > > > > > retain_attached_hw_handler' > > > > > > > > > > > > > hwhandler='1 alua' wp=rw > > > > > > > > > > > > > |-+- policy='round-robin 0' prio=50 status=active > > > > > > > > > > > > > | |- 11:0:0:11 sdan 66:112 active ready running > > > > > > > > > > > > > | `- 13:0:0:11 sdap 66:144 active ready running > > > > > > > > > > > > > `-+- policy='round-robin 0' prio=10 status=enabled > > > > > > > > > > > > > |- 12:0:0:11 sdao 66:128 active ready running > > > > > > > > > > > > > `- 15:0:0:11 sdaq 66:160 active ready running > > > > > > > > > > > > > > > > > > > > > > > > > > The active running are related to running virtual > > > > machines. > > > > > > > > > > > > > The faulty are related to virtual macnines migrated > > on > > > > other > > > > > > kvm > > > > > > > > > > nodes. > > > > > > > > > > > > > Every volume has 4 path because iscsi on unity needs > > two > > > > > > > > different > > > > > > > > > > vlans, > > > > > > > > > > > > > each one with 2 addresses. > > > > > > > > > > > > > I think this issue can be related to os-brick because > > > > when I > > > > > > > > migrate > > > > > > > > > > a > > > > > > > > > > > > > virtual machine from host A host B in the cova > > compute > > > > log on > > > > > > > > host A > > > > > > > > > > I > > > > > > > > > > > > read: > > > > > > > > > > > > > 2020-10-13 10:31:02.769 118727 DEBUG > > > > > > > > > > os_brick.initiator.connectors.iscsi > > > > > > > > > > > > > [req-771ede8c-6e1b-4f3f-ad4a-1f6ed820a55c > > > > > > > > > > > > 66adb965bef64eaaab2af93ade87e2ca > > > > > > > > > > > > > 85cace94dcc7484c85ff9337eb1d0c4c - default default] > > > > > > > > *Disconnecting > > > > > > > > > > from: > > > > > > > > > > > > []* > > > > > > > > > > > > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > That's definitely the right clue!! Though I don't > > fully > > > > agree > > > > > > with > > > > > > > > > > this > > > > > > > > > > > > being an os-brick issue just yet. ;-) > > > > > > > > > > > > > > > > > > > > > > > > Like I mentioned before, RCA is usually non-trivial, > > and > > > > > > > > explaining how > > > > > > > > > > > > to debug these issues over email is close to > > impossible, > > > > but if > > > > > > > > this > > > > > > > > > > > > were my system, and assuming you have tested normal > > > > > > attach/detach > > > > > > > > > > > > procedure and is working fine, this is what I would do: > > > > > > > > > > > > > > > > > > > > > > > > - Enable DEBUG logs on Nova compute node (I believe you > > > > already > > > > > > > > have) > > > > > > > > > > > > - Attach a new device to an instance on that node with > > > > --debug > > > > > > to > > > > > > > > get > > > > > > > > > > > > the request id > > > > > > > > > > > > - Get the connection information dictionary that > > os-brick > > > > > > receives > > > > > > > > on > > > > > > > > > > > > the call to connect_volume for that request, and the > > data > > > > > > that > > > > > > > > > > > > os-brick returns to Nova on that method call > > completion. > > > > > > > > > > > > - Check if the returned data to Nova is a multipathed > > > > device or > > > > > > > > not (in > > > > > > > > > > > > 'path'), and whether we have the wwn or not (in > > > > > > 'scsi_wwn'). It > > > > > > > > > > > > should be a multipath device, and then I would check > > its > > > > > > status > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > multipath daemon. > > > > > > > > > > > > - Now do the live migration (with --debug to get the > > > > request > > > > > > id) > > > > > > > > and > > > > > > > > > > see > > > > > > > > > > > > what information Nova passes in that request to > > > > os-brick's > > > > > > > > > > > > disconnect_volume. > > > > > > > > > > > > - Is it the same? Then it's likely an os-brick issue, > > > > and I > > > > > > can > > > > > > > > have > > > > > > > > > > a > > > > > > > > > > > > look at the logs if you put the logs for that > > os-brick > > > > > > detach > > > > > > > > > > > > process in a pastebin [1]. > > > > > > > > > > > > - Is it different? Then it's either a Nova bug or a > > > > Cinder > > > > > > driver > > > > > > > > > > > > specific bug. > > > > > > > > > > > > - Is there a call from Nova to Cinder, in the > > migration > > > > > > > > request, > > > > > > > > > > for > > > > > > > > > > > > that same volume to initialize_connection > > passing the > > > > > > source > > > > > > > > host > > > > > > > > > > > > connector info (info from the host that is > > currently > > > > > > > > attached)? > > > > > > > > > > > > If there is a call, check if the returned data is > > > > > > different > > > > > > > > from > > > > > > > > > > > > the one we used to do the attach, if that's the > > case > > > > then > > > > > > > > it's a > > > > > > > > > > > > Nova and Cinder driver bug that was solved on the > > > > Nova > > > > > > side > > > > > > > > in > > > > > > > > > > > > 17.0.10 [2]. > > > > > > > > > > > > - If there's no call to Cinder's > > > > initialize_connection, the > > > > > > > > it's > > > > > > > > > > > > most likely a Nova bug. Try to find out if this > > > > > > connection > > > > > > > > info > > > > > > > > > > > > makes any sense for that host (LUN, target, > > etc.) or > > > > if > > > > > > this > > > > > > > > is > > > > > > > > > > > > the one from the destination volume. > > > > > > > > > > > > > > > > > > > > > > > > I hope this somehow helps. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: http://paste.openstack.org/ > > > > > > > > > > > > [2]: https://review.opendev.org/#/c/637827/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno mer 14 ott 2020 alle ore 13:41 Gorka > > Eguileor < > > > > > > > > > > > > geguileo(a)redhat.com> > > > > > > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > > > > > > > > > > > On 09/10, Ignazio Cassano wrote: > > > > > > > > > > > > > > > Hello Stackers, I am using dell emc iscsi driver > > on > > > > my > > > > > > > > centos 7 > > > > > > > > > > > > queens > > > > > > > > > > > > > > > openstack. It works and instances work as well > > but on > > > > > > compute > > > > > > > > > > nodes I > > > > > > > > > > > > > > got a > > > > > > > > > > > > > > > lot a faulty device reported by multipath il > > comand. > > > > > > > > > > > > > > > I do know why this happens, probably attacching > > and > > > > > > detaching > > > > > > > > > > > > volumes and > > > > > > > > > > > > > > > live migrating instances do not close something > > well. > > > > > > > > > > > > > > > I read this can cause serious performances > > problems > > > > on > > > > > > > > compute > > > > > > > > > > nodes. > > > > > > > > > > > > > > > Please, any workaround and/or patch is suggested > > ? > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are many, many, many things that could be > > > > happening > > > > > > > > there, > > > > > > > > > > and > > > > > > > > > > > > > > it's not usually trivial doing the RCA, so the > > > > following > > > > > > > > questions > > > > > > > > > > are > > > > > > > > > > > > > > just me hoping this is something "easy" to find > > out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > What os-brick version from Queens are you running? > > > > Latest > > > > > > > > > > (2.3.9), or > > > > > > > > > > > > > > maybe one older than 2.3.3? > > > > > > > > > > > > > > > > > > > > > > > > > > > > When you say you have faulty devices reported, are > > > > these > > > > > > faulty > > > > > > > > > > devices > > > > > > > > > > > > > > alone in the multipath DM? Or do you have some > > faulty > > > > ones > > > > > > with > > > > > > > > > > some > > > > > > > > > > > > > > that are ok? > > > > > > > > > > > > > > > > > > > > > > > > > > > > If there are some OK and some that aren't, are they > > > > > > consecutive > > > > > > > > > > > > devices? > > > > > > > > > > > > > > (as in /dev/sda /dev/sdb etc). > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Gorka. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

2 2

Re: [kolla] Setting up eth1 on VM for AIO
by Braden, Albert 30 Oct '20

30 Oct '20

From: Mark Goddard <mark(a)stackhpc.com> Sent: Friday, October 30, 2020 4:37 AM To: Radosław Piliszek <radoslaw.piliszek(a)gmail.com> Cc: Braden, Albert <C-Albert.Braden(a)charter.com>; OpenStack Discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [kolla] Setting up eth1 on VM for AIO On Thu, 29 Oct 2020 at 16:20, Radosław Piliszek <radoslaw.piliszek(a)gmail.com<mailto:radoslaw.piliszek@gmail.com>> wrote: Hi Albert! Sorry for top posting but I don't really see which part to quote would be best so they all lose. :D Indeed, it's tricky to handle this if you are not a fan of linux networking. I guess we should document an example way.... Anyhow, the way e.g. Kayobe handles it for users (as it deals with baremetal details) is to use bridges and veths. Basically, you create a bridge, attach it to the expected interface, set any networking you still want on that network for this host on this bridge. Then, you create a veth pair and connect one end to it and the second one is given to neutron. >Alternatively, if this is just for development, you can use a dummy interface. > >ip link add link eth1 type dummy > >It's not persistent, but good enough for testing. If you need external network access from it you could add a MASQUERADE iptables rule and enable the IP >forwarding sysctl. That seems to be working. After googling a bit I was able to make it persistent: [root@adjutant-poc ~]# cat /etc/modules-load.d/dummy.conf dummy [root@adjutant-poc ~]# cat /etc/modprobe.d/dummy.conf install dummy /sbin/modprobe --ignore-install dummy; /sbin/ip link set name eth1 dev dummy0 [root@adjutant-poc ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 NAME=eth1 BOOTPROTO=none DEVICE=eth1 ONBOOT=yes TYPE=Ethernet NM_CONTROLLED=no [root@adjutant-poc ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:22:c4:e2 brd ff:ff:ff:ff:ff:ff inet 192.168.1.83/22 brd 192.168.3.255 scope global dynamic eth0 valid_lft 85821sec preferred_lft 85821sec inet6 fe80::f816:3eff:fe22:c4e2/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 7e:e1:fe:31:c9:2e brd ff:ff:ff:ff:ff:ff inet6 fe80::7ce1:feff:fe31:c92e/64 scope link valid_lft forever preferred_lft forever -yoctozepto On Thu, Oct 29, 2020 at 12:49 PM Braden, Albert <C-Albert.Braden(a)charter.com<mailto:C-Albert.Braden@charter.com>> wrote: > > I’m trying to setup an allinone kolla-ansible Train POC on a VM, and I’m following the instructions here: > > > > https://docs.openstack.org/kolla-ansible/train/user/quickstart.html > > > > Under “Prepare initial configuration” where it talks about globals.yml, it says I need 2 network interfaces: > > > > network_interface: "eth0" > > neutron_external_interface: "eth1" > > > > My VM has a private IP and a floating IP that I use to ssh to it: > > > > | ID | Name | Status | Networks | Image | Flavor | > > | 41f6782a-8ca8-4ab4-8fdc-776584139ba0 | adjutant-poc | ACTIVE | private=<primary IP>, <floating IP> | | s1-standard-08 | > > > > I tried various options for eth1 but I can’t seem to get it to work: > > > > [root@adjutant-poc network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 > > BOOTPROTO=dhcp > > DEVICE=eth1 > > ONBOOT=yes > > TYPE=Ethernet > > NM_CONTROLLED=no > > > > I tried creating another port and adding it but that didn’t seem to make a difference. What am I doing wrong? Do I really need 2 network interfaces on the VM? If so, what’s the correct way to add one? > > > > [root@adjutant-poc network-scripts]# service network restart > > Restarting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details. > > [FAILED] > > > > Log: > > > > http://paste.debian.net/1168982/ > > > > I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

[ironic] SPUC winter season
by Dmitry Tantsur 30 Oct '20

30 Oct '20

Boo! It's SPUC time again! Ironicers, their pet AIs, friendly stackers, as well as significant others and fluffy overladies and overlords, are welcome! For those who don't know, Sanity Preserving Un-Conference is a series of Friday calls for keeping (the remainings) of our sanity in these difficult times. It's goal is to let people virtually socialize and help brighten up lives for each other. Work achievements? A new hobby? A hilarious story? Funny things your kid did at 4am? The first words from your pizza dough that you forgot in the fridge during the previous lockdown? Anything that makes you and others happier is on-topic! Due to an increasing number of APAC contributors, I would like two pick slots: one in APAC evening/Europe morning, the other in Europe morning/US afternoon. Please vote in the doodle: https://doodle.com/poll/tw72utagidz2nac3 Cheers, Dmitry -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

4 6

[openstack][nova] urgent compute in evacuate state
by Ignazio Cassano 30 Oct '20

30 Oct '20

Hello Guys, I have a node in evacuate state....how exit from this state ? Thanks Ignazio

2 6

[kolla] Setting up eth1 on VM for AIO
by Braden, Albert 30 Oct '20

30 Oct '20

I'm trying to setup an allinone kolla-ansible Train POC on a VM, and I'm following the instructions here: https://docs.openstack.org/kolla-ansible/train/user/quickstart.html Under "Prepare initial configuration" where it talks about globals.yml, it says I need 2 network interfaces: network_interface: "eth0" neutron_external_interface: "eth1" My VM has a private IP and a floating IP that I use to ssh to it: | ID | Name | Status | Networks | Image | Flavor | | 41f6782a-8ca8-4ab4-8fdc-776584139ba0 | adjutant-poc | ACTIVE | private=<primary IP>, <floating IP> | | s1-standard-08 | I tried various options for eth1 but I can't seem to get it to work: [root@adjutant-poc network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 BOOTPROTO=dhcp DEVICE=eth1 ONBOOT=yes TYPE=Ethernet NM_CONTROLLED=no I tried creating another port and adding it but that didn't seem to make a difference. What am I doing wrong? Do I really need 2 network interfaces on the VM? If so, what's the correct way to add one? [root@adjutant-poc network-scripts]# service network restart Restarting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details. [FAILED] Log: http://paste.debian.net/1168982/ I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

3 2

[Openstack][cinder] dell unity iscsi faulty devices
by Ignazio Cassano 30 Oct '20

30 Oct '20

Hello Stackers, I am using dell emc iscsi driver on my centos 7 queens openstack. It works and instances work as well but on compute nodes I got a lot a faulty device reported by multipath il comand. I do know why this happens, probably attacching and detaching volumes and live migrating instances do not close something well. I read this can cause serious performances problems on compute nodes. Please, any workaround and/or patch is suggested ? Regards Ignazio

2 18