March 2020 - openstack-discuss - lists.openstack.org

[neutron] How to fix break of connectivity in case of L3 HA after reboot of backup node
by Slawek Kaplonski 20 Mar '20

20 Mar '20

Hi, We have bug [1] to solve. Basically, when node which is backup node for some router, connectivity to external gateway may be broken for some time. It's like that because when host is up and L3 agent is configuring qrouter namespace, it flush all IPv6 addresses from the qg- interface. And due to that some MLDv2 packets are sent from this interface which may cause that ToR switch will learn qg port's mac address on wrong (backup) node. This don't happens every time and for every router because it is a race between L3 agent and OVS agent. When L3 agent creates qg interface in br-int, it sets tag 4095 for it and traffic sent with such vlan tag is always dropped in br-int. So if L3 agent will flush IPv6 addresses before OVS agent wires the port and sets correct tag for it, then all is fine. But if OVS agent is first, then MLDv2 packets are sent to the wire and there is this connectivity break. There are proposed 2 ways of fixing this: - [2] which propsoes to add some kind of "communication" between L3 agent and OVS agent and tell OVS agent that tag can be changed only after IPv6 config is finished by L3 agent. Downside of this solution is that it works for OVS agent only, Linuxbridge agent may still hit the same issue. But plus is that after initial configuration of the router, everything else regarding to failover is handled by keepalived only - in same way like it is now. - [3] which sets qg NIC to be DOWN always on backup nodes. So when keepalived failover active router to new node, L3 agent needs to come and switch interfaces to be UP before it will work. The plus of this solution is that it works for all OVS and Linuxbridge L2 agents (and probably for others too) but downside is that failover process is a bit longer and there may be potentially another race condition between L3 agent and keepalived. Keepalived tries to sent gARP packets after switch node to be active, first attempt will always fail as interface is still DOWN. But keepalived will retry those gARPs after some time and this should be fine if L3 agent will already bring interface to be UP. Both patches are waiting for pretty long time in gerrit and I want to bring more visibility for both of them. Please check them and maybe You will have some opinions about which solution would be better and which we should go with. [1] https://bugs.launchpad.net/neutron/+bug/1859832 [2] https://review.opendev.org/#/c/702856/ [3] https://review.opendev.org/#/c/707406/ -- Slawek Kaplonski Senior software engineer Red Hat

5 4

Neutron RabbitMQ issues
by Grant Morley 20 Mar '20

20 Mar '20

Hi all, We are currently experiencing some fairly major issues with our OpenStack cluster. It all appears to be with Neutron and RabbitMQ. We are seeing a lot of time out messages in responses to replies and because of this instance creation or anything to do with instances and networking is broken. We are running OpenStack Queens. We have already tuned Rabbit for Neutron by doing the following on neutron: heartbeat_timeout_threshold = 0 rpc_conn_pool_size = 300 rpc_thread_pool_size = 2048 rpc_response_timeout = 3600 rpc_poll_timeout = 60 ## Rpc all executor_thread_pool_size = 64 rpc_response_timeout = 3600 What we are seeing in the error logs for neutron for all services (l3-agent, dhcp, linux-bridge etc ) are these timeouts: https://pastebin.com/Fjh23A5a We have manually tried to get everything in sync by forcing fail-over of the networking which seems to get routers in sync. We are also seeing that there are a lot of "unacknowledged" messages in RabbitMQ for 'q-plugin' in the neutron queues. Some times restarting of the services on neutron gets these back acknowledged again, however the timeouts come back. The RabbitMQ servers themselves are not loaded at all. All memory, file descriptors and errlang processes have plenty of resources available. We are also seeing a lot of rpc issues: Timeout in RPC method release_dhcp_port. Waiting for 1523 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 965fa44ab4f6462fa378a1cf7259aad4 2020-03-10 19:02:33.548 16242 ERROR neutron.common.rpc [req-a858afbb-5083-4e21-a309-6ee53582c4d9 - - - - -] Timeout in RPC method release_dhcp_port. Waiting for 3347 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID 7937465f15634fbfa443fe1758a12a9c Does anyone know if there is anymore tuning to be done at all? Upgrading for us at the moment to a newer version isn't really an option unfortunately. Because of our setup, we also have roughly 800 routers enabled and I know that will be putting a load on the system. However these problems have only started to happen roughly 1 week ago and have steadily got worse. If anyone has any use cases for this or any more recommendations that would be great. Many thanks,

3 25

[release] Release countdown for week R-7, March 23-27
by Sean McGinnis 20 Mar '20

20 Mar '20

Development Focus ----------------- We are entering the last weeks of the Ussuri development cycle. From now until the final release, we'll send a countdown email like this every week. It's probably a good time for teams to take stock of their library and client work that needs to be completed yet. The non-client library freeze is coming up, followed closely by the client lib freeze. Please plan accordingly so avoid any last minute rushes to get key functionality in. General Information ------------------- Next week is the Extra-ATC freeze, in preparation for elections. All contributions to OpenStack are valuable, but some are not expressed as Gerrit code changes. Please list active contributors to your project team who do not have a code contribution this cycle, and therefore won't automatically be considered an Active Technical Contributor and allowed to vote. This is done by adding extra-atcs to https://opendev.org/openstack/governance/src/branch/master/reference/projec… before the Extra-ATC freeze on March 26. A quick reminder of the upcoming freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (April 2). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (April 9) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, April 9. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (April 23) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (May 7) Finally, now is also a good time to start planning what highlights you want for your deliverables in the cycle highlights. The deadline to submit an initial version for those is set to Feature freeze (April 9). Background on cycle-highlights: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125613.html Project Team Guide, Cycle-Highlights: https://docs.openstack.org/project-team-guide/release-management.html#cycle… knelson [at] openstack.org/diablo_rojo on IRC is available if you need help selecting or writing your highlights Note that cycle highlights are per-team, not per-deliverable. Upcoming Deadlines & Dates -------------------------- Extra-ATC freeze: March 26 (R-7 week) Non-client library freeze: April 2 (R-6 week) Client library freeze: April 9 (R-5 week) Ussuri-3 milestone: April 9 (R-5 week) Cycle Highlights Due: April 9 (R-5 week)

1 0

[queens][cinder][unity] iscsi driver
by Ignazio Cassano 20 Mar '20

20 Mar '20

Hello All, I would like to suggest to modify the unity iscsi cinder driver documentation. After 1 month of intensive test, we discovered the mutipath.conf needs the following option in multipath.conf kvm nodes: skip_kpartx yes Without the above option instances with more than 2 volumes attached could fail live migration because the os_brick code fails in "multipath -f" command reporting map in use error. Ignazio

1 0

[openstack.org] SEO Improvements
by Jimmy McArthur 19 Mar '20

19 Mar '20

Hi all - We've contracted a professional SEO firm to help improve search placement for all OSF projects. I'd like to crowd source this on each of the project mailing lists, so the community is able to be involved. Could you all help out in providing the below: “Wish List” of keywords: 8-12 big terms you think fit your domain (if you only have 4-5 to share, fewer is okay) - open source ci - ? At least 3 competitors - Jenkins - ? Any other input on positioning, offerings, etc. that you think will help best filter for relevance - ? Cheers, Jimmy

6 12

Re: [Release-job-failures] Tag of openstack/glance_store for ref refs/tags/rocky-em failed
by Sean McGinnis 19 Mar '20

19 Mar '20

On 3/19/20 4:56 PM, zuul(a)openstack.org wrote: > Build failed. > > - publish-openstack-releasenotes-python3 https://zuul.opendev.org/t/openstack/build/66b1173250724747bc9b7630277a5b1a : POST_FAILURE in 4m 24s > > _______________________________________________ > Release-job-failures mailing list > Release-job-failures(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures This failure appears to be from a temporary glitch that we've seen time to time in the gate: "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" This was on publishing release notes. Since no new release notes have been added as part of this (just tagging rocky-em) it is safe to ignore this failure. Sean

1 0

[nova] bug triage
by Balázs Gibizer 19 Mar '20

19 Mar '20

Hi, We surpassed the somewhat magical line of having more than 100 untriaged nova bugs [1]. For me it seems that how we, as a team, are handling the incoming bugs is not sustainable. I'm guilty as well of not doing bug triage in a last two months. So I'm personally trying to change now and dedicate a weekly times lot to look at the bug list. But I also want to open a discussion about bug triage in genera. How can we handle the incoming bugs? I see that neutron does a weekly rotation of bug deputy and for them it works nicely. What do you think? Do we want to try that? Do we have enough volunteers to create a nice rotation period? Cheers, gibi [1] https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New

2 1

[python-openstackclient] --force bug
by jayaditya gupta 19 Mar '20

19 Mar '20

Hello all, new contributor/developer here. I have just submitted a bug for python-openstackclient project : https://storyboard.openstack.org/#!/story/2007440 I am not sure how to provide --force option to openstackclient .. I am comparing quota.py files in nova-client and in openstackclient.but can't seem to understand how to provide --force option. Any suggestions? Best Regards

3 3

[neutron]Drivers meeting cancelled
by Slawek Kaplonski 19 Mar '20

19 Mar '20

Hi, Due to lack of agenda, lets cancel tomorrows (20.03.2020) drivers meeting and lets focus on reviewing patches related to the already accepted RFEs [1]. [1] https://tinyurl.com/vezk6n6 — Slawek Kaplonski Senior software engineer Red Hat

1 0

[all][tc] Moving PTL role to "Maintainers"
by Mohammed Naser 19 Mar '20

19 Mar '20

Hi everyone: We're now in a spot where we have an increasing amount of projects that don't end up with a volunteer as PTL, even if the project has contributors .. no one wants to hold that responsibility alone for many reasons. With time, the PTL role has become far more overloaded with many extra responsibilities than what we define in our charter: https://governance.openstack.org/tc/reference/charter.html#project-team-lea… I think it's time to re-evaluate the project leadership model that we have. I am thinking that perhaps it would make a lot of sense to move from a single PTL model to multiple maintainers. This would leave it up to the maintainers to decide how they want to sort the different requirements/liaisons/contact persons between them. The above is just a very basic idea, I don't intend to diving much more in depth for now as I'd like to hear about what the rest of the community thinks. Thanks, Mohammed

24 34