September 2020 - openstack-discuss

[Neutron] vxlan to vlan bridge
by Ignazio Cassano 16 Sep '20

16 Sep '20

Hello Stackers, is it possibile to create a vxlan to vlan bridge in openstack like vmware nsx does? Thanks Ignazio

3 2

neutron-l3-agent:l3-agent has been creating the "keepalived -P -f..." process
by 宋贺 15 Sep '20

15 Sep '20

we create a '--distributed=True,--ha=True" router in OpenStack Newton. We found that l3-agent cannot read the pid file of keepalived, thus creating the keepalived process again. As a result, a large number of keepalived processes appear on the host node.As picture 1 in the attachment. This problem can be solved when we execute the following content and modify the permissions of the pid file [root@fusion0 ~]# chown neutron:neutron /var/lib/neutron/ha_confs/2a20ab37-ffe8-4955-8148-cefdcb5b6be3.pid [root@fusion0 ~]# chown neutron:neutron /var/lib/neutron/ha_confs/2a20ab37-ffe8-4955-8148-cefdcb5b6be3.pid-vrrp I want to know how to solve this problem completely from the perspective of neutron. neutron-l3-agent.log file is in the attachment.

1 0

[all][elections][ptl][tc] Combined PTL/TC Election Season
by Jeremy Stanley 15 Sep '20

15 Sep '20

Election details: https://governance.openstack.org/election/ The nomination period officially begins Sep 22, 2020 23:45 UTC. Please read the stipulations and timelines for candidates and electorate contained in this governance documentation. Be aware that the Wallaby cycle elections are a few weeks behind schedule[1]. We're still making sure the new TC is confirmed and any PTL runoff polls are closed and resolved before Victoria release day. Due to circumstances of timing, PTL and TC elections for the coming cycle will run concurrently; deadlines for their nomination and voting activities are synchronized but will still use separate ballots. Please note, if only one candidate is nominated as PTL for a project team during the PTL nomination period, that candidate will win by acclaim, and there will be no poll. There will only be a poll if there is more than one candidate stepping forward for a project team's PTL position. If teams do not produce any PTL candidate during the nomination period, or are interested in considering alternatives prior to nominations, the TC may consider requests to switch to the new distributed leadership model they recently documented[2]. In keeping with the established plan[3] to gradually reduce the size of the Technical Committee, we will only fill four (4) seats in this coming election. There will be further announcements posted to the mailing list as action is required from the electorate or candidates. This email is for information purposes only. If you have any questions which you feel affect others please reply to this email thread. If you have any questions that you which to discuss in private please email any of the election officials[4] so that we may address your concerns. [1] https://governance.openstack.org/tc/reference/election-exceptions.html [2] https://governance.openstack.org/tc/resolutions/20200803-distributed-projec… [3] https://governance.openstack.org/tc/reference/charter.html#number-of-seats-… [4] https://governance.openstack.org/election/#election-officials -- Jeremy Stanley on behalf of the OpenStack Technical Election Officials

1 0

[tripleo][ci] Red Tuesday
by Wesley Hayutin 15 Sep '20

15 Sep '20

Greetings, Well if you were working today you know there were several issues upstream. pip issues: https://bugs.launchpad.net/tripleo/+bug/1449136 containers-multinode: https://bugs.launchpad.net/tripleo/+bug/1895290 https://bugs.launchpad.net/tripleo/+bug/1895288 validations: https://launchpad.net/bugs/1895507 Everything I know of atm is either merged on or in the gate. https://review.opendev.org/#/c/751828/ https://review.opendev.org/#/c/751653/ There is a lot going on, please report bugs to launchpad and ping nick-ruck or nick-rover in #tripleo if you see something else. Thanks to everyone helping to push things along. 0/

1 0

Re: [Openstack-mentoring] Neutron subnet with DHCP relay - continued
by Thomas King 15 Sep '20

15 Sep '20

I got the metadata network accessible over a routed subnet! The breakthrough was making sure I pointed 169.254.169.254 to the Neutron DHCP address rather than the physical or bridge IP address, this route added to the physical router (L3 switch) that serves the subnet hosting the Ironic controller. Also, I made sure to add a host route for 169.254.169.254 to the baremetal remote subnet in Neutron so the baremetal nodes will have a proper route to the metadata subnet. There is still some network tweaking left to do. We have a separate subnet for ESXi mgmt vmkernels since they use virtual MAC addresses and neutron doesn't recognize them. However, enabling DHCP relay for that separate subnet breaks routing which also breaks cleaning and deploying, and I need to find out where. This is huge for our metal-as-a-service offering! Thanks, Tom King On Sat, Aug 22, 2020 at 12:53 AM Thomas King <thomas.king(a)gmail.com> wrote: > Ok, thanks. > > On Fri, Aug 21, 2020, 11:39 PM Ruslanas Gžibovskis <ruslanas(a)lpic.lt> > wrote: > >> No, I didn't. In the beginning I wanted to say "yes" but that is for >> overcloud. >> >> Now I am thinking, maybe even I do no have metadata on undercloud.... >> >> Will need to check on Monday. >> >> 2020-08-22, št 00:51, Thomas King <thomas.king(a)gmail.com> rašė: >> >>> Finally got it worked out except for reaching the metadata service IP >>> (169.254.169.254) from a remote network. Did you put specific routes in >>> place for that IP address on your physical network? >>> >>> Tom >>> >>> On Wed, Jul 22, 2020 at 3:19 AM Ruslanas Gžibovskis <ruslanas(a)lpic.lt> >>> wrote: >>> >>>> Ok >>>> >>>> here is a small copy paste: >>>> http://paste.openstack.org/show/o4Uay0DYbAdkcfUJGOLV/ >>>> >>>> also have relaunched previous commands, as I have redeployed overcloud >>>> :) >>>> >>>> and maybe you will be interested, in undercloud containers running: >>>> >>>> (undercloud) [stack@remote-u overcloud]$ sudo podman ps >>>> CONTAINER ID IMAGE >>>> COMMAND CREATED STATUS >>>> PORTS NAMES >>>> 39c684172ce7 >>>> docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo >>>> /usr/sbin/dnsmasq... 7 days ago Up 7 days ago >>>> neutron-dnsmasq-qdhcp-e6d6c50b-fb69-4375-bb7d-0f6e0cfed5cb >>>> 30ccb169e666 >>>> docker.io/tripleomaster/centos-binary-ironic-pxe:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> ironic_pxe_http >>>> 6a62e08e8e10 >>>> docker.io/tripleomaster/centos-binary-ironic-pxe:current-tripleo >>>> /bin/bash -c BIND... 7 days ago Up 7 days ago >>>> ironic_pxe_tftp >>>> e4cbad7f7488 >>>> docker.io/tripleomaster/centos-binary-neutron-l3-agent:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> neutron_l3_agent >>>> d78d3828c420 >>>> docker.io/tripleomaster/centos-binary-neutron-openvswitch-agent:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago neutron_ovs_agent >>>> 3b67b6a69a87 >>>> docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> neutron_dhcp >>>> 46d403293b54 >>>> docker.io/tripleomaster/centos-binary-swift-proxy-server:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_proxy >>>> 15a9792d6f62 >>>> docker.io/tripleomaster/centos-binary-swift-object:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_rsync >>>> 0e6cd094d6dc >>>> docker.io/tripleomaster/centos-binary-swift-object:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_object_updater >>>> 80b7f26e742d >>>> docker.io/tripleomaster/centos-binary-swift-object:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_object_server >>>> 765f79633499 >>>> docker.io/tripleomaster/centos-binary-swift-proxy-server:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_object_expirer >>>> 0e53da61af88 >>>> docker.io/tripleomaster/centos-binary-swift-container:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_container_updater >>>> 8c14cef58eb3 >>>> docker.io/tripleomaster/centos-binary-swift-account:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_account_server >>>> 12930a63dc12 >>>> docker.io/tripleomaster/centos-binary-swift-container:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_container_server >>>> 8a5cd45208c0 >>>> docker.io/tripleomaster/centos-binary-swift-account:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> swift_account_reaper >>>> b871400bfdd9 >>>> docker.io/tripleomaster/centos-binary-mistral-executor:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> mistral_executor >>>> 3e98e5f83f09 >>>> docker.io/tripleomaster/centos-binary-mistral-event-engine:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> mistral_event_engine >>>> 52d711dffca3 >>>> docker.io/tripleomaster/centos-binary-mistral-engine:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> mistral_engine >>>> 23dec12e650f >>>> docker.io/tripleomaster/centos-binary-iscsid:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago iscsid >>>> bafff227d9d7 >>>> docker.io/tripleomaster/centos-binary-haproxy:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago haproxy >>>> 4d17cbd60698 >>>> docker.io/tripleomaster/centos-binary-ironic-inspector:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> ironic_inspector_dnsmasq >>>> f77e49246c35 >>>> docker.io/tripleomaster/centos-binary-ironic-inspector:current-tripleo >>>> kolla_start 7 days ago Up 7 days ago >>>> ironic_inspector >>>> ec70f40bba04 >>>> docker.io/tripleomaster/centos-binary-rabbitmq:current-tripleo >>>> kolla_start 5 weeks ago Up 4 weeks ago rabbitmq >>>> f09c9a129d85 >>>> docker.io/tripleomaster/centos-binary-memcached:current-tripleo >>>> kolla_start 5 weeks ago Up 4 weeks ago >>>> memcached >>>> 0eb1e953dcaa >>>> docker.io/tripleomaster/centos-binary-nova-compute-ironic:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> nova_compute >>>> ac78778d2cb0 >>>> docker.io/tripleomaster/centos-binary-ironic-neutron-agent:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> ironic_neutron_agent >>>> 8660bf80fd9c >>>> docker.io/tripleomaster/centos-binary-ironic-conductor:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> ironic_conductor >>>> 9dead5068168 >>>> docker.io/tripleomaster/centos-binary-mistral-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> mistral_api >>>> f5c1c9d6166c >>>> docker.io/tripleomaster/centos-binary-ironic-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> ironic_api >>>> 48899d782dd5 >>>> docker.io/tripleomaster/centos-binary-nova-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> nova_api_cron >>>> 1b19c94834e8 >>>> docker.io/tripleomaster/centos-binary-nova-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago nova_api >>>> 5b66d128c930 >>>> docker.io/tripleomaster/centos-binary-glance-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> glance_api >>>> 7e63cdb8d6b4 >>>> docker.io/tripleomaster/centos-binary-placement-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> placement_api >>>> 2db105676f8b >>>> docker.io/tripleomaster/centos-binary-zaqar-wsgi:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> zaqar_websocket >>>> d3e5ae7368e6 >>>> docker.io/tripleomaster/centos-binary-zaqar-wsgi:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago zaqar >>>> 6332e26b40ec >>>> docker.io/tripleomaster/centos-binary-nova-scheduler:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> nova_scheduler >>>> a5b0e0904f0c >>>> docker.io/tripleomaster/centos-binary-nova-conductor:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> nova_conductor >>>> 4460302d97c8 >>>> docker.io/tripleomaster/centos-binary-neutron-server:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> neutron_api >>>> d418deb9ea13 >>>> docker.io/tripleomaster/centos-binary-cron:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> logrotate_crond >>>> 7225ff80b26d >>>> docker.io/tripleomaster/centos-binary-heat-engine:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> heat_engine >>>> 84d22b3b1663 >>>> docker.io/tripleomaster/centos-binary-heat-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago >>>> heat_api_cron >>>> 226e0d839772 >>>> docker.io/tripleomaster/centos-binary-heat-api:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago heat_api >>>> 6f7871a64325 >>>> docker.io/tripleomaster/centos-binary-keystone:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago keystone >>>> 8402ff823012 >>>> docker.io/tripleomaster/centos-binary-mariadb:current-tripleo >>>> kolla_start 7 weeks ago Up 4 weeks ago mysql >>>> (undercloud) [stack@remote-u overcloud]$ >>>> >>>

1 0

[Neutron] memchached connections
by Tony Liu 15 Sep '20

15 Sep '20

I restarted neutron-server on all 3 nodes. Those memcached connections from neutron-server are gone. Everything is back to normal. It seems like that memcached connections are not closed properly in neutron-server. The connections pile up over time. Is there any know issue related? Could any Neutron experts comment here? Thanks! Tony > -----Original Message----- > From: Tony Liu <tonyliu0592(a)hotmail.com> > Sent: Friday, September 11, 2020 2:26 PM > To: openstack-discuss <openstack-discuss(a)lists.openstack.org> > Subject: memchached connections > > Hi, > > Is there any guidance or experiences to estimate the number of memcached > connections? > > Here is memcached connection on one of the 3 controllers. > Connection number is the total established connections to all 3 > memcached nodes. > > Node 1: > 10 Keystone workers have 62 connections. > 11 Nova API workers have 37 connections. > 6 Neutron server works have 4304 connections. > 1 memcached has 4973 connections. > > Node 2: > 10 Keystone workers have 62 connections. > 11 Nova API workers have 30 connections. > 6 Neutron server works have 3703 connections. > 1 memcached has 4973 connections. > > Node 3: > 10 Keystone workers have 54 connections. > 11 Nova API workers have 15 connections. > 6 Neutron server works have 6541 connections. > 1 memcached has 4973 connections. > > Before I increase the connection limit for memcached, I'd like to > understand if all the above is expected? > > How Neutron server and memcached take so many connections? > > Any elaboration is appreciated. > > BTW, the problem leading me here is memcached connection timeout, which > results all services depending on memcached stop working properly. > > > Thanks! > Tony

2 3

[tc] weekly update
by Mohammed Naser 15 Sep '20

15 Sep '20

Hi everyone, Here's an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. We've also included a few references to some important mailing list threads that you should check out. # Patches ## Open Reviews - Remove tc:approved-release tag https://review.opendev.org/749363 - Retire the devstack-plugin-zmq project https://review.opendev.org/748731 - Add openstack/osops to Ops Docs and Tooling SIG https://review.opendev.org/749835 - Retire devstack-plugin-pika project https://review.opendev.org/748730 - Reinstate weekly meetings https://review.opendev.org/#/c/749279/ - Resolution to define distributed leadership for projects https://review.opendev.org/744995 - Add assert:supports-standalone https://review.opendev.org/722399 ## Project Updates - Add openstack-helm-deployments to openstack-helm https://review.opendev.org/748302 - Add openstack-ansible/os_senlin role https://review.opendev.org/748677 - kolla-cli: deprecation - Mark as deprecated https://review.opendev.org/749694 ## General Changes - Create starter-kit:kubernetes-in-virt tag https://review.opendev.org/736369 # Other Reminders - PTG Brainstorming: https://etherpad.opendev.org/p/tc-wallaby-ptg Thanks for reading! Mohammed & Kendall -- Mohammed Naser VEXXHOST, Inc.

1 0

[nova][barbican][qa] barbican-tempest-plugin change breaking bfv [ceph]
by Mohammed Naser 15 Sep '20

15 Sep '20

Hi everyone, We just had our gating break due to a change merging inside barbican-tempest-plugin which is the following: https://review.opendev.org/#/c/515210/ It is resulting in an exception in our CI: 2020-08-26 18:04:32.663188 | controller | Response - Headers: {'content-length': '257', 'content-type': 'application/json', 'x-openstack-request-id': 'req-7f55e463-c3de-445e-a814-ef79c5f21235', 'connection': 'close', 'status': '409', 'content-location': 'http://glance.openstack.svc.cluster.local/v2/images/dec14e17-0870-415f-82a6…'} 2020-08-26 18:04:32.663200 | controller | Body: b'{"message": "Image dec14e17-0870-415f-82a6-140c1b7e4a39 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\\n\\n\\n", "code": "409 Conflict", "title": "Conflict"}' This is usually because it's trying to delete the Glance image before deleting an instance that is using it, the specific test that is failing is: barbican_tempest_plugin.tests.scenario.test_certificate_validation.CertificateValidationTest.test_signed_image_invalid_cert_boot_failure[compute,id-6d354881-35a6-4568-94b8-2204bbf67b29,image] This test landed yesterday, we're blacklisting the scenario right now. I do find it quite interesting that in the logs here: http://paste.openstack.org/show/797186/ That the instance reports that it _does_ indeed delete it, so maybe we are trying to delete the image afterwards _too quickly_ and need to wait for Nova to clean up? I'd love to enable that test again and continue full coverage, happy to hear discussion. Thanks Mohammed -- Mohammed Naser VEXXHOST, Inc.

2 1

[ops] picking up some activity
by Chris Morgan 15 Sep '20

15 Sep '20

Greetings! The OpenStack Operators ("ops") meetups team will attempt to have an IRC meeting at the normal time and place (#openstack-operators on freenode at 10am EST/DST) on *Sept 15th*( following a period of complete inactivity for obvious reasons. If you're an official member of the team or even just interested in what we do, please feel free to join us. Whilst we can't yet contemplate resuming in-person meetups during this global pandemic, we can resume attempting to build the openstack operators community, share knowledge and perhaps even do some more virtual get-togethers. See you then Chris on behalf of the openstack ops meetups team -- Chris Morgan <mihalis68(a)gmail.com>

3 2

Need help for stack update feature for scaling out use case or add new resources
by Rahul Kumar 15 Sep '20

15 Sep '20

Hi Team, I have a stack template with 3 nodes , each node having a volume attached ! Now i updated my template with additional node and additional volume for that node ! This results in user_data update of a node in template ! And when i perform stack update feature , it gives me error: Invalid volume: Volume 01e40c6e-4467-42fe-ba9d-ce7012db8978 status must be available or downloading to reserve, but the current status is in-use. This shows for the node (where user_data is changed) and yes it is currently in use ! Then how one can update the stack using stack update feature of openstack with volumes ? Rahul

1 0