October 2021 - openstack-discuss - lists.openstack.org

Octavia loadbalancer status offline
by Md. Hejbul Tawhid MUNNA 26 Oct '21

26 Oct '21

Hi, We have installed openstack ussuri version from ubuntu universe repository. We have installed octavia 6.2.0 version. after creating loadbalancer , listener and pool all are offline. but the LB operation is working as expected. changing the pool member is also working. octavia is installed in compute node. 5555 is listening and allowed in iptables (iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT). amphora to octavia-worker(172.16.0.2) is reachable. Any idea to troubleshoot this issue Please find the log from octavia-worker ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 2021-10-25 18:15:13.482 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540e192be0>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:18.490 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c0ddd00>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:23.507 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c0ddd90>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:28.511 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c0ddbb0>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:33.521 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c064190>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:38.529 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540d111790>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:43.540 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec7a3670>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:48.549 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec74d0d0>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:53.554 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec75a7f0>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:15:58.561 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec75a6a0>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-10-25 18:16:04.707 1192307 INFO octavia.controller.worker.v1.tasks.database_tasks [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ALLOCATED in DB for amphora: c661b828-1690-4866-8152-f745c43e0977 with compute id c9133819-b8e0-42d6-9544-bf83e3ad4b3f for load balancer: c0bd3e21-6983-40c9-8713-859194496b37 2021-10-25 18:16:40.660 1192307 INFO octavia.controller.worker.v1.tasks.database_tasks [req-78e78f29-3bdb-4b12-ae76-b8daa4926c09 - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ACTIVE in DB for load balancer id: c0bd3e21-6983-40c9-8713-859194496b37 2021-10-25 18:16:44.317 1192307 INFO octavia.controller.queue.v1.endpoints [-] Creating listener 'cc45192d-de70-4d59-857b-ac23c4fc8d07'... 2021-10-25 18:16:44.325 1192307 WARNING octavia.controller.worker.v1.controller_worker [-] Failed to fetch listener cc45192d-de70-4d59-857b-ac23c4fc8d07 from DB. Retrying for up to 60 seconds. 2021-10-25 18:17:35.375 1192307 INFO octavia.controller.queue.v1.endpoints [-] Creating pool '9d23855f-d849-4ad9-9de1-66ab5cd268eb'... 2021-10-25 18:17:35.382 1192307 WARNING octavia.controller.worker.v1.controller_worker [-] Failed to fetch pool 9d23855f-d849-4ad9-9de1-66ab5cd268eb from DB. Retrying for up to 60 seconds. 2021-10-25 18:18:02.814 1192307 INFO octavia.controller.queue.v1.endpoints [-] Creating member '29bb41e5-457c-43ba-9149-5af55e73fe38'... 2021-10-25 18:18:02.825 1192307 WARNING octavia.controller.worker.v1.controller_worker [-] Failed to fetch member 29bb41e5-457c-43ba-9149-5af55e73fe38 from DB. Retrying for up to 60 seconds. Regards, Munna

3 4

Re: Openstack magnum
by Vikarna Tathe 26 Oct '21

26 Oct '21

Hi Yasemin, You can find all the stable releases in the below link https://builds.coreos.fedoraproject.org/browser?stream=stable&arch=x86_64 Vikarna On Sun, 24 Oct 2021 at 17:28, Yasemin DEMİRAL (BILGEM BTE) < yasemin.demiral(a)tubitak.gov.tr> wrote: > Hi, > > How can I dowloand fcos 33? I can't find any link for dowloanding it. > > *Yasemin DEMİRAL* > <http://www.tubitak.gov.tr/tr/icerik-sorumluluk-reddi> > > Senior Researcher at TUBITAK BILGEM B3LAB > > Safir Cloud Scrum Master > > > ------------------------------ > *Kimden: *"Vikarna Tathe" <vikarnatathe(a)gmail.com> > *Kime: *"Ammad Syed" <syedammad83(a)gmail.com> > *Kk: *"openstack-discuss" <openstack-discuss(a)lists.openstack.org> > *Gönderilenler: *19 Ekim Salı 2021 16:23:20 > *Konu: *Re: Openstack magnum > > Hi Ammad, > Thanks!!! It worked. > > On Tue, 19 Oct 2021 at 15:00, Vikarna Tathe <vikarnatathe(a)gmail.com> > wrote: > >> Hi Ammad, >> Yes, fcos34. Let me try with fcos33. Thanks >> >> On Tue, 19 Oct 2021 at 14:52, Ammad Syed <syedammad83(a)gmail.com> wrote: >> >>> Hi, >>> >>> Which fcos image you are using ? It looks like you are using fcos 34. >>> Which is currently not supported. Use fcos 33. >>> >>> On Tue, Oct 19, 2021 at 2:16 PM Vikarna Tathe <vikarnatathe(a)gmail.com> >>> wrote: >>> >>>> Hi All, >>>> I was able to login to the instance. I see that kubelet service is in >>>> activating state. When I checked the journalctl, found the below. >>>> >>>> >>>> >>>> >>>> >>>> >>>> *Oct 19 05:18:34 kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: >>>> Started Kubelet via Hyperkube (System Container).Oct 19 05:18:34 >>>> kubernetes-cluster-6cdrblcpckny-master-0 bash[6521]: Error: statfs >>>> /sys/fs/cgroup/systemd: no such file or directoryOct 19 05:18:34 >>>> kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: kubelet.service: Main >>>> process exited, code=exited, status=125/n/aOct 19 05:18:34 >>>> kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: kubelet.service: >>>> Failed with result 'exit-code'.Oct 19 05:18:44 >>>> kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: kubelet.service: >>>> Scheduled restart job, restart counter is at 18.Oct 19 05:18:44 >>>> kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: Stopped Kubelet via >>>> Hyperkube (System Container).* >>>> >>>> Executed the below command to fix this issue. >>>> *mkdir -p /sys/fs/cgroup/systemd* >>>> >>>> >>>> Now I am getiing the below error. Has anybody seen this issue. >>>> >>>> >>>> >>>> *failed to get the kubelet's cgroup: mountpoint for cpu not found. >>>> Kubelet system container metrics may be missing.failed to get the container >>>> runtime's cgroup: failed to get container name for docker process: >>>> mountpoint for cpu not found. failed to run Kubelet: mountpoint for not >>>> found* >>>> >>>> On Mon, 18 Oct 2021 at 14:09, Vikarna Tathe <vikarnatathe(a)gmail.com> >>>> wrote: >>>> >>>>> >>>>>> Hi Ammad, >>>>>> Thanks for responding. >>>>>> >>>>>> Yes the instance is getting created, but i am unable to login >>>>>> though i have generated the keypair. There is no default password for this >>>>>> image to login via console. >>>>>> >>>>>> openstack server list >>>>>> >>>>>> +--------------------------------------+--------------------------------------+---------+------------------------------------+----------------------+----------+ >>>>>> | ID | Name >>>>>> | Status | Networks | Image >>>>>> | Flavor | >>>>>> >>>>>> +--------------------------------------+--------------------------------------+---------+------------------------------------+----------------------+----------+ >>>>>> | cf955a75-8cd2-4f91-a01f-677159b57cb2 | >>>>>> k8s-cluster-01-2nyejxo3hyvb-master-0 | ACTIVE | private1=10.100.0.39, >>>>>> 10.14.20.181 | fedora-coreos-latest | m1.large | >>>>>> >>>>>> >>>>>> ssh -i id_rsa core(a)10.14.20.181 >>>>>> The authenticity of host '10.14.20.181 (10.14.20.181)' can't be >>>>>> established. >>>>>> ECDSA key fingerprint is >>>>>> SHA256:ykEMpwf79/zTMwcELDSI0f66Sxbri56ovGJ+RRwKXDU. >>>>>> Are you sure you want to continue connecting (yes/no/[fingerprint])? >>>>>> yes >>>>>> Warning: Permanently added '10.14.20.181' (ECDSA) to the list of >>>>>> known hosts. >>>>>> core(a)10.14.20.181: Permission denied >>>>>> (publickey,gssapi-keyex,gssapi-with-mic). >>>>>> >>>>>> On Mon, 18 Oct 2021 at 14:02, Ammad Syed <syedammad83(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> Can you check if the master server is deployed as a nova instance ? >>>>>>> if yes, then login to the instance and check cloud-init and heat agent logs >>>>>>> to see the errors. >>>>>>> >>>>>>> Ammad >>>>>>> >>>>>>> On Mon, Oct 18, 2021 at 12:03 PM Vikarna Tathe < >>>>>>> vikarnatathe(a)gmail.com> wrote: >>>>>>> >>>>>>>> Hello All, >>>>>>>> I am trying to create a kubernetes cluster using magnum. Image: >>>>>>>> fedora-coreos. >>>>>>>> >>>>>>>> >>>>>>>> The stack gets stucked in CREATE_IN_PROGRESS. See the output below. >>>>>>>> openstack coe cluster list >>>>>>>> >>>>>>>> +--------------------------------------+----------------+---------+------------+--------------+--------------------+---------------+ >>>>>>>> | uuid | name | keypair | >>>>>>>> node_count | master_count | status | health_status | >>>>>>>> >>>>>>>> +--------------------------------------+----------------+---------+------------+--------------+--------------------+---------------+ >>>>>>>> | 9381cd12-af60-40b1-b1c4-d3dcc93e926e | k8s-cluster-01 | ctrlkey | >>>>>>>> 2 | 1 | CREATE_IN_PROGRESS | None | >>>>>>>> >>>>>>>> +--------------------------------------+----------------+---------+------------+--------------+--------------------+---------------+ >>>>>>>> >>>>>>>> openstack stack resource show k8s-cluster-01-2nyejxo3hyvb >>>>>>>> kube_masters >>>>>>>> >>>>>>>> +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ >>>>>>>> | Field | Value >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> >>>>>>>> +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ >>>>>>>> | attributes | {'refs_map': None, 'removed_rsrc_list': >>>>>>>> [], 'attributes': None, 'refs': None} >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | creation_time | 2021-10-18T06:44:02Z >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | description | >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | links | [{'href': ' >>>>>>>> http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu…', >>>>>>>> 'rel': 'self'}, {'href': ' >>>>>>>> http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu…', >>>>>>>> 'rel': 'stack'}, {'href': ' >>>>>>>> http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu…', >>>>>>>> 'rel': 'nested'}] | >>>>>>>> | logical_resource_id | kube_masters >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | physical_resource_id | 3da2083f-0b2c-4b9d-8df5-8468e0de3028 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | required_by | ['kube_cluster_deploy', >>>>>>>> 'etcd_address_lb_switch', 'api_address_lb_switch', 'kube_cluster_config'] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | resource_name | kube_masters >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | resource_status | CREATE_IN_PROGRESS >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | resource_status_reason | state changed >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | resource_type | OS::Heat::ResourceGroup >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> | updated_time | 2021-10-18T06:44:02Z >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> >>>>>>>> +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ >>>>>>>> >>>>>>>> Vikarna >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Syed Ammad Ali >>>>>>> >>>>>> -- >>> Regards, >>> >>> Syed Ammad Ali >>> >> >

1 0

kolla-ansible wallaby manila ceph pacific
by wodel youchi 26 Oct '21

26 Oct '21

Hi, Has anyone been successful in deploying Manila wallaby using kolla-ansible with ceph pacific as a backend? I have created the manila client in ceph pacific like this : *ceph auth get-or-create client.manila mon 'allow r' mds 'allow r, allow rw path=/' osd 'allow rw pool=cephfs_data' mgr 'allow rw'* When I deploy, I get this error in manila's log file : Bad target type 'mon-mgr' Any ideas? Regards.

3 7

[neutron] October 2021 - PTG Summary
by Lajos Katona 25 Oct '21

25 Oct '21

Hi, Below is my summary of the Neutron sessions during last week's PTG. Here is the etherpad with the agenda: https://etherpad.opendev.org/p/neutron-yoga-ptg # Day 1 (Monday Oct 18.) ## Xena retrospective (I just list here the topics, not every discussion under them): * Good things: ** Less meetings ** New features ** CI jobs have again meaningful names ** Housekeeping - new payload style callbacks switch finished ** Healthy team - we have people from various companies, doing reviews and proposing patches * Bad / not so good / change it ** CI stability could be better ** lack of people involved in the CI meetings and CI issues ** is there a chance to decrease the number of jobs/backends to be * Actions coming out from the above: ** Having video based team meetings every few weeks. ** Video meeting with screen sharing for CI meeting every few weeks. ** ping people before the meetings ## Should we move some parts of neutron devstack plugin to devstack repo? Slaweq brought this topic to rationalize devstack plugin code between neutron and devstack. As a rule of thumb: "what is used/tested by jobs from other repositories (like tempest) should be in devstack repo and not in neutron's devstack plugin". The bad part is that we should loose review velocity if there's nobody to review in devstack. As a consequence some code should be moved back from devstack to neutron's devstack plugin (for drivers that are not that common now like linuxbridge) ## ML2/OVS -> ML2/OVN migration - presentation (continued on Friday) We got an overview from Jakub (jlibosva) of how the migration tool works now, what is finished and which are the possible directions to improve it. https://docs.google.com/presentation/d/1dCdvoi-Cdbl27AL0wjFJ_51NatOxepNOplm… * Problematic parts: ** steps that require the tool to touch the db (no API currently) ** no rollback ** no CI (at least upstream) ** no devstack based tools (related to the previous one) * Actions ** New job for ovs-to-ovn migration *** Now start with tripleo based job and go for a devstack based one ** Don't create new APIs for job type change and other operations during the migration as it seems there is no other usecase for them. ** Add developer documentation for the migration (preparation, steps, rollback....) ** add automatic rollback # Day 2 (Tuesday Oct. 19.) ## tempest_neutron_plugin - a RedHat downstream tests We got an overview of these tests from Eduardo Olivares. * tests requiring advance images ** move them to neutron-tempest-plugin with special tag and run them periodically ** use ubuntu cloud minimal image * tests requiring multinode topology + capturing traffic on hosts/computes ** These tests use a node to run tempest (tripleo based deployment) ** these tests should be changed to run in upstream CI * Tests that restart nodes / services / containers ** better not to run them in upstream CI ** check if it is possible to migrate some to tobiko ( https://opendev.org/x/tobiko ) * sriov tests (macvtap, direct and direct-physical ports) ** No possibility today to run these upstream. ## Edge session (https://etherpad.opendev.org/p/ecg-ptg-october-2021 ) * Thursday we had a session together with the Designate team where we discussed possible action items related to edge use cases. ## RBAC together with Nova * https://etherpad.opendev.org/p/nova-yoga-ptg l108. - l137. # Day 3 (Wednesday Oct. 20.) ## Job reorganization recap * We need fresh data to see where we still have pain points. * Ongoing CI rationalization items: ** Use Ubuntu minimal image where possible. ** Serialize the run of tests with advanced images (reduce memory usage): *** https://review.opendev.org/c/openstack/tempest/+/807790 ## ryu / os-ken Recently we realized that ryu has some activity to fix bugs that can hit Neutron as well (as we forked ryu as os-ken). * regularly checking ryu bugs and fixes (sync on team meeting) * Do not return to ryu as os-ken is adopted for Openstack workflow with CI, etc... ## Nova-Neutron cross-project session * OVN live-migration (https://etherpad.opendev.org/p/ovn_live_migration ) * RBAC continued (https://etherpad.opendev.org/p/nova-yoga-ptg l144 - l165) * libvirt virt driver does not wait for network-vif-plugged event during hard reboot ** document which backend sends what event and when ** workaround in nova for telling whether we should wait for an event ( https://review.opendev.org/c/openstack/nova/+/813419 ) ** long-term solution has to be Neutron providing event information thru the port binding information ** Improved testing of move operations *** Nova will keep one ML2/OVS multinode job running ** Supporting ports with QoS min bw and min pps in multi segment networks with multiple physnets *** finish the ongoing QoS min pps support *** extend placement to support any-trait queries ## Interop with Martin Kopec * Check which tests should be included to refstack * https://etherpad.opendev.org/p/refstack-test-analysis ## Quick introduction of the Tobiko test tool We got an overview of Tobiko from Frederico Ressi, and how it can be used to test scenarios which are destructive. * https://tobiko.readthedocs.io/en/master/ * http://kaplonski.pl/files/Tobiko_-_OpenStack_upgrade_and_disruptive_testing… * Neutron already has a periodic job with tobiko. # Day 4 (Thursday Oct. 21.) ## OVN and BGP * https://opendev.org/x/ovn-bgp-agent ** not just for neutron-dynamic-routing but for networking-bgpvpn as well. ** ovn-bgp-agent is another backend which could be behind neutron-dynamic-routing, and uses FRR. ** In the future it will be a Neutron stadium project ** need more tests, integration with Neutron and better documentation. * Make neutron-dynamic-routing work with OVN ** Frickler and bcafarel checked if the current jobs and tests can work with OVN. ** more investigation is necessary if it works as is now or some change is needed. ** Redhat and Ericsson help in the maintenance of neutron-dynamic-routing. ## Multi segments and provider networks on a host, with 1 subnet per segment * different from the usual routed network use case where we have one segment available in each rack * WIP patch which was never finished: ** https://review.opendev.org/c/openstack/neutron/+/623115 ## Use openstacksdk as client for Heat * Now Heat uses python-neutronclient for python bindings / client (not the CLI part which is deprecated but the python bindings) ** We collected a buch of projects over than heat which uses python-neutronclient: Nova, Horizon, Rally.... ** Actions: *** Write mail about it to have the attention of the community *** Have a migration plan and document it. *** Have a slot during the team meeting to track progress. ## Designate session The topics coming from Designate team were really helpful to translate the requirements from edge session to action items for Neutron. * https://etherpad.opendev.org/p/octavia-designate-neutron-ptg ** dns resolver is not specified for a subnet, (RFE for OVN as it works only for OVS currently) ** initiate a cross-project documentation effort for setting up an edge site ** initiate cross-project job for testing an edge site # Day 5 (Friday Oct. 22.) ## Continued discussion of neutron-dynamic-routing's future * Who will review new features (Both Redhat and Ericsson will increase review attention for this project) * Fixing the scheduler: ** New scheduler whith which the user can select which host/dragent to schedule ** https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/780675 * OVN and neutron-dynamic-routing ** add the missing features to the OVN gap list ( https://docs.openstack.org/neutron/latest/ovn/gaps.html ) ## RBAC discussion continued with TC * https://etherpad.opendev.org/p/tc-yoga-ptg ## ML2/OVS -> ML2/OVN migration - continue We continued the discussion from Monday, on how to integrate the ovs-ovn migration tool to Neutron CI. ## Team photo: https://photos.app.goo.gl/8ERv59XYoFVQz5oT7 I would like to thank you for the great discussions during last week, it was really useful and productive week. Based on these we have some plan what we will work on in the Yoga cycle. Lajos Katona (lajoskatona) Ericsson Software Technologies

1 0

CentOS-9 guests & 'qemu64' CPU model are incompatible; and reasons to avoid 'qemu64' in general
by Kashyap Chamarthy 25 Oct '21

25 Oct '21

Summary ------- RHEL-9 / CentOS-9 (but not Fedora) has switched[1] to a new baseline microarchitecture called "x86-64-v2". This is to bring in support for additioal low-level CPU instructions, among other reasons. Now, if you've explicitly configured "cpu_mode=none" in `nova.conf` on your compute nodes — which results in the guest getting the extremely undesirable "qemu64" CPU model — it will refuse to boot RHEL-9 or CentOS-9 guests. To fix this, please update the CPU model to "Nehalem". It is the oldest CPU model that is compatible with CentOS-9/RHEL-9 "x86-64-v2". Further, Nehalem also works with `virt_type=kvm|qemu`, _and_ on both Intel and AMD hardware. So this is a good alternative. Details ------- Nova has three config attributes to setup various aspect of a guest CPU: `cpu_mode`, `cpu_model[s]`, and `cpu_model_extra_flags`. Examples of how to use these are in the documentation[2]. If you're using `cpu_mode = none` (e.g. upstream DevStack defalts to it for understandable reasons, mainly live-migration compatiblity): [libvirt] cpu_mode = none ... and want to boot CentOS-9, replace the above with the custom model, "Nehalem", which is the oldest CPU model that's compatible with the new x86-64-v2 baseline: [libvirt] cpu_mode = custom cpu_model = Nehalem The same applies if you're using "qemu64" or "kvm64" with, or without any custom CPU flags — i.e. use Nehalem. (Also, please refer to[3] for more fine-grained recommendations of guest CPU configuration. It's a long document, but a patient reader will be rewarded.) Why is "qemu64" model undesirable for production? ------------------------------------------------- For those wondering about it, a few reasons why `qemu64` CPU model is not at all desirable: (1) It is vulnerable to many of the Spectre and other side-channel security flaws. To see this in "action", you can launch a guest with 'qemu64' CPU model, and then run the below: $ cd /sys/devices/system/cpu/vulnerabilities/ $ grep . * l1tf:Mitigation: PTE Inversion mds:Vulnerable: ... no microcode; SMT Host state unknown meltdown:Mitigation: PTI spec_store_bypass:Vulnerable spectre_v1:Mitigation: usercopy/swapgs barriers ... spectre_v2:Mitigation: Full generic retpoline ... Notice the "Vulnerable" entries. (2) "qemu64" does not support several critical CPU features: (a) AES (Advanced Encryption Standard) instruction, which is important for imporved TLS performance and encryption. (b) RDRAND instruction: without this, guests can get starved for entropy. (c) PCID flag: an obscure-but-important flag that'll lower the performance degradation that you incur from the "Meltdown" security fixes. Probably there are more reasons that I don't know of. An understandable reason why CI systems running in a cloud environment go with 'qemu64' is convenience: with 'qemu64', you can live-migrate a guest regardless of its underlying hardware (whether it's Intel or AMD). That's one main reason why upstream DevStack defaults to it. * * * Overall, the thumb-rule here is to either always explicitly specify a "sane" CPU model, based on the recommendations here[3]. Or to use Nova/libvirt's default ("host-model"). [1] https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-l… [2] https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cp… [3] https://www.qemu.org/docs/master/system/i386/cpu.html#recommendations-for-k… [4] https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zu… -- /kashyap

2 2

[cinder] reminder: this week's meeting in video+IRC
by Brian Rosmaita 25 Oct '21

25 Oct '21

Quick reminder that this week's Cinder team meeting on Wednesday 27 October, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. These are the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. * The meeting will be recorded. connection info: https://bluejeans.com/3228528973 meeting agenda: https://etherpad.opendev.org/p/cinder-yoga-meetings cheers, brian

1 0

[magnum] [fcos33]
by Yasemin DEMİRAL (BILGEM BTE) 25 Oct '21

25 Oct '21

Hi, How can I dowloand fcos 33? I can't find any link for dowloanding it. Yasemin DEMİRAL [ http://www.tubitak.gov.tr/tr/icerik-sorumluluk-reddi ] Senior Researcher at TUBITAK BILGEM B3LAB Safir Cloud Scrum Master Kimden: "Vikarna Tathe" <vikarnatathe(a)gmail.com> Kime: "Ammad Syed" <syedammad83(a)gmail.com> Kk: "openstack-discuss" <openstack-discuss(a)lists.openstack.org> Gönderilenler: 19 Ekim Salı 2021 16:23:20 Konu: Re: Openstack magnum Hi Ammad, Thanks!!! It worked. On Tue, 19 Oct 2021 at 15:00, Vikarna Tathe < [ mailto:vikarnatathe@gmail.com | vikarnatathe(a)gmail.com ] > wrote: Hi Ammad, Yes, fcos34. Let me try with fcos33. Thanks On Tue, 19 Oct 2021 at 14:52, Ammad Syed < [ mailto:syedammad83@gmail.com | syedammad83(a)gmail.com ] > wrote: BQ_BEGIN Hi, Which fcos image you are using ? It looks like you are using fcos 34. Which is currently not supported. Use fcos 33. On Tue, Oct 19, 2021 at 2:16 PM Vikarna Tathe < [ mailto:vikarnatathe@gmail.com | vikarnatathe(a)gmail.com ] > wrote: BQ_BEGIN Hi All, I was able to login to the instance. I see that kubelet service is in activating state. When I checked the journalctl, found the below. Oct 19 05:18:34 kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: Started Kubelet via Hyperkube (System Container). Oct 19 05:18:34 kubernetes-cluster-6cdrblcpckny-master-0 bash[6521]: Error: statfs /sys/fs/cgroup/systemd: no such file or directory Oct 19 05:18:34 kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: kubelet.service: Main process exited, code=exited, status=125/n/a Oct 19 05:18:34 kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: kubelet.service: Failed with result 'exit-code'. Oct 19 05:18:44 kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 18. Oct 19 05:18:44 kubernetes-cluster-6cdrblcpckny-master-0 systemd[1]: Stopped Kubelet via Hyperkube (System Container). Executed the below command to fix this issue. mkdir -p /sys/fs/cgroup/systemd Now I am getiing the below error. Has anybody seen this issue. failed to get the kubelet's cgroup: mountpoint for cpu not found. Kubelet system container metrics may be missing. failed to get the container runtime's cgroup: failed to get container name for docker process: mountpoint for cpu not found. failed to run Kubelet: mountpoint for not found On Mon, 18 Oct 2021 at 14:09, Vikarna Tathe < [ mailto:vikarnatathe@gmail.com | vikarnatathe(a)gmail.com ] > wrote: BQ_BEGIN BQ_BEGIN Hi Ammad, Thanks for responding. Yes the instance is getting created, but i am unable to login though i have generated the keypair. There is no default password for this image to login via console. openstack server list +--------------------------------------+--------------------------------------+---------+------------------------------------+----------------------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------------------------------+---------+------------------------------------+----------------------+----------+ | cf955a75-8cd2-4f91-a01f-677159b57cb2 | k8s-cluster-01-2nyejxo3hyvb-master-0 | ACTIVE | private1=10.100.0.39, 10.14.20.181 | fedora-coreos-latest | m1.large | ssh -i id_rsa [ mailto:core@10.14.20.181 | core(a)10.14.20.181 ] The authenticity of host '10.14.20.181 (10.14.20.181)' can't be established. ECDSA key fingerprint is SHA256:ykEMpwf79/zTMwcELDSI0f66Sxbri56ovGJ+RRwKXDU. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '10.14.20.181' (ECDSA) to the list of known hosts. [ mailto:core@10.14.20.181 | core(a)10.14.20.181 ] : Permission denied (publickey,gssapi-keyex,gssapi-with-mic). On Mon, 18 Oct 2021 at 14:02, Ammad Syed < [ mailto:syedammad83@gmail.com | syedammad83(a)gmail.com ] > wrote: BQ_BEGIN Hi, Can you check if the master server is deployed as a nova instance ? if yes, then login to the instance and check cloud-init and heat agent logs to see the errors. Ammad On Mon, Oct 18, 2021 at 12:03 PM Vikarna Tathe < [ mailto:vikarnatathe@gmail.com | vikarnatathe(a)gmail.com ] > wrote: BQ_BEGIN Hello All, I am trying to create a kubernetes cluster using magnum. Image: fedora-coreos. The stack gets stucked in CREATE_IN_PROGRESS. See the output below. openstack coe cluster list +--------------------------------------+----------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+----------------+---------+------------+--------------+--------------------+---------------+ | 9381cd12-af60-40b1-b1c4-d3dcc93e926e | k8s-cluster-01 | ctrlkey | 2 | 1 | CREATE_IN_PROGRESS | None | +--------------------------------------+----------------+---------+------------+--------------+--------------------+---------------+ openstack stack resource show k8s-cluster-01-2nyejxo3hyvb kube_masters +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | {'refs_map': None, 'removed_rsrc_list': [], 'attributes': None, 'refs': None} | | creation_time | 2021-10-18T06:44:02Z | | description | | | links | [{'href': ' [ http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu… | http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu… ] ', 'rel': 'self'}, {'href': ' [ http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu… | http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu… ] ', 'rel': 'stack'}, {'href': ' [ http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu… | http://10.14.20.159:8004/v1/f453bf8a2afa4fbba30030738f4dd216/stacks/k8s-clu… ] ', 'rel': 'nested'}] | | logical_resource_id | kube_masters | | physical_resource_id | 3da2083f-0b2c-4b9d-8df5-8468e0de3028 | | required_by | ['kube_cluster_deploy', 'etcd_address_lb_switch', 'api_address_lb_switch', 'kube_cluster_config'] | | resource_name | kube_masters | | resource_status | CREATE_IN_PROGRESS | | resource_status_reason | state changed | | resource_type | OS::Heat::ResourceGroup | | updated_time | 2021-10-18T06:44:02Z | +------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Vikarna -- Regards, Syed Ammad Ali BQ_END BQ_END BQ_END BQ_END -- Regards, Syed Ammad Ali BQ_END BQ_END

2 3

[Neutron] Bug Deputy Report October 19 - 23
by Jakub Libosvar 25 Oct '21

25 Oct '21

Hi all, I was the bug deputy for the week starting Oct 19. Here is the report, no critical bugs, it was a quiet week, maybe because of the PTG :) Medium ------ * "network_namespace_exists"can timeout in loaded systems Link: https://bugs.launchpad.net/neutron/+bug/1947974 Assigned to Rodolfo Fix proposed: https://review.opendev.org/c/openstack/neutron/+/814868 * Non HA router - missing iptables rule for redirect metadata queries to haproxy Link: https://bugs.launchpad.net/neutron/+bug/1947993 Assigned to Slawek Fix proposed: https://review.opendev.org/c/openstack/neutron/+/814892 * [OVN] Logical router port must have at least one network Edit Link: https://bugs.launchpad.net/neutron/+bug/1948457 Needs an assignee Low --- * qvo ports are not removed correctly when an instance is deleted immediately after creation Link: https://bugs.launchpad.net/neutron/+bug/1948452 Needs an assignee * [OVN] Mech driver fails to delete DHCP options during subnet deletion Link: https://bugs.launchpad.net/neutron/+bug/1948466 Needs an assignee Kuba

2 1

[Glance] Yoga PTG Summary
by Abhishek Kekane 25 Oct '21

25 Oct '21

Hi All, We had our fourth virtual PTG between 18th October to 22nd October 2020. Thanks to everyone who joined the virtual PTG sessions. Using bluejeans app we had lots of discussion around different topics for glance, glance + cinder and Secure RBAC. I have created etherpad [1] with Notes from the session and which also includes the recordings of each discussion. Here is a short summary of the discussions. Tuesday, October 19th # Xena Retrospective On the positive note, we merged a number of useful features this cycle. We managed to implement a project scope of secure RBAC for metadef APIs, Implemented quotas using unified limits and moved policy enforcing closer to API layer. We also manage to wipe out many bugs from our bug backlog. On the other side we need to improve our documentation and API reference guide. Recording: https://bluejeans.com/s/QuCBmIrMuVv - Chapter 1 # Cache API During Xena cycle we managed to start this work and implement the core functionality but failed to merge it due to lack of negative tests and tempest coverage. In the Yoga cycle we are going to focus on adding tempest coverage for the same along with a new API to cache the given image immediately rather than waiting for a periodic job to pre-cache it for us. Recordings: https://bluejeans.com/s/QuCBmIrMuVv - Chapter 2 # Native Image Encryption Unfortunately this topic is sitting in our agenda for the last couple of PTGs. Current update is the core feature is depending on Barbican microversion work and once it is complete then the Consumer API can be functional again. At the moment in Glance we have decided to go ahead and implement the glance side part and instead of having placeholder (barbican consumer API secret register and deletion) code as commented we can have a WIP patch for the same with depending on glance side work. Recordings: https://bluejeans.com/s/QuCBmIrMuVv - Chapter 3 # Default Glance to configure multiple stores Glance has deprecated single stores configuration since Stein cycle and now will start putting efforts to deploy glance using multistore by-default and then remove single store support from glance. This might be going to take a couple of cycles, so in yoga we are going to migrate internal unit and functional tests to use multistore config and also going to modify devstack to deploy glance using multistore configuration for swift and Ceph (for file and cinder it's already supported). Recordings: https://bluejeans.com/s/QuCBmIrMuVv - Chapter 4 # Quotas Usage API In Xena we have implemented quotas for images API using unified limits. This cycle we will add new APIs which will help enduser to get the clear picture of quotas like What is total quota, used quota and remaining quota. So first we are coming up with the spec for the design and then implementation for the same. Recordings: https://bluejeans.com/s/QuCBmIrMuVv - Chapter 5 Wednesday, October 20th 2021 # Policy Refactoring - Part 2 In Xena we have managed to move all policy checks to the API layer. This cycle we need to work on removing dead code of policy and authorization layer. Before removing both the layers we need to make sure that property protection is working as expected and for the same we need to add one job with a POST script to verify that removing auth and policy layer will not break property protection. Recordings: https://bluejeans.com/s/AMZzGObPhK4 - Chapter 1 # Database read only checks We have added new RBAC policy checks which are equivalent to readonly checks in our db layer, e.g. image ownership check, visibility check etc. To kick start work for this dansmith (thanks for volunteering) will work on PoC and abhishekk will work on specs about how we will modify/improve our db layer. # Secure RBAC - System Scope/Project Admin scope In Xena we have managed to move all policy checks to API layer and implemented project scope of metadef APIs. So as of now we have project scope for all glance APIs. During this discussion Security Team has updated us that discussions are still going about how the system scope should be used/implemented and they are planning to introduce a new role 'manager' which will act between 'admin' and 'member' roles. We need to keep an eye on this new development. https://etherpad.opendev.org/p/tc-yoga-ptg - line #446 https://etherpad.opendev.org/p/policy-popup-yoga-ptg - line #122 Recordings: https://bluejeans.com/s/AMZzGObPhK4 - Chapter 3 Thursday, October 21st 2021 # Glance- Interop interlock Due to confusion between timings this discussion didn't happen as planned. The InterOP team has added some questions later to PTG etherpad (refer line no #270). @Glance team please respond to those questions as I will be out for the next couple of weeks. # Upload volume to image in RBD backend In case when we upload a volume as an image to glance's rbd backend, it starts with a 0 size rbd image and performs resize in chunks of 8 MB which makes the operation very slow. Current idea is to pass volume size as image size to avoid these resize operations. As this change will lead to use locations API which are not advisable to use in glance due to security concerns and also checksum and multi-hash will not be available, cinder side will have new config option (default False) to use this optimization if set to True. Also this change needs a glance side to expose ceph pool information, so Rajat (whoami-rajat) will coordinate with the glance team to set up a spec and implement the same. # Upload volume to image when glance backend is cinder Similar to above topic this will have same security concerns and same config option can be used to have this optimization available. Rajat will coordinate with the glance team to implement glance side changes. Note, as we have joined the cinder team, Brian Rosmaita/Rajat will update/share the recording links for above two sessions in glance PTG etherpad once it is available. # Adding multiple tags overrides existing tags We are going to modify the create multiple tags API which will have one boolean parameter (Default to False) in the header to maintain backward compatibility. If it is True then we are going to add new tags to the existing list of tags rather than replacing those. Recordings: https://bluejeans.com/s/@ttsNs8vIFq - Chapter 1 # Delete newly created metadef resource types from DB after deassociating To maintain the consistency with other metadef APIs, we are going to add two new APIs. 1. Create resource types 2. Delete given resource type Recordings: https://bluejeans.com/s/@ttsNs8vIFq - Chapter 2 You will find the detailed information about the same in the PTG etherpad [1] along with the recordings of the sessions. Kindly let me know if you have any questions about the same. [1] https://etherpad.opendev.org/p/yoga-glance-ptg Thank you, Abhishek

2 1

[cyborg][ptg] Meeting time changes
by Brin Zhang(张百林) 25 Oct '21

25 Oct '21

Hi all, At PTG, base the contributors of Cyborg, we discussed the meeting time of Cyborg. By voting on the meeting time on PTG, everyone agreed to adjust the meeting time to Friday 6:00UTC-7:00UTC (Beijing time at China, 2:00-3:00pm). We look forward to your participation in our conference, and you can submit any requirements to us. You can register for the feature on Launchpad [1] at first. Later I will send a summary of our discussion on Cyborg PTG. [1] https://blueprints.launchpad.net/openstack-cyborg brinzhang

1 0