October 2021 - openstack-discuss - lists.openstack.org

[cinder] festival of XS reviews 15 October 2021
by Brian Rosmaita 13 Oct '21

13 Oct '21

Hello Cinder community members, This is a reminder that the most recent edition of the Cinder Festival of XS Reviews will be held at the end of this week on Friday 15 October. who: Everyone! what: The Cinder Festival of XS Reviews when: Friday 15 October 2021 from 1400-1600 UTC where: https://meetpad.opendev.org/cinder-festival-of-reviews This recurring meeting can be placed on your calendar by using this handy ICS file: http://eavesdrop.openstack.org/calendars/cinder-festival-of-reviews.ics See you there! brian

1 0

[nova][placement] No meeting next week
by Sylvain Bauza 13 Oct '21

13 Oct '21

As agreed during yesterday's meeting [1], Tuesday Oct 19th's meeting is *CANCELLED* as all of us will be attending the virtual PTG. I'm more than happy tho to see all of you next week thru video ! I'll add the PTG connection details in https://etherpad.opendev.org/p/nova-yoga-ptg -Sylvain [1] https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-12-16.00.log.h…

1 0

[tripleo] Unable to execute pre-introspection and pre-deployment command
by Anirudh Gupta 13 Oct '21

13 Oct '21

3 6

[docs] has_project_guide key
by Peter Matulis 13 Oct '21

13 Oct '21

Does the has_project_guide key in the www/project-data/<RELEASE>.yaml file have any meaning? One of my projects has been dragging this key along from release to release and I do not see it documented [1]. I want to avoid unintended results if that key is removed. Thanks. [1]: https://docs.openstack.org/doc-contrib-guide/doc-tools/template-generator.h… Peter

1 0

[cinder] Bug deputy report for week of 10-13-2021
by Sofia Enriquez 13 Oct '21

13 Oct '21

This is a bug report from 10-06-2021-15-09 to 10-13-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1946736 'Powerflex driver: update supported storage versions'. - https://bugs.launchpad.net/cinder/+bug/1946350 '[gate-failure] nova-live-migration evacuation failure due to slow lvchange -a command in c-vol during volume attachment update'. - https://bugs.launchpad.net/cinder/+bug/1946340 '[gate-failure] Unable to stack on fedora34 due to cinder pulling in oslo.vmware and suds-jurko that uses use_2to3 that is invalid with setuptools >=58.0.0'. - https://bugs.launchpad.net/cinder/+bug/1946263 'NetApp ONTAP Failing migrating volume from/to FlexGroup pool'. Low - https://bugs.launchpad.net/cinder/+bug/1946618 'Add same volume to the group-update does not show proper error'. Wishlist - https://bugs.launchpad.net/cinder/+bug/1946645 '[doc] Install and configure a storage node in cinder'. Cheers -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[KEYSTONE][POLICIES] - Overrides that don't work?
by Gaël THEROND 13 Oct '21

13 Oct '21

Hi team, I'm having a weird behavior with my Openstack platform that makes me think I may have misunderstood some mechanisms on the way policies are working and especially the overriding. So, long story short, I've few services that get custom policies such as glance that behave as expected, Keystone's one aren't. All in all, here is what I'm understanding of the mechanism: This is the keystone policy that I'm looking to override: https://paste.openstack.org/show/bwuF6jFISscRllWdUURL/ This policy default can be found in here: https://opendev.org/openstack/keystone/src/branch/master/keystone/common/po… Here is the policy that I'm testing: https://paste.openstack.org/show/bHQ0PXvOro4lXNTlxlie/ I know, this policy isn't taking care of the admin role but it's not the point. >From my understanding, any user with the project-manager role should be able to add any available user on any available group as long as the project-manager domain is the same as the target. However, when I'm doing that, keystone complains that I'm not authorized to do so because the user token scope is 'PROJECT' where it should be 'SYSTEM' or 'DOMAIN'. Now, I wouldn't be surprised of that message being thrown out with the default policy as it's stated on the code with the following: https://opendev.org/openstack/keystone/src/branch/stable/ussuri/keystone/co… So the question is, if the custom policy doesn't override the default scope_types how am I supposed to make it work? I hope it was clear enough, but if not, feel free to ask me for more information. PS: I've tried to assign this role with a domain scope to my user and I've still the same issue. Thanks a lot everyone!

2 4

[kolla-ansible][neutron] configuration question
by Ignazio Cassano 13 Oct '21

13 Oct '21

Hello Everyone, I need to know if it possible configure kolla neutron ovs with more than one bridge mappings, for example: bridge_mappings = physnet1:br-ex,physnet2:br-ex2 I figure out that in standard configuration ansible playbook create br-ex and add the interface with variable "neutron_external_interface" under br-ex. What can I do if I need to do if I wand more than one bridge ? How kolla ansible playbook can help in this case ? I could use multiple bridges in /etc/kolla/config neutron configuration files, but I do not know how ansible playbook can do the job. because I do not see any variable can help me in /etc/kolla/globals.yml Thanks Ignazio

2 3

Re: [neutron] OVN and dynamic routing
by Francois 13 Oct '21

13 Oct '21

...forgot to add the mailing list in the reply On Wed, 13 Oct 2021 at 10:01, Francois <rigault.francois(a)gmail.com> wrote: > > On Tue, 12 Oct 2021 at 21:46, Dan Sneddon <dsneddon(a)redhat.com> wrote: > > > > On 10/12/21 07:03, Francois wrote: > > > Hello Neutron! > > > I am looking into running stacks with OVN on a leaf-spine network, and > > > have some floating IPs routed between racks. > > > > > > Basically each rack is assigned its own set of subnets. > > > Some VLANs are stretched across all racks: the provisioning VLAN used > > > by tripleo to deploy the stack, and the VLANs for the controllers API > > > IPs. However, each tenant subnet is local to a rack: for example each > > > OVN chassis has a ovn-encap-ip with an IP of the tenant subnet of its > > > own rack. Traffic between 2 racks is sent to a spine, and leaves and > > > spines run some eVPN-like thing: each pair of ToR is a vtep, traffic > > > is encapsulated as VXLAN, and routes between vteps are exchanged with > > > BGP. > > > > > > > There has been a lot of work put into TripleO to allow you to provision > > hosts across L3 boundaries using DHCP relay. You can create a routed > > provisioning network using "helper-address" or vendor-specific commands > > on your top-of-rack switches, and a different subnet and DHCP address > > pool per rack. > Yes I saw that in the doc. I was not planning on using this for reasons I mentioned in another reply (this provisioning network is ""useless most of the time"" since there is almost no provisioning happening :D ) If any, I would love to work on Ironic DHCP-less deployments which was almost working last time I tried and I saw Ironic team contributing fixes since then. > > > > > > I am looking into supporting floating IPs in there: I expect floating > > > IPs to be able to move between racks, as such I am looking into > > > publishing the route for a FIP towards an hypervisor, through BGP. > > > Each fip is a /32 subnet with an hypervisor tenant's IP as next hop. > > > > This is becoming a very common architecture, and that is why there are > > several projects working to achieve the same goal with slightly > > different implementations. > > > > > > > > It seems there are several ideas to achieve this (it was discussed > > > [before][1] in ovs conference) > > > - using [neutron-dynamic-routing][2] - that seems to have some gaps > > > for OVN. It uses os-ken to talk to switches and exchange routes > > > - using [OVN BGP agent][3] that uses FRR, it seems there is a related > > > [RFE][4] for integration in tripleo > > > > > > There is btw also a [BGPVPN][5] project (it does not match my usecase > > > as far as I tried to understand it) that also has some code that talks > > > BGP to switches, already integrated in tripleo. > > > > > > For my tests, I was able to use the neutron-dynamic-routing project > > > (almost) as documented, with a few changes: > > > - for traffic going from VMs to outside the stack, the hypervisor was > > > trying to resolve the "gateway of fIPs" with ARP request which does > > > not make any sense. I created a dummy port with the mac address of the > > > virtual router of the switches: > > > ``` > > > $ openstack port list --mac-address 00:1c:73:00:00:11 -f yaml > > > - Fixed IP Addresses: > > > - ip_address: 10.64.254.1 > > > subnet_id: 8f37 > > > ID: 4028 > > > MAC Address: 00:1c:73:00:00:11 > > > Name: lagw > > > Status: DOWN > > > ``` > > > this prevent the hypervisor to send ARP requests to a non existent gateway > > > - for traffic coming back, we start the neutron-bgp-dragent agent on > > > the controllers. We create the right bgp speaker, peers, etc. > > > - neutron-bgp-dragent seems to work primarily with ovs ml2 plugin, it > > > selects fips and join with ports owned by a "floatingip_agent_gateway" > > > which does not exist on OVN. We can define ourselves some ports so > > > that the dragent is able to find the tenant IP of a host: > > > ``` > > > openstack port create --network provider --device-owner > > > network:floatingip_agent_gateway --host cpu35d.cloud --fixed-ip > > > ip-address=10.64.245.102 ag2 > > > ``` > > > - when creating a floating IP and assigning a port to it, Neutron > > > reads changes from OVN SB and fills the binding information into the > > > port: > > > ``` > > > $ openstack port show -c binding_host_id `openstack floating ip show > > > 10.64.254.177 -f value -c port_id` > > > +-----------------+----------------------------------------+ > > > | Field | Value | > > > +-----------------+----------------------------------------+ > > > | binding_host_id | cpu35d.cloud | > > > +-----------------+----------------------------------------+ > > > ``` > > > this allows the dragent to publish the route for the fip > > > ``` > > > $ openstack bgp speaker list advertised routes bgpspeaker > > > +------------------+---------------+ > > > | Destination | Nexthop | > > > +------------------+---------------+ > > > | 10.64.254.177/32 | 10.64.245.102 | > > > +------------------+---------------+ > > > ``` > > > - traffic reaches the hypervisor but (for reason I don't understand) I > > > had to add a rule > > > ``` > > > $ ip rule > > > 0: from all lookup local > > > 32765: from all iif vlan1234 lookup ovn > > > 32766: from all lookup main > > > 32767: from all lookup default > > > $ ip route show table ovn > > > 10.64.254.177 dev vlan1234 scope link > > > ``` > > > so that the traffic coming for the fip is not immediately discarded by > > > the hypervisor (it's not an ideal solution but it is a workaround that > > > makes my one fIP work!) > > > > > > So all in all it seems it would be possible to use the > > > neutron-dynamic-routing agent, with some minor modifications (eg: to > > > also publish the fip of the OVN L3 gateway router). > > > > > > I am wondering whether I have overlooked anything, and if such kind of > > > deployment (OVN + neutron dynamic routing or similar) is already in > > > use somewhere. Does it make sense to have a RFE for better integration > > > between OVN and neutron-dynamic-routing? > > > > I have been helping to contribute to integrating FRR with OVN in order > > to advertise FIPs and provider network IPs into BGP. The OVN BGP Agent > > is very new, and I'm pretty sure that nobody is using it in production > > yet. However the initial implementation is fairly simple and hopefully > > it will mature quickly. > > > > As you discovered, the solution that uses neutron-bgp-dragent and os-ken > > is not compatible with OVN > Pretty much the contrary, it basically worked. There are a few differences but the gap seems very tiny (unless I overlooked something and I'm fundamentally wrong) I don't understand why a new project would be needed to make it work for OVN. > > >, that is why ovs-bgp-agent is being > > developed. You should be able to try the ovs-bgp-agent with FRR and > > properly configured routing switches, it functions for the basic use case. > > > > The OVN BGP Agent will ensure that FIP and provider network IPs are > > present in the kernel as a /32 or /128 host route, which is then > > advertised into the BGP fabric using the FRR BGP daemon. If the default > > route is received from BGP it will be installed into the kernel by the > > FRR zebra daemon which syncs kernel routes with the FRR BGP routing > > table. The OVN BGP agent installs flows for the Neutron network gateways > > that hand off traffic to the kernel for routing. Since the kernel > > routing table is used, the agent isn't compatible with DPDK fast > > datapath yet. > > > > We don't have good documentation for the OVN BGP integration yet. I've > > only recently been able to make it my primary priority, and some of the > > other engineers which have done the initial proof of concept are moving > > on to other projects. There will be some discussions at the upcoming > > OpenStack PTG about this work, but I am hopeful that the missing pieces > > for your use case will come about in the Yoga cycle. > I did not try to run the OVN BGP agent but I saw your blog posts and I think it's enough to get started with. I still don't get why an extra OVN BGP agent would be needed. One thing I was wondering from the blog posts (and your reply here) is whether every single compute would need connectivity to the physical switches to publish the routes - as the dragent runs on the controller node you only need to configure connectivity between the controllers and the physical switches while in the FRR case you need to open much more. > Would the developments in the Yoga cycle be focused on the OVN BGP agent only, and so there is no interest in improving the neutron-dynamic-routing project ? > Thanks for your insightful comments :) > > > > > > > > > Thanks > > > Francois

1 0

[TripleO] Timeout while introspecting Overcloud Node
by Anirudh Gupta 13 Oct '21

13 Oct '21

Hi Team, We were trying to provision Overcloud Nodes using the Tripleo wallaby release. For this, on Undercloud machine (Centos 8.4), we downloaded the ironic-python and overcloud images from the following link: https://images.rdoproject.org/centos8/wallaby/rdo_trunk/current-tripleo/ After untarring, we executed the command *openstack overcloud image upload* This command setted the images at path /var/lib/ironic/images folder successfully. Then we uploaded our instackenv.json file and executed the command *openstack overcloud node introspect --all-manageable* On the overcloud node, we are getting the Timeout error while getting the agent.kernel and agent.ramdisk image. *http://10.0.1.10/8088/agent.kernel......Connection <http://10.0.1.10/8088/agent.kernel......Connection> timed out (http://ipxe.org/4c0a6092 <http://ipxe.org/4c0a6092>)* *http://10.0.1.10/8088/agent.kernel......Connection <http://10.0.1.10/8088/agent.kernel......Connection> timed out (http://ipxe.org/4c0a6092 <http://ipxe.org/4c0a6092>)* However, from another test machine, when I tried *wget http://10.0.1.10/8088/agent.kernel <http://10.0.1.10/8088/agent.kernel>* - It successfully worked Screenshot is attached for the reference Can someone please help in resolving this issue. Regards Anirudh Gupta

1 1

[TripleO] Issue in running Pre-Introspection
by Anirudh Gupta 13 Oct '21

13 Oct '21

Hi Team, I tried installing Undercloud using the below link: https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deploym… I am getting the following error: (undercloud) [stack@undercloud ~]$ openstack tripleo validator run --group pre-introspection Selected log directory '/home/stack/validations' does not exist. Attempting to create it. +--------------------------------------+---------------------------------+--------+------------+-----------------+-------------------+-------------+ | UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration | +--------------------------------------+---------------------------------+--------+------------+-----------------+-------------------+-------------+ | 7029c1f6-5ab4-465d-82d7-3f29058012ce | check-cpu | PASSED | localhost | localhost | | 0:00:02.531 | | db059017-30f1-4b97-925e-3f55b586d492 | check-disk-space | PASSED | localhost | localhost | | 0:00:04.432 | | e23dd9a1-90d3-4797-ae0a-b43e55ab6179 | check-ram | PASSED | localhost | localhost | | 0:00:01.324 | | 598ca02d-258a-44ad-b78d-3877321cdfe6 | check-selinux-mode | PASSED | localhost | localhost | | 0:00:01.591 | | c4435b4c-b432-4a1e-8a99-00638034a884 | *check-network-gateway | FAILED* | undercloud | *No host matched* | | | | cb1eed23-ef2f-4acd-a43a-86fb09bf0372 | *undercloud-disk-space | FAILED* | undercloud | *No host matched* | | | | abde5329-9289-4b24-bf16-c4d82b03e67a | *undercloud-neutron-sanity-check | FAILED* | undercloud | *No host matched* | | | | d0e5fdca-ece6-4a37-b759-ed1fac31a10f | *ctlplane-ip-range | FAILED* | undercloud | No host matched | | | | 91511807-225c-4852-bb52-6d0003c51d49 | *dhcp-introspection | FAILED* | undercloud | No host matched | | | | e96f7704-d2fb-465d-972b-47e2f057449c |* undercloud-tokenflush | FAILED *| undercloud | No host matched | | | As per the validation link, https://docs.openstack.org/tripleo-validations/wallaby/validations-pre-intr… check-network-gateway If gateway in undercloud.conf is different from local_ip, verify that the gateway exists and is reachable Observation - In my case IP specified in local_ip and gateway, both are pingable, but still this error is being observed ctlplane-ip-range¶ <https://docs.openstack.org/tripleo-validations/wallaby/validations-pre-intr…> Check the number of IP addresses available for the overcloud nodes. Verify that the number of IP addresses defined in dhcp_start and dhcp_end fields in undercloud.conf is not too low. - ctlplane_iprange_min_size: 20 Observation - In my case I have defined more than 20 IPs Similarly for disk related issue, I have dedicated 100 GB space in /var and / Filesystem Size Used Avail Use% Mounted on devtmpfs 12G 0 12G 0% /dev tmpfs 12G 84K 12G 1% /dev/shm tmpfs 12G 8.7M 12G 1% /run tmpfs 12G 0 12G 0% /sys/fs/cgroup /dev/mapper/cl-root 100G 2.5G 98G 3% / /dev/mapper/cl-home 47G 365M 47G 1% /home /dev/mapper/cl-var 103G 1.1G 102G 2% /var /dev/vda1 947M 200M 747M 22% /boot tmpfs 2.4G 0 2.4G 0% /run/user/0 tmpfs 2.4G 0 2.4G 0% /run/user/1000 Despite setting al the parameters, still I am not able to pass pre-introspection checks. *"NO Host Matched" *is found in the table. Regards Anirudh Gupta

2 3