February 2020 - openstack-discuss

[Tempest] OpenSack Powered * vs OpeStack Compatible
by prakash RAMCHANDRAN 16 Mar '20

16 Mar '20

Hi all, I am curious to find out if any Distribution or Products based on Openstack Train or Usuri are seeking the latest certifications based on 2019.02. Similarly does any Hardware Driver of Software application seeking OpenStack compatibility Logo? Finally does anyone think that Open Infra Distro like Airship or StarlingX should promote Open Infra Airship or Open Infra StarlingX powered as a new way to promote eco system surrounding them similar to OpenStack compatible drivers and software. Will then Argo, customize, Metal3.io or Ironic be qualified as Open Infra Airship compatible? If so how tempest can help in testing the above comments? Refer to this market place below as how Distos and Products leverage OpenStack logos and branding programs. https://www.openstack.org/marketplace/distros/ Discussions and feedback are welcome. A healthy debate as how k8s modules used in Open Infra can be certified will be a good start. ThanksPrakash Sent from Yahoo Mail on Android

6 11

[tripleo][RDO] Version of Ansible in RDO CentOS8 repository
by Alfredo Moralejo Alonso 16 Mar '20

16 Mar '20

Hi all, During CentOS 8 dependencies preparation we've built ansible 2.9 in RDO dependencies repo which was released on Oct 2019, While testing TripleO with CentOS8 it has been discovered that the latest release of ceph-ansible does not support ansible 2.9 but only 2.8, so I'm opening discussion about the best way to move on in CentOS 8: - Make ceph-ansible 4.0 to work with ansible 2.8 *and* 2.9 so that the same releases can be used in CentOS7 with Stein and Train and CentOS8 Train and Ussuri. - Maintain separated ceph-ansible releases and builds for centos7/ansible 2.8 and centos8/ansible 2.9 able to deploy Nautilus. - Move ansible back to 2.8 in CentOS 8 Ussuri repository. I wonder if TripleO or other projects using ansible from RDO repositories has any requirement or need to move to ansible 2.9 in Ussuri cycle or can stay in 2.8 until next release, any thoughts? Best regards, Alfredo

1 2

[neutron] Why network performance is extremely bad and linearly related with number of VMs?
by Yi Yang (杨�D)-云服务集团 11 Mar '20

11 Mar '20

Hi, All Anybody has noticed network performance between VMs is extremely bad, it is basically linearly related with numbers of VMs in same compute node. In my case, if I launch one VM per compute node and run iperf3 tcp and udp, performance is good, it is about 4Gbps and 1.7Gbps, for 16 bytes small UDP packets, it can reach 180000 pps (packets per second), but if I launch two VMs per compute node (note: they are in the same subnet) and only run pps test case, that will be decrease to about 90000 pps, if I launch 3 VMs per compute node, that will be about 50000 pps, I tried to find out the root cause, other VMs in this subnet (they are in the same compute node as iperf3 client) can receive all the packets iperf3 client VM sent out although destination MAC isn’t broadcast MAC or multicast MAC, actually it is MAC of iperf3 server VM in another compute node, by further check, I did find qemu instances of these VMs have higher CPU utilization and corresponding vhost kernel threads also also higher CPU utilization, to be importantly, I did find ovs was broadcasting these packets because all the ovs bridges didn’t learn this destination MAC. I tried this in Queens and Rocky, the same issue is there. By the way, we’re using linux bridge for security group, so VM tap interface is attached into linux bridge which is connected to br-int by veth pair. Here is output of “ovs-appctl dpif/dump-flows br-int” after I launched many VMs: recirc_id(0),in_port(12),eth(src=fa:16:3e:49:26:51,dst=fa:16:3e:a7:0a:3a),et h_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:11012944, bytes:726983412, used:0.000s, flags:SP., actions:push_vlan(vid=1,pcp=0),2,set(tunnel(tun_id=0x49,src=10.3.2.17,dst=10 .3.2.16,ttl=64,tp_dst=4789,flags(df|key))),pop_vlan,9,8,11,13,14,15,16,17,18 ,19 $ sudo ovs-appctl fdb/show br-floating | grep fa:16:3e:49:26:51 $ sudo ovs-appctl fdb/show br-tun | grep fa:16:3e:49:26:51 $ sudo ovs-appctl fdb/show br-bond1 | grep fa:16:3e:49:26:51 $ sudo ovs-appctl fdb/show br-int | grep fa:16:3e:49:26:51 All the bridges can’t learn this MAC. My question is why ovs bridges can’t learn MACs of other compute nodes, is this common issue of all the Openstack versions? Is there any known existing way to fix it? Look forward to hearing your insights and solutions, thank you in advance and have a good day.

4 5

[manila] share group replication spike/questions
by Dong.Ding＠dell.com 11 Mar '20

11 Mar '20

Hi, guys, As we talked about the topic in a virtual PTG few months ago. https://etherpad.openstack.org/p/shanghai-ptg-manila-virtual (Support promoting several shares in group (DELL EMC: dingdong) I'm trying to write a manila-spec for it. It's my first experience to implement such feature in framework. I need to double check with you something, and hope you can give me some guides like: 1. Where is the extra-spec defined for group/group type, it's in Manila repo, right? (like manila.db.sqlalchemy.models....) 2. The command cli should be implemented for 'python-manilaclinet' repo, right? (I have never touched this repo before) 3. Where is the rest-api should be implemented? 4. And more tips you have? like any other related project should be changed? Just list what I know, and more details questions will be raised when implementing, I think. FYI Thanks, Ding Dong

2 7

Re: [CINDER] Distributed storage alternatives
by Burak Hoban 06 Mar '20

06 Mar '20

Hi guys, We use Dell EMC VxFlex OS, which in its current version allows for free use and commercial (in version 3.5 a licence is needed, but its perpetual). It's similar to Ceph but more geared towards scale and performance etc (it use to be called ScaleIO). Other than that, I know of a couple sites using SAN storage, but a lot of people just seem to use Ceph. Cheers, Burak ------------------------------ Message: 2 Date: Thu, 13 Feb 2020 18:20:29 +0100 From: Ignazio Cassano <ignaziocassano(a)gmail.com> To: Alfredo De Luca <alfredo.deluca(a)gmail.com> Cc: openstack-discuss <openstack-discuss(a)lists.openstack.org> Subject: Re: [CINDER] Distributed storage alternatives Message-ID: <CAB7j8cXLQWh5fx-E9AveUEa6OncDwCL6BOGc-Pm2TX4FKwnUKg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello Alfredo, I think best opensource solution is ceph. As far as commercial solutions are concerned we are working with network appliance (netapp) and emc unity. Regards Ignazio Il Gio 13 Feb 2020, 13:48 Alfredo De Luca <alfredo.deluca(a)gmail.com> ha scritto: > Hi all. > we 'd like to explore storage back end alternatives to CEPH for > Openstack > > I am aware of GlusterFS but what would you recommend for distributed > storage like Ceph and specifically for block device provisioning? > Of course must be: > > 1. *Reliable* > 2. *Fast* > 3. *Capable of good performance over WAN given a good network back > end* > > Both open source and commercial technologies and ideas are welcome. > > Cheers > > -- > *Alfredo* > > _____________________________________________________________________ The information transmitted in this message and its attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information, by persons or entities other than the intended recipient is prohibited. If you have received this in error, please contact the sender and delete this e-mail and associated material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files, with the permission of the sender. This message has been scanned for viruses. _____________________________________________________________________

3 3

[neutron] security group list regression
by James Denton 04 Mar '20

04 Mar '20

Hello all, We recently upgraded an environment from Newton -> Rocky, and have noticed a pretty severe regression in the time it takes the API to return the list of security groups. This environment has roughly 8,000+ security groups, and it takes nearly 75 seconds for the ‘openstack security group list’ command to complete. I don’t have actual data from the same environment running Newton, but was able to replicate this behavior with the following lab environments running a mix of virtual and baremetal machines: Newton (VM) Rocky (BM) Stein (VM) Train (BM) Number of sec grps vs time in seconds: # Newton Rocky Stein Train 200 4.1 3.7 5.4 5.2 500 5.3 7 11 9.4 1000 7.2 12.4 19.2 16 2000 9.2 24.2 35.3 30.7 3000 12.1 36.5 52 44 4000 16.1 47.2 73 58.9 5000 18.4 55 90 69 As you can see (hopefully), the response time increased significantly between Newton and Rocky, and has grown slightly ever since. We don't know, yet, if this behavior can be seen with other 'list' commands or is limited to secgroups. We're currently verifying on some intermediate releases to see where things went wonky. There are some similar recent reports out in the wild with little feedback: https://bugzilla.redhat.com/show_bug.cgi?id=1788749 https://bugzilla.redhat.com/show_bug.cgi?id=1721273 I opened a bug here, too: https://bugs.launchpad.net/neutron/+bug/1865223 Bottom line: Has anyone else experienced similar regressions in recent releases? If so, were you able to address them with any sort of tuning? Thanks in advance, James

4 7

[release][tc][horizon] xstatic repositories marked as deprecated
by Akihiro Motoki 04 Mar '20

04 Mar '20

Hi, I noticed some xstatic repositories [1] under the horizon team are marked as deprecated. They are marked as deprecated in [2] as they were never released, but the actual situation looks different. They are used in horizon requirements and the contents of these repositories are same as those in the corresponding PyPI modules. I guess they were released to PyPI based on the repositories but the horizon team just forgot to push the corresponding tags. I would like to drop 'deprecated' mark from the repositories and move them to the proper situation. Does it make sense? If so, what is the right solution to do this? In my understanding, we need to add tags corresponding to the deliverables and to add deliverable files in the releases repo. What I am not sure is whether we can push a tag to a repository without pushing deliverables to PyPI? Or do we need to create another release to tag them in their repositories? [1] Such xstatic repositories are: - xstatic-bootstrap-datepicker - xstatic-hogan - xstatic-jquery-migrate - xstatic-jquery.quicksearch - xstatic-jquery.tablesorter - xstatic-rickshaw - xstatic-spin [2] https://review.opendev.org/#/c/629889/ Thanks, Akihiro Motoki (irc: amotoki)

5 8

[glance] Different checksum between CLI and curl
by gaetan.trellu＠incloudus.com 03 Mar '20

03 Mar '20

Hi guys, Does anyone know why the md5 checksum is different between the "openstack image save" CLI and "curl" commands? During the image creation a checksum is computed to check the image integrity, using the "openstack" CLI match the checksum generated but when "curl" is used by following the API documentation[1] the checksum change at every "download". Any idea? Thanks, Gaëtan [1] https://docs.openstack.org/api-ref/image/v2/index.html?expanded=download-bi…

12 21

[TaskFlow] running multiple engines on a shared thread
by Sachin Laddha 02 Mar '20

02 Mar '20

Hi, We are using taskflow to execute workflows. Each workflow is executed by a separate thread (using engine.run() method). This is limiting our capability to execute maximum number of workflows that can run in parallel. It is limited by the number of threads there in the thread pool. Most of the time, the workflow tasks are run by agents which could take some time to complete. Each engine is alive and runs on a dedicated thread. Is there any way to reuse or run multiple engines on one thread. The individual tasks of these engines can run in parallel. I came across iter_run method of the engine class. But not sure if that can be used for this purpose. Any help is highly appreciated.

2 3

[all] Moving towards OpenDev
by Mohammed Naser 29 Feb '20

29 Feb '20

Hi everyone, The OpenStack infrastructure team has helped build an extremely powerful set of tools which we use on a daily basis in order to build OpenStack. You may have noticed that Open Infrastructure projects (such as Zuul, StarlingX, Airship, etc) have leveraged these tools as well as they found them to be extremely powerful and useful to build software by leveraging our four opens. As you might have noticed, there has been many references to OpenDev and it's time that we split off the infrastructure team to it's own separate community with governance and the OpenStack infrastructure team will continue to help work with the OpenDev team in order to keep our lights running (and most likely, those two teams will be the same at the start, but the idea is hoping that one or both grow separately). I'd like to invite our community to please look at the following two patches and leave your comments. Regardless how you feel about this, we'd really appreciate if you can leave a review (-1 or +1) just to show some sort of acknowledgment and help signal that the community is aware of this upcoming change. I'm certainly very excited about it. :) https://review.opendev.org/#/c/710020/ https://review.opendev.org/#/c/703488/ Thanks everyone! Regards, Mohammed

3 3