April 2020 - openstack-discuss - lists.openstack.org

[all][PTL] U Community Goal: Project PTL & Contrib Docs Update #8
by Kendall Nelson 28 Apr '20

28 Apr '20

Hello Everyone! More excellent progress[1] has been made since last week! Again, thank you for all the hard work! Projects missing progress/merged status according to the story[4] are as follows: - Adjutant - Barbican - Cloudkitty - Congress - Cyborg - Ec2-API - Freezer - Heat - Horizon - Karbor - Kuryr - LOCI - Masakari - Mistral - Monasca - Murano - OpenStack Charms - openstack-chef - OpenStackAnsible - OpenStackClient - Oslo - Packaging-RPM - Placement - Puppet OpenStack - Rally - Release Management - Requirements - Senlin - Solum - Storlets - Swift - Tacker - Telemetry - Tricircle - TripleO - Winstackers - Zaqar If you are on this list, please get started IMMEDIATELY. All told, if you are doing the bare minimum of only filling out the template[2] it shouldn't take more than 5-10 minutes per repo. Please add me if you need help with reviews and let me know if there is any other help I can provide. If you have questions about the goal itself, here is the link to that[3]. And as you push patches, please be sure to update the StoryBoard story[4]. Thanks! -Kendall (diablo_rojo) [1] Progress: https://review.opendev.org/#/q/topic:project-ptl-and-contrib-docs+(status:o… [2] Cookiecutter Template: https://opendev.org/openstack/cookiecutter/src/branch/master/%7b%7bcookiecu… [3] Accepted Goal: https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-c… [4] Task Tracking: https://storyboard.openstack.org/#!/story/2007236

1 0

[scientific-sig] IRC Meeting today 2100 UTC - Open Infra labs, Train upgrade experiences
by Stig Telfer 28 Apr '20

28 Apr '20

Hi All, We have a Scientific SIG IRC meeting in #openstack-meeting at 2100 UTC (about 45 minutes). Everyone is welcome. Today’s agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_April_28th_2020 <https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_April_28th_2020> This week we have a discussion on the Open Infra Labs project, plus we are looking to share operational experiences on OpenStack Train. Cheers, Stig

1 0

[tripleo] ptg
by Wesley Hayutin 28 Apr '20

28 Apr '20

Greetings, Just a reminder to review the TripleO time slots for PTG [1] Looking for feedback on the times. We'll start a schedule etherpad for folks to align their topics to a particular time slot soon. Note, the time slot for Q&A or really anything for folks not able to join the 13-17UTC time slot. Thanks [1] https://ethercalc.openstack.org/126u8ek25noy [2] https://etherpad.opendev.org/p/tripleo-victoria-topics

1 0

Announcing the general availability of Debian packages for Ussuri
by Thomas Goirand 28 Apr '20

28 Apr '20

Hi, Even after 9 years packaging OpenStack, I'm still excited to announce the availability of the last release of OpenStack in Debian. Indeed, today, I was able to launch my first VM using Ussuri, running on top of Debian Buster. Even if we haven't run Tempest on this release yet, it's already usable, and I wanted everyone to know about it. Packages are available either from Debian Experimental (they will be uploaded to Unstable the day of the official release, then hopefully migrate automatically to Debian Testing) or from the debian.net backports to Buster, from both of these repos: deb http://buster-ussuri.debian.net/debian buster-ussuri-backports main deb-src http://buster-ussuri.debian.net/debian buster-ussuri-backports main deb http://buster-ussuri.debian.net/debian buster-ussuri-backports-nochange main deb-src http://buster-ussuri.debian.net/debian buster-ussuri-backports-nochange main *OR* from this server (which contains all release of OpenStack since the Kilo release): deb http://osbpo.debian.net/debian buster-ussuri-backports main deb-src http://osbpo.debian.net/debian buster-ussuri-backports main deb http://osbpo.debian.net/debian buster-ussuri-backports-nochange main deb-src http://osbpo.debian.net/debian buster-ussuri-backports-nochange main Please make sure to configure your servers to use *both* buster-ussuri-backports and buster-ussuri-backports-nochange suites. The first one contains packages maintained by the Debian OpenStack team, while the later are backports of packages maintained by other people in Debian (like those maintained in the Debian Python Module Team, but not only). Still missing from this release... - vitrage-dashboard: There's some new XStatic packages to make, work is ongoing. - some neutron plugins: Many aren't released yet... - heat: It just got released a few hours ago, and I didn't have time yet to figure out what's wrong with these 3 unit tests: http://paste.openstack.org/show/792781/ Hopefully, this will be addressed tomorrow. Note that OCI, the Debian cluster installer for OpenStack, already supports that version (that's how I launched my first instance), and is running the latest version of all puppet modules of puppet-openstack: https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer Some fun statistics: 187 packages were updated in the OpenStack team's Git repository to make this release (plus some updates of packages not within the team). Note that the last piece of software (cinder) was converted to uwsgi, so this release comes with absolutely all API servers running under uwsgi. Please try these packages and report bugs to the Debian BTS as usual, so we can fix as many bugs as possible before the final release. Some packages are still waiting for the FTP master's approval before they can reach Debian proper. Noticeably, many Tempest plugins (as I decided to package them all in Debian). 6 of these are pending, and probably more will be packaged. Stay tuned. Cheers, Thomas Goirand (zigo)

5 5

[neutron] Low-throughput over high-latency connections
by Eric K. Miller 28 Apr '20

28 Apr '20

Hi, I'm at a loss after quite a bit of testing, so thought I'd see if someone has any ideas... We have a Stein cluster that was/is deployed with Kolla Ansible with DVR without OVS+DPDK - rather just the standard kernel OVS. Everything appears to work well with low-latency local and Internet destinations, but as the latency increases, throughput drops significantly. I would have blamed the standard bandwidth-delay product and TCP window size not being large enough, etc., but that definitely isn't the issue, since we have shown that VMs on our VMware clusters (VLAN-based, not NSX-based) do not have the issue and use the same many-10Gbps-connected ISPs to our routers as the OpenStack configuration. I am using a Vultr machine in London as a test destination since the latency is around 86ms - a good high-latency test. The good environment we are testing against: Using iperf3 with 1 thread on a VMware VM results in an average of 155Mbits/sec sent to Vultr London with a few retries recorded that lower the window size to around 1.78MiBytes. This is roughly what we expect, if not more, with faster machines. The problem: Performing the same test on a "much" faster server (E-2288G based, without hyperthreading enabled, so 4.5GHz on all 8 cores), with no other VMs on it, a VM with iperf3 with 1 thread results in an average of only around 32Mbits/sec with a window size that drops significantly at the start and increases to only about 386KiBytes before retries start to happen. This average does not change when using other machines in our cluster, including some Xeon Gold and EPYC processor machines. The only thing I can point to is OVS and/or something in Linux. However, both tests (VMware and KVM) were performed with the same fresh installation and configuration (CentOS 7 with all updates and the standard kernel). So, my doubts are that it has something to do with Linux. On the physical host, the vhost process barely registers any CPU usage, and neither does the VM - as expected on such a fast machine. I even tried to enable multiqueue, since we had it disabled. With a 2 vCPU VM, ethtool -l eth0 displays that 2 queues are in use. Same results - 32Mbits/sec throughput to Vultr London. All machines are connected via 100Gbps or 25Gbps connections with Mellanox Connect-X 4 and 5 cards with Mellanox switches, and iperf can saturate these links between the physical hosts when run on the physical hosts' CentOS. No errors (tx errors, rx errors, packet loss, etc.) are being reported on any of the logical or physical links. I thought a queue got full somewhere, so I looked for where queues were used and found that virtio has a tx_queue_size that can be adjusted, but unfortunately, it appears this can only be done with vhost-user, which I believe is specific to OVS+DPDK. The default tx_queue_size in virtio is 256 elements. I'm not sure what this represents, though, much less how to measure the queue usage statistics. Running iperf3 in "Reverse mode", so the Vultr London machine sends data back to the KVM VM, bandwidth averages around 254Mbits/sec! So the issue appears to be only in one direction - sending network traffic outbound. Running the same iperf3 test from the KVM VM to a Vultr machine that is very close to us (latency of 2ms or so), results in an average bandwidth of 373Mbits/sec, but with 39 retries per second on average. The window size never adjusts low enough to stop the retries - it averages 146KiBytes in size. Using "Reverse mode" results in an average of 1.21Gbits/sec! So, the OVS software isn't slow, but something is causing issues when transmitting data when latency is large - something that doesn't happen with the VMware VMs with the same network and Internet connections. Also, pings from the VMware and KVM VMs are nearly identical - so there isn't some crazy amount of latency caused by OVS - at least when it is essentially idle. I'm not sure where to go from here in terms of diagnostics. Any ideas? Thanks for any help, in advance! Eric

1 0

[tc][puppet][Glare] Propose retirement of puppet-glare
by Takashi Kajinami 28 Apr '20

28 Apr '20

Hi, Currently Glare[1] project is located under x instead of openstack, and IIUC this means that Glare is no longer maintained and has been retired already. [1] https://opendev.org/x/glare Regarding its status, I'll propose to retire puppet-glare[2] because it doesn't make sense to maintain a puppet module for the retired project. [2] https://github.com/openstack/puppet-glare May I ask TC to confirm my understanding about Glare project status ? Also, please let me know if anybody has concern with retiring puppet-glare. Thank you, Takashi Kajinami

4 4

[all] Virtual PTG Tooling Requirements & Suggestions
by Kendall Nelson 28 Apr '20

28 Apr '20

Hello Everyone :) We want to get your input on tooling for the PTG. We have a general idea of what we have for requirements, but wanted to get the community's input- i.e. make sure we aren't missing any. Here is the list of requirements we have right now[1]. Also, if anyone has suggestions of tools that meet these criteria, we would love to hear them! We are gathering and vetting options for the event and want to make sure nothing gets overlooked. -the Kendalls (diablo_rojo & wendallkaters) [1] https://etherpad.opendev.org/p/virt-ptg-requirements

4 3

Openstacksdk
by Tom Schoonjans 28 Apr '20

28 Apr '20

Hi all, I am trying to use the openstacksdk to get a list of my projects. I have not been successful at doing this, as it always tries to get all projects, which doesn’t work for me as I am not an admin: the error message I get is "Policy doesn't allow os_compute_api:servers:detail:get_all_tenants to be performed." This is certainly possible though, because it works fine with the client: openstack project list --my-projects Any thoughts on how I can accomplish this? Many thanks in advance, Tom

1 0

trove image
by 会水的鸟 28 Apr '20

28 Apr '20

您好：     我在使用openstack trove镜像时，一直处于build状态，我应该如何解决。      镜像链接：      https://tarballs.opendev.org/openstack/trove/images/trove-train-mysql-ubuntu-xenial-dev.qcow2                                                                                                                         谢谢 hello：     When I was using the openstack trove image，Always in build state，What should I do。     https://tarballs.opendev.org/openstack/trove/images/trove-train-mysql-ubunt…                                                                                                                             thank you

2 4

[ironic]: multiple compute nodes
by fsbiz＠yahoo.com 28 Apr '20

28 Apr '20

Hi folks, We are seeing some weird issues with multiple compute nodes and would appreciate your thoughts. Background:We are on stable Queens.As part of an upgrade to accomodate 3X more servers, we decided to add three more compute nodes+ three more ICs for a total of 6 compute nodes and 6 ICs.As soon as we added these in preparation for the 3X increase in servers I am seeing weirdbehaviour. A general question to everyone:How many of you run your baremetal clouds with 5+ computes and ICs?Are things stable with the setup ? Logs and Analysis:all compute and conductor services are up and running. 1) Baremetal node c1bda753-d46c-4379-8d07-7787c2a4a7f2 mapped to sc-ironic08root@stg-cl1-dev-001:~# openstack hypervisor show c1bda753-d46c-4379-8d07-7787c2a4a7f2 | grep ironic | | service_host | sc-ironic08.nvc.nvidia.com 2)Mac address is 6c:b3:11:4f:8a:c0root@stg-cl1-dev-001:~# openstack baremetal port list --node c1bda753-d46c-4379-8d07-7787c2a4a7f2+--------------------------------------+-------------------+| UUID | Address |+--------------------------------------+-------------------+| a517fb41-f977-438d-8c0d-21046e2918d9 | 6c:b3:11:4f:8a:c0 |+--------------------------------------+-------------------+ 2)Provisioning starts: ironic06 receives the VIF update: WHY ?2020-04-21 15:05:47.509 71431 INFO ironic.conductor.manager VIF 657fea31-3218-4f10-b6ad-8b6a0fa7bab8 successfully attached to node c1bda753-d46c-4379-8d07-7787c2a4a7f2 ironic08 (correct one) also receives updates.[root@sc-ironic08 master_images]# tail -f /var/log/ironic/ironic-conductor.log | grep c1bda753-d46c-4379-8d07-7787c2a4a7f22020-04-21 15:08:04.943 27542 INFO ironic.conductor.task_manager [req-259b0175-65bc-4707-8c88-a65189a29954 - - - - -] Node c1bda753-d46c-4379-8d07-7787c2a4a7f2 moved to provision state "deploying" from state "wait call-back"; target provision state is "active" For now we have backed down to 3 and are stable again but I would really like to overprovision our computes and conductors if possible. Please let me know your thoughts and if anything rings a bell. thanks,Fred.

3 4