November 2020 - openstack-discuss

[qa][cinder][glance][swift] Gate status for swift test failing Bug#1905432
by Ghanshyam Mann 25 Nov '20

25 Nov '20

Hello Everyone, As you have seen that test_create_object_with_transfer_encoding is failing consistently since 20th Nov which is failing integrated-gate-storage and integrated-gate-object-storage jobs. The issue is that Tempest is getting 500 from swift API, I could not find the root cause yet so decided to disable this test for now to unlock the gate. Avoid recheck for this failure until this is merge - https://review.opendev.org/c/openstack/tempest/+/764061 I have filled bug against Tempest and Swift to track it - https://bugs.launchpad.net/swift/+bug/1905432 -gmann

2 1

[ironic] Proposing Steve Baker (stevebaker) for ironic-core
by Julia Kreger 25 Nov '20

25 Nov '20

Greetings everyone, I would like to propose Steve Baker for ironic-core. Steve shifted into focusing on Ironic contribution this year and he has been actively reviewing changes during the Victoria and now Wallaby cycle. He has brought insight, drive, and a curiosity to ask questions. Plus he took on the challenge of removing wsme from Ironic's API which is no small feat. I've run this nomination past the existing cores and everyone responded in full support of doing so. So if there are no objections, I'll add Steve to ironic-core next week. Otherwise, Congratulations Steve! -Julia

3 2

[qa][all] New hacking 4.0.0 is available
by Ghanshyam Mann 25 Nov '20

25 Nov '20

Hello Everyone, Hacking 4.0.0 is available to use now which includes many common checks used in various projects. While upgrading to new hacking you can remove those checks definition from your repo hacking checks. Check release notes to find what all common checks are being moved from the project side to hacking: - https://docs.openstack.org/releasenotes/hacking/unreleased.html#relnotes-4-… -gmann

1 0

[cinder][kayobe][kolla][OSA][tripleo][release] Victoria cycle-trailing release deadline
by Herve Beraud 25 Nov '20

25 Nov '20

Hello teams with deliverables following the cycle-trailing release model! This is just a reminder about wrapping those Victoria trailing deliverables up. A few cycles ago we extended the deadline for cycle-trailing to give more time, so the actual deadline isn't until January 14, 2021: https://releases.openstack.org/wallaby/schedule.html#w-cycle-trail If things are ready sooner than that though, all the better for our downstream consumers. Just for awareness, the following cycle-trailing deliverables will need their final releases at some point in the next few months: cinderlib kayobe kolla-ansible kolla openstack-ansible os-collect-config os-refresh-config tripleo-ipsec Thanks! -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

[tripleo][ci] Issues w/ the quay container registry
by Wesley Hayutin 25 Nov '20

25 Nov '20

FYI.. This is impacting jobs atm. https://bugs.launchpad.net/tripleo/+bug/1905592 Will follow up when we're clear. 0/

1 0

[magnum] Can not deploy a cluster using Fedora CoreOS 32
by monadt 25 Nov '20

25 Nov '20

I am trying to use Magnum on Openstack that was deployed using kolla-ansible. The instance is completely private (so floating IPs are only accessible inside a VPN network) - not sure if this is a problem. The deployment looks as follows: Openstack Nodes: Ubuntu 20.04 Kolla-Ansible version: '10.1.0' kolla_base_distro: "ubuntu" openstack_release: "ussuri" I could successfully deploy the required magnum services and created a cluster template using the magnum cli as follows: openstack coe cluster template create kubernetes-cluster-template --image fedora-coreos-32 --external-network public-external --dns-nameserver 1.1.1.1 --master-flavor m1.small --flavor m1.small --docker-storage-driver overlay --coe kubernetes I am using the Fedora CoreOS 32 release (and uploaded the image like this): openstack image create --disk-format=qcow2 --container-format=bare --file=fedora-coreos-32.20201104.3.0-openstack.x86_64.qcow2 --property os_distro='fedora-coreos' fedora-coreos-32 I made sure to set the os_distro as stated in the repository when using fedora coreos. When I now try to create a cluster (openstack coe cluster create kubernetes-cluster --cluster-template kubernetes-cluster-template --master-count 1 --node-count 1 --keypair mykey) all the heat resources start spinning up, but the kube-masters resource will never come up. While debugging it seems to hang during the following step inside the heat-container-agent: + '[' ok = ok ']' + kubectl patch node kubernetes-cluster-flfrufngma6x-master-0 --patch '{"metadata": {"labels": {"node-role.kubernetes.io/master": ""}}}' error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable + echo 'Trying to label master node with node-role.kubernetes.io/master=""' + sleep 5s Trying to label master node with node-role.kubernetes.io/master="" I also didn't find a way to set the KUBERNETES_MASTER environment variable from the outside. I assume there is no other supported distribution given that Fedora Atomic and non-Fedora CoreOS do no longer exist? Sent with [ProtonMail](https://protonmail.com) Secure Email.

3 2

[cinder] wallaby R-18 virtual mid-cycle on 9 december
by Brian Rosmaita 25 Nov '20

25 Nov '20

As voted upon at today's weekly meeting, the Cinder Wallaby R-18 virtual mid-cycle will be held: DATE: WEDNESDAY 9 DECEMBER 2020 TIME: 1400-1600 UTC LOCATION: https://bluejeans.com/3228528973 The meeting will be recorded. Please add topics to the mid-cycle etherpad: https://etherpad.opendev.org/p/cinder-wallaby-mid-cycles cheers, brian

1 0

[neutron][operators][all] Watch out for updates of stable/train and stable/stein releases in Neutron
by Slawek Kaplonski 25 Nov '20

25 Nov '20

Hi, I want to warn all of You about terrible mistake which we made in Neutron some time ago. We backported to stable releases patch [1] which broke update workflow. So if You are now updating Your Stein or Train Neutron to latest version and You will do it as it should be done, so first neutron-server and then agents, Your neutron-ovs-agents will not work properly with newer neutron-server. Details are in reported bug [2] Broken versions are: * for Train 15.2.0 and 15.3.0 * for Stein 14.3.1, 14.4.0 and 14.4.1 We proposed reverts of [1] and those reverts are now in gate. As soon as they will be merged we will release new, fixed versions for both Stein and Train. So if You didn't update to those broken versions yet, please don't do it now and wait for next version with fix. If You already updated and fixed that issue on Your clusters - You will have exactly same problem during next update again. I know it's very bad but unfortunately we don't have any other way to fix that issue. [1] https://review.opendev.org/#/c/744133/ [2] https://bugs.launchpad.net/neutron/+bug/1903531 -- Slawek Kaplonski Principal Software Engineer Red Hat

5 7

[neutron][openstack-ansible] Instances can only connect to provider-net via tenant-net but not directly
by Oliver Wenz 25 Nov '20

25 Nov '20

Hi, Starting instances which are connected directly to a provider network results in an error in my Ussuri cloud. However, I can associate floating IPs for this provider network when I connect instances to it via a tenant network and it works fine. Both nova-compute and neutron-linuxbridge-agent services don't show errors, I only get an error in the instance status: 'code': 500, 'created': '2020-11-25T12:05:42Z', 'message': 'Build of instance 274e0a7d-fb33-430a-986e-74fceae6a36d aborted: Failed to allocate the network(s), not rescheduling.', 'details': 'Traceback (most recent call last): File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6549, in _create_domain_and_network post_xml_callback=post_xml_callback) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/nova/compute/manager.py", line 513, in wait_for_instance_event actual_event = event.wait() File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/eventlet/event.py", line 125, in wait result = hub.switch() File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch return self.greenlet.switch() eventlet.timeout.Timeout: 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/nova/compute/manager.py", line 2378, in _build_and_run_instance accel_info=accel_info) File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3683, in spawn cleanup_instance_disks=created_disks) File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6572, in _create_domain_and_network raise exception.VirtualInterfaceCreateException() nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/nova/compute/manager.py", line 2200, in _do_build_and_run_instance filter_properties, request_spec, accel_uuids) File "/openstack/venvs/nova-21.1.0/lib/python3.6/site-packages/nova/compute/manager.py", line 2444, in _build_and_run_instance reason=msg) nova.exception.BuildAbortException: Build of instance 274e0a7d-fb33-430a-986e-74fceae6a36d aborted: Failed to allocate the network(s), not rescheduling. ' Any ideas what could cause this? Kind regards, Oliver On 2020-11-25 13:00, openstack-discuss-request(a)lists.openstack.org wrote: > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. Re: > [nova][tripleo][rpm-packaging][kolla][puppet][debian][osa] Nova > enforces that no DB credentials are allowed for the nova-compute > service (Balázs Gibizer) > 2. Re: [ironic] [infra] Making Glean work with IPA for static IP > assignment (Dmitry Tantsur) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 25 Nov 2020 11:13:23 +0100 > From: Balázs Gibizer <balazs.gibizer(a)est.tech> > To: Thomas Goirand <zigo(a)debian.org> > Cc: openstack maillist <openstack-discuss(a)lists.openstack.org> > Subject: Re: > [nova][tripleo][rpm-packaging][kolla][puppet][debian][osa] Nova > enforces that no DB credentials are allowed for the nova-compute > service > Message-ID: <BEKCKQ.PYGQ12VO6AF23(a)est.tech> > Content-Type: text/plain; charset=iso-8859-1; format=flowed > > > > On Mon, Nov 23, 2020 at 13:47, Thomas Goirand <zigo(a)debian.org> wrote: >> On 11/23/20 11:31 AM, Balázs Gibizer wrote: >>> It is still a security problem if nova-compute ignores the config >>> as the >>> config still exists on the hypervisor node (in some deployment >>> scenarios) >> >> Let's say we apply the patch you're proposing, and that nova-compute >> isn't loaded anymore with the db credentials, because it's on a >> separate >> file, and nova-compute doesn't load it. >> >> In such scenario, the /etc/nova/nova-db.conf could still be present >> with >> db credentials filled-in. So, the patch you're proposing is still not >> effective for wrong configuration of nova-compute hosts. > > Obviously we cannot prevent that the deployer stores the DB creds on a > compute host as we cannot detect it in general. But we can detect it if > it is stored in the config the nova-compute reads. I don't know why > should we not make sure to tell the deployer not to do that as it is > generally considered unsafe. > >> >>> From the nova-compute perspective we might be able to >>> replace the [api_database]connection dependency with some hack. E.g >>> to >>> put the service name to the global CONF object at the start of the >>> service binary and depend on that instead of other part of the >>> config. >>> But I feel pretty bad about this hack. >> >> Because of the above, I very much think it'd be the best way to go, >> but >> I understand your point of view. Going to the /etc/nova/nova-db.conf >> and >> nova-api-db.conf thing is probably good anyways. >> >> As for the nova-conductor thing, I very much would prefer if we had a >> clean and explicit "superconductor=true" directive, with possibly some >> checks to display big warnings in the nova-conductor.log file in case >> of >> a wrong configuration. If we don't have that, then at least things >> must >> be extensively documented, because that's really not obvious what's >> going on. > > I agree that superconductor=true would be a more explicit config option > than [api_database]connection. However this would also enforce that > deployers need a separate config file for nova-compute as there neither > superconductor=true nor superconductor=false (meaning it is a cell > conductor) make sense. > >> >> Cheers, >> >> Thomas Goirand (zigo) >> > > > > > > ------------------------------ > > Message: 2 > Date: Wed, 25 Nov 2020 11:54:13 +0100 > From: Dmitry Tantsur <dtantsur(a)redhat.com> > To: Ian Wienand <iwienand(a)redhat.com> > Cc: openstack-discuss <openstack-discuss(a)lists.openstack.org> > Subject: Re: [ironic] [infra] Making Glean work with IPA for static IP > assignment > Message-ID: > <CACNgkFwVhMMxVRK2PkFEkHORRwY4wY49g7G3CpzYwaFzC27Bjw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > Thank you for your input! > > On Wed, Nov 25, 2020 at 3:09 AM Ian Wienand <iwienand(a)redhat.com> wrote: > >> On Tue, Nov 24, 2020 at 11:54:55AM +0100, Dmitry Tantsur wrote: >>> The problem is, I cannot make Glean work with any ramdisk I build. The >> crux >>> of the problem seems to be that NetworkManager (used by default in RHEL, >>> CentOS, Fedora and Debian at least) starts very early, creates the >> default >>> connection and ignores whatever files Glean happens to write afterwards. >> On >>> Debian running `systemctl restart networking` actually helped to pick the >>> new configuration, but I'm not sure we want to do that in Glean. I >> haven't >>> been able to make NetworkManager pick up the changes on RH systems so >> far. >> >> So we do use NetworkManager in the OpenDev images, and we do not see >> NetworkManager starting before glean. >> > > Okay, thanks for confirming. Maybe it's related to how IPA is built? It's > not exactly a normal image after all, although it's pretty close to one. > > >> >> The way it should work is that simple-init in dib installs glean to >> the image. That runs the glean install script (use --use-nm argument >> if DIB_SIMPLE_INIT_NETWORKMANAGER, which is default on centos/fedora) >> which installs two things; udev rules and a systemd handler. >> > > I have checked that these are installed, but I don't know how to verify a > udev rule. > > >> >> The udev is pretty simple [1] and should add a "Wants" for each net >> device; e.g. eth1 would match and create a Wants glean(a)eth1.service, >> which then matches [2] which should write out the ifcfg config file. >> After this, NetworkManager should start, notice the config file for >> the interface and bring it up. >> > > Yeah, I definitely see logging from NetworkManager DHCP before this service > is run (i.e. before the output from Glean). > > >> >>> Do you maybe have any hints how to proceed? I'd be curious to know how >>> static IP assignment works in the infra setup. Do you have images with >>> NetworkManager there? Do you use the simple-init element? >> >> As noted yes we use this. Really only in two contexts; it's Rackspace >> that doesn't have DHCP so we have to setup the interface statically >> from the configdrive data. Other clouds all provide DHCP, which is >> used when there's no configdrive data. >> >> Here is a systemd-analyze from one of our Centos nodes if it helps: >> > >> graphical.target @18.403s >> └─multi-user.target @18.403s >> └─unbound.service @5.467s +12.934s >> └─network.target @5.454s >> └─NetworkManager.service @5.339s +112ms >> └─network-pre.target @5.334s >> └─glean(a)ens3.service @4.227s +1.102s >> └─basic.target @4.167s >> └─sockets.target @4.166s >> └─iscsiuio.socket @4.165s >> └─sysinit.target @4.153s >> └─systemd-udev-settle.service @1.905s +2.245s >> └─systemd-udev-trigger.service @1.242s +659ms >> └─systemd-udevd-control.socket @1.239s >> └─system.slice >> > > # systemd-analyze critical-chain > multi-user.target @2min 6.301s > └─tuned.service @1min 32.273s +34.024s > └─network.target @1min 31.590s > └─network-pre.target @1min 31.579s > └─glean(a)enp1s0.service @36.594s +54.952s > └─system-glean.slice @36.493s > └─system.slice @4.083s > └─-.slice @4.080s > > # systemd-analyze critical-chain NetworkManager.service > NetworkManager.service +9.287s > └─network-pre.target @1min 31.579s > └─glean(a)enp1s0.service @36.594s +54.952s > └─system-glean.slice @36.493s > └─system.slice @4.083s > └─-.slice @4.080s > > # cat /etc/sysconfig/network-scripts/ifcfg-enp1s0 > # Automatically generated, do not edit > DEVICE=enp1s0 > BOOTPROTO=static > HWADDR=52:54:00:1f:79:7e > IPADDR=192.168.122.42 > NETMASK=255.255.255.0 > ONBOOT=yes > NM_CONTROLLED=yes > > # ip addr > ... > 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state > UP group default qlen 1000 > link/ether 52:54:00:1f:79:7e brd ff:ff:ff:ff:ff:ff > inet 192.168.122.77/24 brd 192.168.122.255 scope global dynamic > noprefixroute enp1s0 > valid_lft 42957sec preferred_lft 42957sec > inet6 fe80::f182:7fb4:7a39:eb7b/64 scope link noprefixroute > valid_lft forever preferred_lft forever > > >> At a guess; I feel like the udev bits are probably not happening >> correctly in your case? That's important to get the glean@<interface> >> service in the chain to pre-create the config file >> > > It seems that the ordering is correct and the interface service is > executed, but the IP address is nonetheless wrong. > > Can it be related to how long glean takes to run in my case (54 seconds vs > 1 second in your case)? > > Dmitry > > >> >> -i >> >> [1] >> https://opendev.org/opendev/glean/src/branch/master/glean/init/glean-udev.r… >> [2] >> https://opendev.org/opendev/glean/src/branch/master/glean/init/glean-nm@.se… >> >> >

3 2

[nova][tripleo][rpm-packaging][kolla][puppet][debian][osa] Nova enforces that no DB credentials are allowed for the nova-compute service
by Balázs Gibizer 25 Nov '20

25 Nov '20

Dear packagers and deployment engine developers, Since Icehouse nova-compute service does not need any database configuration as it uses the message bus to access data in the database via the conductor service. Also, the nova configuration guide states that the nova-compute service should not have the [api_database]connection config set. Having any DB credentials configured for the nova-compute is a security risk as well since that service runs close to the hypervisor. Since Rocky[1] nova-compute service fails if you configure API DB credentials and set upgrade_level config to 'auto'. Now we are proposing a patch[2] that makes nova-compute fail at startup if the [database]connection or the [api_database]connection is configured. We know that this breaks at least the rpm packaging, debian packaging, and puppet-nova. The problem there is that in an all-in-on deployment scenario the nova.conf file generated by these tools is shared between all the nova services and therefore nova-compute sees DB credentials. As a counter-example, devstack generates a separate nova-cpu.conf and passes that to the nova-compute service even in an all-in-on setup. The nova team would like to merge [2] during Wallaby but we are OK to delay the patch until Wallaby Milestone 2 so that the packagers and deployment tools can catch up. Please let us know if you are impacted and provide a way to track when you are ready with the modification that allows [2] to be merged. There was a long discussion on #openstack-nova today[3] around this topic. So you can find more detailed reasoning there[3]. Cheers, gibi [1] https://github.com/openstack/nova/blob/dc93e3b510f53d5b2198c8edd22528f0c899… [2] https://review.opendev.org/#/c/762176 [3] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.… -- http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.…

11 53