February 2020 - openstack-discuss

[operators][cinder][nova][glance] possible data loss situation (bug 1852106)
by Brian Rosmaita 19 Feb '20

19 Feb '20

tl;dr: If you are running the OpenStack Train release, add the following to the [DEFAULT]non_inheritable_image_properties configuration option [0] in your nova.conf: cinder_encryption_key_id,cinder_encryption_key_deletion_policy About Your nova.conf ==================== - if you already have a value set for non_inheritable_image_properties, add the above to what you currently have - if non_inheritable_image_properties is *not* set in your current nova.conf, you must set its value to include BOTH the above properties AND the default values (which you can see when you generate a sample nova configuration file [1]) NOTE: At a minimum, the non_inheritable_image_properties list should contain: - the properties used for image-signature-validation (these are not transferrable from image to image): * img_signature_hash_method * img_signature * img_signature_key_type * img_signature_certificate_uuid - the properties used to manage the keys for images of cinder encrypted volumes: * cinder_encryption_key_id * cinder_encryption_key_deletion_policy - review the documentation to determine whether you want to include the other default properties: * cache_in_nova * bittorrent Details ======= This issue is being tracked as Launchpad bug 1852106 [2]. This is probably a low-occurrence situation, because in order for the issue to occur, all of the following must happen: (0) using the OpenStack Train release (or code from master (Ussuri development)) (1) cinder_encryption_key_id and cinder_encryption_key_deletion_policy are NOT included in the non_inheritable_image_properties setting in nova.conf (which they aren't, by default) (2) a user has created a volume of an encrypted volume-type in the Block Storage service (cinder). Call this Volume-1 (3) using the Block Storage service, the user has uploaded the encrypted volume as an image to the Image service (glance). Call this Image-1 (4) using the Compute service (nova), the user has attempted to directly boot a server from the image. (Note: this is an unsupported action, the supported workflow is to use the image to boot-from-volume.) (5) although an unsupported action, if a user does (4), it currently results in a server in status ACTIVE but which is unusable because the operating system can't be found (6) using the Compute service, the user requests the createImage action on the unusable (yet ACTIVE) server, resulting in Image-2 (7) using the Image service, the user deletes Image-2 (which has inherited the cinder_encryption_key_* properties from Image-1) upon which the encryption key is deleted, thereby rendering Image-1 non-decryptable so that it can no longer be used in the normal boot-from-volume workflow NOTE 1: the cinder_encryption_key_deletion_policy image property was introduced in Train. In pre-Train releases, deleting the useless Image-2 in step (7) does NOT result in encryption key deletion. NOTE 2: Volume-1 created in step (2) has a *different* encryption key ID than the one associated with Image-1. Thus, even in the scenario where Image-1 becomes non-decryptable, Volume-1 is not affected. Workaround ========== When cinder_encryption_key_id,cinder_encryption_key_deletion_policy are added to the non_inheritable_image_properties setting in nova.conf, the useless Image-2 created in step (6) above will not have the image properties on it that enable Glance to delete the encryption key still in use by Image-1. This does not, however, protect images for which steps (4)-(6) have been performed before the deployment of this workaround. The safest way to deal with images created before the workaround is deployed is to remove the cinder_encryption_key_deletion_policy image property from any image that has it (or to change its value to 'do_not_delete'). While it is possible to use other image properties to identify images created by Nova as opposed to images created by Cinder, this is not guaranteed to be reliable because image properties may have been modified or removed by the image owner. Proposed Longer-term Fixes ========================== - In the Ussuri release, the unsupported action in step (4) above will result in a 400 rather than an active yet unusable server. Hence it will no longer be possible to create the image of the unusable server that causes the issue. [3] - Additionally, given that the image properties associated with cinder encrypted volumes and image signature validation are specific to a single image and should not be inherited by server snapshots under any circumstances, in Ussuri these "absolutely non-inheritable image properties" will no longer be required to appear in the non_inheritable_image_properties configuration setting in order to prevent them from being propagated to server snapshots. [4] References ========== [0] https://docs.openstack.org/nova/train/configuration/config.html#DEFAULT.non… [1] https://docs.openstack.org/nova/train/configuration/sample-config.html [2] https://bugs.launchpad.net/nova/+bug/1852106 [3] https://review.opendev.org/#/c/707738/ [4] https://review.opendev.org/#/c/708126/

2 1

[operators][cinder] driver removal policy update
by Brian Rosmaita 19 Feb '20

19 Feb '20

The Cinder driver removal policy has been updated [0]. Drivers eligible for removal, at the discretion of the team, may remain in the code repository as long as they continue to pass OpenStack CI testing. When such a driver blocks the CI check or gate, it will be removed immediately. The intent of the policy revision is twofold. First, it gives vendors a longer grace period in which to make the necessary changes to have their drivers reinstated as ‘supported’. Second, keeping these drivers in-tree longer should make life easier for operators who have deployed storage backends with drivers that have been marked as ‘unsupported’. Please see the full statement of the policy [0] for details. [0] https://docs.openstack.org/cinder/latest/drivers-all-about.html#driver-remo…

1 0

[cinder] volume-local-cache spec - last call
by Brian Rosmaita 19 Feb '20

19 Feb '20

The volume-local-cache spec [0], which was granted a Cinder spec-freeze exception, currently has two +2s. It's been reviewed by a lot of people, and as far as I can tell, everyone's concerns have been addressed at this point. However, I'd like to give people one last opportunity to verify this for themselves. Unless a substantive objection is raised, I intend to approve the spec around 12:30 UTC tomorrow (Thursday) so that it can be merged before the Nova meeting at 14:00 UTC. cheers, brian [0] https://review.opendev.org/#/c/684556/

1 0

[kolla] Following Nova with XenAPI deprecation
by Radosław Piliszek 19 Feb '20

19 Feb '20

Hello fellow OpenStackers! There exists some logic in Kolla Ansible that handles XenAPI-specific overrides and bootstrapping with os-xenapi. It is one of those murky, untested features that we prefer to remove as soon as possible. :-) Hence we follow Nova in deprecating this functionality for removal. We will do the removal in Victoria. If you are using XenAPI with recent Kolla-Ansible, please let us know. I will be proposing deprec changes later today. As far as other deployment projects are concerned, DevStack, OpenStack-Helm and Puppet-OpenStack seem to still handle XenAPI specifics. -yoctozepto

1 0

[OSSA-2020-001] Nova can leak consoleauth token into log files (CVE-2015-9543)
by Jeremy Stanley 19 Feb '20

19 Feb '20

============================================================= OSSA-2020-001: Nova can leak consoleauth token into log files ============================================================= :Date: February 19, 2020 :CVE: CVE-2015-9543 Affects ~~~~~~~ - Nova: <18.2.4,>=19.0.0<19.1.0,>=20.0.0<20.1.0 Description ~~~~~~~~~~~ Paul Carlton from HP reported a vulnerability in Nova. An attacker with read access to the serviceâ€™s logs may obtain tokens used for console access. All Nova setups using novncproxy are affected. Patches ~~~~~~~ - https://review.opendev.org/707845 (Queens) - https://review.opendev.org/704255 (Rocky) - https://review.opendev.org/702181 (Stein) - https://review.opendev.org/696685 (Train) - https://review.opendev.org/220622 (Ussuri) Credits ~~~~~~~ - Paul Carlton from HP (CVE-2015-9543) References ~~~~~~~~~~ - https://launchpad.net/bugs/1492140 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-9543 Notes ~~~~~ - The stable/queens branch is under extended maintenance and will receive no new point releases, but a patch for it is provided as a courtesy. -- Jeremy Stanley, on behalf of OpenStack Vulnerability Management

1 0

[oslo][release] oslo.limit mistakenly released as 1.0.0
by Ben Nemec 19 Feb '20

19 Feb '20

Hi, We seem to have created a bit of a problem with the latest oslo.limit release. In keeping with our general policy of bumping the major version when we release libraries without py2 support, oslo.limit got bumped to 1.0. Unfortunately, being a pre-1.0 library it should have only had the minor version bumped. This puts us in an awkward situation since the library is still under heavy development and we expect the API will change, possibly multiple times, before we're ready to commit to a stable API. We also need the ability to continue doing releases during development so we can test the library with consuming projects. I can think of a few options, although I won't guarantee all of these are even possible: * Unpublish 1.0.0 and do further pre-1.0 development on a feature branch cut from before we released 1.0.0. Once we're ready for "1.0", we merge the feature branch to master and release it as 2.0.0. * Stick a big disclaimer in the 1.0.0 docs that it is still under development and proceed to treat 1.0 the same as we would have treated a pre-1.0 library. Again, when ready for "1.0" we tag it 2.0.0. * Make our breaking changes as needed and just continue bumping the major version every release. This unfortunately makes it hard to communicate via the version when the library is ready for use. :-/ * [some better idea that you suggest :-)] Any input on the best way to handle this is greatly appreciated. Thanks. -Ben

9 22

[vmware][hyperv][kolla] Is there any interest? (or should we deprecate and remove?)
by Radosław Piliszek 19 Feb '20

19 Feb '20

Hello Fellow Stackers, In Kolla and Kolla-Ansible we have some support for VMware (both hypervisor and networking controller stuff) and Hyper-V. The issue is the relevant code is in pretty bad shape and we had no recent reports about these being used nor working at all for that matter and we are looking into dropping support for these. Please respond if you are interested in these. Long term we would require access to some CI running these to really keep things in shape. -yoctozepto

1 1

Re: [openstack-dev][kuryr] Not working in ARM64 (as Node)
by mdulko＠redhat.com 19 Feb '20

19 Feb '20

Hi, So from this run you need the kuryr-controller logs. Apparently the pod never got annotated with an information about the VIF. Thanks, Michał On Fri, 2020-02-07 at 05:14 +0000, VeeraReddy wrote: > Hi mdulko, > Thanks for your support. > > As you mention i removed readinessProbe and > livenessProbe from Kuryr pod definitions. Still i am facing issue , unable to create pod. > > > > > Attached kubelet and kuryr-cni logs. > > > > Regards, > Veera. > > > On Thursday, 6 February, 2020, 05:19:12 pm IST, <mdulko(a)redhat.com> wrote: > > > Hm, nothing too troubling there too, besides Kubernetes not answering > on /healthz endpoint. Are those full logs, including the moment you > tried spawning a container there? It seems like you only pasted the > fragments with tracebacks regarding failures to read /healthz endpoint > of kube-apiserver. That is another problem you should investigate - > that causes Kuryr pods to restart. > > At first I'd disable the healthchecks (remove readinessProbe and > livenessProbe from Kuryr pod definitions) and try to get fresh set of > logs. > > On Thu, 2020-02-06 at 10:46 +0000, VeeraReddy wrote: > > Hi mdulko, > > Please find kuryr-cni logs > > http://paste.openstack.org/show/789209/ > > > > > > Regards, > > Veera. > > > > > > On Thursday, 6 February, 2020, 04:08:35 pm IST, <mdulko(a)redhat.com> wrote: > > > > > > Hi, > > > > The logs you provided doesn't seem to indicate any issues. Please > > provide logs of kuryr-daemon (kuryr-cni pod). > > > > Thanks, > > Michał > > > > On Thu, 2020-02-06 at 07:09 +0000, VeeraReddy wrote: > > > Hi, > > > I am trying to run kubelet in arm64 platform > > > 1. Generated kuryr-cni successfullly. using kur-cni Dockerfile > > > 2. Generated kuryr-cni-arm64 container. > > > 3. my kube-kuryr-arm64.yml (http://paste.openstack.org/show/789208/) > > > > > > My master node in x86 installed successfully using devstack > > > > > > While running kubelet in arm platform , not able to create kubelet interface (kubelet logs: http://paste.openstack.org/show/789206/) > > > > > > COntroller logs: http://paste.openstack.org/show/789209/ > > > > > > Please help me to fix the issue > > > > > > Veera. > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > Veera. > > > > > >

2 6

Virtio memory balloon driver
by Albert Braden 18 Feb '20

18 Feb '20

When we build a Centos 7 VM with 1.4T RAM it fails with "[ 17.797177] BUG: unable to handle kernel paging request at ffff988b19478000" I asked in #centos and they asked me to show a list of devices from a working VM (if I use 720G RAM it works). This is the list: [root@alberttest1 ~]# lspci 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device 00:04.0 SCSI storage controller: Red Hat, Inc. Virtio block device 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device 00:06.0 SCSI storage controller: Red Hat, Inc. Virtio block device 00:07.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [root@alberttest1 ~]# lsusb Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub They suspect that the "Virtio memory balloon" driver is causing the problem, and that we should disable it. I googled around and found this: http://www.linux-kvm.org/page/Projects/auto-ballooning It looks like memory ballooning is deprecated. How can I get rid of the driver? Also they complained about my host bridge device; they say that we should have a newer one: 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) Where can I specify the host bridge? <bugs_> ok ozzzo one of the devices is called "virtio memory balloon" [13:18:12] <bugs_> do you see that? [13:18:21] <ozzzo> yes [13:18:47] <bugs_> i suggest you google that and read about what it does - i think it would [13:19:02] <bugs_> be worth trying to disable that device on your larger vm to see what happens [13:19:18] <ozzzo> ok I will try that, thank you [13:19:30] * Altiare (~Altiare@unaffiliated/altiare) has quit IRC (Quit: Leaving) [13:21:45] * Sheogorath[m] (sheogora1(a)gateway/shell/matrix.org/x-uiiwpoddodtgrwwz) joins #centos [13:22:06] <@TrevorH> I also notice that the VM seems to be using the very old 440FX and there's a newer model of hardware available that might be worth checking [13:22:21] <@TrevorH> 440FX chipset is the old old pentium Pro chipset! [13:22:32] <@TrevorH> I had one of those in about 1996

4 11

ops meetups team meeting 2020-2-18
by Chris Morgan 18 Feb '20

18 Feb '20

The OpenStack Ops Meetups team meeting was held today on IRC, minutes linked below. Key links vancouver opendev+ptg : https://www.openstack.org/events/opendev-ptg-2020/ South Korea Ops Meetup proposal : https://etherpad.openstack.org/p/ops-meetup-2nd-korea-2020 OpenInfra summit 2020 announced for Berlin, Oct 19-23 : https://superuser.openstack.org/articles/inside-open-infrastructure-the-lat… The meetups team will be proposing content for Vancouver and will shortly solicit feedback on preferred dates for the Ops Meetups even in South Korea. Please follow https://twitter.com/osopsmeetup for more announcements. Minutes: <•openstack> Meeting ended Tue Feb 18 15:37:42 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) 10:37 AM Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2020/ops_meetup_tea… 10:37 AM Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2020/ops_meetup_tea… 10:37 AM Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2020/ops_meetup_tea… Chris Morgan - on behalf of the OpenStack Ops Meetups team ( https://wiki.openstack.org/wiki/Ops_Meetups_Team) -- Chris Morgan <mihalis68(a)gmail.com>

1 0