October 2022 - openstack-discuss - lists.openstack.org

[openstack-helm] No Weekly IRC Meeting - PTG
by Gage Hugo 17 Oct '22

17 Oct '22

Hey team, This week is the PTG so the weekly IRC meeting is cancelled. We will be meeting for an hour at the same time slot for our PTG session: https://ptg.opendev.org/ptg.html

1 0

[0SSN-0090] Best practices when configuring Glance with COW backends
by Brian Rosmaita 17 Oct '22

17 Oct '22

Best practices when configuring Glance with COW backends --- ### Summary ### When deploying Glance in a popular configuration where Glance shares a common storage backend with Nova and/or Cinder, it is possible to open some known attack vectors by which malicious data modification can occur. This note reviews the known issues and suggests a Glance deployment configuration that can mitigate such attacks. ### Affected Services / Software ### Glance, all supported releases (Queens through Zed) ### Discussion ### This note applies to you if you are operating an end-user-facing glance-api service with the 'show_multiple_locations' option set to True (the default value is False) or if your end-user-facing glance-api has the 'show_image_direct_url' option set to True (default value is False). Our recommendation is that the image "locations" and "direct_url" fields [0] *never* be displayed to end users in a cloud. This can be accomplished by running two glance-api services: - A "user-facing" glance-api that is accessible to end users and which appears in users' service catalogs. - An "internal-only-facing" glance-api that is accessible only to those services that require access to the 'direct_url' or image location fields, and which is protected by firewalls from access by end users. (Nova, Cinder, and Ironic all have configuration options to specify the Glance API endpoint each service uses [1].) This dual glance-api deployment was suggested in "Known Issues" in Glance release notes in the Rocky [2] through Ussuri releases, but it seems that the idea has not received sufficient attention. Hence this security note. The attack vector that becomes available when image locations are exposed to end users was originally outlined in OSSN-0065 [3], though that note was not clear about the attack surface or mitigation, and contained some forward-looking statements that were not fulfilled. The attack vector is: [A] malicious user could create an image in Glance, set an additional location on that image pointing to an altered image, then delete the original location, so that consumers of the original image would unwittingly be using the malicious image. Note, however, that this attack vector cannot change the original image's checksum, and it is limited to images that are owned by the attacker. OSSN-0065 suggests that this is only an issue when users do not checksum their image data. It neglects the fact that in some popular deployment configurations in which Nova creates a root disk snapshot, data is never uploaded to Glance, but instead a snapshot is created directly in the backend and Nova creates a Glance image record with "size" 0 and an empty "os_hash_value" [4], making it impossible to compare the hash of downloaded image data to the value maintained by Glance. Further, when Nova is configured to use the same storage for ephemeral disks that is used as a Glance image store, Nova efficiently creates a server root disk directly in the backend without checksumming the image data. This is an intentional design choice to optimize storage space and host resources, but an implication is that even if the image record has a recorded hash, it is not being checked at the point of image consumption. Similarly, when using a shared backend, or a cinder glance_store, Cinder will efficiently clone a volume created from an image directly in the backend without checksumming the image data. Again, this is done intentionally in order to optimize resources, but it is important to be aware of the security tradeoff being made by this configuration. In other words, if the image data is not going to be checked at the point of image consumption, then extra care needs to be taken to ensure the integrity of the data path. OSSN-0065 suggested that the attack vector of substituting image data by modifying the image locations could be addressed by using policies, but that has turned out not to be the case. The only way currently to mitigate this vector is to deploy glance-api in a dual configuration as described above, namely with an internal-only-facing glance-api used by Nova and Cinder (that has show_multiple_locations enabled), and an end-user-facing glance-api (that has show_multiple_locations disabled). So far the focus has been on 'show_multiple_locations'. When that setting is disabled in Glance, it is not possible to manipulate the locations via the OpenStack Images API. Keep in mind, however, that in any Glance/Nova/Cinder configuration where Nova and/or Cinder do copy-on-write directly in the image store, image data transfer takes place outside Glance's image data download path, and hence the os_hash_value is *not* checked. Thus, if the backend store is itself compromised and image data is replaced directly in the backend, the substitution will *not* be detected. This brings us to the 'show_image_direct_url' option, which includes a "direct_url" field in the image-show response that can be used by various OpenStack services to consume images directly from the storage backend. Exposing the 'direct_url' to end users leaks information about the storage backend. What exactly that information consists of depends upon the backend in use and how it is configured, but in general, it is not a good idea to provide hints that could be useful to malicious actors in their attempts to compromise the backend storage by some type of independent exploit. The 'direct_url', being read-only, may appear innocuous, but its use by services is usually to perform some kind of optimized image data access that most likely does not include computing a hash of the image data. We therefore recommend that OpenStack services that require exposure of the 'direct_url' image property be similarly configured to use an internal-only-facing glance-api. It is worth nothing that end users who wish to download an image do not require access to the 'direct_url' image property because they can simply use the image-data-download API call [5]. ### Recommended Actions ### A glance-api service with 'show_multiple_locations' enabled should *never* be exposed directly to end users. This setting should only be enabled on an internal-only-facing glance-api that is used by OpenStack services that require access to image locations. This could be done, for example, by running two glance-api services with different configuration files and using the appropriate configuration options for each service to specify the Image API endpoint to access, and making sure the special internal endpoint is firewalled in such a way that only the appropriate OpenStack services can contact it. Similarly, enabling 'show_image_direct_url' exposes information about the storage backend that could be of use to malicious actors in as yet unknown exploits, so it should likewise only be enabled on an internal-only-facing glance-api. ### Notes / References ### [0] https://docs.openstack.org/api-ref/image/v2/index.html#show-image-schema [1] Nova and Ironic use 'endpoint_override' in the '[glance]' section of the configuration file; Cinder uses 'glance_api_servers' in the '[DEFAULT]' section. [2] OSSN-0065: https://wiki.openstack.org/wiki/OSSN/OSSN-0065 [3] The Glance "multihash" metadata pair of 'os_hash_algo' and 'os_hash_value' were introduced in Rocky to replace the legacy md5 'checksum' field. The python-glanceclient has used multihash checksumming for download verification since version 2.13.0. [4] https://docs.openstack.org/releasenotes/glance/rocky.html#known-issues [5] https://docs.openstack.org/api-ref/image/v2/index.html?#download-binary-ima… ### Contacts / References ### Author: Brian Rosmaita, Red Hat This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0090 Original LaunchPad Bug : https://bugs.launchpad.net/ossn/+bug/1990157 Mailing List : [Security] tag on openstack-discuss(a)lists.openstack.org OpenStack Security Project : https://launchpad.net/~openstack-ossg CVE: none

1 0

[swift] upgrade newton to train in containers
by Marcel 17 Oct '22

17 Oct '22

I'm planning an upgrade for our newton based swift cluster (OOO vm based) to a (kolla image based containers) train cluster So far I have tested the upgrade for the proxies, account_servers and container servers and it looks promising. I have in a test environment: - switched to the new proxies with the old ring files and it looks like everything works normally - Added the new (train) account and container servers to the rings and it looks like all is fine - Removed the old account and container servers, still fine - tested fall back, also fine My question actually is: Did the account and container database format change between newton and train in such a way that I might run into troubles trying the upgrade as tested above in ways that I did not yet foresee? Thanks Marcel

1 0

[Neutron] Bug Deputy Report Oct 10 - 17
by Jakub Libosvar 17 Oct '22

17 Oct '22

Hi all, I was the bug deputy last week, here is the report. Nothing that requires immediate attention. Medium ------ - [ovn-octavia-provider] Detach OVN-LB LS from the LR breaks OVN-LB connectivity https://bugs.launchpad.net/neutron/+bug/1992363 assigned to Luis Fix proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/860781 - [scale] Setting a gateway on router is killing database https://bugs.launchpad.net/neutron/+bug/1992950 Assigned to Arnaud Morin Fix proposed: https://review.opendev.org/c/openstack/neutron/+/861322 - Disable in-band flow management in the ovs bridges https://bugs.launchpad.net/neutron/+bug/1992953 Assigned to Slawek Fix proposed: https://review.opendev.org/c/openstack/neutron/+/861351 Wishlist ------- - fip will loss when it migrate between dvr-sant agent and dvr_no_external in Rocky https://bugs.launchpad.net/neutron/+bug/1992542 The reported bug uses unsupported configuration dvr_snat agent mode on the compute nodes. Cheers Kuba

1 0

[kolla][ptg] Using Meetpad instead of Zoom
by Michał Nasiadka 17 Oct '22

17 Oct '22

Hello, If you’re planning on joining the Kolla PTG - please use Meetpad - https://meetpad.opendev.org/kolla-antelope-ptg Sorry for last minute changes Best regards, Michal

1 0

[ptg][nova][placement] Nova/Placement at the PTG, howto
by Sylvain Bauza 17 Oct '22

17 Oct '22

Hey stackers, Got some questions about how/when joining the Nova community at the PTG during this week, so I'll provide you a howto ;) *What is the main document for the Nova PTG ?* This : https://etherpad.opendev.org/p/nova-antelope-ptg Please, please, don't try to translate this etherpad directly as you'll see. *I'm an operator and I want to tell you how terrible you folks are* There : https://etherpad.opendev.org/p/oct2022-ptg-operator-hour-nova Again, don't try to translate this document or you'll create problems for people who don't know your language. *When do you folks discuss ?* Monday : 14:00-15:00 UTC Ironic/Nova cross-project session Tuesday : 13:00 - 15:00 UTC Operator hours. If you are an operator, please join us ! :-) 15:00 - 17:00 UTC Nova sessions for contributors 15:10 UTC (possibly) : Nova team photo Wednesday : 13:00 - 16:00 UTC Nova sessions for contributors 16:00 - 17:00 Operator hour. If you are an operator, please join us ! :-) Thursday : 13:00 - 15:00 UTC Neutron/Nova cross-project session 15:00 - 17:00 UTC Nova sessions for contributors Friday : 13:00 - 17:00 UTC Nova sessions for contributors *How to join ?* Simple. You can look at https://ptg.opendev.org/ptg.html if you want so you'll see what topic we are currently discussing. If you want to arrive, just click either on "operator-hour" nova tag or "nova-placement" tag and it will automatically open the Google Meet client for joining the Nova meeting. If you also want, you can directly join us by using this link https://www.openstack.org/ptg/rooms/bexar (we'll use the same virtual room for all our sessions) *What if you can't join the whole days ?* No worries ! Just make sure you use an IRC client and just go to the #openstack-nova channel. Once you're there, add your IRC in the etherpad for every topic you'd like to be around and then I'll ping you when we start discussing for those topics you added your nick. *What if I want to add some topic I'd like to discuss ?* Easy peasy. Two possibilities : - you are a contributor and you want to add a new feature in nova. Just add your topic in the main etherpad (link above). Make sure your name is marked in the topic title (like every other topic) and I'll ping you when we start discussing your topic. - you are an operator and you want to discuss wishlists or bugs or whatever else. Good news, we have specific timeslots for you ! See the agenda above, we have *operator hours* that are intended for operators to join and engage on specific deployment and operational topics. Just then look at https://etherpad.opendev.org/p/oct2022-ptg-operator-hour-nova and add your points, we'll discuss them during the operator hours (given above). Hope you'll find your way and you'll appreciate the PTG as much productive as we do. In any case, my IRC nick is bauzas. Don't hesitate to bug me over IRC (#openstack-dev, #openstack-nova or #openinfra-events) or reach me by email (sbauza(a)redhat.com), I'll try to answer you as much as I can. HTH and thanks, -Sylvain

1 0

Re: Installing out-of-tree ML2 neutron plugins with kolla-ansible
by Radosław Piliszek 17 Oct '22

17 Oct '22

Hi Aditya, First of all, please remember to always CC the mailing list so that other users can benefit from the answers. Regarding the question at hand - the same documentation (starting from the beginning of the page) describes the kolla-build tool, including the usual way to install it. Remember to install the version for your OpenStack release - the right source of versions is [1]. The kolla-build tool is the tool to build the images. I assume so far you have used the prebuilt ones. This time you will have to build your own because you want to add software inside of them. You might also be interested in deploying your own local registry. The simplest way to do so is via [2] but understand it is not production-ready unless in a strictly isolated network (for other use cases I generally recommend Harbor [3]). [1] https://docs.openstack.org/releasenotes/kolla/ [2] https://docs.openstack.org/kolla-ansible/yoga/user/multinode.html#deploy-a-… [3] https://goharbor.io/ Kind regards, Radek -yoctozepto On Sun, 16 Oct 2022 at 21:43, Aditya Sathish <saditya(a)vt.edu> wrote: > > Hi Radek, > > This was incredibly helpful. However, I couldn't find where the kolla-build.conf file is and should I create my own, where I have to add this. > > Regards, > Aditya. > > On Sun, Oct 16, 2022 at 3:10 PM Radosław Piliszek <radoslaw.piliszek(a)gmail.com> wrote: >> >> I believe you want to first install the plugin in the container image. >> The section on plugins might be of interest to you. [1] >> >> [1] https://docs.openstack.org/kolla/yoga/admin/image-building.html#plugin-func… >> >> Kind regards, >> Radek >> -yoctozepto >> >> On Sun, 16 Oct 2022 at 20:31, Aditya Sathish <saditya(a)vt.edu> wrote: >> > >> > Hello, >> > >> > I am trying to use an out-of-tree ML2 neutron plugin with OpenStack with Kolla-Ansible but I am having a hard time figuring out how to go about it. >> > >> > For example, this is my repository for the plugin: https://github.com/adityasathis/networking-onos. >> > >> > I have made changes to the deployment YML files to allow configuration for this new neutron plugin from the global.yml file. However, I am not able to figure out how to copy over my plugin files and install them on the controller node. >> > >> > I came across the commit: https://opendev.org/openstack/kolla-ansible/commit/418cb52767270d85e28a6f30… which, I think, does what I'm looking to do and so I kept the networking-onos directory (with the setup.py in it) in the /etc/kolla/config/neutron/plugins directory. The deploy script is able to detect the file in the "Checking for ML2 plugins" step however, I'm not able to copy it anywhere in the "Copying ML2 plugin" step. Am I missing something? >> > >> > Regards, >> > Aditya.

3 2

[ptg][cinder] Finalized Schedule
by Rajat Dhasmana 17 Oct '22

17 Oct '22

Hello Everyone, I've finalized the schedule for cinder PTG[1] based on the topics, number of hours of PTG and as per the date and time suited to the authors. Please go through it and let me know if any changes are needed. I've also included a *Courtesy Ping* section with each topic so the people interested in a particular topic will get a ping when it is being discussed. Please include your IRC nick in front of it if you would like to be notified. The schedule includes a Day wise topic list where, except for monday, we've PTG from 1300-1700 UTC (where the last hour acts as a buffer time for extended topic discussions or other planned activities). Following are some of the highlight events of each day, Monday: TC + PTL session: 1500-1700 UTC Tuesday: First day of cinder PTG Operator hour from 1500-1600 UTC Wednesday: Team photo at 1400 UTC (timing could be changed based on the topic duration before it) Thursday: Drivers Day! Friday: Festival of XS reviews (If we've enough time left) [1] https://etherpad.opendev.org/p/zed-ptg-cinder Thanks and regards Rajat Dhasmana

1 1

[TripleO][PTG]TripleO Antelope PTG schedule and other relevant Information
by Rabi Mishra 17 Oct '22

17 Oct '22

Hi All, As you all would already know, 2023.1(Antelope) PTG will start in a few hours from now. TripleO PTG schedule has been published @ https://etherpad.opendev.org/p/oct2022-ptg-tripleo for your reference. We would have sessions spanning across 2 days i.e. 17th Oct(Monday) and 18th Oct(Tuesday) from 13:00-17:00 UTC @https://www.openinfra.dev/ptg/rooms/ocata . Hope to see you there. Some other relevant information: - Please register for the event, if you've not already done so @ https://openinfra-ptg.eventbrite.com - PTG Page https://ptg.opendev.org/ptg.html - PTG Bot: https://opendev.org/openstack/ptgbot -- Regards, Rabi Mishra

1 0

[Glance]Weekly Meeting cancelled for Next 2 Weeks
by Pranali Deore 17 Oct '22

17 Oct '22

Hello, There will be no weekly meeting *this week* due to PTG sessions and *next week of PTG *as well, as most of the team members will not be around during that week. Next meeting will be directly on Nov 3rd. See you in the PTG !! Thanks, Pranali Deore

1 0