October 2020 - openstack-discuss - lists.openstack.org

[Trove] Project udpate
by Lingxian Kong 06 Oct '20

06 Oct '20

Hi there, As the official Victoria release is approaching and it has been a long time silence for Trove in the upstream, I think it's good time for me as the Trove PTL for the last 3 dev cycles to have a project update. The things that will be described below have not been achieved in one single dev cycle, but are some significant changes since the 'dark time' of Trove project in the past. Tips hat to those who have made contributions to Trove project ever before. ## Service tenant configuration Service tenant configuration was added in Stein release, before that, it's impossible to deploy Trove in the public cloud (even not for some private cloud due to security concerns) because the user may have access to the guest instance which contains sensitive data in the config files, the users can also perform operations towards either storage or networking resources which may bring much management overhead and make it easy to break the database functionality. With service tenant configuration (which is currently the default setting in devstack), almost all the cloud resources(except the Swift objects for backup data) created for a Trove instance are only visible to the Trove service user. As Trove users, they can only see a Trove instance, but know nothing about the Nova VM, Cinder volume, Neutron management network, and security groups under the hood. The only way to operate Trove instances is to interact with Trove API. ## Message queue security concerns To do database operations, trove controller services communicate with trove-guestagent service inside the instance via message queue service (i.e. RabbitMQ in most environments). In the meantime, trove-guestagent periodically sends status update information to trove-conductor through the same messaging system. In the current design, the RabbitMQ username and password need to be configured in the trove-guestagent config file, which brought significant security concern for the cloud deployers in the past. If the guest instance is compromised, then guest credentials are compromised, which means the messaging system is compromised. As part of the solution, a security enhancement was introduced in the Ocata release, using encryption keys to protect the messages between the control plane and the guest instances. First, the rabbitmq credential should only have access to trove services. Second, even with the rabbitmq credential and the message encryption key of the particular instance, the communication from the guest agent and trove controller services are restricted in the context of that particular instance, other instances are not affected as the malicious user doesn't know their message encryption keys. Additionally, since Ussuri, trove is running in service tenant model in devstack by default which is also the recommended deployment configuration. Most of the cloud resources(except the Swift objects for backup data) created for a trove instance should only be visible to the trove service user, which also could decrease the attack surface. ## Datastore images Before Victoria, Trove provided a bunch of diskimage-builder elements for building different datastore images. As contributors were leaving, most of the elements just became unmaintained except for MySQL and MariaDB. To solve the problem, database containerization was introduced in Victoria dev cycle, so that the database service is running in a docker container inside the guest instance, trove guest agent is pulling container image for a particular datastore when initializing guest instance. Trove is not maintaining those container images. That means, since Victoria, the cloud provider only needs to maintain one single datastore image which only contains common code that is datastore independent. However, for backward compatibility, the cloud provider still needs to create different datastores but using the same Glance image ID. Additionally, using database container also makes it much easier for database operations and management. To upgrade from the older version to Victoria onwards, the Trove user has to create backups before upgrading, then create instances from the backup, so downtime is expected. ## Supported datastores Trove used to support several datastores such as MySQL, MariaDB, PostgreSQL, MongoDB, CouchDB, etc. Most of them became unmaintained because of a lack of maintainers in the community. Currently, only MySQL and MariaDB drivers are fully supported and tested in the upstream. PostgreSQL driver was refactored in Victoria dev cycle and is in experimental status again. Adding extra datastores should be quite easy by implementing the interfaces between trove task manager and guest agent. Again, no need to maintain separate datastore images thanks to the container technology. ## Instance backup At the same time as we were moving to use container for database services, we also moved the backup and restore functions out of trove guest agent code because the backup function is usually using some 3rd party software which we don't want to pre-install inside the datastore image. As a result, we are using container as well for database backup and restore. For more information about the backup container image, see https://lingxiankong.github.io/2020-04-14-database-backup-docker-image.html. ## Others There are many other improvements not mentioned above added to Trove since Train, e.g. * Access configuration for the instance. * The swift backend customization for backup. * Online volume resize support. * XFS disk format for database data volume. * API documentation improvement. * etc. By the way, Catalyst Cloud has already deployed Trove (in Alpha) in our public cloud in New Zealand, we are getting feedback from customers. I believe there are other deployers already have Trove in their production but running an old version because of previous upstream situation in the past. If you are one of them and interested in upgrading to the latest, please either reply to this email or send personal email to me, I would be very happy to provide any help or guidance. For those who are still in evaluation phase, you are also welcome to reach out for any questions. I'm always in the position to help in #openstack-trove IRC channel. --- Lingxian Kong Senior Software Engineer Catalyst Cloud www.catalystcloud.nz

2 1

[tripleo][ci] RFC, supported releases of TripleO
by Wesley Hayutin 06 Oct '20

06 Oct '20

Greetings, I just wanted to give folks the opportunity to speak up before we make any changes. Please review https://releases.openstack.org/ As we are branching master -> victoria it looks like we can definitely remove stable/rocky from both upstream and RDO software factory. Any comments about rocky? Stein is end of maintenance on 2020 November 11th. Any thoughts on dropping stein from upstream and rdo software factory? Thanks! Please be specific if you have comments as to which release you are referring to.

4 5

Modify devstack after basic deployment
by Md. Hejbul Tawhid MUNNA 06 Oct '20

06 Oct '20

Hi, I have deployed a basic openstack using devstack. now I want to install additional component like designate or magnum. What is the way to do that? Regards, Munna

2 1

[nova]: Separation of VM size and extra specs in flavors
by Csatari, Gergely (Nokia - FI/Espoo) 06 Oct '20

06 Oct '20

Hi, During some discussions about CNTT<https://cntt.readthedocs.io/>, where several flavor sizes and extra specs are specified we realized, that the combination of all flavor sizes and the different extra specs results in a very large number of flavors. One idea was to separate the size and the extra specs on the creation of flavors and just define some kind of rules do define how the different sizes and extra specs can be combined. The other idea was just to leave things as they are. Are there any opinions on which idea is better? Thanks, Gerg0

2 1

[tripleo][ussuri][release] bugfix stable/ussuri release for tripleo 'cycle-with-intermediary'
by Marios Andreou 06 Oct '20

06 Oct '20

o/ tripleo On request I made a bugfix stable/ussuri release for the tripleo 'cycle-with-intermediary' repos. If you are interested see [1] for the latest hashes. If you need a release for a different branch reach out to me. thanks, marios [1] https://review.opendev.org/#/c/755718/

1 0

[Neutron] Cancelled QoS team meeting October 6
by Rodolfo Alonso Hernandez 06 Oct '20

06 Oct '20

Hello: Due to the lack of agenda, the Neutron QoS meeting will be cancelled today. Next meeting will be held on October 20. Regards.

1 0

qemu ussuri can't launch instance
by ops＠clustspace.com 06 Oct '20

06 Oct '20

Hello, Does anyone know where to find the problem? Searched few days in google :) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [req-f16a5b6c-3475-4322-a664-45f3bd9c041a cb4c181d25a14dda958e40792968f97e 5242bcab73714ab192cca8d909b72e9a - default default] [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] Instance failed to spawn: libvirt.libvirtError: internal error: cannot load AppArmor$ 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] Traceback (most recent call last): 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2614, in _build_resources 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] yield resources 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2378, in _build_and_run_instance 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] accel_info=accel_info) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3580, in spawn 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] power_on=power_on) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6438, in _create_domain_and_network 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] destroy_disks_on_failure) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] self.force_reraise() 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] six.reraise(self.type_, self.value, self.tb) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] raise value 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6410, in _create_domain_and_network 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] post_xml_callback=post_xml_callback) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6351, in _create_domain 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] guest.launch(pause=pause) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 142, in launch 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] self._encoded_xml, errors='ignore') 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] self.force_reraise() 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] six.reraise(self.type_, self.value, self.tb) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] raise value 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 137, in launch 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] return self._domain.createWithFlags(flags) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] result = proxy_call(self._autowrap, f, *args, **kwargs) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] rv = execute(f, *args, **kwargs) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] six.reraise(c, e, tb) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] raise value 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] rv = meth(*args, **kwargs) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] File "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in createWithFlags 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f] libvirt.libvirtError: internal error: cannot load AppArmor profile 'libvirt-63bb4205-fcbb-4f2b-9ade-e055e7ff966f' 2020-10-05 23:11:04.054 1585 ERROR nova.compute.manager [instance: 63bb4205-fcbb-4f2b-9ade-e055e7ff966f]

3 2

[neutron] Security groups with SR-IOV as a second ML2 mechanism driver
by GABRIEL OMAR GAMERO MONTENEGRO 05 Oct '20

05 Oct '20

Dear all, I'm planning to use the SR-IOV Networking L2 Agent with another L2 Agent as Open vSwitch or Linux Bridge (a configuration with multiple ML2 mechanism drivers). Does anybody know if I can use the Open vSwitch or Linux Bridge L2 agents with security group feature (implemented with iptables firewall driver or Native Open vSwitch firewall driver)? Or am I restricted to apply no security to my instances because SR-IOV L2 agent is being used as a second mechanism driver in the same OpenStack deployment? Thanks in advance, Gabriel Gamero

3 2

[tc] weekly update
by Mohammed Naser 05 Oct '20

05 Oct '20

Hi everyone, Here's an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. # Patches ## Open Reviews - Select Prvisep as the Wallaby Goal https://review.opendev.org/755590 - Add assert:supports-standalone https://review.opendev.org/722399 - Add Ironic charms to OpenStack charms https://review.opendev.org/754099 - Clarify impact on releases for SIGs https://review.opendev.org/752699 - Add election schedule exceptions in charter https://review.opendev.org/751941 ## Project Updates - Retire devstack-plugin-pika project https://review.opendev.org/748730 - Retire openstack/os-loganalyze https://review.opendev.org/753834 ## General Changes - Define TC-approved release in a resolution https://review.opendev.org/752256 - Reorder repos alphabetically https://review.opendev.org/754097 - Remove tc:approved-release tag https://review.opendev.org/749363 - Migrate rpm-packaging to a SIG https://review.opendev.org/752661 # Other Reminders - PTG Brainstorming: https://etherpad.opendev.org/p/tc-wallaby-ptg - PTG Registration: https://october2020ptg.eventbrite.com Thanks for reading! Mohammed & Kendall -- Mohammed Naser VEXXHOST, Inc.

1 0

[tc][all][interop] Removed the 'tc:approved-release' tag in favor of projects.yaml
by Ghanshyam Mann 05 Oct '20

05 Oct '20

Hello Everyone, TC has merged the recent change to remove the 'tc:approved-release' tag[1]. This tag was used as the superset of projects used by the OpenStack Foundation when creating commercial trademark programs. Basically, this tag was used to know if the project under OpenStack governance is mature and follows the defined release model so that it can be considered as a possible candidate for trademark programs. This situation was in the early stage of OpenStack when we used to have incubated vs integrated project status but now everything in projects.yaml[2] file is considered as mature, following release, active. We do not need any changes in the bylaw as all projects listed in projects.yaml file is applicable as bylaw term "OpenStack Technical Committee Approved Release" which is made clear in the resolution also[3]. Interop group can refer to the OpenStack governed projects list for possible candidates for the trademark program. [1] https://review.opendev.org/#/c/749363/ [2] https://governance.openstack.org/tc/reference/projects/ [3] https://governance.openstack.org/tc/resolutions/20200920-tc-approved-releas… -gmann

1 0