- openstack-discuss - lists.openstack.org

senlin auto scaling question
by Satish Patel 03 Sep '20

03 Sep '20

Folks, I have created very simple cluster using following command openstack cluster create --profile myserver --desired-capacity 2 --min-size 2 --max-size 3 --strict my-asg It spun up 2 vm immediately now because the desired capacity is 2 so I am assuming if any node dies in the cluster it should spin up node to make count 2 right? so i killed one of node with "nove delete <instance-foo-1>" but senlin didn't create node automatically to make desired capacity 2 (In AWS when you kill node in ASG it will create new node so is this senlin different then AWS?)

3 5

[kolla] questions when using external mysql
by sosogh 03 Sep '20

03 Sep '20

Hi list: I want to use kolla-ansible to deploy openstack , but using external mysql. I am following these docs: https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html https://docs.openstack.org/kolla-ansible/latest/reference/databases/externa…. I have some questions: ################ ## Question 1 ## ################ According to the offical doc , if setting it in inventory file(multinode), kolla-ansible -i ./multinode deploy will throw out error: I guest when kolla-ansible running the playbook against myexternalmariadbloadbalancer.com , the """register: find_custom_fluentd_inputs""" in """TASK [common : Find custom fluentd input config files]""" maybe null . ################ ## Question 2 ## ################ According to the offical doc , If the MariaDB username is not root, set database_username in /etc/kolla/globals.yml file: But in kolla-ansible/ansible/roles/xxxxxx/tasks/bootstrap.yml , they use ''' login_user: "{{ database_user }}" ''' ， for example : So at last , I took the following steps: 1. """not""" setting [mariadb] in inventory file(multinode) 2. set "database_user: openstack" for "privillegeduser" PS: My idea is that if using an external ready-to-use mysql (cluster), it is enough to tell kolla-ansible only the address/user/password of the external DB. i.e. setting them in the file /etc/kolla/globals.yml and passwords.yml , no need to add it into inventory file(multinode) Finally , it is successful to deploy openstack via kolla-ansible . So far I have not found any problems. Are the steps what I took good ( enough ) ? Thank you !

2 4

[tacker] Wallaby vPTG
by Yasufumi Ogawa 03 Sep '20

03 Sep '20

Hi team, I booked our slots, 27-29 Oct 6am-8am UTC, for the next vPTG[1] as we agreed in previous irc meeting. I also prepared an etherpad [2], so please add your name and suggestions. [1] https://ethercalc.openstack.org/7xp2pcbh1ncb [2] https://etherpad.opendev.org/p/Tacker-PTG-Wallaby Thanks, Yasufumi

2 2

[ovn][neutron][ussuri] How to configure for overlay in DPDK with OVN ?
by 한승진 03 Sep '20

03 Sep '20

Hi, I’m trying to implement dpdk-ovs with ovn for our openstack environment. I create a bond interface(tunbound) with ovs and tun0 and tun1 are slave interfaces of the bond interface named tunbond. Bridge br-int fail_mode: secure datapath_type: netdev Port tunbond Interface tun0 type: dpdk options: {dpdk-devargs="0000:1a:00.1", n_rxq="2"} Interface tun1 type: dpdk options: {dpdk-devargs="0000:d8:00.1", n_rxq="2"} Port br-int Interface br-int type: internal Now, I need to configure remote-ovn options to the ovs. *I am confused how can I define the IP address of the bonded interface with ovs?* Theses are commands for connect remote ovn services. ovs-vsctl set open . external-ids:ovn-remote=tcp:{controller ip}:6642 ovs-vsctl set open . external-ids:ovn-encap-type=geneve *1* This below command make it me confused. ovs-vsctl set open . external-ids:ovn-encap-ip={local ip} How should I resolve this issue? Regards, John Haan

2 2

[tripleo, ironic] Error: Could not retrieve ... pxelinux.0
by Pete Zaitcev 02 Sep '20

02 Sep '20

Hello: I wanted to give the TripleO a try, so started follow our installation guide for Ussuri, and eventually made it to "openstack undercloud install". It fails with something like this: Aug 28 10:10:53 undercloud puppet-user[48657]: Error: /Stage[main]/Ironic::Pxe/File[/var/lib/ironic/tftpboot/ipxe.efi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/share/ipxe/ipxe-x86_64.efi Aug 27 20:05:42 undercloud puppet-user[37048]: Error: /Stage[main]/Ironic::Pxe/Ironic::Pxe::Tftpboot_file[pxelinux.0]/File[/var/lib/ironic/tftpboot/pxelinux.0]: Could not evaluate: Could not retrieve information from environment production source(s) file:/tftpboot/pxelinux.0 Does anyone have an idea what it wants? I added a couple of packages on the host system that provided the files mentioned in the message, but it made no difference. Ussuri is conteinerized anyway. Since I'm very new to this, I have no clue where to look at all. The nearest task is a wrapper of some kind, so the install-undercloud.log looks like this: 2020-08-28 14:11:31.397 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] TASK [Run container-puppet tasks (generate config) during step 1 with paunch] *** 2020-08-28 14:11:31.397 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Friday 28 August 2020 14:11:31 -0400 (0:00:00.302) 0:06:28.734 ********* 2020-08-28 14:11:32.223 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] changed: [undercloud] 2020-08-28 14:11:32.325 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] 2020-08-28 14:11:32.326 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] TASK [Wait for container-puppet tasks (generate config) to finish] ************* 2020-08-28 14:11:32.326 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Friday 28 August 2020 14:11:32 -0400 (0:00:00.928) 0:06:29.663 ********* 2020-08-28 14:11:32.948 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] WAITING FOR COMPLETION: Wait for container-puppet tasks (generate config) to finish (1200 retries left). . . . If anyone could tell roughly what is supposed to be going on here, it would be great. I may be able figure out the rest. Greetings, -- Pete

2 3

Re: [neutron][ovn] OVN Performance
by Apsey, Christopher 02 Sep '20

02 Sep '20

All, Just wanted to loop back here and give an update. For reference, [1] (blue means successful action, red means failed action) is the result we got when booting 5000 instances in rally [2] before the Red Hat OVN devs poked around inside our environment, and [3] is the result after. The differences are obviously pretty significant. I think the biggest change was setting metadata_workers = 2 in neutron_ovn_metadata_agent.ini on the compute nodes per https://bugs.launchpad.net/neutron/+bug/1893656. We have 64C/128T on all compute nodes, so the default neutron calculation of scaling metadata workers based on available cores created 900+ connections to the southbound db at idle; after the control plane got loaded up it just quit around 2500 instances (my guess is it hit the open file limit, although I don’t think increasing it would have made it better for much longer since the number of connections were increasing exponentially). Capping the number of metadata workers decreased open southbound connections by 90%. Even more telling was that rally was able to successfully clean up after itself after we made that change, whereas previously it wasn’t even able to successfully tear down any of the instances that were made, indicating that the control plane was completely toast. Note that the choppiness towards the end of [3] had nothing to do with OVN – our compute nodes had a loadavg approaching 1000 at that point, so they were just starved for cpu cycles. This would have scaled even better with additional compute nodes. The other piece was RAFT. Currently, RDO is shipping with ovs 2.12, but 2.13 has a bunch of RAFT fixes in it that improve stability and knock out some bugs. We were having issues with chassis registration on 2.12, but after using the 2.13 package from cbs, all those issues went away. Big thanks to the great people at Red Hat on the cc line for volunteering their valuable time to take a look. I’m now significantly more comfortable with defaulting to OVN as the backend of choice as the performance delta is now gone. That said, should the community consider dropping linuxbridge as the backend in the official upstream docs and jump straight to OVN rather than ml2/OVS? I think that would increase the test base and help shine light on other issues as time goes on. My org can devote some time to doing this work if the community agrees that it’s the right action to take. Hope that’s helpful! [1] https://ibb.co/GTjZP2y [2] https://pastebin.com/5pEDZ7dY [3] https://ibb.co/pfB9KTV Chris Apsey GEORGIA CYBER CENTER From: Apsey, Christopher Sent: Thursday, August 27, 2020 11:33 AM To: Assaf Muller <amuller(a)redhat.com> Cc: openstack-discuss(a)lists.openstack.org; Lucas Alvares Gomes Martins <lmartins(a)redhat.com>; Jakub Libosvar <jlibosva(a)redhat.com>; Daniel Alvarez Sanchez <dalvarez(a)redhat.com> Subject: RE: [EXTERNAL] Re: [neutron][ovn] OVN Performance Assaf, We can absolutely support engineering poking around in our environment (and possibly an even larger one at my previous employer that was experiencing similar issues during testing). We can take this offline so we don’t spam the mailing list. Just let me know how to proceed, Thanks! Chris Apsey GEORGIA CYBER CENTER From: Assaf Muller <amuller(a)redhat.com<mailto:amuller@redhat.com>> Sent: Thursday, August 27, 2020 11:18 AM To: Apsey, Christopher <CAPSEY(a)augusta.edu<mailto:CAPSEY@augusta.edu>> Cc: openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>; Lucas Alvares Gomes Martins <lmartins(a)redhat.com<mailto:lmartins@redhat.com>>; Jakub Libosvar <jlibosva(a)redhat.com<mailto:jlibosva@redhat.com>>; Daniel Alvarez Sanchez <dalvarez(a)redhat.com<mailto:dalvarez@redhat.com>> Subject: [EXTERNAL] Re: [neutron][ovn] OVN Performance CAUTION: EXTERNAL SENDER This email originated from an external source. Please exercise caution before opening attachments, clicking links, replying, or providing information to the sender. If you believe it to be fraudulent, contact the AU Cybersecurity Hotline at 72-CYBER (2-9237 / 706-722-9237) or 72CYBER(a)augusta.edu<mailto:72CYBER@augusta.edu> The most efficient way about this is to give one or more of the Engineers working on OpenStack OVN upstream (I've added a few to this thread) temporary access to an environment that can reproduce issues you're seeing, we could then document the issues and work towards solutions. If that's not possible, if you could provide reproducer scripts, or alternatively sharpen the reproduction method, we'll take a look. What you've described is not something that's 'acceptable', OVN should definitely not scale worse than Neutron with the Linux Bridge agent. It's possible that the particular issues you ran in to is something that we've already seen internally at Red Hat, or with our customers, and we're already working on fixes in future versions of OVN - I can't tell you until you elaborate on the details of the issues you're seeing. In any case, the upstream community is committed to improving OVN scale and fixing scale issues as they pop up. Coincidentally, Red Hat scale engineers just published an article [1] about work they've done to scale RH-OSP 16.1 (== OpenStack Train on CentOS 8, with OVN 2.13 and TripleO) to 700 compute nodes. [1] https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-…<https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-…> On Thu, Aug 27, 2020 at 10:44 AM Apsey, Christopher <CAPSEY(a)augusta.edu<mailto:CAPSEY@augusta.edu>> wrote: > > All, > > > > I know that OVN is going to become the default neutron backend at some point and displace linuxbridge as the default configuration option in the docs, but we have noticed a pretty significant performance disparity between OVN and linuxbridge on identical hardware over the past year or so in a few different environments[1]. I know that example is unscientific, but similar results have been borne out in many different scenarios from what we have observed. There are three main problems from what we see: > > > > 1. OVN does not handle large concurrent requests as well as linuxbridge. Additionally, linuxbridge concurrent capacity grows (not linearly, but grows nonetheless) by adding additional neutron API endpoints and RPC agents. OVN does not really horizontally scale by adding additional API endpoints, from what we have observed. > > 2. OVN gets significantly slower as load on the system grows. We have observed a soft cap of about 2000-2500 instances in a given deployment before ovn-backed neutron stops responding altogether to nova requests (even for booting a single instance). We have observed linuxbridge get to 5000+ instances before it starts to struggle on the same hardware (and we think that linuxbridge can go further with improved provider network design in that particular case). > > 3. Once the southbound database process hits 100% CPU usage on the leader in the ovn cluster, it’s game over (probably causes 1+2) > > > > It's entirely possible that we just don’t understand OVN well enough to tune it [2][3][4], but then the question becomes how do we get that tuning knowledge into the docs so people don’t scratch their heads when their cool new OVN deployment scales 40% as well as their ancient linuxbridge-based one? > > > > If it is ‘known’ that OVN has some scaling challenges, is there a plan to fix it, and what is the best way to contribute to doing so? > > > > We have observed similar results on Ubuntu 18.04/20.04 and CentOS 7/8 on Stein, Train, and Ussuri. > > > > [1] https://pastebin.com/kyyURTJm<https://pastebin.com/kyyURTJm> > > [2] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb<https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb> > > [3] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron<https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron> > > [4] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute<https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute> > > > > Chris Apsey > > GEORGIA CYBER CENTER > >

2 1

[manila] Victoria Collab Review next Tuesday (Sep 1st)
by Douglas 02 Sep '20

02 Sep '20

Hi everybody We will have a new edition of our collaborative review next Tuesday, September 1st, where we'll go through the code and review the proposed feature Share Server Migration[1][2]. This meeting is scheduled for two hours, starting at 5:00PM UTC. Meeting notes and videoconference links will be available here[3]. Feel free to attend if you are interested and available. Hoping to see you there, - dviroel [1] https://opendev.org/openstack/manila-specs/src/branch/master/specs/victoria… [2] https://review.opendev.org/#/q/topic:bp/share-server-migration+(status:open) [3] https://etherpad.opendev.org/p/manila-victoria-collab-review

1 1

Graceful stopping of RabbitMQ AMQP notification listener
by Anil Jangam 02 Sep '20

02 Sep '20

Hi, I have coded OpenStack AMQP listener following the example and it is working fine. https://github.com/gibizer/nova-notification-demo/blob/master/ws_forwarder.… The related code snippets of the NotificationHandler class are shown as follows. # Initialize the AMQP listener def init(self, cluster_ip, user, password, port): cfg.CONF() cluster_url = "rabbit://" + user + ":" + password + "@" + cluster_ip + ":" + port + "/" transport = oslo_messaging.get_notification_transport(cfg.CONF, url=cluster_url) targets = [ oslo_messaging.Target(topic='versioned_notifications'), ] endpoints = [self.__endpoint] # Initialize the notification listener try: self.__amqp_listener = oslo_messaging.get_notification_listener(transport, targets, endpoints, executor='threading') except NotImplementedError as err: LOGGER.error("Failed to initialize the notification listener {}".format(err)) return False LOGGER.debug("Initialized the notification listener {}".format(cluster_url)) return True # Arm the compute event listeners def start_amqp_event_listener(self): # Start the notification handler LOGGER.debug("Started the OpenStack notification handler") self.__amqp_listener.start() # Disarm the compute event listeners def stop_amqp_event_listener(self): LOGGER.debug("Stopping the OpenStack notification handler") if self.__amqp_listener is not None: self.__amqp_listener.stop() I am using this interface from a new process handler function, however, when I invoke the stop_amqp_eent_listener() method, my process hangs. It does not terminate naturally. I verified that the self.__amqp_listener.stop() function is not returning. Is there anything missing in this code? Is there any specific consideration when calling the listener from a new process? Can someone provide a clue? # Stop the worker def stop(self): # Stop the AMQP notification handler self.__amqp_handler.stop_amqp_event_listener() LOGGER.debug("Stopped the worker for {}".format(self.__ops_conn_info.cluster_ip)) /anil.

1 1

[Magnum][Kayobe] Magnum Kubernetes clusters failing to be created (bugs?)
by Tony Pearce 02 Sep '20

02 Sep '20

Hi guys, I hope you are all keeping safe and well at the moment. I am trying to launch Kubernetes clusters into Openstack Train which has been deployed via Kayobe (Kayobe as I understand is a wrapper for kolla-ansible). There have been a few strange issues here and I've struggled to isolate them. These issues started recently after a fresh Openstack deployment some months ago (around February 2020) to give some context. This Openstack is not "live" as I've been trying to get to the bottom of the issues: Issue 1. When trying to launch a cluster we get error "Resource Create Failed: Forbidden: Resources.Kube Masters.Resources[0].Resources.Kube-Master: Only Volume-Backed Servers Are Allowed For Flavors With Zero Disk. " Issue 2. After successfully creating a cluster of a smaller node size, the "resize cluster" is failing (however update the cluster is working). Some background on this specific environment: Deployed via Kayobe, with these components: Cinder, Designate, iscsid, Magnum, Multipathd, neutron provider networks The Cinder component integrates with iSCSI SAN storage using the Nimble driver. This is the only storage. In order to prevent Openstack from allocating Compute node local HDD as instance storage, I have all flavours configured with root disk / ephemeral disk / swap disk = "0MB". This then results in all instance data being stored on the backend Cinder storage appliance. I was able to get a cluster deployed by first creating the template as needed, then when launching the cluster Horizon prompts you for items already there in the template such as number of nodes, node flavour and labels etc. I re-supplied all of the info (as to duplicate it) and then tried creating the cluster. After many many times trying over the course of a few weeks to a few months it was successful. I was then able to work around the issue #2 above to get it increased in size. When looking at the logs for issue #2, it looks like some content is missing in the API but I am not certain. I will include a link to the pastebin below [1]. When trying to resize the cluster, Horizon gives error: "Error: Unable to resize given cluster id: 99693dbf-160a-40e0-9ed4-93f3370367ee". I then searched the controller node /var/log directory for this ID and found "horizon.log [:error] [pid 25] Not Found: /api/container_infra/clusters/99693dbf-160a-40e0-9ed4-93f3370367ee/resize". Going to the Horizon menu "update cluster" allows you to increase the number of nodes and then save/apply the config which does indeed resize the cluster. Regarding issue #1, we've been unable to deploy a cluster in a new project and the error is hinting it relates to the flavours having 0MB disk specified, though this error is new and we've been successful previously with deploying clusters (albeit with the hit-and-miss experiences) using the flavour with 0MB disk as described above. Again I searched for the (stack) ID after the failure, in the logs on the controller and I obtained not much more than the error already seen with Horizon [2]. I was able to create new flavours with root disk = 15GB and then successfully deploy a cluster on the next immediate try. Update cluster from 3 nodes to 6 nodes was also immediately successful. However I see the compute nodes "used" disk space increasing after increasing the cluster size which is an issue as the compute node has very limited HDD capacity (32GB SD card). At this point I also checked 1) previously installed cluster using the 0MB disk flavour and 2) new instances using the 0MB disk flavour. I notice that the previous cluster is having host storage allocated but while the new instance is not having host storage allocated. So the cluster create success is using flavour with disk = 0MB while the result is compute HDD storage being consumed. So with the above, please may I clarify on the following? 1. It seems that 0MB disk flavours may not be supported with magnum now? Could the experts confirm? :) Is there another way that I should be configuring this so that compute node disk is not being consumed (because it is slow and has limited capacity). 2. The issue #1 looks like a bug to me, is it known? If not, is this mail enough to get it realised? Pastebin links as mentioned [1] http://paste.openstack.org/show/797316/ [2] http://paste.openstack.org/show/797318/ Many thanks, Regards, Tony Pearce

2 4

[ptg][glance] Wallaby PTG planning etherpad
by Abhishek Kekane 02 Sep '20

02 Sep '20

Hi All, The Wallaby PTG will be held next month from Oct 26 to Oct 30. I have booked some time slots in the ethercalc[0], however these are not fixed, if the booked time does not suit your time, please let me know. I have also created a planning etherpad [1] where you can add your topics for discussion during PTG. [0]https://ethercalc.openstack.org/7xp2pcbh1ncb. [1] https://etherpad.opendev.org/p/Glance-Wallaby-PTG-planning Thank you, Abhishek Kekane

1 0

Issue with heat and magnum
by Alexander Dibbo - UKRI STFC 02 Sep '20

02 Sep '20

Hi, I am having an issue with magnum creating clusters when I have multiple active heat-engine daemons running. I get the following error in the heat engine logs: 2020-08-14 10:36:30.237 598383 INFO heat.engine.resource [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] signal SoftwareDeployment "master_config_deployment" [67ba9ce2-aba5-4c15-a7ea -6b774659a0e2] Stack "kubernetes-test-26-3uzjqqob47fh-kube_masters-mhctjio2b4gh-0-pbhumflm5mn5" [dc66e4d9-0c9b-4b18-a2c6-dd9724fa51a9] : Authentication cannot be scoped to multiple target s. Pick one of: project, domain, trust or unscoped 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource Traceback (most recent call last): 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2462, in _handle_signal 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource signal_result = self.handle_signal(details) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py", line 514, in handle_signal 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource timeutils.utcnow().isoformat()) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 788, in signal_software_deployment 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource version='1.6') 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 89, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 165, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource msg_ctxt = self.serializer.serialize_context(ctxt) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/messaging.py", line 46, in serialize_context 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource _context = ctxt.to_dict() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 185, in to_dict 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource 'roles': self.roles, 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 315, in roles 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource self._load_keystone_data() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 292, in wrapped_f 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self.call(f, *args, **kw) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 358, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource do = self.iter(retry_state=retry_state) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 319, in iter 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return fut.result() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self.__get_result() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 361, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource result = fn(*args, **kwargs) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 306, in _load_keystone_data 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource auth_ref = self.auth_plugin.get_access(self.keystone_session) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource self.auth_ref = self.get_auth_ref(session) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self._plugin.get_auth_ref(session, **kwargs) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 144, in get_auth_ref 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource message='Authentication cannot be scoped to multiple' 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource AuthorizationFailure: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] Unhandled error in asynchronous task: ResourceFailure: AuthorizationFailure: resources.master_config_deployment: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service Traceback (most recent call last): 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 132, in log_exceptions 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service gt.wait() 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 181, in wait 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service return self._exit_event.wait() 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 132, in wait 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service current.throw(*self._exc) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 221, in main 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service result = function(*args, **kwargs) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 123, in _start_with_trace 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service return func(*args, **kwargs) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1871, in _resource_signal 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service needs_metadata_updates = rsrc.signal(details, need_check) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2500, in signal 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service self._handle_signal(details) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2480, in _handle_signal 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service raise failure 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service ResourceFailure: AuthorizationFailure: resources.master_config_deployment: Authentication cannot be scoped to multiple targets. Pi ck one of: project, domain, trust or unscoped 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service Each of the individual heat-engine daemons create magnum clusters correctly when they are the only ones online. Attached are the heat and magnum config files. Any ideas where to look would be appreciated? Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support(a)gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

2 1

[ptg][cyborg] Cyborg Wallaby PTG schedule and resources
by yumeng bao 02 Sep '20

02 Sep '20

Hi team, The next virtual Wallaby PTG will be held next month from Oct 26 to Oct 30. I have booked the same time slots in the ethercalc[0] as our last PTG did. However, these are not fixed, if the booked time does not suit your time, Please don't hesitate to add your availability to the doodle[1]. BTW, PTG Topics are open to collect and you can add to the etherpad[2]. [0]https://ethercalc.openstack.org/7xp2pcbh1ncb. [1]https://doodle.com/poll/mrudbrn87vh48ayq [2]https://etherpad.opendev.org/p/cyborg-wallaby-goals Regards， Yumeng

1 0

[scientific-sig] SIG Meeting: Ironic kexec, CANOPIE-HPC and the PTG
by Stig Telfer 01 Sep '20

01 Sep '20

Hi All - We have a Scientific SIG meeting on IRC channel #openstack-meeting at 2100 UTC (about 45 minutes). Everyone is welcome. Agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_September_1st_20… Today we'd like to talk about supporting the development of kexec support for Ironic deployments. Also some up-coming CFPs and PTG participation. Cheers, Stig

1 0

[nova] providing a local disk to a Nova instance
by Thomas Goirand 01 Sep '20

01 Sep '20

Hi Nova team! tl;dr: we would like to contribute giving instances access to physical block devices directly on the compute hosts. Would this be accepted? Longer version: About 3 or 4 years ago, someone wrote a spec, so we'd be able to provide a local disk of a compute, directly to a VM to use. This was then rejected, because at the time, Cinder had the blockdevice drive, which was more or less achieving the same thing. Unfortunately, because nobody was maintaining the blockdevice driver in Cinder, and because there was no CI that could test it, the driver got removed. We've investigated how we could otherwise implement it, and one solution would be to use Cinder, but then we'd be going through an iSCSI export, which would drastically reduce performances. Another solution would be to manage KVM instances by hand, not touching anything to libvirt and/or OpenVSwitch, but then we would loose the ease of using the Nova API, so we would prefer to avoid this direction. So we (ie: employees in my company) need to ask the Nova team: would you consider a spec to do what was rejected before, since there's now no other good enough alternative? Our current goal is to be able to provide a disk directly to a VM, so that we could build Ceph clusters with an hyper-converged model (ie: storage hosted on the compute nodes). In this model, we wouldn't need live-migration of a VM with an attached physical block device (though the feature could be added on a later stage). Before we start investigating how this can be done, I need to know if this has at least some chances to be accepted or not. If there is, then we'll probably start an experimental patch locally, then write a spec to properly start this project. So please let us know. Cheers, Thomas Goirand (zigo)

4 3

[neutron] Bug deputy report August 24 - 30
by Rodolfo Alonso Hernandez 01 Sep '20

01 Sep '20

Hello: This is the list of bugs in the period from August 24 to August 30. Critical: (none) High: - https://bugs.launchpad.net/neutron/+bug/1892861: [neutron-tempest-plugin] If paramiko SSH client connection fails because of authentication, cannot reconnect (unassigned) - https://bugs.launchpad.net/neutron/+bug/1893314: set neutron quota not take effect (ZhouHeng, https://review.opendev.org/#/c/748578/) Medium: - https://bugs.launchpad.net/neutron/+bug/1892496: 500 on SG deletion: Cannot delete or update a parent row (unassigned) - https://bugs.launchpad.net/neutron/+bug/1888878: Data not present in OVN IDL when requested (fix released) - https://bugs.launchpad.net/neutron/+bug/1892846: AttributeError in l3-agent changing HA router to non-HA (Slawek, https://review.opendev.org/#/c/747922/) Low: - https://bugs.launchpad.net/neutron/+bug/1892741: neutron-sriov-nic-agent cannot be disabled (unassigned) - https://bugs.launchpad.net/neutron/+bug/1893188: [tempest] Randomize subnet CIDR to avoid test clashes (Lajos, https://review.opendev.org/#/c/749041/) Wishlist: - https://bugs.launchpad.net/neutron/+bug/1893625: Implement router gateway IP QoS in OVN backend (zhanghao, https://review.opendev.org/#/c/749012/) Incomplete: - https://bugs.launchpad.net/neutron/+bug/1893015: ping with large package size fails Regards.

3 2

2020 Virtual Summit: Forum Submissions Now Accepted
by Jimmy McArthur 01 Sep '20

01 Sep '20

Hello Everyone! We are now accepting Forum [1] submissions for the 2020 Virtual Open Infrastructure Summit [2]. Please submit your ideas through the Summit CFP tool [3] through September 14th. Don't forget to put your brainstorming etherpad up on the Shanghai Forum page [4]. This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) discuss the topics they want to cover and get alignment on and we welcome your participation. The Forum is your opportunity to help shape the development of future project releases. More information about the Forum [1]. The timeline for submissions is as follows: Aug 31st | Formal topic submission tool opens: https://cfp.openstack.org. Sep 14th | Deadline for proposing Forum topics. Scheduling committee meeting to make draft agenda. Sep 21st | Draft Forum schedule published. Crowd sourced session conflict detection. Forum promotion begins. Sept 28th | Forum schedule final Oct 19th | Forum begins! If you have questions or concerns, please reach out to speakersupport(a)openstack.org. Cheers, Jimmy [1] https://wiki.openstack.org/wiki/Forum [2] https://www.openstack.org/summit/2020/ [3] https://cfp.openstack.org [4]https://wiki.openstack.org/wiki/Forum/Virtual2020

1 1

[tripleo][ussuri][undercloud][tripleo_rabbitmq] do not start after reboot
by Ruslanas Gžibovskis 01 Sep '20

01 Sep '20

Hi all, I think this issue might be known to RHOSP16 [1], but it is same in RDO - ussuri with centos8. if anything is needed, I could try to help/test. my workaround is to run openstack undercloud install --force-stack-update [1] https://access.redhat.com/solutions/5269241 -- Ruslanas Gžibovskis +370 6030 7030

1 0

[horizon] horizon-integration-tests job is broken
by Ivan Kolodyazhny 01 Sep '20

01 Sep '20

Hi team, Please, do not trigger recheck if horizon-integration-tests failed like [1]: ______________ TestDashboardHelp.test_dashboard_help_redirection _______________ 'NoneType' object is not iterable While I'm trying to figure out what is happening there [2], any help with troubleshooting is welcome. [1] https://51bb980dc10c72928109-9873e0e5415ff38d9f1a5cc3b1681b19.ssl.cf1.rackc… [2] https://review.opendev.org/#/c/749011 Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

1 1

[ops] Restructuring OSOPS tools
by Sean McGinnis 01 Sep '20

01 Sep '20

Hello everyone, We recently expanded the scope of the Ops Docs SIG to also include any ops tooling. I think it's now time to move on to the next step of actually getting some of the old tooling in place and organized how we want it. We have several semi-abandoned repos from back when there was more work being done on ops tooling. During the great rebranding, those all were moved under the x/ namespace: https://opendev.org/x/?tab=&sort=recentupdate&q=osops Since these are now owned by an official SIG, we can move this content back under the openstack/ namespace. That should help increase visibility somewhat, and make things look a little more official. It will also allow contributors to tooling to get recognition for contributing to an import part of the OpenStack ecosystem. I do think it's can be a little more difficult to find things spread out over several repos though. For simplicity with finding tooling, as well as watching for reviews and helping with overall maintenance, I would like to move all of these under a common openstack/osops. Under that repo, we can then have a folder structure with tools/logging, tools/monitoring, etc. Then with everything in one place, we can have docs published in one place that helps find everything and easily links between tools. We can also capture some metadata about the tools, and use that to reflect their state in those docs. Please let me know if there are any objects to this plan. Otherwise, I will start cleaning things up and getting it staged in a new repo to be imported as an official repo owned by the SIG. Thanks! Sean

3 2

[TripleO][Ussuri] image prepare, cannot download containers image prepare recently
by Ruslanas Gžibovskis 01 Sep '20

01 Sep '20

Hi all, I have noticed, that recently my undercloud is not able to download images [0]. I have provided newly generated containers-prepare-parameter.yaml and outputs from container image prepare providing --verbose and later beginning of --debug (in the end) [0] were there any changes? As "openstack tripleo container image prepare default --output-env-file containers-prepare-parameter.yaml --local-push-destination" have prepared a bit different file, compared what was previously: NEW # namespace: docker.io/tripleou VS namespace: docker.io/tripleomaster # OLD [0] - http://paste.openstack.org/show/rBCNAQJBEe9y7CKyi9aG/ -- Ruslanas Gžibovskis +370 6030 7030

2 1

[neutron] Team meeting - Tuesday 01.09.2020
by Slawek Kaplonski 01 Sep '20

01 Sep '20

Hi, I will be on PTO on Tuesday and I will not be able to run our team meeting. So lets cancel it this week and see You all on meeting next week. For this week there are 3 important things from me: * please check doodle: https://doodle.com/poll/2ppmnua2nuva5nyp and put there time slots which works the best for You for the PTG in October - please do that this week as next week I need to book some slots for us, * as we are close to the Victoria-3 milestone (next week) which is also feature freeze week, please focus now on reviewin patches for BPs targeted for this cycle: https://wiki.openstack.org/wiki/Network/Meetings#Blueprints * There is new bug deputy role starting this week. Please check new schedule at https://wiki.openstack.org/wiki/Network/Meetings#Bug_deputy and let me know if it don't works for You. That's all from me for this week. See You all online :) -- Slawek Kaplonski Principal software engineer Red Hat

1 0

[tc] weekly update
by Mohammed Naser 31 Aug '20

31 Aug '20

Hi everyone, Here’s an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. We've also included a few references to some important mailing list threads that you should check out. # Patches ## Open Reviews - Move ansible-role-XXX-hsm projects to Barbican team https://review.opendev.org/748027 - Retire the devstack-plugin-zmq project https://review.opendev.org/748731 - Retire devstack-plugin-pika project https://review.opendev.org/748730 - Add openstack-ansible/os_senlin role https://review.opendev.org/748677 - Drop all exceptions for legacy validation https://review.opendev.org/745403 - Add openstack-helm-releases to openstack-helm https://review.opendev.org/748302 - Add assert:supports-standalone. https://review.opendev.org/722399 ## Project Updates - Add etcd3gw to Oslo https://review.opendev.org/747188 ## General Changes - Update and simplify comparison of working groups https://review.opendev.org/746763 - Move towards dual office hours in diff TZ https://review.opendev.org/746167 ## Abandoned Changes - Drop requirement of 1/3 positive TC votes to land https://review.opendev.org/746711 - Move towards single office hour https://review.opendev.org/745200 # Email Threads - vPTG October 2020 Signup: http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016497.h… Thanks for reading! Mohammed & Kendall -- Mohammed Naser VEXXHOST, Inc.

1 0

Setuptools 48 and Devstack Failures
by Clark Boylan 31 Aug '20

31 Aug '20

Hello, Setuptools has made a new version 48 release. This appears to be causing problems for devstack because `pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past. `pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected. Devstack is failing because keystone-manage cannot currently be found at the specific /usr/local/bin/ path. Potential workarounds for this include not using `pip install -e` or relying on $PATH to find the commands rather than specifying rooted paths to them. I'll defer to the QA team on how they want to address this. While we can have devstack install an older setuptools version as well, generally this is not considered to be a good idea because anyone doing pip installs outside of devstack may get the newer behavior. It is actually important for us to try and keep up with setuptools changes as a result. Fungi indicated that setuptools expected this to be a bumpy upgrade. I'm not sure if they would consider `pip install -e` and `pip install` installing to different paths as a bug, and if they did which behavior is correct. It would probably be a good idea to file a bug upstream if we debug this further. Clark

5 6

[octavia] usage of SELECT .. FOR UPDATE
by Mohammed Naser 31 Aug '20

31 Aug '20

Hi everyone, We're being particularly hit hard across different deployments where Octavia has several SELECT .. FOR UPDATE queries which are causing load balancers to fail to provision properly. - spare_pools: This usually hits on rolling restarts of o-housekeeping as they all seem to try to capture a lock -- https://github.com/openstack/octavia/blob/73fbc05386b512aa1dd86a0ed6e8455cc… - quota: This hits when provisioning a lot of load balancers in parallel. For example in cases when using Heat -- https://github.com/openstack/octavia/blob/bf3d5372b9fc670ecd08339fa989c9b73… These hurt quite a lot in a busy deployment and result in a poor user experience unfortunately. We're trying to off-load Octavia to it's own database server but that is more of a "throw power at the problem" solution. I can imagine that we can probably likely look into a better/cleaner alternative that avoids this entirely? I'm happy to try and push for some of this work on our side. Thanks, Mohammed -- Mohammed Naser VEXXHOST, Inc.

2 1

Openstack zun ui
by mahdi abbasi 30 Aug '20

30 Aug '20

Hi, I want install zun ui, first i installed horizon with package and then install zun ui with pip and then python horizon_path/manage.py collectstatic But i receive error in httpd finaly. Error: ImportError: No module named zun_ui Please help me Best Regards Mahdi

2 4

[all] READMEs of zuul roles not rendered properly - missing content
by Martin Kopec 30 Aug '20

30 Aug '20

Hello everyone, I've noticed that READMEs of zuul roles within openstack projects are not rendered properly on opendev.org - ".. zuul:rolevar::" syntax seems to be the problem. Although it's rendered well on github.com, see f.e. [1] [2]. I wonder if there were some changes in the supported README syntax. Also the ".. zuul:rolevar::" syntax throws errors on online rst formatters I was testing on, however, it's rendered fine by md online formatters - maybe opendev.org is more rst strict in case of .rst files than github? Any ideas? [1] https://opendev.org/openstack/tempest/src/branch/master/roles/run-tempest [2] https://github.com/openstack/tempest/tree/master/roles/run-tempest Thanks, -- Martin Kopec Quality Engineer Red Hat EMEA <https://www.redhat.com> <https://red.ht/sig>

4 4

[tc][telemetry][gnocchi] The future of Gnocchi in OpenStack
by Adrian Turjak 29 Aug '20

29 Aug '20

Hey OpenStackers, We're currently in the process of discussing what to do with OpenStack's reliance on Gnocchi, and at present it is looking like we are most likely to just fork it back under a new name (currently Farfalle to stick with the pasta theme). The discussion is mostly happening here: https://review.opendev.org/#/c/744592/ But for those running Gnocchi in prod, this is likely something you may want to know about and we'd like to hear from you. A bit of history: Gnocchi started off as a new backend for Ceilometer in OpenStack, and eventually become the defacto API for telemetry samples when that was removed from Ceilometer (as backed by MongoDB). Gnocchi was eventually spun off outside of OpenStack, but still essentially remained our API for telemetry despite not being an official part of OpenStack anymore. Since then the development around it seems to have stalled, with pull requests left unreviewed, CI broken, and even the domain for the docs lapsing once. They have essentially said the project is unmaintained themselves: https://github.com/gnocchixyz/gnocchi/issues/1049 Given that OpenStack telemetry relies on it, we needed to decide what to do. We tried talking to the devs which spun it off outside of OpenStack, but they seem disinclined to interact with the OpenStack community, or move the project back to our infra/governance despite OpenStack looking like the only consumers of Gnocchi as a project. We want to find a solution, and the feeling is that they don't. So we've opted to fork it back and now the discussion is how to approach that fork. The OpenStack community doesn't want to maintain a time series database, but our telemetry API is part of it. We are putting it under non-OpenStack namespace to start, but we need to decide what the long term place for it should be. Do we want to make it an official project again? Do we keep it just as an API and drop the time series DB part for another DB? Do we build a new API back into Ceilometer and switch to a different backend like InfluxDB? We don't know yet, and we want some input from people who use the service so we can hopefully work with OpenStack telemetry as a whole and figure out what the long term picture is. If Gnocchi matters to you at all, or you use it, we want to hear from you. Cheers, Adrian Turjak

3 3

[release] Release countdown for week R-6 Aug 31 - Sept 4
by Sean McGinnis 28 Aug '20

28 Aug '20

Development Focus ----------------- Work on libraries should be wrapping up, in preparation for the various library-related deadlines coming up. Now is a good time to make decisions on deferring feature work to the next development cycle in order to be able to focus on finishing already-started feature work. General Information ------------------- We are now getting close to the end of the cycle, and will be gradually freezing feature work on the various deliverables that make up the OpenStack release. This coming week is the deadline for general libraries (except client libraries): their last feature release needs to happen before "Non-client library freeze" on September 3. Only bugfix releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/victoria branching request with the review (as an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/… In the next weeks we will have deadlines for: * Client libraries (think python-*client libraries), which need to have their last feature release before "Client library freeze" (Sept 10) * Deliverables following a cycle-with-rc model (that would be most services), which observe a Feature freeze on that same date, Sept 10. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. As we are getting to the point of creating stable/victoria branches, this would be a good point for teams to review membership in their victoria-stable-maint groups. Once the stable/victoria branches are cut for a repo, the ability to approve any necessary backports into those branches for Victoria will be limited to the members of that stable team. If there are any questions about stable policy or stable team membership, please reach out in the #openstack-stable channel. Upcoming Deadlines & Dates -------------------------- Non-client library freeze: September 3 (R-6 week) Client library freeze: September 10 (R-5 week) Victoria-3 milestone: September 10 (R-5 week) Cycle Highlights Due: September 10 (R-5 week) Victoria release: October 14

1 0

[all][goals] Switch legacy Zuul jobs to native - update #3
by Luigi Toscano 28 Aug '20

28 Aug '20

Hi, it's time for another status report on this goal. A lot of reviews have been merged in the past 10 days, and several projects are not on the list anymore. This is a very good news, but we still have some work to complete. Please keep pushing! Status ====== The number of the project with legacy jobs is now limited, so I'm going to explain the status in more details for each of them. The links to the patches can be found in the etherpad [3] (see below for the links). cinder ------ There is just one test left, and it is definitely tricky, because it implements a cycle of "change tempest configuration"/"run tempest"/"repeat", which is not the usual pattern. In the worst case I will "port" by adding a simple bash wrapper, but I'd like to have a clean ansible solution. designate --------- A patch for the only legacy job is under review after some forth and back, but there are some open questions. heat ------ Only one legacy job left, the heat cores are aware of it. infra ----- Only one devstack-gate job in the os-loganalyze repository, which should be probably retired. There are 2 other legacy jobs, but not devstack-gate, so less urgent. ironic ------ There has been an open review with the full port of the last legacy job, but it is failing. As it is has been failing even before the porting, the patch could be probably merged as it is. karbor ------ A patch for the only legacy job is under review, but it still has some issues. manila ------ There is only one legacy-base job (not devstack-gate), so less urgent, but there is a patch for it. monasca ------- One job in the monasca-transform repository, which is most likely due for a retirement. There are 3 legacy (non devstack-gate) jobs in other repositories. murano ------ There are two legacy jobs left. I'm not sure whether murano-apps-refstackclient-unittest is still needed. murano-dashboard-sanity-check is a bit tricky, the tests still use nose and the corresponding code in horizon has seen several changes. neutron ------- There are three types of legacy jobs: * all jobs in networking-midonet, whose retirement is under discussion, but the final decision is not clear, so a porting may be needed anyway: * two grenade jobs are being worked on; * the remaining legacy job could be maybe dropped. nova ---- The team is trying to port the two legacy job left with some refactoring, but it may require some effort yet. oslo ---- Only one legacy job left, but it is part of the soon-to-be-retired devstack-plugin-zmq repository. senlin ------ A patch for the only legacy job has been proposed and it is working, needs reviews. trove ----- The trove-grenade job should be ported, but on the other hand, trove has no grenade plugin. At this point it is unlikely to be implemented before Victoria, so maybe the job can be dropped for now. zaqar ----- A few patches have been proposed and working. One of them is failing (python-zaqarclient) but it does a bit more than a simple porting, so it may be simply changed to do exactly what the old job was doing (input needed). References ========== [1] the goal: https://governance.openstack.org/tc/goals/selected/victoria/native-zuulv3-j… [2] the up-to-date Zuul v3 porting guide: https://docs.openstack.org/project-team-guide/zuulv3.html [3] the etherpad which tracks the current status: https://etherpad.opendev.org/p/goal-victoria-native-zuulv3-migration [4] the previous reports: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016058.html http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016561.h… Ciao -- Luigi

1 0

[neutron][gate] verbose q-svc log files and e-r indexing
by melanie witt 28 Aug '20

28 Aug '20

Hi all, Recently we've noticed elastic search indexing is behind 115 hours [1] and we looked for abnormally large log files being generated in the gate. We found that the q-svc log is very large, one example being 71.6M [2]. There is a lot of Time-Cost profiling output in the log, like this: Aug 17 14:22:23.210076 ubuntu-bionic-ovh-bhs1-0019298855 neutron-server[5168]: DEBUG neutron_lib.utils.helpers [req-75719db1-4abf-4500-bb0a-6d24e82cd4fd req-d88e7052-7da9-4bc9-8b35-5730ae76dcad service neutron] Time-cost: call 48e628cc-8c3a-408d-a36f-b219524480e0 function apply_funcs start {{(pid=5554) wrapper /usr/local/lib/python3.6/dist-packages/neutron_lib/utils/helpers.py:218}} We saw that there was a recent-ish change to remove some of the profiling output [3] but it was only for the get_objects method. Looking at the total number of lines in the file vs the number of lines without apply_funcs Time-Cost output: $ wc -l screen-q-svc.txt 186387 screen-q-svc.txt $ grep -v "function apply_funcs" screen-q-svc.txt|wc -l 102593 Would it be possible to remove this profiling output from the gate log to give elastic search indexing a better chance at keeping up? Or is there something else I've missed that could be made less verbose in the logging? Thanks for your help. Cheers, -melanie [1] http://status.openstack.org/elastic-recheck [2] https://b6ba3b9af8fd7de57099-18aa39cea11f738aa67ebd6bc9fb5e4c.ssl.cf2.rackc… [3] https://review.opendev.org/741540

3 5

[all][tc][policy] Progress report of consistent and secure default policies pop-up team
by Ghanshyam Mann 28 Aug '20

28 Aug '20

Hello Everyone, This is a regular update on progress in 'Consistent and Secure Default Policies Popup Team'. We will try to make it a monthly report form now onwards. Progress so far: ============ * Popup team meet twice in a month and discuss and work on progress and pre-work to do. - https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popu… * Pre-work to provide a smooth migration path to the new policy <here we will add any pre-work we need to do before more project start moving towards new policy> ** Migrate Default Policy Format from JSON to YAML - This involves oslo side + each project side works. - oslo side work to provide tool and utils method are merged (one patch is in gate). - The new tool 'oslopolicy-convert-json-to-yaml' is available now to convert your existing JSON formatted policy file to YAML formatted in a backward-compatible way. - I have started to do it in Nova (need to update the patch though) to give example work for other projects: https://review.opendev.org/#/c/748059/ - all work is tracked here: https://review.opendev.org/#/q/topic:bp/policy-json-to-yaml+(status:open+OR… ** Improving documentation about target resources (oslo.policy) - https://bugs.launchpad.net/oslo.policy/+bug/1886857 - raildo pushed the patch which is under review: https://review.opendev.org/#/c/743318/ * Team Progress: (list of a team interested or have volunteer to work) ** Keystone (COMPLETED; use as a reference) ** Nova (COMPLETED; use as a reference) - All APIs except deprecated APIs were done in the Ussuri cycle and deprecated APIs also done now. ** Cyborg (in-progress) - Spec is merged, code under review. ** Barbican (not started) ** Neutron (not started) ** Cinder (not started) ** Manila (not started) Why This Is Important ================= (I have copied it from Colleen email which is nicely written) Separating system, domain, and project-scope APIs and providing meaningful default roles is critical to facilitating secure cloud deployments and to fulfilling OpenStack's vision as a fully self-service infrastructure provider[1]. Until all projects have completed this policy migration, the "reader" role that exists in keystone is dangerously misleading, and the `[oslo_policy]/enforce_scope` option has limited usefulness as long as projects lack uniformity in how an administrator can use scoped APIs. How You Can Help ================ Contributor: - You can help by starting the work in your (or any other you would like to help) project and attend popup team meeting in case of any question, review request etc. Cloud operator: - Please help review the proposed policy rule changes to sanity-check the new scope and role defaults. - Migrate your JSON formatted policy file to YAML JSON formatted file can be problematic in the various way as described here[2]. You can use 'oslopolicy-convert-json-to-yaml' tool [3] to convert your existing JSON formatted policy file to YAML formatted in a backward-compatible way. [1] https://governance.openstack.org/tc/reference/technical-vision.html#self-se… [2] https://specs.openstack.org/openstack/oslo-specs/specs/victoria/policy-json… [3] https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-t… -gmann & raildo

1 0

Horizon installation problem
by mahdi abbasi 28 Aug '20

28 Aug '20

Hi openstack development team, When i want install horizon with pip3 i recieve and error: Could not satisfy constraints for horizon: installation from path or url cannot be constrained to a version. Please help me Best regards Mahdi

2 1

[Kolla] re-create container
by Tony Liu 28 Aug '20

28 Aug '20

Hi, Is Kolla container created by playbook only or there is something like docker-compose to re-create container in case it's deleted after initial deployment? Thanks! Tony

2 1

[tc] office hours update
by Mohammed Naser 28 Aug '20

28 Aug '20

Hi everyone, In order to be able to have more folks available during office hours, we have the following newly available times once this patch lands: https://review.opendev.org/746167 * 01:00 UTC on Tuesdays: http://www.timeanddate.com/worldclock/fixedtime.html?hour=01&min=00&sec=0 * 15:00 UTC on Wednesdays: http://www.timeanddate.com/worldclock/fixedtime.html?hour=15&min=00&sec=0 We look forward to seeing our community present at those office hours with anything they have. Thank you, Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

[kuryr] vPTG October 2020
by Michał Dulko 28 Aug '20

28 Aug '20

Hello all, There's a vPTG October 2020 project signup process going on and I'd like to ask if you want me to reserve an hour or two there for a sync up on the priorities and plans of various parts of the team. Thanks, Michał

1 1

[Heat] Reg Creation of resource based on another resource attribute value
by NANTHINI A A 28 Aug '20

28 Aug '20

Hi Team , I want to create the openstack subnet resource based on the openstack network resource's attribute STATUS value. i.e Create neutron subnet only when the neutron network status is ACTIVE . I can see currently the support of get_Attr function is not there in conditions section .Also the depends_on function accepts input as resource ids only .I cant pass a condition there . Is there any other way to implement the same .Please suggest . Thanks, A.Nanthini

2 1

per user quota not applign in openstack train
by its-openstack＠zohocorp.com 28 Aug '20

28 Aug '20

Dear openstack, We are facing a peculiar issue with regards to users quota of resources. e.g: +------------------------------------------------------------------------------------------------------+ | project | user | instance quota | no: of instance created | | -----------|------------|-----------------------------------|------------------------------------------| | tes | - | 10 | - | | test | user1 | 2 | 2 | | test | user2 | 2 | error "quota over" | | test | user3 | 3 | only 1 instance allowed | | test | user4 | no user quota defined | able to create 10 instance| +-------------------------------------------------------------------------------------------------------+ As you see from mentioned table. when user1,user2, has instance quota of 2 and when user1 has created 2 instance, user2 unable to create instance. but user3 able to create only 1 more instance, user 4 has no quota applied so project quota 10 will be applied and he can create 10 instance. the quota is applied to each user but not tracked for each user, so this defeats the purpose of per user quota. Please help us with resolving this issue. Regards, sysadmin team

3 2

[kolla] Focal upgrade
by Mark Goddard 28 Aug '20

28 Aug '20

Hi, For the Victoria release we will be moving our Ubuntu support from Bionic 18.04 to the Focal 20.04 LTS release. This applies to both the base container image and host OS. We would like to request feedback from any Ubuntu users about how they typically deal with a distro upgrade like this. I would assume that the following workflow would be used: 1. start with a Ussuri release on Bionic 2. distro upgrade to Focal 3. OpenStack upgrade to Victoria However, that would imply that it would not be possible to make any more changes to the Ussuri deploy after the Focal upgrade, since Kolla Ansible Ussuri release does not support Focal (it is blocked by prechecks). An alternative approach is: 1. start with a Ussuri release on Bionic 2. OpenStack upgrade to Victoria 3. distro upgrade to Focal This implies that Victoria must support both Bionic and Focal as a host OS, which it currently does. This flow matches more closely what we are currently testing in CI (steps 1 and 2 only). In both cases, Victoria container images are based on Focal. Feedback on this would be appreciated. Thanks, Mark

2 1

Do you want to render ANSI in Zuul console?
by Sorin Sbarnea 27 Aug '20

27 Aug '20

At this moment Zuul web interfaces displays output of commands as raw, so any ANSI terminal output will display ugly artifacts. I tried enabling ANSI about half a year ago but even after providing two different implementations, I was not able to popularize it enough. As this is a UX related feature, I think would like more appropriate to ask for feedback from openstack-discuss, likely the biggest consumer of zuul web interface. Please comment/+/- on review below even if you are not a zuul core. At least it should show if this is a desired feature to have or not: https://review.opendev.org/#/c/739444/ ✅ This review also includes a screenshot that shows how the rendering looks (an alternative for using the sitepreview) Thanks Sorin Sbarnea

3 5

[neutron] Drivers meeting 28.08.2020 cancelled
by Slawek Kaplonski 27 Aug '20

27 Aug '20

Hi, There is no any new RFEs in the agenda for tomorrow's drivers team meeting so lets cancel it and see You all next week. Have a great weekend. -- Slawek Kaplonski Principal software engineer Red Hat

1 0

[tc] Monthly meeting
by Mohammed Naser 27 Aug '20

27 Aug '20

Hi everyone, Our monthly TC meeting is scheduled for next Thursday, September 3rd, at 1400 UTC. If you would like to add topics for discussion, please go to https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting and fill out your suggestions by Wednesday, September 2nd, at 1900 UTC. Thank you, Regards, Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

[all] Wallaby Release Schedule
by Sean McGinnis 27 Aug '20

27 Aug '20

Hey everyone, We have officially published the schedule for the Wallaby development cycle. That can now be found on the release.openstack.org site here: https://releases.openstack.org/wallaby/schedule.html PTLs, feel free to propose any updates if there are important project-specific deadlines you would like to include on the schedule. Just ping me if you need any examples of how that is done. Thanks! Sean

2 2

[neutron][ovn] OVN Performance
by Apsey, Christopher 27 Aug '20

27 Aug '20

All, I know that OVN is going to become the default neutron backend at some point and displace linuxbridge as the default configuration option in the docs, but we have noticed a pretty significant performance disparity between OVN and linuxbridge on identical hardware over the past year or so in a few different environments[1]. I know that example is unscientific, but similar results have been borne out in many different scenarios from what we have observed. There are three main problems from what we see: 1. OVN does not handle large concurrent requests as well as linuxbridge. Additionally, linuxbridge concurrent capacity grows (not linearly, but grows nonetheless) by adding additional neutron API endpoints and RPC agents. OVN does not really horizontally scale by adding additional API endpoints, from what we have observed. 2. OVN gets significantly slower as load on the system grows. We have observed a soft cap of about 2000-2500 instances in a given deployment before ovn-backed neutron stops responding altogether to nova requests (even for booting a single instance). We have observed linuxbridge get to 5000+ instances before it starts to struggle on the same hardware (and we think that linuxbridge can go further with improved provider network design in that particular case). 3. Once the southbound database process hits 100% CPU usage on the leader in the ovn cluster, it’s game over (probably causes 1+2) It's entirely possible that we just don’t understand OVN well enough to tune it [2][3][4], but then the question becomes how do we get that tuning knowledge into the docs so people don’t scratch their heads when their cool new OVN deployment scales 40% as well as their ancient linuxbridge-based one? If it is ‘known’ that OVN has some scaling challenges, is there a plan to fix it, and what is the best way to contribute to doing so? We have observed similar results on Ubuntu 18.04/20.04 and CentOS 7/8 on Stein, Train, and Ussuri. [1] https://pastebin.com/kyyURTJm [2] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb [3] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron [4] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute Chris Apsey GEORGIA CYBER CENTER

3 5

Re: [cyborg] Temporary treatment plan for the 3rd-party driver
by yumeng bao 27 Aug '20

27 Aug '20

Brin, thanks for bringing this up! > Hi all： > This release we want to introduce some 3rd party drivers (e.g. Intel QAT, Inspur FPGA, and Inspur SSD etc.) in Cyborg, and we discussed the handling of 3rd-party driver CI in Cyborg IRC meeting [1]. > Due to the lack of CI test environment supported by hardware, we reached a temporary solution in two ways, as follows: > 1. Provide a CI environment and provide a tempest test for Cyborg, this method is recommended; > 2. If there is no CI environment, please provide the test results of this driver in the master branch or in the designated branch, which should be as complete as possible, sent to the Cyborg team, or pasted in the implementation of the commit. Providing test result can be our option. The test result can be part of the driver documentation[0] as this is public to users. And from my understanding, the test result should work as the role of tempest case and clarify at least: necessary configuration,test operations and test results. [0] https://docs.openstack.org/cyborg/latest/reference/support-matrix.html#driv… > [1] http://eavesdrop.openstack.org/meetings/openstack_cyborg/2020/openstack_cyb… Regards， Yumeng

4 4

DB Prune
by Szabo, Istvan (Agoda) 27 Aug '20

27 Aug '20

Hi, We have a cluster where the user continuously spawn and delete servers which makes the db even in compressed state 1.1GB. I'm sure it has a huge amount of trash because this is a cicd environment and the prod just uses 75MB. How is it possible to cleanup the db on a safe way, what should be the steps? Best regards, Istvan ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4

[ironic] next Victoria meetup
by Riccardo Pittau 27 Aug '20

27 Aug '20

Hello everyone! The time for the next Ironic virtual meetup is close! It will be an opportunity to review what has been done in the last months, exchange ideas and plan for the time before the upcoming victoria release, with an eye towards the future. We're aiming to have the virtual meetup the first week of September (Monday August 31 - Friday September 4) and split it in two days, with one two-hours slot per day. Please vote for your best time slots here: https://doodle.com/poll/pi4x3kuxamf4nnpu We're planning to leave the vote open at least for the entire week until Friday August 21, so to have enough time to announce the final slots and planning early next week. Thanks! A si biri Riccardo

1 2

Re: [openstack-community] Error add member to pool ( OCTAVIA ) when using SSL to verify
by Amy Marrich 27 Aug '20

27 Aug '20

Adding the OpenStack discuss list. Amy (spotz) > On Aug 24, 2020, at 11:14 PM, Vinh Nguyen Duc <vinhducnguyen1708(a)gmail.com> wrote: > > > Dear Openstack community, > > My name is Duc Vinh, I am newer in Openstack > I am deploy Openstack Ussuri on Centos8 , I am using three nodes controller with High Availability topology and using HAproxy to verify cert for connect HTTPS, > I have trouble with project Octavia, I cannot add member in a pool after created Loadbalancer, listener, pool ( everything is fine). > Here is my log and configuration file: > > LOGS: > > 2020-08-25 10:55:42.872 226250 DEBUG octavia.network.drivers.neutron.base [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Neutron extension security-group found enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66 > 2020-08-25 10:55:42.892 226250 DEBUG octavia.network.drivers.neutron.base [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Neutron extension dns-integration is not enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:70 > 2020-08-25 10:55:42.911 226250 DEBUG octavia.network.drivers.neutron.base [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66 > 2020-08-25 10:55:42.933 226250 DEBUG octavia.network.drivers.neutron.base [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Neutron extension allowed-address-pairs found enabled _check_extension_enabled /usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py:66 > 2020-08-25 10:55:43.068 226250 WARNING keystoneauth.identity.generic.base [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Failed to discover available identity versions when contacting https://192.168.10.150:5000. Attempting to parse version from URL.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://192.168.10.150:5000: HTTPSConnectionPool(host='192.168.10.150', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Error retrieving subnet (subnet id: 035f3183-f469-415f-b536-b4a81364e814.: keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://192.168.10.150:5000: HTTPSConnectionPool(host='192.168.10.150', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last): > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base chunked=chunked) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base self._validate_conn(conn) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base conn.connect() > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base ssl_context=context) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 367, in ssl_wrap_socket > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base return context.wrap_socket(sock) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base _context=self, _session=session) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base self.do_handshake() > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base self._sslobj.do_handshake() > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base self._sslobj.do_handshake() > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred: > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last): > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base timeout=timeout > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base _stacktrace=sys.exc_info()[2]) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base raise MaxRetryError(_pool, url, error or ResponseError(cause)) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.10.150', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred: > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last): > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base resp = self.session.request(method, url, **kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base resp = self.send(prep, **send_kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base r = adapter.send(request, **kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 514, in send > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base raise SSLError(e, request=request) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.10.150', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred: > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last): > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base authenticated=False) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base authenticated=authenticated) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1452, in get_discovery > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base disc = Discover(session, url, authenticated=authenticated) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 536, in __init__ > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base authenticated=authenticated) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 102, in get_version_data > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base resp = session.get(url, headers=headers, authenticated=authenticated) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1123, in get > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base return self.request(url, 'GET', **kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base resp = send(**kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1008, in _send_request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base raise exceptions.SSLError(msg) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://192.168.10.150:5000: HTTPSConnectionPool(host='192.168.10.150', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base During handling of the above exception, another exception occurred: > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base Traceback (most recent call last): > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py", line 193, in _get_resource > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base resource_type)(resource_id) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 869, in show_subnet > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base return self.get(self.subnet_path % (subnet), params=_params) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 354, in get > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base headers=headers, params=params) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base headers=headers, params=params) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 282, in do_request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base headers=headers) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/neutronclient/client.py", line 339, in do_request > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base self._check_uri_length(url) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/neutronclient/client.py", line 332, in _check_uri_length > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base uri_len = len(self.endpoint_url) + len(url) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/neutronclient/client.py", line 346, in endpoint_url > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base return self.get_endpoint() > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 282, in get_endpoint > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base return self.session.get_endpoint(auth or self.auth, **kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1225, in get_endpoint > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base return auth.get_endpoint(self, **kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base allow_version_hack=allow_version_hack, **kwargs) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base service_catalog = self.get_access(session).service_catalog > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base self.auth_ref = self.get_auth_ref(session) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base self._plugin = self._do_create_plugin(session) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base 'auth_url is correct. %s' % e) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://192.168.10.150:5000: HTTPSConnectionPool(host='192.168.10.150', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),)) > 2020-08-25 10:55:43.070 226250 ERROR octavia.network.drivers.neutron.base > 2020-08-25 10:55:43.074 226250 DEBUG wsme.api [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Client-side error: Subnet 035f3183-f469-415f-b536-b4a81364e814 not found. format_exception /usr/lib/python3.6/site-packages/wsme/api.py:222 > 2020-08-25 10:55:43.076 226250 DEBUG octavia.common.keystone [req-57c5b37c-e50f-4d50-b535-b0a3d19db1d5 - 8259463ce052437396afa845933afe4b - default default] Request path is / and it does not require keystone authentication process_request /usr/lib/python3.6/site-packages/octavia/common/keystone.py:77 > 2020-08-25 10:55:43.080 226250 DEBUG octavia.common.keystone [req-5091d326-0cb4-4ae1-bf4b-9ef6b9313dca - - - - -] Request path is / and it does not require keystone authentication process_request /usr/lib/python3.6/site-packages/octavia/common/keystone.py:77 > > Configuration: > [root@controller01 ~]# cat /etc/octavia/octavia.conf > [DEFAULT] > > log_dir = /var/log/octavia > debug = True > transport_url = rabbit://openstack:4ychZAT5VrWlk6KFfgAmpXvGdzfdV8hEpIgOLhyF@192.168.10.178:5672,openstack:4ychZAT5VrWlk6KFfgAmpXvGdzfdV8hEpIgOLhyF@192.168.10.179:5672,openstack:4ychZAT5VrWlk6KFfgAmpXvGdzfdV8hEpIgOLhyF@192.168.10.28:5672 > > [api_settings] > api_base_uri = https://192.168.10.150:9876 > bind_host = 192.168.10.178 > bind_port = 9876 > auth_strategy = keystone > healthcheck_enabled = True > allow_tls_terminated_listeners = True > > [database] > connection = mysql+pymysql://octavia:FUkbii8AY4G6H9LxbJ2RRlOzHN61X8PI8FrMcuXQ@192.168.10.150/octavia > max_retries = -1 > > [health_manager] > bind_port = 5555 > bind_ip = 192.168.10.178 > controller_ip_port_list = 192.168.10.178:5555, 192.168.10.179:5555, 192.168.10.28:5555 > heartbeat_key = insecure > > [keystone_authtoken] > service_token_roles_required = True > www_authenticate_uri = https://192.168.10.150:5000 > auth_url = https://192.168.10.150:5000 > region_name = Hanoi > memcached_servers = 192.168.10.178:11211,192.168.10.179:11211,192.168.10.28:11211 > auth_type = password > project_domain_name = Default > user_domain_name = Default > project_name = service > username = octavia > password = esGn3rN3iJOAD2HXmqznFPI9oAY2wQNDWYwqJaCH > cafile = /etc/ssl/private/haproxy.pem > insecure = false > > > [certificates] > cert_generator = local_cert_generator > #server_certs_key_passphrase = insecure-key-do-not-use-this-key > ca_private_key_passphrase = esGn3rN3iJOAD2HXmqznFPI9oAY2wQNDWYwqJaCH > ca_private_key = /etc/octavia/certs/server_ca.key.pem > ca_certificate = /etc/octavia/certs/server_ca.cert.pem > region_name = Hanoi > ca_certificates_file = /etc/ssl/private/haproxy.pem > endpoint_type = internal > > [networking] > #allow_vip_network_id = True > #allow_vip_subnet_id = True > #allow_vip_port_id = True > > [haproxy_amphora] > #bind_port = 9443 > server_ca = /etc/octavia/certs/server_ca.cert.pem > client_cert = /etc/octavia/certs/client.cert-and-key.pem > base_path = /var/lib/octavia > base_cert_dir = /var/lib/octavia/certs > connection_max_retries = 1500 > connection_retry_interval = 1 > > [controller_worker] > amp_image_tag = amphora > amp_ssh_key_name = octavia > amp_secgroup_list = 80f44b73-dc9f-48aa-a0b8-8b78e5c6585c > amp_boot_network_list = 04425cb2-5963-48f5-a229-b89b7c6036bd > amp_flavor_id = 200 > network_driver = allowed_address_pairs_driver > compute_driver = compute_nova_driver > amphora_driver = amphora_haproxy_rest_driver > client_ca = /etc/octavia/certs/client_ca.cert.pem > loadbalancer_topology = SINGLE > amp_active_retries = 9999 > > [task_flow] > [oslo_messaging] > topic = octavia_prov > rpc_thread_pool_size = 2 > > [house_keeping] > [amphora_agent] > [keepalived_vrrp] > > [service_auth] > auth_url = https://192.168.10.150:5000 > auth_type = password > project_domain_name = default > user_domain_name = default > project_name = admin > username = admin > password = F35sXAYW5qDlMGfQbhmexIx12DqrQdpw6ixAseTd > cafile = /etc/ssl/private/haproxy.pem > region_name = Hanoi > memcached_servers = 192.168.10.178:11211,192.168.10.179:11211,192.168.10.28:11211 > #insecure = true > > > [glance] > ca_certificates_file = /etc/ssl/private/haproxy.pem > region_name = Hanoi > endpoint_type = internal > insecure = false > > [neutron] > ca_certificates_file = /etc/ssl/private/haproxy.pem > region_name = Hanoi > endpoint_type = internal > insecure = false > > [cinder] > ca_certificates_file = /etc/ssl/private/haproxy.pem > region_name = Hanoi > endpoint_type = internal > insecure = false > > [nova] > ca_certificates_file = /etc/ssl/private/haproxy.pem > region_name = Hanoi > endpoint_type = internal > insecure = false > > [oslo_policy] > #policy_file = /etc/octavia/policy.json > > [oslo_messaging_notifications] > transport_url = rabbit://openstack:4ychZAT5VrWlk6KFfgAmpXvGdzfdV8hEpIgOLhyF@192.168.10.178:5672,openstack:4ychZAT5VrWlk6KFfgAmpXvGdzfdV8hEpIgOLhyF@192.168.10.179:5672,openstack:4ychZAT5VrWlk6KFfgAmpXvGdzfdV8hEpIgOLhyF@192.168.10.28:5672 > > _______________________________________________ > Community mailing list > Community(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/community

3 2

[Diversity] Diversity & Inclusion WG Meeting 8/31 - Removing Divisive Language
by Amy Marrich 26 Aug '20

26 Aug '20

The Diversity & Inclusion WG has taken on the task from this week's Board meeting to assist with the development of the OSF's stance on the removal of Divisive Language within the OSF projects. The WG invites members of all OSF projects to participate in this effort and to join us at our next meeting Monday, August 31, at 17:00 UTC which will be held at https://meetpad.opendev.org/osf-diversity-and-inclusion. <https://meetpad.opendev.org/osf-diversity-and-inclusion> The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. If you have any questions please let me and the team know here, on #openstack-diversity on IRC, or you can email me directly. Thanks, Amy Marrich (spotz)

1 0

openstack-discuss
by ARUNKUMAR PALANISAMY 26 Aug '20

26 Aug '20

Hello Team, My name is ARUNKUMAR PALANISAMY, As part of our project requirement, we are evaluating trove components and need your support for experimental datastore Image for testing cluster. (Redis, Cassandra, MongoDB, Couchbase) 1.) We are running devstack enviorment with Victoria Openstack release and with this image (trove-master-guest-ubuntu-bionic-dev.qcow2<https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubun…>), we are able to deploy mysql instance and and getting below error while creating mongoDB instances. "ModuleNotFoundError: No module named 'trove.guestagent.datastore.experimental' " 2.) While tried creating mongoDB image with diskimage-builder<https://opendev.org/openstack/diskimage-builder> tool, but we are getting "Block device " element error. Regards, Arunkumar Palanisamy =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you

1 0

Re: [tc][masakari] Project aliveness (was: [masakari] Meetings)
by Jegor van Opdorp 26 Aug '20

26 Aug '20

We're also using masakari and willing to help maintain it! ________________________________ From: Mark Goddard <mark(a)stackhpc.com> Sent: Monday, August 17, 2020 12:12 PM To: Jegor van Opdorp <jegor(a)greenedge.cloud> Subject: Fwd: [tc][masakari] Project aliveness (was: [masakari] Meetings) ---------- Forwarded message --------- From: Radosław Piliszek <radoslaw.piliszek(a)gmail.com> Date: Fri, 14 Aug 2020 at 08:53 Subject: [tc][masakari] Project aliveness (was: [masakari] Meetings) To: openstack-discuss <openstack-discuss(a)lists.openstack.org> Cc: Sampath Priyankara (samP) <sam47priya(a)gmail.com>, Tushar Patil (tpatil) <tushar.vitthal.patil(a)gmail.com> Hi, it's been a month since I wrote the original (quoted) email, so I retry it with CC to the PTL and a recently (this year) active core. I see there have been no meetings and neither Masakari IRC channel nor review queues have been getting much attention during that time period. I am, therefore, offering my help to maintain the project. Regarding the original topic, I would opt for running Masakari meetings during the time I proposed so that interested parties could join and I know there is at least some interest based on recent IRC activity (i.e. there exist people who want to use and discuss Masakari - apart from me that is :-) ). -yoctozepto On Mon, Jul 13, 2020 at 9:53 PM Radosław Piliszek <radoslaw.piliszek(a)gmail.com> wrote: > > Hello Fellow cloud-HA-seekers, > > I wanted to attend Masakari meetings but I found the current schedule unfit. > Is there a chance to change the schedule? The day is fine but a shift > by +3 hours would be nice. > > Anyhow, I wanted to discuss [1]. I've already proposed a change > implementing it and looking forward to positive reviews. :-) That > said, please reply on the change directly, or mail me or catch me on > IRC, whichever option sounds best to you. > > [1] https://blueprints.launchpad.net/masakari/+spec/customisable-ha-enabled-ins… > > -yoctozepto

6 13