November 2019 - openstack-discuss

deployment failed
by SHAMBHU KUMAR 12 Nov '19

12 Nov '19

Dear Sir/mam i'm facing the issue while deployment of triple o no valid host found error code 500 Can you please eloborate in this matter because i'm stuck heere Your support will be highly appriciated..

2 1

[scientific-sig] IRC meeting today - Shanghai roundup and Supercomputing 2019
by Stig Telfer 12 Nov '19

12 Nov '19

Greetings all - We have a Scientific SIG meeting in about an hour’s time (2100 UTC) in channel #openstack-meeting. Everyone is welcome. Agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_12th_20… <https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_12th_20…> We are going to cover a trip report from Shanghai and planning for the many activities coming up next week in Supercomputing. Cheers, Stig

1 0

[nova][ironic][ptg] Resource tracker scaling issues
by Balázs Gibizer 12 Nov '19

12 Nov '19

* COMPUTE_RESOURCE_SEMAPHORE blocks instance creation on all nodes (on the same host) while the _update_available_resource runs on all nodes. On 3500 baremetal nodes _update_available_resource takes 1.5 hour. * Do we still need _update_available_resource periodic task to run for ironic nodes? * Reduce the scope of the COMPUTE_RESOURCE_SEMAPHORE lock * https://review.opendev.org/#/c/682242/ * https://review.opendev.org/#/c/677790/ * changing a locking scheme is frightening => we need more testing Agreement: * Do a tempest test with a lot of fake ironic node records to have a way to test if changing the locking scheme breaks anything * Log a bug and propose a patch for having a per-node lock instead of the same object for all the ResourceTrackers * See also whether concurrency helps * Propose a spec if you really want to pursue the idea of being somehow inconsistent with data by not having a lock Cheers, gibi

8 11

[tripleo][ptg]
by Wesley Hayutin 12 Nov '19

12 Nov '19

Greetings, Thanks for all those Red Hatters who attended the OpenStack Summit and PTG in Shanghai! A special thanks to those who presented their topics and discussed work items with the folks in attendance. As the current PTL for TripleO I will do my best here to summarize those conversations and items others should be made aware of. Over the course of Thursday and Friday roughly 7-10 folks discussed the identified topics [1], at the following times with my raw notes attached [2]. My apologies if I did not accurately represent your topic here, please feel free to correct me. Thursday, Giulio Fidente: Edge ( SDS storage ) Guilio walked us through some background work with regards to support for storage in remote sites / edge deployments. Working through support for Cinder was straight forward enough with no real collaboration required. Support for ceph copy on write for nova guests was also added with the glance image added to remote sites. Where Guilio needed input was with regards to having change the ctrl plane config for glance for each remote site [3]. This ctrl plane update would force operators to put the cloud in maintenance mode for a stack update. It was determined this could not be avoided at this time. It was noted that the TripleO simplification project and rework puppet-apply, please help us achieve that by reviewing the following two topics [4][5]. Thanks Giulio! Thursday, Guilio Fidente: Virtual IP / Storage Guilio walked us through some challenges with hosting a shared file system on remote/edge sites using manilla. The idea was to use Ganesha translation with CephFS. The proposal was that Ganesha and pacemaker would be managed in the ctrl plane but there was a question with regards to the virtual ip on edge sites. This was an interesting conversation that ended up with a suggestion from Kevin Carter to use a host-only local route on the edge to properly route the ip. This seemed to everyone to be a very clever solution to the problem :) Thanks Guilio, Thanks Kevin! Thursday, Martin Schuppert: Nova CellV2 multicell Martin walked the group through the current status and plans for the multicell implementation. Background: Nova multicells are used to help scale a cloud and partition it in such a way to get the messaging queue closer to the compute cell, essentially rabbit, galera, collector, vnc proxy and a number of compute nodes. This architecture is already in use but with only one default cell, pike was the switch to cellv2. The work started in Stein and continued through train using a similar approach as DCN. Some of the specs are that there is one cell per stack that is initially created from an export of the central stack, more ansible is place for the deployment as well. Two different architectures were noted, all cells in one heat stack [6], and one that splits the cell controllers and computes into different heat stacks w/ multiple stacks on the edge sites [7]. The development work for updates is complete and upgrades is still a WIP. Plans for the future included integrating TLS everywhere and enabling storage in the cell ( cinder, ceph, glance). Tony Breeds pointed out this architecture should just work in multiarch but would like the teams help in designing / advice while creating a test environment. Please review the following patches [13] Thanks Martin!! We tried to get more folks to switch to their topics to Thursday but were not able to. On to Friday. Friday, Edge ( DCN ) roadmap: David Paterson This conversation was informally walked through Thursday mainly with Arkady and Guilio and was followed up on Friday with a joint edge session regarding booting edge nodes. Several questions were raised on Thursday regarding the networking and connectivity for edge sites as it relates to provisioning. Validations were discussed as a way to address the minimum requirements for booting edge nodes. David did not end up presenting here, but was available at the joint session. See the “edge booting” section later in the document for details. Friday, Backup and Restore: Carlos Camacho The project started in Newton. Initially the backup consisted of a database dump and files being backed up for a defined set of use cases. In the field it was discovered that customers had many different kinds of deployments and the feature did not work well for all the intended use cases. An improved plan included to move to full disk image backups utilizing REAR [8]. Carlos also noted that customers are now trying to use ( or misuse ) this feature to perform baremetal to virt migrations. One of the current issues with the current solution include that it’s not clear how services behave after backup and restore.. E.G. OSD mons. Wes Hayutin noted that we have an opportunity to test the full image backup and restore solution by moving to a more image based internal CI system currently being designed by Jesse Pretorious and others. Thanks Carlos!! Friday, Failure Domains: Kevin Carter Unfortunately Kevin was in high demand across PTG events and was unable to present this topic. This should be discussed in a mid-cycle ( virtual or in person ) and written up as a blueprint. Essentially Kevin is proposing in large deployments to allow some number or percentage of nodes to fail a deployment while not failing the entire deployment. If a few non-critical nodes fail a large scale deployment TripleO should be better able to handle that, report back and move on. It was pointed out to me there is a related customer bug <https://bugzilla.redhat.com/show_bug.cgi?id=1721089> as well. Thanks Kevin!! Friday, Cross project: Edge Booting: Julia Kreger You can find notes on this session here [9]. I will only summarize the questions proposed on earlier edge ( DCN ) topic. With regards to when does TripleO need to support redfish there was no immediate or extremely urgent requests ( please correct me if I do not have the correct information there). Redfish IMHO did seem to be a nice improvement when compared to IPMI. This was my first introduction to Redfish, and I of course curious what steps we had to take in order to CI it. Luckily after doc diving I found several helpful links that include steps with setting Redfish up with our own OVB tooling ( hooray \0/ ). Links can be found here [10], and it seems like others have done some hard work to make that possible so thank you!! Thank you Julia!! Friday, Further TripleO Ansible Integration: Wes Hayutin The idea here would be allow the TripleO project to govern how TripleO is deployed with ansible as an operator. The TripleO project would ship ansible modules and roles that directly import python-tripleoclient to support ansible to cli parity [12]. Using or modeling a new repo, perhaps called tripleo-operator-ansible would be used to host these modules and roles and include the same requirements and features of tripleo-ansible’s linting, molecule tests, and auto documentation. This could tie in well with an initiative from Dan Macpherson to ship Ansible playbooks as part of our OSP documentation. Julia Kreger noted that we should not ignore the Ansible OpenStackSDK for part of the deployment process which is a very valid point. Most everyone at the PTG agreed this was a good direction moving forward and would help consolidate the public and internal tooling around TripleO’s CLI in ansible. Thanks Dan, Julia!! Friday, TLS-E standalone gate: Ade Lee Ade Lee walked us through a proposal to help test and CI TLS upstream which has been very difficult to date ( I can personally vouch for this ). Using a two node setup upstream with one node as the IPA server and the other a TripleO standalone deployment. The keystone team is setting the right example for other projects and teams that are finding it difficult to keep outside patches from breaking their code, and that is to find a way to get something voting and gating upstream even if it’s not installed and deployed in the same exact ways customers may use it. Please help by reviewing the keystone / security teams patches here [14] Thanks Ade!! Friday, Octavia tempest plugin support in TripleO-CI: Chandan Kumar Chandan was off fighting battles with the infra team and other projects. Here are some of his notes: Have a rdo third party standalone job with full octavia tempest triggered against octavia patches from stein onwards. FS062 Look into multinode job for queens releases as a third party from queens and rocky. FS038 Have a support of octavia tempest plugin in os_tempest. We certainly should have a conversation offline regarding these topics. I’ll note the TripleO-CI community meeting or the #tripleo meeting on Tuesdays are a good way to continue collaborating here. Thanks Chandan!! Friday, Replace clouds.yaml with an openrc ansible module: Chandan Kumar Open Question: is this module [15] from the openstack-ansible project something we can reuse in TripleO via tripleo-ansible? Friday, Zuul jobs and ansible roles to handle rpm packaging: Chandan Kumar The background and context can be found here https://pagure.io/zuul-distro-jobs - collection of ansible roles to deal with rpm packaging https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/r… -> creating rpms from different projects depends-on in commit message Proposal: Move it to zuul-jobs Making the rpm packaging more generic for centos/fedora/rhel and move it to zuul jobs * Move mock and rpmbuild related roles to zuul jobs repo * Adding a mention of third party zuul jobs to main zuul jobs doc * Build-set_registry setup http server and start the job * Details are here <https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019> Thanks Chandan!! This is indeed a very interesting and powerful proposal. We should definitely continue this conversation with the broader community. Did you make it all the way down here? Well done! I should add an easter egg :) Links: [1] https://etherpad.openstack.org/p/tripleo-ussuri-topics [2] https://etherpad.openstack.org/p/tripleo-ptg-ussuri [3] https://blueprints.launchpad.net/tripleo/+spec/split-controlplane-glance-ca… [4 ] https://review.opendev.org/#/q/topic:disable/paunch+(status:open+OR+status:… [5] https://review.opendev.org/#/q/topic:deconstruct/container-puppet+(status:m… [6] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/feature… [7] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/feature… https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/feature… [8] https://access.redhat.com/solutions/2115051 [9] https://etherpad.openstack.org/p/PVG-ECG-PTG [10] https://github.com/openstack/sushy https://docs.openstack.org/sushy/latest/contributor/index.html#contributing https://docs.openstack.org/sushy-tools/latest/ https://docs.openstack.org/sushy-tools/latest/user/dynamic-emulator.html#sy… [12] https://hackmd.io/caRlGha7SueZxDRcyq9eGA?both [13] https://review.opendev.org/#/q/topic:cellv2+(status:open+OR+status:merged) [14] https://review.opendev.org/#/q/status:open+project:openstack/tripleo-heat-t… [15] https://opendev.org/openstack/openstack-ansible-openstack_openrc [16] https://etherpad.openstack.org/p/PVG-keystone-forum-policy [17 https://datko.pl/zuul.pdf [18] https://github.com/openstack/tripleo-heat-templates/blob/master/README.rst#… [19] https://github.com/openstack/openstack-virtual-baremetal Thanks all!! Wes Hayutin TripleO-PTL

1 0

[ops] next ops meetup : London 7,8 Jan 2020 - approved!
by Chris Morgan 12 Nov '19

12 Nov '19

https://twitter.com/osopsmeetup/status/1194281816468938752?s=20 -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[all][doc] Patches to add --keep-going to sphinx-build (and patches proposed to many many repositories)
by Akihiro Motoki 12 Nov '19

12 Nov '19

Hi, As you may notice, we see a lot of patches which try to add --keep-going to sphinx-build. [0] I have suggestions and questions. 1) First, when reviewing them, keep the following in your mind. * --keep-going is added even when -W option is not used in the sphinx-build command line. -W is recommended in the PTI [1], so ensure to have -W. * Some of them ignores cases where "python setup.py build_sphinx" is still used. If it is a good chance to clean them up and use "sphinx-build" consistently. 2) Why do we need to remove the build directory for releasenotes? Some of them propose to add "rm -rf releasenotes/build"? (for example [2]) I cannot understand why this needs to be added. Do we really want to call "rm -rf <build dir>"? I know it is needed in some repositories due to various reasons, but generally speaking it make the documentation build longer and they are unnecessary. I tried to get the reason from the authors in reviews but they just say it is simple and it can be added at the same time. Thus, I would like to ask it more broadly in the list. 3) What is the recommended way to get a consensus to make this kind of patches which affects many many repositories? It is not productive to ask questions in individual reviews. Some patches are approved fast and questions are pop-up in other patches. It makes it difficult to discuss the real needs and keep many repositories consistent. I am not against all of these changes, but I would like to see more organized way. Thought? Thanks, Akihiro [0] https://review.opendev.org/#/q/message:keeping [1] https://governance.openstack.org/tc/reference/project-testing-interface.htm… [2] https://review.opendev.org/#/c/690956/2/tox.ini

2 1

[nova][ptg] Resurrecting NUMA topology in Placement
by Sylvain Bauza 12 Nov '19

12 Nov '19

We discussed about a long known story https://review.openstack.org/#/c/552924/ The whole agreement during the PTG was to keep things simple with baby steps : - only supporting a few NUMA queries and defer others as unsupported (still supported by legacy NUMATopologyFilter) - The to-be-resurrected spec would be only focus on VCPU/PCPU/MEMORY_MB resource classes and not handle PCI or GPU devices (ie. level-1 tree, no children under the NUMA RPs) Agreement was also there for saying that functional tests should be enough since PlacementFixture already works perfectly. -S

2 3

[infra] Etherpad problem
by Slawek Kaplonski 12 Nov '19

12 Nov '19

Hi, Just at the end of the ptg sessions, neutron etherpad was got broken somehow. Now when I try to open [1] I see only something like: An error occurred The error was reported with the following id: 'igzOahZ6ruH0eSUAWKaj' Please press and hold Ctrl and press F5 to reload this page, if the problem persists please send this error message to your webmaster: 'ErrorId: igzOahZ6ruH0eSUAWKaj URL: https://etherpad.openstack.org/p/Shanghai-Neutron-Planning UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0 TypeError: r.dropdowns is undefined in https://etherpad.openstack.org/javascripts/lib/ep_etherpad-lite/static/js/p… at line 18' We can open one of the previous versions which is available at [2] but I don't know how we can fix original etherpad or restore version from [2] to be original etherpad and make it working again. Can someone from infra team check that for us maybe? Thx in advance for any help. [1] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning [2] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning/timeslider#16117 -- Slawek Kaplonski Senior software engineer Red Hat

4 4

[ops] meetups team meeting and meetups venue for Jan 2020
by Chris Morgan 12 Nov '19

12 Nov '19

There wll be an OpenStack Ops Meetups team meeting in just under 45 minutes on #openstack-operators. The agenda is here: https://etherpad.openstack.org/p/ops-meetups-team We'd like to see if we can agree to formally accept the offer to host the next OpenStack Operators Meetup in London next January (see https://etherpad.openstack.org/p/ops-meetup-1st-2020) Please attend the meeting if you have feedback for or against this proposal. So far all feedback has been positive and there are no other offers. Note (full disclosure) that this proposal is from my employer. We'll also go over material from last week's summit in Shanghai. Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[ptg][nova][glance] x-p meeting minutes
by Sylvain Bauza 12 Nov '19

12 Nov '19

We only discussed about nova snapshots to dedicated glance stores. Reference is https://review.opendev.org/#/c/641210/ We had concerns on implementing glance storage location strategies in Nova. A counter-proposal was made to only pass the original image ID when calling Glance for a snapshot so then Glance would then use any store the operator wants (eg. a location strategy like "please store the snapshot to a place close to where the original image is"). The agreement was there for the counter-proposal so the Nova change would only be to pass the original glance image ID to the new glanceclient that would support it. Glance folks will provide a new spec revision. -S

1 0