April 2019 - openstack-discuss - lists.openstack.org

Re: [openstack-dev] Are we ready to put stable/ocata into extended maintenance mode?
by Alex Schultz 03 Mar '21

03 Mar '21

On Tue, Sep 18, 2018 at 1:30 PM Alex Schultz <aschultz(a)redhat.com> wrote: > > On Tue, Sep 18, 2018 at 1:27 PM, Matt Riedemann <mriedemos(a)gmail.com> wrote: > > The release page says Ocata is planned to go into extended maintenance mode > > on Aug 27 [1]. There really isn't much to this except it means we don't do > > releases for Ocata anymore [2]. There is a caveat that project teams that do > > not wish to maintain stable/ocata after this point can immediately end of > > life the branch for their project [3]. We can still run CI using tags, e.g. > > if keystone goes ocata-eol, devstack on stable/ocata can still continue to > > install from stable/ocata for nova and the ocata-eol tag for keystone. > > Having said that, if there is no undue burden on the project team keeping > > the lights on for stable/ocata, I would recommend not tagging the > > stable/ocata branch end of life at this point. > > > > So, questions that need answering are: > > > > 1. Should we cut a final release for projects with stable/ocata branches > > before going into extended maintenance mode? I tend to think "yes" to flush > > the queue of backports. In fact, [3] doesn't mention it, but the resolution > > said we'd tag the branch [4] to indicate it has entered the EM phase. > > > > 2. Are there any projects that would want to skip EM and go directly to EOL > > (yes this feels like a Monopoly question)? > > > > I believe TripleO would like to EOL instead of EM for Ocata as > indicated by the thead > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.ht… > Bringing this backup to see what we need to do to get the stable/ocata branches ended for the TripleO projects. I'm bringing this up because we have https://review.openstack.org/#/c/647009/ which is for the upcoming rename but CI is broken and we have no interest in continue to keep the stable/ocata branches alive (or fix ci for them). Thanks, -Alex > Thanks, > -Alex > > > [1] https://releases.openstack.org/ > > [2] > > https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… > > [3] > > https://docs.openstack.org/project-team-guide/stable-branches.html#extended… > > [4] > > https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… > > > > -- > > > > Thanks, > > > > Matt > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 11

[openstack-ansible] dropping suse support
by Mohammed Naser 05 May '20

05 May '20

Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct… [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct…

6 9

[nova] critical bug around reload/upgrades
by Mohammed Naser 24 Aug '19

24 Aug '19

Hello: I've discussed this for quite sometime with Dan over IRC and a bit with Zane as well, but basically, Nova thinks that when it gets a reload (aka SIGHUP), nothing else has occurred. However, oslo.service actually calls stop(), reload() then start() again, which potentially kills all RPC. This has caused a pretty big issue in our gates and it also means that the whole idea behind 'reload nova-compute while upgrading to refresh info' concept is fundamentally broken. I tried to do some work on this here, however, I wasn't really able to get to the bottom of it. There seems to be a decision that needs to be taken in terms of .. do we change what reload() actually means in oslo_service (it actually is more like a restart, not a reload) or does Nova (and other projects) change their implementation in assuming what reload() does? https://review.openstack.org/#/c/641907/ This seems to have been floating around for a really long time, so I'd be happy to work with someone to find the fix (and we can totally test it inside OpenStack Ansible by reloading instead of restarting). Thanks! Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

4 4

[stackalytics] Reported numbers seem inaccurate
by Mark Goddard 26 Jun '19

26 Jun '19

Hi, I've heard a couple of people say they don't feel the numbers reported by Stackalytics accurately reflect reality. I've been trying to gather a few stats for my Kolla update session in Denver, and am finding the same. I'll try to give some concrete examples. Reviews for all kolla deliverables in Stein [1]. Here the company stats don't reflect the individual stats. Also, the total reviews in the 'Kolla Official' module does not equal the sum of the reviews of its submodules (kolla, kolla-ansible, kolla-cli). If I look at the contribution summary for Kolla Official in the last 90 days [2], they are actually greater than those for the last 180 days [3]! There are also similar issues with commit metrics, and none seem to match what I see in git. Thanks, Mark [1] https://www.stackalytics.com/?metric=marks&module=kolla-group [2] https://www.stackalytics.com/report/contribution/kolla-group/90 [3] https://www.stackalytics.com/report/contribution/kolla-group/180

10 11

Re: [nova][placement] Need for a nova-manage placement audit command
by Matt Riedemann 19 Jun '19

19 Jun '19

Re-posting this since it came up again in IRC today. On 9/20/2018 6:09 PM, Matt Riedemann wrote: > mnaser wrote a simple placement audit tool today and dumped his script > into a bug report [1]. It turns out several operators have already > written a variant of this same tool. The idea is to compare what's in > placement for allocations and (compute node) resource providers against > what's in nova for instances and compute nodes, look for orphans in > placement and then report them, or maybe also clean them up. > > I think this could go into a "nova-manage placement audit" command and > should be pretty easy to write for minimum functionality (start with > just the report). > > I'm advertising the need here in case someone wants to work on this. I'd > like to myself, but just can't justify the time right now. > > [1] https://bugs.launchpad.net/nova/+bug/1793569 > -- Thanks, Matt

2 1

[kolla][magnum] Cluster creation failed due to "Waiting for Kubernetes API..."
by Giuseppe Sannino 29 May '19

29 May '19

Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled --------------- VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out" I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5 Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE) Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver. [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1. May I ask for an help on this ? Many thanks /Giuseppe

5 11

Re: [stackalytics] Reported numbers seem inaccurate
by Edward Ionel 24 May '19

24 May '19

Hello All, I first want to thank each and everyone of you for reaching out and sharing the inaccuracies you all have found within Stackalytics, this has helped us tremendously with the efforts to correct these flaws. I am writing this email to directly address each individual that has reached out and shared their concerns with our Stackalytics data. Mark Goddard Michael Johnson Julia Kreger Collen Murphy Clark Boylan Sean Mooney Thierry Carrez Jermey Stanley Arkady Kanvesky (Hopefully, I did not miss anyone) I want to insure everyone that the team is working hard to fix the inaccuracies in some of our data. We are still very much involved in OpenStack and we do read each and every message regarding Stackalytics. As Boris and Sergey mentioned, we found that this occurred due to an outdated production environment. We are currently in the progress of migrating to a new environment, and the data should be corrected. The team and I are sorry for these discrepancies and appreciate you all for bringing this to our attention. Thank you all, for providing links and examples of the inaccuracies. This has helped our efforts greatly in correcting the current issues. It would be great if the community can continue reviewing Stackalytics data and help us provide a better service to all. Once again, we are all very sorry for this and will continue to fix these issues. Cheers, Edward > On Apr 16, 2019, at 4:05 PM, Edward Ionel <eionel(a)mirantis.com> wrote: > > > >> On Tue, Apr 16, 2019 at 2:56 PM Boris Renski <brenski(a)mirantis.com> wrote: >> Folks - more specifically, we plan to roll out an update by tomorrow mid-day PT which should fix all the numbers + add aggregate details on Airship project contributions. >> >> Would really appreciate the help of the community to poke around tomorrow afternoon and see if there are still discrepancies. >> >> Sorry about this and thank you for catching it and brining it to our attention. >> >> -Boris >> >>> On Tue, Apr 16, 2019 at 1:33 PM Sergey Nikitin <snikitin(a)mirantis.com> wrote: >>> Hello everyone, >>> As you mentioned there are some inaccuracies in stats on stackalytics.com. It happened because of outdated Prod env. At this moment we are in a process of migration to the new env. All data on this env is fresh and accurate. So the problem will be solved soon. >>> >>> -- >>> Best Regards, >>> Sergey Nikitin > > > -- > Edward Ionel > Marketing Operations Manager > Mirantis, Inc. > 900 E Hamilton Ave, Suite 650 > Campbell, CA 95008 > Eionel(a)mirantis.com > www.mirantis.com

4 14

[heat][neutron] improving extraroute support
by Bence Romsics 24 May '19

24 May '19

Hi Heat and Neutron folks, I'm wondering how could I improve support for extra routes for one of our users. In this email I'd like to get your input (from both Heat and Neutron folks) in which direction should I start before I sit down to write the relevant blueprints and RFEs. This will be a bit long, please bear with me. As you know a neutron router can have extra static routing table entries as the 'routes' attribute like this (please see the api-ref in [1]): $ openstack router show router1 -f value -c routes destination='10.1.0.0/24', gateway='10.0.0.11' destination='10.2.0.0/24', gateway='10.0.0.12' ... To my best knowledge the whole set of extra routes must be updated at once which easily leads to lost updates if multiple clients modify this attribute concurrently (imagine client A and B working concurrently, for example both reads routes [r1], client A writes [r1, r2], client B writes [r1, r3]; depending on timing either r2 or r3 will be lost). Heat has an unsupported resource OS::Neutron::ExtraRoute [2] to front Neutron extra routes. For example: heat_template_version: '2015-04-30' resources: Extra_route_1: type: OS::Neutron::ExtraRoute properties: destination: '10.1.0.0/24' nexthop: '10.0.0.11' router_id: 'd544b5c5-d4d2-4b52-aaa8-ef4b9fa3e154' The Heat resource represents a single extra route out of the many in the router's routes attribute. If you have multiple OS::Neutron::ExtraRoute resources Heat by default handles them concurrently leading to the lost update problem. You can easily reproduce the problem just by having 10 extra routes in a template. The stack creation will succeed, but the router in fact will have only 6-7 (the exact number changes based on timing) extra routes. AFAIU this is exactly why this was marked unsupported. (Please let me know if you know other reasons.) As a workaround I suggested our user a hack to serialize the OS::Neutron::ExtraRoute resources by 'depends_on'. But which way should I go if I want to fix this properly? way #1 - changes in Heat only: Introduce a new Heat resource OS::Neutron::ExtraRoutes (please notice the plural) which can represent the whole routes attribute of a router (i.e. multiple extra routes in a single resource). This will not solve the design shortcomings of the neutron extraroute API, but it can at least express what can be done in the Neutron API as is. Lost updates of multiple stacks editing the same router's routes are still possible. way #2 - changes in Heat and Neutron: Introduce a new Neutron extension with a better API (either extraroutes as a top level resource, each object with a router_id or an action map a'la add/remove_router_interface to edit extra routes on the server side). This new API would work on the same DB tables and backends as the old. The old API would be kept for backwards compatibility. But if a new user exclusively uses the new API it can work concurrently. Then introduce a new Heat resource (or change the existing unsupported one) to use this new Neutron API. This is obviously more work than the first option. Which one do you prefer and why? Or let me know please if I missed a better alternative. In this thread I hope to get a cross-project agreement of this between Heat and Neutron so I can go ahead and open the relevant Heat and Neutron blueprints where we can focus only on each projects' internal questions. Thanks in advance, Bence Romsics irc: rubasov [1] https://developer.openstack.org/api-ref/network/v2/?expanded=update-router-… [2] https://docs.openstack.org/heat/latest/template_guide/unsupported.html#OS::…

4 11

[all][docs] season of docs
by Mohammed Naser 23 May '19

23 May '19

Hi there: It seems like Google has come up with a new somewhat-GSoC-like idea but focused on documentation. I think it could be a good opportunity for the documentation team (or any specific team actually, coordinated with docs) to be part of this. https://opensource.googleblog.com/2019/03/introducing-season-of-docs.html I'm not sure if the team has the amount of resources, but it seems they should be able to apply to this. Does this seem like something that might help the team more (or perhaps a specific project, coordinating with the docs team) to apply for this? Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

7 10

[glance][interop] standardized image "name" ?
by Mohammed Naser 23 May '19

23 May '19

Hi everyone, Something that I've been dealing with is the fact that regardless of what cloud you're using, the naming structure is very much different. While we aim to be interoperable, if I want to "boot an Debian Stretch server", the image name will probably be different all over the places. Is there some sort of way we can eliminate this by implementing a standard naming convention which would allow us to use the same images across all systems? I'm just opening the idea for discussion, I'd love to hear what others think about this. Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

15 22

[openstack-ansible] Retirement of unused OpenStack Ansible roles
by Guilherme Steinmüller 20 May '19

20 May '19

Hello all, We have been discussing about some of our roles that are not maintained for a while, some of them are not even integrated with our main repository. In order to the team to focus on performance and stability of the roles that has been actively used, we are discussing about retiring the unused ones. The roles are the following: openstack-ansible-os_monasca openstack-ansible-os_monasca-agent openstack-ansible-os_zaqar openstack-ansible-os_watcher openstack-ansible-os_almanach openstack-ansible-os_molteniron openstack-ansible-os_searchlight We would love to help if someone wants to pick one or some of them to keep our variety of roles. If we don't hear back, we'll retire those roles out of OpenStack Ansible. Thank you Guilherme Steinmüller

6 11

OpenStack User Survey 2019
by Jimmy Mcarthur 16 May '19

16 May '19

It's time for the OpenStack User Survey <https://www.openstack.org/user-survey> [1]! In an effort to create less work for our community, in 2018 we moved to an annual, rather than a bi-annual report. If you run an OpenStack deployment, now is the time to share details about how you run the software and your thoughts on ways that the community can improve the software. Your feedback matters and it's shared directly with the PTLs and community leaders that work to build OpenStack! We had some good community feedback on the last survey and ended up with only minor updates from 2018. Below you can see a list of changes that we've incorporated: * "Which new/emerging technologies are you interested in using or seeing integrated with OpenStack?" o Added options for "AI/ML" and "5G" * Deleted the question "Are you interested in helping our interoperability efforts by using RefStack to test your cloud capabilities?" * "Which tools are you using to deploy / manage this cluster?" o Added the word "manage" to the question as tools can be used for both o Add "TripleO" and "Container-based orchestration" as options o Created a new "Please specify" if "Other tools" is chosen * "Which container orchestration or Platform-as-a-Service (PaaS) tools are you using to manage applications on this OpenStack deployment?" o Added "container orchestration or " to question * "Which OpenStack Network (Neutron) drivers are you using?" o Add Calico as an answer * "Which compatibility APIs does this deployment support?" o Added the option "This deployment doesn't use compatibility APIs" * Please note: This year, we will not be including the hardware and software vendor questions in the report or analytics dashboard due to the potential for the data to be misinterpreted. * *In addition to the question modifications above, we reached out to PTLs to ask for their feedback and guidance on the project specific questions of the survey. We've modified or added those questions as noted below: *Keystone* This will be a question where respondents can rank their choices and will follow their existing question: * Which of the following initiatives would be most beneficial for your deployment? You can rank items by priority with items at the top being most important and items at the bottom being least important. o Improved performance o Better support for synchronizing multiple keystone regions o Read-only RBAC roles o More flexible federated mappings o Improved fine-grained access control o Better support for domain administrators o Improved usability of keystone outside of an OpenStack environment Glance Changed existing question to: * "Multi back-ends feature will stay Experimental in Stein release. How badly this affect your deployment plans? Any details how you’re planning to use it would be helpful.” *Ironic* Changed existing question to: * "What would you find most useful if it was part of Ironic?" *Swift* Changed existing question to: * What features of Swift are you currently using (or have plans to use in the next 12 months)? (select all that apply o S3 compatibility o Dynamic Large Objects o Static Large Objects o Temporary URLs o Data-at-rest encryption o Multiple storage policies o Multi-region clusters o Explicit data placement (composite rings) o Container sync o Static websites or CORS o Symlinks o Quotas o Object versioning If you have further questions, concerns, please let us know. Cheers, Jimmy and Allison [1] https://www.openstack.org/user-survey

7 14

[PTL][SIG][WG] PTG Team Photos
by Kendall Nelson 15 May '19

15 May '19

Hello! If your team is attending the PTG and is interested in having a team photo taken, here is the signup[1]! There are slots Thursday and Friday from 10:00 AM to 4:30 PM. The location is TBD but will likely be close to where registration will be. I'll send an email out the day before with a reminder of your time slot and an exact location. -Kendall (diablo_rojo) [1] https://docs.google.com/spreadsheets/d/1DgsRHVWW2YLv7ewfX0M21zWJRf4wUfPG4ff…

5 10

[devstack][stein][masakari]Using Masakari in multihost devstack stein setup
by Sanjay K 13 May '19

13 May '19

Hi all, I have been trying to setup Masakari in a 3 node setup - 1 controller + 2 computes with minimal openstack services installed using devstack stable/stein version. All these 3 nodes are Ubuntu 16.04 VMs. I have included Masakari plugin in controller's local.conf file and included masakari-monitor in both compute node's local.conf file. I want to test out VM/process failure in my test environment. To do so, when I kill one of the qemu process created for one instance (cirros 256 flavor VMs) on one of the compute with root user login, I did not see any notification under my horizon/instance-ha section for this (I have already created Segments, Hosts under instance-ha in horizon). Also the killed process is not restarted by Masakari. I have asked the same question in openstack forum, but not got a single response. masakari-notification-on-process-failure <https://ask.openstack.org/en/question/121490/masakari-notification-on-proce…> Whether any additional services related to Masakari need to be configured on compute hosts to detect failures since I did not find exactly the documentation related inside devstack for this. Please let me know if I am missing anything my set up. Any helps and pointers are most appreciated. thank you for your reply. best regards

2 4

[placement][nova][ptg] resource provider affinity
by Chris Dent 09 May '19

09 May '19

Spec: https://review.openstack.org/650476 >From the commit message: To support NUMA and similar concepts, this proposes the ability to request resources from different providers nested under a common subtree (below the root provider). There's much in the feature described by the spec and the surrounding context that is frequently a source of contention in the placement group, so working through this spec is probably going to require some robust discussion. Doing most of that before the PTG will help make sure we're not going in circles in person.k Some of the areas of potential contention: * Adequate for limited but maybe not all use case solutions * Strict trait constructionism * Evolving the complexity of placement solely for the satisfaction of hardware representation in Nova * Inventory-less resource providers * Developing new features in placement before existing features are fully used in client services * Others? I list this not because they are deal breakers or the only thing that matters, but because they have presented stumbling blocks in the past and we may as well work to address them (or make an agreement to punt them until later) otherwise there will be lingering dread. And, beyond all that squishy stuff, there is the necessary discussion over the solution described in the spec. There are several alternatives listed in the spec, and a few more in the comments. We'd like to figure out the best solution that can actually be done in a reasonable amount of time, not the best solution in the absolute. Discuss! -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

10 37

[nova] Stein regressions
by Matt Riedemann 08 May '19

08 May '19

Hey cool cats, I know we just released Stein last week so what better time to write an email about a bunch of Stein regressions?! I wanted to dump this in the list so (a) I don't lose track of them all and (b) other people are aware and get involved. In no particular order: 1. https://bugs.launchpad.net/nova/+bug/1822801 There is a patch being worked: https://review.openstack.org/#/c/649345/ The workaround for this would be to disable the _heal_instance_info_cache periodic in nova-compute services that manage baremetal nodes by setting the heal_instance_info_cache_interval value to 0. 2. https://bugs.launchpad.net/nova/+bug/1824435 We have a re-create scenario for this in devstack (noted in the bug report). I've been unable to recreate in a functional test, possibly because of how sqlite deals with NULL values in unique constraints differently from mysql. I would have thought this regression test [1] would have hit it but it didn't. There are some options for fixing this, some more hacky than others, e.g.: https://review.openstack.org/#/c/653065/ Need some input from others on fixing that bug in maybe a less hacky way. 3. https://bugs.launchpad.net/nova/+bug/1825034 I found this when writing [1] to try and recreate the previous bug 1824435. I have a series up for this starting with a recreate patch [1]. The fix on top is one of a few options I detailed in the bug report. It's not pretty but it works. The workaround for this would be to archive deleted records but you'd have to do that after each run of the online data migration which is not great. 4. https://bugs.launchpad.net/nova/+bug/1825020 I just triaged this a couple of hours ago. I plan on writing a functional regression recreate test since the recreate scenario seems pretty easy - try to resize a volume-backed server where the new flavor has a smaller disk than the old flavor. The fix should also be relatively straight-forward. -- Once we get fixes landed and backported to stable/stein for these bugs we should do a patch release ASAP. [1] https://review.openstack.org/#/c/653098/ -- Thanks, Matt

1 2

[oslo][oslo-messaging][nova] Stein nova-api AMQP issue running under uWSGI
by iain.macdonnell＠oracle.com 08 May '19

08 May '19

Is anyone running Stein nova-api under uWSGI? I'm trying to do this, in exactly the same way as I did with Rocky, but encountering a new problem... The symptom is intermittent failure of API operations that involve RPC calls - notably getting the noVNC console URL for an instance. The nova-api log shows RPC timeouts and "connection reset by peer" for AMQP. I've ruled out environmental issues, as a Rocky installation on the same hosts works fine. With oslo debug enabled, I see that the first API call causes a AMQP connections to be established, and the RPC call gets its reply, and all is well... but there are no heartbeats logged after that. 3 minutes later, rabbitmq-server reports that it has received no heartbeats, and drops the connection. On a subsequent API call (but not before), nova-api logging "connection reset by peer" and the RPC call times out. With the Rocky code, once the first API call causes the AMQP connections to be established, regular heartbeats start happening, the connections are maintained, and subsequent API calls work as expected. It seems like a lockup in a thread that's supposed to maintain the AMQP connection, or something. I tried to use SIGUSR2 to trigger guru meditation, but it seems that uWSGI has its own interpretation of SIGUSR2, so nova-api never sees it (anyone know a way around this?). Running nova-api as a standalone service, instead of under uWSGI, the problem does not occur (I see the heartbeats, and everything works fine). I've tried updating oslo-messaging (and python-amqp) on a Rocky installation to the Stein version, but the problem does not reproduce. Ideas? ~iain

8 18

[octavia][taskflow] Adaption of persistence/jobboard (taskflow expert review needed)
by Anna Taraday 07 May '19

07 May '19

Hello everyone! I was looking at the topic of usage taskflow persistence and jobboard in Octavia [1]. I created a simple PoC to check what should be done to enable this functionality [2] . >From what I see, taskflow expects that data, which will be stored in persitence backend/jobboard backend, is a dict or an object easily converted to dicts [3] (error [3.1]) Also functions that creates flow should be importable [4] (error [4.1]). These two points lead to refactor required for Octavia to enable taskflow persistence and jobboard: 1) Convert data which is passed between flows in dicts, at this moment it is db objects with links to other db objects. 2) Create importable flow functions. As far as I see the only OpenStack project which adapted taskflow persistence is poppy [5] I'm looking for taskflow expect to take a look at all this and give some comments - whether I am correct or missing something. Thank you for your time in advance! [1] - https://storyboard.openstack.org/#!/story/2005072 [2] - https://review.openstack.org/#/c/647406 [3] - https://github.com/openstack/taskflow/blob/master/taskflow/persistence/back… [3.1] - http://paste.openstack.org/show/749530/ [4] - https://docs.openstack.org/taskflow/latest/_modules/taskflow/engines/helper… [4.1] - http://paste.openstack.org/show/749527/ [5] - https://github.com/openstack/poppy -- Regards, Ann Taraday Mirantis, Inc

2 3

[kolla] Virtual PTG scheduling poll
by Mark Goddard 07 May '19

07 May '19

Hi, Since most of the Kolla team aren't attending the PTG, we've agreed to hold a virtual PTG. We agreed to start with two 4 hour sessions. We can finish early or schedule another session, depending on how we progress. We'll use some video conferencing software TBD. I've created a Doodle poll here [2], please fill it in if you hope to attend. Times are in UTC. Please continue to fill out the planning Etherpad [1]. Thanks, Mark [1] https://etherpad.openstack.org/p/kolla-train-ptg [2] https://doodle.com/poll/adk2smds76d8db4u

1 2

[blazar] Scheduling a new Blazar IRC meeting for the Americas
by Pierre Riteau 06 May '19

06 May '19

Hello, Contributors to the Blazar project are currently mostly from Europe or Asia. Our weekly IRC meeting at 0900 UTC is a good match for this group. To foster more contributions, I would like to schedule another IRC meeting in the morning for American timezones, probably every two weeks to start with. I am thinking of proposing 1600 UTC on either Monday, Tuesday, or Thursday, which doesn't appear to conflict with closely related projects (Nova, Placement, Ironic). If there is anyone who would like to join but cannot make this time, or has a preference on which day, please let me know. I will wait for a few days before requesting a meeting slot. Thanks, Pierre

1 1

[tripleo] Nominate Cédric Jeanneret (Tengu) for tripleo-validations core
by Gaël Chamoulaud 04 May '19

04 May '19

Hi TripleO devs, The new Validation Framework is a big step further for the tripleo-validations project. In many ways, it improves the way of detecting & reporting potential issues during a TripleO deployment. As the mastermind of this new framework, Cédric brought a new lease of life to the tripleo-validations project. That's why we would highly benefit from his addition to the core reviewer team. Assuming that there are no objections, we will add Cédric to the core team next week. Thanks, Cédric, for your excellent work! =Gaël

10 10

[Vitrage] add datasource kapacitor for vitrage
by hoanq13＠viettel.com.vn 04 May '19

04 May '19

Hi, In our system, we use monitor by TICK stack (include: Telegraf for collect metric, InfluxDB for storage metric, Chronograf for visualize and Kapacitor alarming), which is popular monitor solution. We hope can integrate vitrage in, so we decide to write kapacitor datasource contribute for vitrage. The work is almost done , you can review in: https://review.opendev.org/653416 So i send this mail hope for more review, ideal,... Appreciate it. also ask: have any step i miss in pipeline of contribute datasource vitrage? like create blueprints, vitrage-spec,vv.. Should i do it?

2 2

[keystone][heat] security_compliance options and auto-created users
by Pavlo Shchelokovskyy 03 May '19

03 May '19

Hi all, I am currently looking at options defined in [security_compliance] section of keystone.conf [0] and trying to understand how enabling those security features may affect other services. The first thing I see is that any project that auto-creates some temporary users may be affected. Of the top of my head I can recall only Heat and Tempest doing this. For Tempest situation is easier as a) tempest can use static credentials instead of dynamic ones so it is possible to craft appropriate users beforehand and b) those users are relatively short-lived (required for limited time). In case of Heat though those users are used for deferred auth (like in autoscaling) which for long lived stacks can happen at arbitrary time in future - which is a problem. Below is breakdown of options/features possible to set and what problems that may pose for Heat and ideas on how to work those around: - disable_user_account_days_inactive - this may pose a problem for deferred auth, and it seems is not possible to override it via user "options". IMO there's a need to add a new user option to Keystone to ignore this setting for given user, and then use it in Heat to create temporary users. - lockout failure options (lockout_failure_attempts, lockout_duration) - can be overridden by user option, but Heat has to set it first. Also the question remains how realistically such problem may arise for an auto-created internal user and whether Heat should set this option at all - password expiry options (password_expires_days, unique_last_password_count, minimum_password_age) - poses a problem for deferred auth, but can be overridden by user option, so Heat should definitely set it IMO for users it creates - change_password_upon_first_use - poses problem for auto-generated users, can be overridden by a user option, but Heat must set it for its generated users - password strength enforcement (password_regex, password_regex_description) - now this is an interesting one. Currently Heat creates passwords for its temporary users with this function [1] falling back to empty password if a resource is not generating one for itself. Depending on regex_password setting in keystone, it may or may not be enough to pass the password strength check. I've looked around and (as expected) generating a random string which satisfies a pre-defined arbitrary regex is quite a non-trivial task, couple of existing Python libraries that can do this note that they support only a limited subset of full regex spec. So it seems that a most simple solution would be to add yet another user option to Keystone to ignore password strength enforcement for this given user, and amend Heat to set this option as well for internal users it creates. We in Heat may also think as to whether it would have any benefit to also set the 'lock_password' user option for the auto-created users which will prohibit such users to change their passwords via API themselves. I'd very like to hear opinion from Keystone community as most solutions I named are 'add new user option to Keystone' :-) [0] https://opendev.org/openstack/keystone/src/branch/master/keystone/conf/secu… [1] https://opendev.org/openstack/heat/src/branch/master/heat/common/password_g… Cheers, - Pavlo -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

4 4

[oslo] Proposing Hervé Beraud for Oslo Core
by Ben Nemec 02 May '19

02 May '19

Hi, Hervé has been working on Oslo for a while now and in that time has shown tremendous growth in his understanding of Oslo and OpenStack. I think he would make a good addition to the general Oslo core team. Existing Oslo team members (+Keystone, Castellan, and anyone else we co-own libraries with) please respond with +1/-1. If there are no objections I'll add him to the ACL next week and we can celebrate in person. :-) Thanks. -Ben

9 9

properly sizing openstack controlplane infrastructure
by Hartwig Hauschild 02 May '19

02 May '19

Hi everyone, A colleague and I have more or less recently been tasked with redesigning my employers openstack-infrastructure and it turned out that starting over will be easier than fixing the existing stack as we've managed to lock ourselves in a corner quite good. The requirements we've got are basically "here's 50 compute-nodes, make sure whatever you're building scales upwards from there". We have the existing pike-stack as a reference but we don't really know how the different services scale up with more compute-nodes to handle. The pike-stack has three servers as control-plane, each of them with 96G of RAM and they don't seem to have too much room left when coordinating 14 compute-nodes. We're thinking about splitting the control-nodes into infrastructure (db/rabbit/memcache) and API. What would I want to look for when sizing those control-nodes? I've not been able to find any references for this at all, just rather nebulous '8G RAM should do' which is around what our rabbit currently inhales. Also: We're currently running Neutron in OVS-DVR-VXLAN-Configuration. Does that properly scale up and above 50+ nodes or should we look into offloading the networking onto something else? I'm aware that the questions may sound a bit funny, but we have to spec out the controlplane-hardware before we can start testing and we'd prefer not to retrofit the servers because we goofed up. Thanks for your time, -- Cheers, Hardy

4 7