April 2019 - openstack-discuss - lists.openstack.org

Re: [openstack-dev] Are we ready to put stable/ocata into extended maintenance mode?
by Alex Schultz 03 Mar '21

03 Mar '21

On Tue, Sep 18, 2018 at 1:30 PM Alex Schultz <aschultz(a)redhat.com> wrote: > > On Tue, Sep 18, 2018 at 1:27 PM, Matt Riedemann <mriedemos(a)gmail.com> wrote: > > The release page says Ocata is planned to go into extended maintenance mode > > on Aug 27 [1]. There really isn't much to this except it means we don't do > > releases for Ocata anymore [2]. There is a caveat that project teams that do > > not wish to maintain stable/ocata after this point can immediately end of > > life the branch for their project [3]. We can still run CI using tags, e.g. > > if keystone goes ocata-eol, devstack on stable/ocata can still continue to > > install from stable/ocata for nova and the ocata-eol tag for keystone. > > Having said that, if there is no undue burden on the project team keeping > > the lights on for stable/ocata, I would recommend not tagging the > > stable/ocata branch end of life at this point. > > > > So, questions that need answering are: > > > > 1. Should we cut a final release for projects with stable/ocata branches > > before going into extended maintenance mode? I tend to think "yes" to flush > > the queue of backports. In fact, [3] doesn't mention it, but the resolution > > said we'd tag the branch [4] to indicate it has entered the EM phase. > > > > 2. Are there any projects that would want to skip EM and go directly to EOL > > (yes this feels like a Monopoly question)? > > > > I believe TripleO would like to EOL instead of EM for Ocata as > indicated by the thead > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.ht… > Bringing this backup to see what we need to do to get the stable/ocata branches ended for the TripleO projects. I'm bringing this up because we have https://review.openstack.org/#/c/647009/ which is for the upcoming rename but CI is broken and we have no interest in continue to keep the stable/ocata branches alive (or fix ci for them). Thanks, -Alex > Thanks, > -Alex > > > [1] https://releases.openstack.org/ > > [2] > > https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… > > [3] > > https://docs.openstack.org/project-team-guide/stable-branches.html#extended… > > [4] > > https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… > > > > -- > > > > Thanks, > > > > Matt > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 11

[openstack-ansible] dropping suse support
by Mohammed Naser 06 May '20

06 May '20

Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct… [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct…

6 9

[nova] critical bug around reload/upgrades
by Mohammed Naser 24 Aug '19

24 Aug '19

Hello: I've discussed this for quite sometime with Dan over IRC and a bit with Zane as well, but basically, Nova thinks that when it gets a reload (aka SIGHUP), nothing else has occurred. However, oslo.service actually calls stop(), reload() then start() again, which potentially kills all RPC. This has caused a pretty big issue in our gates and it also means that the whole idea behind 'reload nova-compute while upgrading to refresh info' concept is fundamentally broken. I tried to do some work on this here, however, I wasn't really able to get to the bottom of it. There seems to be a decision that needs to be taken in terms of .. do we change what reload() actually means in oslo_service (it actually is more like a restart, not a reload) or does Nova (and other projects) change their implementation in assuming what reload() does? https://review.openstack.org/#/c/641907/ This seems to have been floating around for a really long time, so I'd be happy to work with someone to find the fix (and we can totally test it inside OpenStack Ansible by reloading instead of restarting). Thanks! Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

4 4

[stackalytics] Reported numbers seem inaccurate
by Mark Goddard 27 Jun '19

27 Jun '19

Hi, I've heard a couple of people say they don't feel the numbers reported by Stackalytics accurately reflect reality. I've been trying to gather a few stats for my Kolla update session in Denver, and am finding the same. I'll try to give some concrete examples. Reviews for all kolla deliverables in Stein [1]. Here the company stats don't reflect the individual stats. Also, the total reviews in the 'Kolla Official' module does not equal the sum of the reviews of its submodules (kolla, kolla-ansible, kolla-cli). If I look at the contribution summary for Kolla Official in the last 90 days [2], they are actually greater than those for the last 180 days [3]! There are also similar issues with commit metrics, and none seem to match what I see in git. Thanks, Mark [1] https://www.stackalytics.com/?metric=marks&module=kolla-group [2] https://www.stackalytics.com/report/contribution/kolla-group/90 [3] https://www.stackalytics.com/report/contribution/kolla-group/180

10 11

Re: [nova][placement] Need for a nova-manage placement audit command
by Matt Riedemann 19 Jun '19

19 Jun '19

Re-posting this since it came up again in IRC today. On 9/20/2018 6:09 PM, Matt Riedemann wrote: > mnaser wrote a simple placement audit tool today and dumped his script > into a bug report [1]. It turns out several operators have already > written a variant of this same tool. The idea is to compare what's in > placement for allocations and (compute node) resource providers against > what's in nova for instances and compute nodes, look for orphans in > placement and then report them, or maybe also clean them up. > > I think this could go into a "nova-manage placement audit" command and > should be pretty easy to write for minimum functionality (start with > just the report). > > I'm advertising the need here in case someone wants to work on this. I'd > like to myself, but just can't justify the time right now. > > [1] https://bugs.launchpad.net/nova/+bug/1793569 > -- Thanks, Matt

2 1

[kolla][magnum] Cluster creation failed due to "Waiting for Kubernetes API..."
by Giuseppe Sannino 30 May '19

30 May '19

Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled --------------- VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out" I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5 Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE) Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver. [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1. May I ask for an help on this ? Many thanks /Giuseppe

5 11

Re: [stackalytics] Reported numbers seem inaccurate
by Edward Ionel 25 May '19

25 May '19

Hello All, I first want to thank each and everyone of you for reaching out and sharing the inaccuracies you all have found within Stackalytics, this has helped us tremendously with the efforts to correct these flaws. I am writing this email to directly address each individual that has reached out and shared their concerns with our Stackalytics data. Mark Goddard Michael Johnson Julia Kreger Collen Murphy Clark Boylan Sean Mooney Thierry Carrez Jermey Stanley Arkady Kanvesky (Hopefully, I did not miss anyone) I want to insure everyone that the team is working hard to fix the inaccuracies in some of our data. We are still very much involved in OpenStack and we do read each and every message regarding Stackalytics. As Boris and Sergey mentioned, we found that this occurred due to an outdated production environment. We are currently in the progress of migrating to a new environment, and the data should be corrected. The team and I are sorry for these discrepancies and appreciate you all for bringing this to our attention. Thank you all, for providing links and examples of the inaccuracies. This has helped our efforts greatly in correcting the current issues. It would be great if the community can continue reviewing Stackalytics data and help us provide a better service to all. Once again, we are all very sorry for this and will continue to fix these issues. Cheers, Edward > On Apr 16, 2019, at 4:05 PM, Edward Ionel <eionel(a)mirantis.com> wrote: > > > >> On Tue, Apr 16, 2019 at 2:56 PM Boris Renski <brenski(a)mirantis.com> wrote: >> Folks - more specifically, we plan to roll out an update by tomorrow mid-day PT which should fix all the numbers + add aggregate details on Airship project contributions. >> >> Would really appreciate the help of the community to poke around tomorrow afternoon and see if there are still discrepancies. >> >> Sorry about this and thank you for catching it and brining it to our attention. >> >> -Boris >> >>> On Tue, Apr 16, 2019 at 1:33 PM Sergey Nikitin <snikitin(a)mirantis.com> wrote: >>> Hello everyone, >>> As you mentioned there are some inaccuracies in stats on stackalytics.com. It happened because of outdated Prod env. At this moment we are in a process of migration to the new env. All data on this env is fresh and accurate. So the problem will be solved soon. >>> >>> -- >>> Best Regards, >>> Sergey Nikitin > > > -- > Edward Ionel > Marketing Operations Manager > Mirantis, Inc. > 900 E Hamilton Ave, Suite 650 > Campbell, CA 95008 > Eionel(a)mirantis.com > www.mirantis.com

4 14

[heat][neutron] improving extraroute support
by Bence Romsics 24 May '19

24 May '19

Hi Heat and Neutron folks, I'm wondering how could I improve support for extra routes for one of our users. In this email I'd like to get your input (from both Heat and Neutron folks) in which direction should I start before I sit down to write the relevant blueprints and RFEs. This will be a bit long, please bear with me. As you know a neutron router can have extra static routing table entries as the 'routes' attribute like this (please see the api-ref in [1]): $ openstack router show router1 -f value -c routes destination='10.1.0.0/24', gateway='10.0.0.11' destination='10.2.0.0/24', gateway='10.0.0.12' ... To my best knowledge the whole set of extra routes must be updated at once which easily leads to lost updates if multiple clients modify this attribute concurrently (imagine client A and B working concurrently, for example both reads routes [r1], client A writes [r1, r2], client B writes [r1, r3]; depending on timing either r2 or r3 will be lost). Heat has an unsupported resource OS::Neutron::ExtraRoute [2] to front Neutron extra routes. For example: heat_template_version: '2015-04-30' resources: Extra_route_1: type: OS::Neutron::ExtraRoute properties: destination: '10.1.0.0/24' nexthop: '10.0.0.11' router_id: 'd544b5c5-d4d2-4b52-aaa8-ef4b9fa3e154' The Heat resource represents a single extra route out of the many in the router's routes attribute. If you have multiple OS::Neutron::ExtraRoute resources Heat by default handles them concurrently leading to the lost update problem. You can easily reproduce the problem just by having 10 extra routes in a template. The stack creation will succeed, but the router in fact will have only 6-7 (the exact number changes based on timing) extra routes. AFAIU this is exactly why this was marked unsupported. (Please let me know if you know other reasons.) As a workaround I suggested our user a hack to serialize the OS::Neutron::ExtraRoute resources by 'depends_on'. But which way should I go if I want to fix this properly? way #1 - changes in Heat only: Introduce a new Heat resource OS::Neutron::ExtraRoutes (please notice the plural) which can represent the whole routes attribute of a router (i.e. multiple extra routes in a single resource). This will not solve the design shortcomings of the neutron extraroute API, but it can at least express what can be done in the Neutron API as is. Lost updates of multiple stacks editing the same router's routes are still possible. way #2 - changes in Heat and Neutron: Introduce a new Neutron extension with a better API (either extraroutes as a top level resource, each object with a router_id or an action map a'la add/remove_router_interface to edit extra routes on the server side). This new API would work on the same DB tables and backends as the old. The old API would be kept for backwards compatibility. But if a new user exclusively uses the new API it can work concurrently. Then introduce a new Heat resource (or change the existing unsupported one) to use this new Neutron API. This is obviously more work than the first option. Which one do you prefer and why? Or let me know please if I missed a better alternative. In this thread I hope to get a cross-project agreement of this between Heat and Neutron so I can go ahead and open the relevant Heat and Neutron blueprints where we can focus only on each projects' internal questions. Thanks in advance, Bence Romsics irc: rubasov [1] https://developer.openstack.org/api-ref/network/v2/?expanded=update-router-… [2] https://docs.openstack.org/heat/latest/template_guide/unsupported.html#OS::…

4 11

[all][docs] season of docs
by Mohammed Naser 24 May '19

24 May '19

Hi there: It seems like Google has come up with a new somewhat-GSoC-like idea but focused on documentation. I think it could be a good opportunity for the documentation team (or any specific team actually, coordinated with docs) to be part of this. https://opensource.googleblog.com/2019/03/introducing-season-of-docs.html I'm not sure if the team has the amount of resources, but it seems they should be able to apply to this. Does this seem like something that might help the team more (or perhaps a specific project, coordinating with the docs team) to apply for this? Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

7 10

[glance][interop] standardized image "name" ?
by Mohammed Naser 24 May '19

24 May '19

Hi everyone, Something that I've been dealing with is the fact that regardless of what cloud you're using, the naming structure is very much different. While we aim to be interoperable, if I want to "boot an Debian Stretch server", the image name will probably be different all over the places. Is there some sort of way we can eliminate this by implementing a standard naming convention which would allow us to use the same images across all systems? I'm just opening the idea for discussion, I'd love to hear what others think about this. Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

15 22