September 2019 - openstack-discuss

Re: [openstack-dev] Are we ready to put stable/ocata into extended maintenance mode?
by Alex Schultz 03 Mar '21

03 Mar '21

On Tue, Sep 18, 2018 at 1:30 PM Alex Schultz <aschultz(a)redhat.com> wrote: > > On Tue, Sep 18, 2018 at 1:27 PM, Matt Riedemann <mriedemos(a)gmail.com> wrote: > > The release page says Ocata is planned to go into extended maintenance mode > > on Aug 27 [1]. There really isn't much to this except it means we don't do > > releases for Ocata anymore [2]. There is a caveat that project teams that do > > not wish to maintain stable/ocata after this point can immediately end of > > life the branch for their project [3]. We can still run CI using tags, e.g. > > if keystone goes ocata-eol, devstack on stable/ocata can still continue to > > install from stable/ocata for nova and the ocata-eol tag for keystone. > > Having said that, if there is no undue burden on the project team keeping > > the lights on for stable/ocata, I would recommend not tagging the > > stable/ocata branch end of life at this point. > > > > So, questions that need answering are: > > > > 1. Should we cut a final release for projects with stable/ocata branches > > before going into extended maintenance mode? I tend to think "yes" to flush > > the queue of backports. In fact, [3] doesn't mention it, but the resolution > > said we'd tag the branch [4] to indicate it has entered the EM phase. > > > > 2. Are there any projects that would want to skip EM and go directly to EOL > > (yes this feels like a Monopoly question)? > > > > I believe TripleO would like to EOL instead of EM for Ocata as > indicated by the thead > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.ht… > Bringing this backup to see what we need to do to get the stable/ocata branches ended for the TripleO projects. I'm bringing this up because we have https://review.openstack.org/#/c/647009/ which is for the upcoming rename but CI is broken and we have no interest in continue to keep the stable/ocata branches alive (or fix ci for them). Thanks, -Alex > Thanks, > -Alex > > > [1] https://releases.openstack.org/ > > [2] > > https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… > > [3] > > https://docs.openstack.org/project-team-guide/stable-branches.html#extended… > > [4] > > https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… > > > > -- > > > > Thanks, > > > > Matt > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 11

[Syntribos] Does anyone still use this?
by Gage Hugo 08 May '20

08 May '20

The Security SIG has recently been looking into updating several sites/documentation and one part we discussed the last couple meetings was the security tools listings[0]. One of the projects listed there, Syntribos, doesn't appear to have much activity[1] outside of zuul/python/docs maintenance and the idea of retiring the project was mentioned. However, there does seem to be an occasional bug-fix submitted, so we are sending this email out to see if anyone is still utilizing Syntribos. If so, please either respond here, reach out to us in the #openstack-security irc channel, or fill out the section in the Security SIG Agenda etherpad[2]. Thanks! [0] https://security.openstack.org/#security-tool-development [1] https://review.opendev.org/#/q/project:openstack/syntribos [2] https://etherpad.openstack.org/p/security-agenda

2 2

[openstack-ansible] dropping suse support
by Mohammed Naser 05 May '20

05 May '20

Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct… [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct…

6 9

[cinder] consistency group not working
by Francois Scheurer 16 Jan '20

16 Jan '20

Dear Cinder Experts We are running the rocky release. |We can create a consistency group: openstack consistency group create --volume-type b9f67298-cf68-4cb2-bed2-c806c5f83487 fsc-consgroup Bug 1: but adding volumes is not working: openstack consistency group add volume c3f49ef0-601e-4558-a75a-9b758304ce3b b48752e3-641f-4a49-a892-6cb54ab6b74d c0022411-59a4-4c7c-9474-c7ea8ccc7691 0f4c6493-dbe2-4f75-8e37-5541a267e3f2 => Invalid volume: Volume is not local to this node. (HTTP 400) (Request-ID: req-7f67934a-5835-40ef-b25c-12591fd79f85) Bug 2: deleting consistency group is also not working (silently failing): openstack consistency group delete c3f49ef0-601e-4558-a75a-9b758304ce3b |||=> AttributeError: 'RBDDriver' object has no attribute 'delete_consistencygroup'| See details below. Using the --force option makes no difference and the consistency group is not deleted. Do you think this is a bug or a configuration issue? Thank you in advance. | Cheers Francois |Details: ==> /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-api-access.log <== 10.0.129.17 - - [27/Sep/2019:12:16:24 +0200] "POST /v3/f099965b37ac41489e9cac8c9d208711/consistencygroups/3706bbab-e2df-4507-9168-08ef811e452c/delete HTTP/1.1" 202 - 109720 "-" "python-cinderclient" ==> /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-volume.log <== 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server [req-9010336e-d569-47ad-84e2-8dd8b729939c b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Exception during message handling: AttributeError: 'RBDDriver' object has no attribute 'delete_consistencygroup' 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 3397, in delete_group 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server vol_obj.save() 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 3362, in delete_group 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server self.driver.delete_consistencygroup(context, cg, 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server AttributeError: 'RBDDriver' object has no attribute 'delete_consistencygroup' 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server| -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

2 2

[ospurge] looking for project owners / considering adoption
by Adam Harwell 04 Nov '19

04 Nov '19

I haven't seen much activity on this project in a while, and it's been moved to opendev/x since the opendev migration... Who is the current owner of this project? Is there anyone who actually is maintaining it, or would mind if others wanted to adopt the project to move it forward? Thanks, --Adam Harwell

5 10

[nova]review guide for the policy default refresh spec
by Ghanshyam Mann 30 Oct '19

30 Oct '19

Hello Everyone, As many of you might know that in Train, we are doing Nova policy changes to adopt the keystone's new defaults and scope type[1]. There are multiple changes required per policy as mentioned in spec. I am writing this review guide for the patch sequence and at the end how each policy will look like. I have prepared the first set of patches. I would like to get the feedback on those so that we can modify the other policy also on the same line. My plan is to start the other policy work after we merge the first set of policy changes. Patch sequence: Example: os-services API policy: ------------------------------------------------------------- 1. Cover/Improve the test coverage for existing policies: This will be the first patch. We do not have good test coverage of the policy, current tests are not at all useful and do not perform the real checks. Idea is to add the actual tests coverage for each policy as the first patch. new tests try to access the API with all possible context and check for positive and negative cases. - https://review.opendev.org/#/c/669181/ 2. Introduce scope_types: This will add the scope_type for policy. It will be either 'system', 'project' or 'system and project'. In the same patch, along with existing test working as it is, new tests of scope type will be added which will run with [oslo_policy] enforce_scope=True so that we can capture the real scope checks. - https://review.opendev.org/#/c/645427/ 3. Add new default roles: This will add new defaults which can be SYSTEM_ADMIN, SYSTEM_READER, PROJECT_MEMBER_OR_SYSTEM_ADMIN, PROJECT_READER_OR_SYSTEM_READER etc depends on Policy. Test coverage of new defaults, as well as deprecated defaults, are covered in same patch. This patch will add the granularity in policy if needed. Without policy granularity, we cannot add new defaults per rule. - https://review.opendev.org/#/c/648480/ (I need to add more tests for deprecated rules) 4. Pass actual Targets in policy: This is to pass the actual targets in context.can(). Main goal is to remove the defaults targets which is nothing but context'user_id,project_id only. It will be {} if no actual target data needed in check_str. - https://review.opendev.org/#/c/676688/ Patch sequence: Example: Admin Action API policy: 1. https://review.opendev.org/#/c/657698/ 2. https://review.opendev.org/#/c/657823/ 3. https://review.opendev.org/#/c/676682/ 4. https://review.opendev.org/#/c/663095/ There are other patches I have posted in between for common changes or fix or framework etc. [1] https://specs.openstack.org/openstack/nova-specs/specs/train/approved/polic… -gmann

1 2

[cinder][tooz]Lock-files are remained
by Rikimaru Honjo 19 Oct '19

19 Oct '19

Hi, I'm using Queens cinder with the following setting. --------------------------------- [coordination] backend_url = file://$state_path --------------------------------- As a result, the files like the following were remained under the state path after some operations.[1] cinder-63dacb3d-bd4d-42bb-88fe-6e4180164765-delete_volume cinder-32c426af-82b4-41de-b637-7d76fed69e83-delete_snapshot In my understanding, these are lock-files created for synchronization by tooz. But, these lock-files were not deleted after finishing operations. Is this behaviour correct? [1] e.g. Delete volume, Delete snapshot -- _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/ Rikimaru Honjo E-mail:honjo.rikimaru@ntt-tx.co.jp

7 15

[metrics] [telemetry] [stein] cpu_util
by Budai Laszlo 18 Oct '19

18 Oct '19

Hello all, the release release announce of ceilometer rocky is deprecating the cpu_util and *.rate metrics "* cpu_util and *.rate meters are deprecated and will be removed in future release in favor of the Gnocchi rate calculation equivalent." so we don't have them in Stein. Can you direct me to some document that describes how to achieve these with Gnocchi rate calculation? Thank you, Laszlo

5 4

[aodh] [heat] Stein: How to create alarms based on rate metrics like CPU utilization?
by Bernd Bausch 18 Oct '19

18 Oct '19

Prior to Stein, Ceilometer issued a metric named /cpu_util/, which I could use to trigger alarms and autoscaling when CPU utilization was too high. cpu_util doesn't exist anymore. Instead, we are asked to use Gnocchi's /rate/ feature. However, when using rates, alarms on a group of resources require more parameters than just one metric: Both an aggregation and a reaggregation method are needed. For example, a group of instances that implement "myapp": gnocchi measures aggregation -m cpu --reaggregation mean --aggregation rate:mean --query server_group=myapp --resource-type instance Actually, this command uses a deprecated API (but from what I can see, Aodh still uses it). The new way is like this: gnocchi aggregates --resource-type instance '(aggregate rate:mean (metric cpu mean))' server_group=myapp If rate:mean is in the archive policy, it also works the other way around: gnocchi aggregates --resource-type instance '(aggregate mean (metric cpu rate:mean))' server_group=myapp Without reaggregation, I get quite unexpected numbers, including negative CPU rates. If you want to understand why, see this discussion with one of the Gnocchi maintainers [1]. *My problem*: Aodh allows me to set an aggregation method, but not a reaggregation method. How can I create alarms based on rates? The problem extends to Heat and autoscaling. Thanks much, Bernd. [1] https://github.com/gnocchixyz/gnocchi/issues/1044

4 3

[nova][ptg] Ussuri scope containment
by Eric Fried 16 Oct '19

16 Oct '19

Nova developers and maintainers- Every cycle we approve some number of blueprints and then complete a low percentage [1] of them. Which blueprints go unfinished seems to be completely random (notably, it appears to have nothing to do with our declared cycle priorities). This is especially frustrating for consumers of a feature, who (understandably) interpret blueprint/spec approval as a signal that they can reasonably expect the feature to land [2]. The cause for non-completion usually seems to fall into one of several broad categories: == Inadequate *developer* attention == - There's not much to be done about the subset of these where the contributor actually walks away. - The real problem is where the developer thinks they're ready for reviewers to look, but reviewers don't. Even things that seem obvious to experienced reviewers, like failing CI or "WIP" in the commit title, will cause patches to be completely ignored -- but unseasoned contributors don't necessarily understand even that, let alone more subtle issues. Consequently, patches will languish, with each side expecting the other to take the next action. This is a problem of culture: contributors don't understand nova reviewer procedures and psychology. == Inadequate *reviewer* attention == - Upstream maintainer time is limited. - We always seem to have low review activity until the last two or three weeks before feature freeze, when there's a frantic uptick and lots gets done. - But there's a cultural rift here as well. Getting maintainers to care about a blueprint is hard if they don't already have a stake in it. The "squeaky wheel" concept is not well understood by unseasoned contributors. The best way to get reviews is to lurk in IRC and beg. Aside from not being intuitive, this can also be difficult logistically (time zone pain, knowing which nicks to ping and how) as well as interpersonally (how much begging is enough? too much? when is it appropriate?). == Multi-release efforts that we knew were going to be multi-release == These may often drag on far longer than they perhaps should, but I'm not going to try to address that here. ======== There's nothing new or surprising about the above. We've tried to address these issues in various ways in the past, with varying degrees of effectiveness. I'd like to try a couple more. (A) Constrain scope, drastically. We marked 25 blueprints complete in Train [3]. Since there has been no change to the core team, let's limit Ussuri to 25 blueprints [4]. If this turns out to be too few, what's the worst thing that happens? We finish everything, early, and wish we had done more. If that happens, drinks are on me, and we can bump the number for V. (B) Require a core to commit to "caring about" a spec before we approve it. The point of this "core liaison" is to act as a mentor to mitigate the cultural issues noted above [5], and to be a first point of contact for reviews. I've proposed this to the spec template here [6]. Thoughts? efried [1] Like in the neighborhood of 60%. This is anecdotal; I'm not aware of a good way to go back and mine actual data. [2] Stuff happens, sure, and nobody expects 100%, but 60%? Come on, we have to be able to do better than that. [3] https://blueprints.launchpad.net/nova/train [4] Recognizing of course that not all blueprints are created equal, this is more an attempt at a reasonable heuristic than an actual expectation of total size/LOC/person-hours/etc. The theory being that constraining to an actual number, whatever the number may be, is better than not constraining at all. [5] If you're a core, you can be your own liaison, because presumably you don't need further cultural indoctrination or help begging for reviews. [6] https://review.opendev.org/685857

11 21

[tripleo] Deprecating paunch CLI?
by Emilien Macchi 16 Oct '19

16 Oct '19

With our long-term goal to simplify TripleO and focus on what people actually deploy and how they operate their clouds, it appears that the Paunch CLI hasn't been a critical piece in our project and I propose that we deprecate it; create an Ansible module to call Paunch as a library only. I've been playing with it a little today: https://review.opendev.org/#/c/682093/ https://review.opendev.org/#/c/682094/ Here is how you would call paunch: - name: Start containers for step {{ step }} paunch: config: "/var/lib/tripleo-config/hashed-container-startup-config-step_{{ step }}.json" config_id: "tripleo_step{{ step }}" action: apply container_cli: "{{ container_cli }}" managed_by: "tripleo-{{ tripleo_role_name }}" A few benefits: - Deployment tasks in THT would call the new module instead of a shell command - More Pythonic and clean for Ansible, to interact with the actual task during the run - Removing some code in Paunch, make it easier to maintain for us For now, the Ansible module only covers "paunch apply", we will probably cover "delete" and "cleanup" eventually. Please let me know if you have any questions or concerns, -- Emilien Macchi

9 18

[oslo] New courtesy ping list for Ussuri
by Ben Nemec 14 Oct '19

14 Oct '19

As we discussed at the beginning of the cycle, I'll be clearing the current ping list in the next few weeks. This is to prevent courtesy pinging people who are no longer active on the project. If you wish to continue receiving courtesy pings at the start of the Oslo meeting please add yourself to the new list on the agenda template [0]. Note that the new list is above the template, called "Courtesy ping list for Ussuri". If you add yourself again to the end of the existing list I'll assume you want to be left on though. :-) Thanks. -Ben 0: https://wiki.openstack.org/wiki/Meetings/Oslo#Agenda_Template

1 1

Port creation times out for some VMs in large group
by Albert Braden 12 Oct '19

12 Oct '19

When I create 100 VMs in our prod cluster: openstack server create --flavor s1.tiny --network it-network --image cirros-0.4.0-x86_64 --min 100 --max 100 alberttest Most of them build successfully in about a minute. 5 or 10 will stay in BUILD status for 5 minutes and then fail with " BuildAbortException: Build of instance <UUID> aborted: Failed to allocate the network(s), not rescheduling." If I build smaller numbers, I see less failures, and no failures if I build one at a time. This does not happen in dev or QA; it appears that we are exhausting a resource in prod. I tried reducing various config values in dev but am not able to duplicate the issue. The neutron servers don't appear to be overloaded during the failure. What config variables should I be looking at? Here are the relevant log entries from the HV: 2019-09-26 10:10:43.001 57008 INFO os_vif [req-dea54d9a-3f3e-4d47-b901-a4f41b1947a8 d28c3871f61e4c8c8f8c7600417f7b14 e9621e3b105245ba8660f434ab21016c - default 4fb72165eee4468e8033cdc7d506ddf0] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:8b:45:07,bridge_name='brq49cbe55d-51',has_traffic_filtering=True,id=18f4e419-b19c-4b62-b6e4-152ec78e72bc,network=Network(49cbe55d-5188-4183-b5ad-e65f9b46f8f2),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap18f4e419-b1') 2019-09-26 10:15:44.029 57008 WARNING nova.virt.libvirt.driver [req-dea54d9a-3f3e-4d47-b901-a4f41b1947a8 d28c3871f61e4c8c8f8c7600417f7b14 e9621e3b105245ba8660f434ab21016c - default 4fb72165eee4468e8033cdc7d506ddf0] [instance: dc58f154-00f9-4c45-8986-94b10821cbc9] Timeout waiting for [('network-vif-plugged', u'18f4e419-b19c-4b62-b6e4-152ec78e72bc')] for instance with vm_state building and task_state spawning.: Timeout: 300 seconds More logs and data: http://paste.openstack.org/show/779524/

4 17

Release Cycle Observations
by Donny Davis 09 Oct '19

09 Oct '19

As many of you know I am fairly new to public participation in Openstack, but I am a long time lurker and operator. This is my first cycle and I wanted to capture an observation on the release cycle of Openstack. *this is only an observation for discussion* Over the last week or so I have watch people just about kill themselves to get features into Openstack for this release. Everyone in this community appears to be giving it their all. And way to go everyone for pushing so hard. Seriously! So this brings me to my observation. Can the release cycle be modified to ease this push at the end? I think what we all want is a smooth and steady stream of features and bug fixes, but a 6 month release cycle seems to do directly the opposite. Also if everyone has gate issues at the end of the cycle, is it the gate or the cycle? The gate seemed to work fine early on... Maybe if we went to a Major / Minor release sort of system we could remove some burden to *get that feature into X release* and we could steady out that flow of features / bug fixes. Maybe something like a 9 month major, one month minor release cycle. It would also probably help with micro api versioning. I am sure the more experienced developers here can highlight pros / cons, etc of all of the release types. Just a thought. Tell me yours. ~/DonnyD

10 19

[all][PTG] Strawman Schedule
by Kendall Nelson 09 Oct '19

09 Oct '19

Hello Everyone! In the attached picture or link [0] you will find the proposed schedule for the various tracks at the Shanghai PTG in November. We did our best to avoid the key conflicts that the track leads (PTLs, SIG leads...) mentioned in their PTG survey responses, although there was no perfect solution that would avoid all conflicts especially when the event is three-ish days long and we have over 40 teams meeting. If there are critical conflicts we missed or other issues, please let us know, by October 6th at 7:00 UTC! -Kendall (diablo_rojo) [0] https://usercontent.irccloud-cdn.com/file/00mZ3Q3M/pvg_ptg_schedule.png

9 20

[dev][taskflow] Accepting any decision
by Jiří Rája 08 Oct '19

08 Oct '19

Hi, I wrote the code in the attachment and I would like to ask if it's possible to execute next task (step3) even if one decider returns True (link from step 1 to step 3) and one returns False (link from step 2 to step 3). If it is possible could someone alter the code? Or is there any other way to do it? And if the task wouldn't have to wait for all of the links, it would be great. Thank you! All the best, Rája

2 1

[oslo][nova] Revert of oslo.messaging JSON serialization change
by Ben Nemec 04 Oct '19

04 Oct '19

Hi, I've just proposed https://review.opendev.org/#/c/685724/ which reverts a change that recently went in to make the fake driver in oslo.messaging use jsonutils for message serialization instead of json.dumps. As explained in the commit message on the revert, this is problematic because the rabbit driver uses kombu's default serialization method, which is json.dumps. By changing the fake driver to use jsonutils we've made it more lenient than the most used real driver which opens us up to merging broken changes in consumers of oslo.messaging. We did have some discussion of whether we should try to override the kombu default and tell it to use jsonutils too, as a number of other drivers do. The concern with this was that the jsonutils handler for things like datetime objects is not tz-aware, which means if you send a datetime object over RPC and don't explicitly handle it you could lose important information. I'm open to being persuaded otherwise, but at the moment I'm leaning toward less magic happening at the RPC layer and requiring projects to explicitly handle types that aren't serializable by the standard library json module. If you have a different preference, please share it here. Thanks. -Ben

4 5

[nova][kolla] questions on cells
by Mark Goddard 03 Oct '19

03 Oct '19

Hi, I'm currently working (with dougsz) on adding support for nova cells v2 in Kolla Ansible. I have a few questions, and things that I'd like confirming. 1. Is there any benefit to not having a superconductor? Presumably it's a little more efficient in the single cell case? Also IIUC it only requires a single message queue so is a little simpler? 2. Do console proxies need to live in the cells? This is what devstack does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct? 3. Should I upgrade the superconductor or conductor service first? 4. Does the cell conductor need access to the API DB? 5. What DB configuration should be used in nova.conf when running online data migrations? I can see some migrations that seem to need the API DB, and others that need a cell DB. If I just give it the API DB, will it use the cell mappings to get to each cell DB, or do I need to run it once for each cell? 6. After an upgrade, when can we restart services to unpin the compute RPC version? Looking at the compute RPC API, it looks like the super conductor will remain pinned until all computes have been upgraded. For a cell conductor, it looks like I could restart it to unpin after upgrading all computes in that cell, correct? 7. Which services require policy.{yml,json}? I can see policy referenced in API, conductor and compute. I'm trying to design it such that it's possible to deploy a new cell in an existing environment, and upgrade one cell at a time. Getting the ordering right for this is fiddly, and I expect I'll be back to get some eyes on my plan. Cheers, Mark

6 16

[i18n] Request to be added as Vietnamese translation group coordinators
by Trinh Nguyen 03 Oct '19

03 Oct '19

Hi i18n team, Dai and I would like to volunteer as the coordinators of the Vietnamese translation group. If you find us qualified, please let us know. Bests, -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

5 7

[all] [tc] [docs] [release] [ptls] Docs as SIG: Ownership of docs.openstack.org
by Alexandra Settle 03 Oct '19

03 Oct '19

Hi all, Quick recap: The documentation team is set to disband as an official project, and make the leap to becoming a SIG (Special Interest Group). This decision is not one we have made lightly, but with the changes of direction since Pike (project documentation belonging with the projects, etc) the need for a centralised documentation team as an official project is no longer integral to producing the software. The transition of the docs team to a SIG has already begun [1] and [2]. The openstackdocstheme, openstack-doc-tools, os-api-ref and whereto have been placed within the remit of the Oslo team and approved. The remaining individuals working on the docs team are sorting through what's left and where it is best placed. In [1], Doug has rightfully pointed out that whilst the documentation team as it stands today is no longer "integral", the docs.openstack.org web site is. An owner is required. The suggestion is for the Release Management team is the "least worst" (thanks, tonyb) place for the website manaagement to land. As Tony points out, this requires learning new tools and processes but the individuals working on the docs team currently have no intention to leave, and are around to help manage this from the SIG. Open to discussion and suggestions, but to summarise the proposal here: docs.openstack.org ownership is to transition to be the responsibility of the Release Management team officially provided there are no strong objections. Thanks, Alex IRC: asettle [1] https://review.opendev.org/#/c/657142/ [2] https://review.opendev.org/#/c/657141/

5 11

[kolla-ansible] migration
by Ignazio Cassano 02 Oct '19

02 Oct '19

Hello, Anyone have tried to migrate an existing openstack installation to kolla containers? Thanks Ignazio

5 16

issues creating a second vm with numa affinity
by Manuel Sopena Ballesteros 02 Oct '19

02 Oct '19

Dear Openstack user community, I have a compute node with 2 numa nodes and I would like to create 2 vms, each one using a different numa node through numa affinity with cpu, memory and nvme pci devices. pci passthrough whitelist [root@zeus-53 ~]# tail /etc/kolla/nova-compute/nova.conf [notifications] [filter_scheduler] enabled_filters = enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter available_filters = nova.scheduler.filters.all_filters [pci] passthrough_whitelist = [ {"address":"0000:06:00.0"}, {"address":"0000:07:00.0"}, {"address":"0000:08:00.0"}, {"address":"0000:09:00.0"}, {"address":"0000:84:00.0"}, {"address":"0000:85:00.0"}, {"address":"0000:86:00.0"}, {"address":"0000:87:00.0"} ] alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"} Openstack flavor openstack flavor create --public xlarge.numa.perf.test --ram 200000 --disk 700 --vcpus 20 --property hw:cpu_policy=dedicated --property hw:emulator_threads_policy=isolate --property hw:numa_nodes='1' --property pci_passthrough:alias='nvme:4' The first vm is successfully created openstack server create --network hpc --flavor xlarge.numa.perf.test --image centos7.6-image --availability-zone nova:zeus-53.localdomain --key-name mykey kudu-1 However the second vm fails openstack server create --network hpc --flavor xlarge.numa.perf --image centos7.6-kudu-image --availability-zone nova:zeus-53.localdomain --key-name mykey kudu-4 Errors in nova compute node 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [req-b5a25c73-8c7d-466c-8128-71f29e7ae8aa 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] Instance failed to spawn: libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-09-27T06:45:19.118089Z qemu-kvm: kvm_init_vcpu failed: Cannot allocate memory 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] Traceback (most recent call last): 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2369, in _build_resources 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] yield resources 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2133, in _build_and_run_instance 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] block_device_info=block_device_info) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3142, in spawn 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] destroy_disks_on_failure=True) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5705, in _create_domain_and_network 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] destroy_disks_on_failure) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] self.force_reraise() 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] six.reraise(self.type_, self.value, self.tb) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5674, in _create_domain_and_network 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] post_xml_callback=post_xml_callback) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5608, in _create_domain 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] guest.launch(pause=pause) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] self._encoded_xml, errors='ignore') 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] self.force_reraise() 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] six.reraise(self.type_, self.value, self.tb) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] return self._domain.createWithFlags(flags) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] result = proxy_call(self._autowrap, f, *args, **kwargs) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] rv = execute(f, *args, **kwargs) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] six.reraise(c, e, tb) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] rv = meth(*args, **kwargs) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-09-27T06:45:19.118089Z qemu-kvm: kvm_init_vcpu failed: Cannot allocate memory 2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] Numa cell/node 1 (the one assigned on kudu-4) has enough cpu, memory, pci devices and disk capacity to fit this vm. NOTE: below is the information relevant I could think of that shows resources available after creating the second vm. [root@zeus-53 ~]# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 28 29 30 31 32 33 34 35 36 37 38 39 40 41 node 0 size: 262029 MB node 0 free: 52787 MB node 1 cpus: 14 15 16 17 18 19 20 21 22 23 24 25 26 27 42 43 44 45 46 47 48 49 50 51 52 53 54 55 node 1 size: 262144 MB node 1 free: 250624 MB node distances: node 0 1 0: 10 21 1: 21 10 NOTE: this is to show that numa node/cell 1 has enough resources available (also nova-compute logs shows that kudu-4 is assigned to cell 1) [root@zeus-53 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/md127 3.7T 9.1G 3.7T 1% / ... NOTE: vm disk files goes to root (/) partition [root@zeus-53 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 59.6G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 16G 0 part [SWAP] loop0 7:0 0 100G 0 loop └─docker-9:127-6979358884-pool 253:0 0 100G 0 dm ├─docker-9:127-6979358884-4301cee8d0433729cd6332ca2b6111afc85f14c48d4ce2d888a1da0ef9b5ca01 253:1 0 10G 0 dm ├─docker-9:127-6979358884-d59208adcb7cee3418f810f24e6c3a55d39281f713c8e76141fc61a8deba8a2b 253:2 0 10G 0 dm ├─docker-9:127-6979358884-106bc0838e37442eca84eb9ab17aa7a45308b7e3a38be3fb21a4fa00366fe306 253:3 0 10G 0 dm ├─docker-9:127-6979358884-7e16b5d012ab8744739b671fcdc8e47db5cc64e6c3d5a5fe423bfd68cfb07b20 253:4 0 10G 0 dm ├─docker-9:127-6979358884-f1c2545b4edbfd7b42d2a492eda8224fcf7cefc3e3a41e65d307c585acffe6a8 253:5 0 10G 0 dm ├─docker-9:127-6979358884-e7fd6c7b3f624f387bdb3746a7944a30c92d8ee5395e75c76288b281bd009d90 253:6 0 10G 0 dm ├─docker-9:127-6979358884-95a818cc7afd9867385bb9a9ea750d4cc6e162916c6ae3a157097af74578e1e4 253:7 0 10G 0 dm ├─docker-9:127-6979358884-9a7f28d396c149119556f382bf5c19f5925eed5d18b94407649244c7adabb4b3 253:8 0 10G 0 dm ├─docker-9:127-6979358884-b25941b6f115300caea977911e2d7fd3541ef187c9aa5736fe10fad638ecd0d1 253:9 0 10G 0 dm ├─docker-9:127-6979358884-122b201c6ad24896a205f8db4a64759ba8fbd5bbe245d0f98984268a01e6a0c4 253:10 0 10G 0 dm └─docker-9:127-6979358884-bc04120ba59a1b393f338a1cef64b16d920cf4e73400198e4b999bb72a42ff90 253:11 0 10G 0 dm loop1 7:1 0 2G 0 loop └─docker-9:127-6979358884-pool 253:0 0 100G 0 dm ├─docker-9:127-6979358884-4301cee8d0433729cd6332ca2b6111afc85f14c48d4ce2d888a1da0ef9b5ca01 253:1 0 10G 0 dm ├─docker-9:127-6979358884-d59208adcb7cee3418f810f24e6c3a55d39281f713c8e76141fc61a8deba8a2b 253:2 0 10G 0 dm ├─docker-9:127-6979358884-106bc0838e37442eca84eb9ab17aa7a45308b7e3a38be3fb21a4fa00366fe306 253:3 0 10G 0 dm ├─docker-9:127-6979358884-7e16b5d012ab8744739b671fcdc8e47db5cc64e6c3d5a5fe423bfd68cfb07b20 253:4 0 10G 0 dm ├─docker-9:127-6979358884-f1c2545b4edbfd7b42d2a492eda8224fcf7cefc3e3a41e65d307c585acffe6a8 253:5 0 10G 0 dm ├─docker-9:127-6979358884-e7fd6c7b3f624f387bdb3746a7944a30c92d8ee5395e75c76288b281bd009d90 253:6 0 10G 0 dm ├─docker-9:127-6979358884-95a818cc7afd9867385bb9a9ea750d4cc6e162916c6ae3a157097af74578e1e4 253:7 0 10G 0 dm ├─docker-9:127-6979358884-9a7f28d396c149119556f382bf5c19f5925eed5d18b94407649244c7adabb4b3 253:8 0 10G 0 dm ├─docker-9:127-6979358884-b25941b6f115300caea977911e2d7fd3541ef187c9aa5736fe10fad638ecd0d1 253:9 0 10G 0 dm ├─docker-9:127-6979358884-122b201c6ad24896a205f8db4a64759ba8fbd5bbe245d0f98984268a01e6a0c4 253:10 0 10G 0 dm └─docker-9:127-6979358884-bc04120ba59a1b393f338a1cef64b16d920cf4e73400198e4b999bb72a42ff90 253:11 0 10G 0 dm nvme0n1 259:8 0 1.8T 0 disk └─nvme0n1p1 259:9 0 1.8T 0 part └─md127 9:127 0 3.7T 0 raid0 / nvme1n1 259:6 0 1.8T 0 disk └─nvme1n1p1 259:7 0 1.8T 0 part └─md127 9:127 0 3.7T 0 raid0 / nvme2n1 259:2 0 1.8T 0 disk nvme3n1 259:1 0 1.8T 0 disk nvme4n1 259:0 0 1.8T 0 disk nvme5n1 259:3 0 1.8T 0 disk NOTE: this is to show that there are 4 nvme disks (nvme2n1, nvme3n1, nvme4n1, nvme5n1) available for the second vm What "emu-kvm: kvm_init_vcpu failed: Cannot allocate memory" means in this context? Thank you very much NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

3 3

[qa][stable] tempest.api.volume.test_versions.VersionsTest.test_show_version fails on stable/pike
by Matt Riedemann 02 Oct '19

02 Oct '19

A new test was recently added to tempest [1] which uses the volume API v3 client to list and show versions and compare those against the response schema. It's failing against stable/pike changes [2] presumably because the volume v3 API does not show the (deprecated) v1 API, though I'm a bit confused because the test is doing a GET / and then for each version in the response it's doing a GET /{version}/ so I'm not sure what's up with cinder's pike API version interface where it would list a version but not be able to show it. Anyway, it sounds like this is another case where we're going to have to pin tempest to a tag in devstack on stable/pike to continue running tempest jobs against stable/pike changes, similar to what recently happened with stable/ocata [3]. [1] https://review.opendev.org/#/c/680790/ [2] https://145b0d5be77f0cbe05ff-d67a6226f6eb6d34ab4efd951744a16f.ssl.cf5.rackc… [3] https://review.opendev.org/#/c/681950/ -- Thanks, Matt

3 5

[OSA][openstack-ansible] Stepping down from core reviewer
by Jean-Philippe Evrard 02 Oct '19

02 Oct '19

Hello OSA friends, It's with great sadness that announcing I will be stepping down from OpenStack-Ansible's core role. OSA has been the place where I grew from contributor to OpenStack core for the first time, so it will always keep a special place in my mind :) It's also where I met contributors I can now consider personal friends. It's a project I've helped grow and prosper the last 4 years. I am very happy of what we have all achieved. My last goals in OSA were to simplify it further and a focus on bare metal. Those efforts are either merged or advanced enough nowadays. I consider I have achieved what I wanted to: OSA is now easier than ever to manage, contribute, and deliver. With more experienced people leaving and new people joining, I sure hope the DNA of the project will stay the same: An always welcoming and friendly community, with a no-bullshit and not-too-serious attitude. A project focusing on operator issues and use cases, mentoring contributors to be great members of the OpenStack community. Again, I want to thank you for being an amazing community, and it's been great working with all of you. I think there are still plenty of things OSA can achieve. If you want my opinion, you shouldn't hesitate to contact me. I just don't have enough time to keep up with the reviews, nor am I actively contributing enough to stay at core. All the best, Jean-Philippe Evrard (evrardjp)

5 4

Re: [Release-job-failures] release-post job for openstack/releases for ref refs/heads/master failed
by Sean McGinnis 30 Sep '19

30 Sep '19

On Mon, Sep 30, 2019 at 06:23:47PM +0000, zuul(a)openstack.org wrote: > Build failed. > > - tag-releases https://zuul.opendev.org/t/openstack/build/9ee941bea3b8476c98fe6fc17328f1c5 : FAILURE in 15m 10s > - publish-tox-docs-static https://zuul.opendev.org/t/openstack/build/None : SKIPPED > > _______________________________________________ > Release-job-failures mailing list > Release-job-failures(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures I'm not entirely sure what caused the failure here, but it is not really an issue. This was a docs-only change, so there wasn't anything to be tagged. Due to the tag-releases job failing, the publish-tox-docs-static job was skipped. There are other patches going through though, so the docs will get picked up and published very soon, if they haven't been already, so nothing to worry about there. Sean

1 0