- openstack-discuss - lists.openstack.org

[senlin] New meeting time for odd weeks
by Duc Truong 01 Dec '18

01 Dec '18

As discussed in our last Senlin meeting [1], I would like to propose holding the weekly meeting during odd weeks at a time that is more convenient for US and Europe. The proposal is to have the meeting at 19:00 UTC (11am PST) for odd weeks, while the meeting time for even weeks stays at the current time of 5:30 UTC. If accepted, the meeting times for upcoming weeks would be as follows: - Thursday, December 6, 2018 at 19:00:00 UTC - Friday, December 14, 2018 at 05:30:00 UTC - Thursday, December 20, 2018 at 19:00:00 UTC ... Please reply with any feedback. Regards, Duc (dtruong) [1] http://eavesdrop.openstack.org/meetings/senlin/2018/senlin.2018-11-30-05.30…

1 0

[openstack-dev] [nova] Would an api option to create an instance without powering on be useful?
by Matthew Booth 01 Dec '18

01 Dec '18

I have a request to do $SUBJECT in relation to a V2V workflow. The use case here is conversion of a VM/Physical which was previously powered off. We want to move its data, but we don't want to be powering on stuff which wasn't previously on. This would involve an api change, and a hopefully very small change in drivers to support it. Technically I don't see it as an issue. However, is it a change we'd be willing to accept? Is there any good reason not to do this? Are there any less esoteric workflows which might use this feature? Matt -- Matthew Booth Red Hat OpenStack Engineer, Compute DFG Phone: +442070094448 (UK) __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 4

[tripleo] Expose healthchecks via systemd for podman
by Jill Rouleau 01 Dec '18

01 Dec '18

The healthchecks currently in use in tripleo depend on the the docker healthcheck implementation, so podman/oci containers will not expose them. The /openstack/healthcheck script is still available in the images, we just need a way to run the checks and expose the result. https://review.openstack.org/#/c/620372/ proposes having paunch write out a $service-healthcheck systemd unit and corresponding timer. Interested in what folks think about the approach. -Jill

2 2

[openstack-dev][tripleo] TripleO CI Summary: Sprint 22
by Rafael Folco 30 Nov '18

30 Nov '18

Greetings, The TripleO CI team has just completed Sprint 22 (Nov 08 thru Nov 28). The following is a summary of completed work during this sprint cycle: - Converted multinode scenarios(1-4) to standalone job. This effort will allow us to replace multinode scenario jobs that are consuming most of upstream resources. Replacement of multinode scenario jobs is starting immediately and should be complete by Jan 2019. - Improved python and ansible support on Fedora 28 job, which is successfully deploying TripleO, but still requires further work on Tempest packaging before becoming a voting job. This is the first step in using upstream CI to validate python3 and RHEL8 work. - Investigated alternative solutions for the reproducer to approximate with the zuulv3 workflow. A PoC has been developed to run Zuul in a container as the reproducer script for TripleO CI. - Tested a new CI environment for OVB jobs on RDO cloud. Experiments for refectoring ovb deployment to be incorporated into CI code without te-broker bits. - https://tree.taiga.io/project/tripleo-ci-board/epic/298 - Initial testing of CentOS 7.6 looks good - https://tree.taiga.io/project/tripleo-ci-board/issue/309?kanban-status=2027… The sprint task board for CI team is tracked on Taiga [1] and the Ruck and Rover notes for this sprint has been tracked in the etherpad [2]. The planned work for the next sprint focuses in completing the upstream standalone job for Fedora 28 by eliminating all related issues. It also includes moving the multinode scenarios to the standalone jobs that has started in the previous sprint. The team continues to work on the reproducer and preparing a CI environment for OVB. The only new item for the sprint is to start building the production pipeline promotion jobs for Fedora 28. The Ruck and Rover for this sprint are Marios Andreou (marios) and Sorin Sbarnea (ssbarnea). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Thanks, Folco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/sprint-23unified-s… [2] https://review.rdoproject.org/etherpad/p/ruckrover-sprint22 -- Rafael Folco Senior Software Engineer

1 0

[placement] update 18-48
by Chris Dent 30 Nov '18

30 Nov '18

HTML: https://anticdent.org/placement-update-18-48.html Here's a placement update. It's been a rather busy week in placement-land: Gibi and I, with the help of others, have been getting the functional tests in nova working with an external placement. It is working now, which is great, but much work was done, more details within. # Most Important Progress continues on the work in ansible, puppet/tripleo, kolla, loci to package placement up and establish upgrade processes. All of these things need review (see below). Work on GPU reshaping in virt drivers is getting close. # What's Changed * Devstack installs placement from openstack/placement, not nova. * Placement now runs tempest and grenade (in py3) in CI. This is great because real integration tests and sad because our once really fast CI runs are gone. * The placement service can now start without a conf file. This is basically fixing a bug: we required a single config file in a specific location, rather than the defaults (which includes no file). We require an explicit database connection. # Bugs * Placement related [bugs not yet in progress](https://goo.gl/TgiPXb): 19. +2. * [In progress placement bugs](https://goo.gl/vzGGDQ) 14. +1 # Specs Spec freeze is milestone 2, the week of January 7th. None of the specs listed last week have merged. * <https://review.openstack.org/#/c/544683/> Account for host agg allocation ratio in placement (Still in rocky/) * <https://review.openstack.org/#/c/595236/> Add subtree filter for GET /resource_providers * <https://review.openstack.org/#/c/597601/> Resource provider - request group mapping in allocation candidate * <https://review.openstack.org/#/c/549067/> VMware: place instances on resource pool (still in rocky/) * <https://review.openstack.org/#/c/555081/> Standardize CPU resource tracking * <https://review.openstack.org/#/c/599957/> Allow overcommit of dedicated CPU (Has an alternative which changes allocations to a float) * <https://review.openstack.org/#/c/591037/> Modelling passthrough devices for report to placement * <https://review.openstack.org/#/c/603955/> Nova Cyborg interaction specification. * <https://review.openstack.org/#/c/601596/> supporting virtual NVDIMM devices * <https://review.openstack.org/#/c/603352/> Spec: Support filtering by forbidden aggregate * <https://review.openstack.org/#/c/552924/> Proposes NUMA topology with RPs * <https://review.openstack.org/#/c/569011/> Count quota based on resource class * <https://review.openstack.org/#/c/141219/> Adds spec for instance live resize * <https://review.openstack.org/#/c/612497/> Provider config YAML file * <https://review.openstack.org/#/c/509042/> Propose counting quota usage from placement and API database * <https://review.openstack.org/603545> Resource modeling in cyborg. * <https://review.openstack.org/#/c/609960/> Support filtering of allocation_candidates by forbidden aggregates # Main Themes ## Making Nested Useful Progress is being made on gpu-reshaping for libvirt and xen: * <https://review.openstack.org/#/q/topic:bp/reshape-provider-tree+status:open> Also making use of nested is bandwidth-resource-provider: * <https://review.openstack.org/#/q/topic:bp/bandwidth-resource-provider> Eric's in the process of doing lots of cleanups to how often the ProviderTree in the resource tracker is checked against placement, and a variety of other "let's make this more right" changes in the same neighborhood: * Stack at: <https://review.openstack.org/#/c/615646/> ## Extraction (There's an [etherpad](https://etherpad.openstack.org/p/placement-extract-stein-4) which tracks some of the work related to extraction. Please refer to that for additional information.) Several deployment tools are working on making things work with the extracted placement: * [TripleO](https://review.openstack.org/#/q/topic:tripleo-placement-extractio… * [OpenStack Ansible](https://review.openstack.org/#/q/project:openstack/openstack-ansib… * [Kolla](https://review.openstack.org/#/c/613589/) * [Kolla Ansible](https://review.openstack.org/#/c/613629/) * [Loci](https://review.openstack.org/#/c/617273/) A replacement for `placeload` performance testing that was in the `nova-next` job: <https://review.openstack.org/#/c/619248/>. This might be of interest to people trying to do testing of live services without devstack. It starts with a basic node, turns on mysql, runs placement with uwsgi, and does the placeload testing. Note that this found a pretty strange [bug in _ensure_aggregates](https://bugs.launchpad.net/nova/+bug/1804453). Documentation tuneups: * Release-notes: <https://review.openstack.org/#/c/618708/> This is blocked until we refactor the release notes to reflect _now_ better. * The main remaining task here is participating in [openstack-manuals](https://docs.openstack.org/doc-contrib-guide/doc-index.h…. **The functional tests in nova** excitement this week has led to some interesting changes. The biggest of these is changing placement to [no longer use global config](https://review.openstack.org/#/c/619121/). That patch is an aggregate of the many things done to remove a race condition in the [nova functional tests](https://review.openstack.org/#/c/617941/). Trying to use the config of two different projects in the same process space is hard if they are both global. So now placement doesn't. This is a win in many ways. Getting those two patches merged _soon_ is important. We've been putting off making a decision about os-resource-classes. Anyone have strong opinions? # Other There are 12 or so [open changes](https://review.openstack.org/#/q/status:open+project:openstack/pla… in placement itself. Of those these two are the most important: * <https://review.openstack.org/#/c/619121/> Stop using global oslo_config * <https://review.openstack.org/#/c/614559/> Correct lower-constraints.txt and the related tox job Outside of placement itself, here are some related changes: * <https://review.openstack.org/#/q/topic:bp/initial-allocation-ratios> Improve handling of default allocation ratios * <https://review.openstack.org/#/q/topic:minimum-bandwidth-allocation-placeme…> Neutron minimum bandwidth implementation * <https://review.openstack.org/#/c/602160/> Add OWNERSHIP $SERVICE traits * <https://review.openstack.org/#/c/586960/> zun: Use placement for unified resource management * <https://review.openstack.org/#/q/project:openstack/blazar+topic:bp/placemen…> Blazar using the placement-api * <https://review.openstack.org/619626> Tenks doing some node management, with a bit of optional placement. * <https://review.openstack.org/#/c/620485/> Sync placement database to the current version (in grenade) * <https://review.openstack.org/#/c/619107/> Address nits on lib/placement for extracted placement (in devstack) # End That was the November that was. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

2 1

Horizon Logging Question
by sam.bouwari＠verizon.com 30 Nov '18

30 Nov '18

Hello Horizon Proj Developers. In previous (older) Openstack/Horizon versions (version 7 or 8), the /var/log/horizon.log file used to reveal "remote address" information when a user attempts to login to Horizon GUI with wrong password, or if someone is trying to guess a password, then a log entry similar to this would be logged: 2018-11-01 19:39:35,728 3711 WARNING openstack_auth.forms Login failed for user "admin", remote address 192.168.1.12. In newer OSP/Horizon versions, the "remote address" field seems to have been truncated from the log and all we see now is a login entry in /var/log/horizon.log similar to this: 2018-11-30 14:23:10,912 473292 WARNING openstack_auth.forms Login failed for user "admin". Is there any valid reason as to why this feature was removed from newer version of Horizon? Logging the "remote address" information is very useful from security/IR and log monitoring standpoint where an analyst can identify if someone is trying to guess passwords, or trying to brute force the Horizon GUI, and other useful scenarios. Is there a configuration parameter in Horizon that can be turned on to log the "remote address" infromation again? Or is this feature just not available anymore in the product? Your help is greatly appreciated! Thanks Sam Bouwari

1 0

[dev][keystone]
by Colleen Murphy 30 Nov '18

30 Nov '18

# Keystone Team Update - Week of 26 November 2018 ## News ### Forum Recap Two weeks ago some of us attended the OpenStack Summit in Berlin, where we had a couple of keystone-specific sessions and had some discussions with Edge computing operators and public cloud operators about keystone use cases. Lance wrote up a recap[1]. [1] https://www.lbragstad.com/blog/openstack-summit-berlin-recap ### Refreshable application credentials We had a short discussion on how refreshable application credentials will be implemented[2]. The summary is that new fields should be added to the existing model, renewed_at and renewable_roles. We had discussed adding a new expiration field or reusing the existing expiration field, but decided that for the refresh use case, application credentials shouldn't be using expiration but instead a static comparison of the last refresh time and a configurable TTL tied to the IdP. The existing expiration field will remain user-configurable. [2] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-k… ## Open Specs Stein specs: https://bit.ly/2Pi6dGj Ongoing specs: https://bit.ly/2OyDLTh The JWT spec[3] and the domain support for limits spec[4] are ready to go, please have a look. There are also several specs not proposed for Stein but instead proposed as "ongoing" that need reviews. [3] https://review.openstack.org/541903 [4] https://review.openstack.org/599491 ## Recently Merged Changes Search query: https://bit.ly/2pquOwT We merged 20 changes this week. ## Changes that need Attention Search query: https://bit.ly/2RLApdA There are 82 changes that are passing CI, not in merge conflict, have no negative reviews and aren't proposed by bots. Most of them are from Lance to incorporate our new default roles into our own policies. ## Bugs This week we opened 14 new bugs and closed 4. Bugs opened (14) Bug #1805400 (keystone:High) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805400 Bug #1805363 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805363 Bug #1805366 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805366 Bug #1805368 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805368 Bug #1805369 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805369 Bug #1805371 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805371 Bug #1805372 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805372 Bug #1805402 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805402 Bug #1805403 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805403 Bug #1805406 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805406 Bug #1805880 (keystone:Medium) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805880 Bug #1805409 (keystone:Wishlist) opened by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1805409 Bug #1805165 (keystone:Undecided) opened by Boris Bobrov https://bugs.launchpad.net/keystone/+bug/1805165 Bug #1805817 (keystone:Undecided) opened by Robert Duncan https://bugs.launchpad.net/keystone/+bug/1805817 Bugs fixed (4) Bug #1735250 (keystone:High) fixed by wangxiyuan https://bugs.launchpad.net/keystone/+bug/1735250 Bug #1792026 (keystone:Medium) fixed by Lance Bragstad https://bugs.launchpad.net/keystone/+bug/1792026 Bug #1781536 (keystone:Low) fixed by Gage Hugo https://bugs.launchpad.net/keystone/+bug/1781536 Bug #1802357 (oslo.policy:Undecided) fixed by no one https://bugs.launchpad.net/oslo.policy/+bug/1802357 ## Milestone Outlook https://releases.openstack.org/stein/schedule.html There is just over a month left until the Stein spec freeze. Please have a look at the currently proposed specs for Stein. ## Shout-outs Thanks Jens Harbott for helping us out with our Zuul job issues[5][6]! [5] https://review.openstack.org/611563 [6] https://review.openstack.org/620553 ## Help with this newsletter Help contribute to this newsletter by editing the etherpad: https://etherpad.openstack.org/p/keystone-team-newsletter Dashboard generated using gerrit-dash-creator and https://gist.github.com/lbragstad/9b0477289177743d1ebfc276d1697b67

1 1

Re: [Openstack] [Pike][Neutron] L3 metering with DVR doesn't work
by Alexandru Sorodoc 30 Nov '18

30 Nov '18

Hello Brian, Thanks for the info. I looked into the code for the metering agent and discovered the following: 1. The metering agent on compute nodes doesn't get notified about DVR routers running on the node. 2. For DVR routers it tries to meter the rfp- interface(for floating ips) on the snat- namespace. This is wrong. The rfp- interface lies on the qrouter- namespace on compute nodes. Also, there is a qg- interface on the snat- namespace on network nodes (for performing NAT) which should be metered too. 3. There is a race condition whereby the metering agent is notified about a router before its namespaces are created. The agent ends up not adding the metering rules for those namespaces and this leads to unrecorded traffic. I addressed those issues in a change: https://review.openstack.org/#/c/621165/. Any feedback is appreciated. Best regards, Alex On 26/10/2018 21:49, Brian Haley wrote: > On 10/25/2018 08:06 AM, Alexandru Sorodoc wrote: >> Hello, >> >> I'm trying to set up metering for neutron in Pike. I tested it with a >> centralized router and it works, but when I try with a distributed >> router it >> doesn't record any usage samples. I have one compute node and one >> network node >> and I've created an instance with a floating ip. > > The metering agent isn't very well maintained, and I don't see any > open bugs similar to this issue. The only thing I can remember is > this abandoned change regarding traffic counters for DVR routers - > https://review.openstack.org/#/c/486493/ but there was no follow-on > from the author. > > The best thing to do would be to try and reproduce it on the master > branch (or Rocky) and file a bug. > > > I think this is because the router is running on network1. Why is it > > running on > > network1 and why does it seem that the l3 agent on compute1 does the > actual > > routing? > > The compute node will do all the routing when a floating IP is > associated, the router on network1 is for default snat traffic when > there is no floating IP and the instance tries to communicate out the > external network. > > -Brian > >> >> openstack router show public-router2 >> +-------------------------+----------------------------------------------------+ >> >> | Field | >> Value | >> +-------------------------+----------------------------------------------------+ >> >> | admin_state_up | >> UP | >> | availability_zone_hints >> | | >> | availability_zones | >> nova | >> | created_at | >> 2018-10-05T12:07:32Z | >> | description | | >> | distributed | >> True | >> | external_gateway_info | {"network_id": >> "b96473ce- | >> | | 94f6-464f-a703-5285fb8ff3d3", >> "enable_snat": true, | >> | | "external_fixed_ips": >> [{"subnet_id": | >> | | >> "6c08c3d9-7df1-4bec-b847-19f80b9d1764", | >> | | "ip_address": >> "192.168.252.102"}]} | >> | flavor_id | >> None | >> | ha | >> False | >> | id | >> 37c1794b-58d1-4d0d-b34b-944ca411b86b | >> | name | >> public-router2 | >> | project_id | >> fe203109e67f4e39b066c9529f9fc35d | >> | revision_number | >> 5 | >> | routes | | >> | status | >> ACTIVE | >> | tags | | >> | updated_at | >> 2018-10-05T12:09:36Z | >> +-------------------------+----------------------------------------------------+ >> >> >> openstack network agent list >> +-----------+------------+-----------+-------------------+-------+-------+--------------+ >> >> | ID | Agent Type | Host | Availability Zone | Alive | >> State | Binary | >> +-----------+------------+-----------+-------------------+-------+-------+--------------+ >> >> | 14b9ea75- | L3 agent | compute1. | nova | :-) | UP | >> neutron-l3-a | >> | 1dc1-4e37 | | localdoma | | | | gent | >> | -a2b0-508 | | in | | | | | >> | 3d336916d | | | | | | | >> | 26139ec1- | Metering | compute1. | None | :-) | UP | >> neutron- | >> | f4f9-4bb3 | agent | localdoma | | | | metering- | >> | -aebb-c35 | | in | | | | agent | >> | 3a36ed79c | | | | | | | >> | 2a54971f- | DHCP agent | network1. | nova | :-) | UP | >> neutron- | >> | 9849-4ed2 | | localdoma | | | | dhcp-agent | >> | -b009-00e | | in | | | | | >> | 45eb4d255 | | | | | | | >> | 443c0b49- | Open | compute1. | None | :-) | UP | >> neutron- | >> | 4484-44d2 | vSwitch | localdoma | | | | openvswitch- | >> | -a704-32a | agent | in | | | | agent | >> | 92ffe6982 | | | | | | | >> | 5d00a219 | L3 agent | network1. | nova | :-) | UP | >> neutron-vpn- | >> | -abce- | | localdoma | | | | agent | >> | 48ca- | | in | | | | | >> | ba1e-d962 | | | | | | | >> | 01bd7de3 | | | | | | | >> | bc3458b4 | Open | network1. | None | :-) | UP | >> neutron- | >> | -250e- | vSwitch | localdoma | | | | openvswitch- | >> | 4adf-90e0 | agent | in | | | | agent | >> | -110a1a7f | | | | | | | >> | 6ccb | | | | | | | >> | c29f9da8- | Metering | network1. | None | :-) | UP | >> neutron- | >> | ca58-4a11 | agent | localdoma | | | | metering- | >> | -b500-a25 | | in | | | | agent | >> | 3f820808e | | | | | | | >> | cdce667d- | Metadata | network1. | None | :-) | UP | >> neutron- | >> | faa4 | agent | localdoma | | | | metadata- | >> | -49ed- | | in | | | | agent | >> | 83ee-e0e5 | | | | | | | >> | a352d482 | | | | | | | >> | cf5ae104- | Metadata | compute1. | None | :-) | UP | >> neutron- | >> | 49d7-4c85 | agent | localdoma | | | | metadata- | >> | -a252-cc5 | | in | | | | agent | >> | 9a9a12789 | | | | | | | >> +-----------+------------+-----------+-------------------+-------+-------+--------------+ >> >> >> If I check the node on which my distributed router is running it >> tells me that >> it's running on the network node: >> >> neutron l3-agent-list-hosting-router >> 37c1794b-58d1-4d0d-b34b-944ca411b86b >> +--------------------------------------+----------------------+----------------+-------+----------+ >> >> | id | host | >> admin_state_up | alive | ha_state | >> +--------------------------------------+----------------------+----------------+-------+----------+ >> >> | 5d00a219-abce-48ca-ba1e-d96201bd7de3 | network1.localdomain | >> True | :-) | | >> +--------------------------------------+----------------------+----------------+-------+----------+ >> >> >> If I check the iptable rules for the router on the compute and >> network nodes by running: >> >> ip netns exec qrouter-37c1794b-58d1-4d0d-b34b-944ca411b86b iptables >> -nv -L >> >> I see that compute1 records the traffic while network1 doesn't. Also, >> I did some >> debugging and found out that the metering agent on compute1 receives >> an empty >> list of routers when querying the routers that it should monitor. >> >> Source: >> >> https://github.com/openstack/neutron/blob/stable/pike/neutron/services/mete… >> >> >> https://github.com/openstack/neutron/blob/stable/pike/neutron/db/metering/m… >> >> >> I think this is because the router is running on network1. Why is it >> running on >> network1 and why does it seem that the l3 agent on compute1 does the >> actual >> routing? >> >> Thanks, >> Alex >> >> >> >> _______________________________________________ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack(a)lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> > _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack(a)lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

1 0

[oslo][nova][stable][requirements] Fixing a high CPU usage from oslo.service into stable/rocky branch
by Herve Beraud 30 Nov '18

30 Nov '18

Hey all! Here is a thread to coordinate all the teams (oslo, nova, stable, requirements) working on the update of the oslo.service constraint in the Rocky requirements. # Summary Usage of threading event with eventlet caused inefficient code (causing many useless system calls and high CPU usage). This issue was already fixed on oslo.service master and we also want to fix it in stable/rocky. Our main issue is how to fix the high CPU usage on stable/rocky without break the nova CI. Indeed, we already have backported the eventlet related fix to oslo.service but this fix requires also a nova update to avoid nova CI errors due to threading removal on oslo.service that introduce the nova CI errors. A fix was proposed and merged on oslo.service master to introduce a new feature (fixture) that avoid the nova CI errors, but backporting the master fix to Rocky introduces a new feature into a stable branch so this is also an issue. So we need to discuss with all the teams to find a proper solution. # History A few weeks ago this issue was opened on oslo.service ( https://bugs.launchpad.net/oslo.service/+bug/1798774) and it was fixed by this submited patch on the master branch ( https://review.openstack.org/#/c/611807/ ). This change use the proper event primitive to fix the performance issue. A new version of oslo.service was released (1.32.1) Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master. Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release. When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version). Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error. # Issue Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements. In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers. The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue. Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes. # Questions and proposed solutions This thread try to summarize the current situation. We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix. 1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ? 2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_comp…) ? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above. Personaly I think we need to find an another approach like the mocking remplacement (c.f 2). We need to decide which way we use and to discuss about other solutions. -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

8 20

Re: [openstack-dev] [vitrage] I have some problems with Prometheus alarms in vitrage.
by Won 30 Nov '18

30 Nov '18

Hi, I attached four log files. I collected the logs from about 17:14 to 17:42. I created an instance of 'deltesting3' at 17:17. 7minutes later, at 17:24, the entity graph showed the dentesting3 and vitrage colletor and graph logs are appeared. When creating an instance in ubuntu server, it appears immediately in the entity graph and logs, but when creating an instance in computer1 (multi node), it appears about 5~10 minutes later. I deleted an instance of 'deltesting3' around 17:26. > After ~20minutes, there was only Apigateway. Does it make sense? did you > delete the instances on ubuntu, in addition to deltesting? > I only deleted 'deltesting'. After that, only the logs from 'apigateway' and 'kube-master' were collected. But other instances were working well. I don't know why only two instances are collected in the log. NOV 19 In this log, 'agigateway' and 'kube-master' were continuously collected in a short period of time, but other instances were sometimes collected in long periods. In any case, I would expect to see the instances deleted from the graph at > this stage, since they were not returned by get_all. > Can you please send me the log of vitrage-graph at the same time (Nov 15, > 16:35-17:10)? > Information 'deldtesting3' that has already been deleted continues to be collected in vitrage-graph.service. Br, Won 2018년 11월 15일 (목) 오후 10:13, Ifat Afek <ifatafekn(a)gmail.com>님이 작성: > > On Thu, Nov 15, 2018 at 10:28 AM Won <wjstk16(a)gmail.com> wrote: > >> Looking at the logs, I see two issues: >>> 1. On ubuntu server, you get a notification about the vm deletion, while >>> on compute1 you don't get it. >>> Please make sure that Nova sends notifications to >>> 'vitrage_notifications' - it should be configured in /etc/nova/nova.conf. >>> 2. Once in 10 minutes (by default) nova.instance datasource queries all >>> instances. The deleted vm is supposed to be deleted in Vitrage at this >>> stage, even if the notification was lost. >>> Please check in your collector log for the a message of "novaclient.v2.client >>> [-] RESP BODY" before and after the deletion, and send me its content. >> >> >> I attached two log files. I created a VM in computer1 which is a >> computer node and deleted it a few minutes later. Log for 30 minutes from >> VM creation. >> The first is the log of the vitrage-collect that grep instance name. >> The second is the noovaclient.v2.clinet [-] RESP BODY log. >> After I deleted the VM, no log of the instance appeared in the collector >> log no matter how long I waited. >> >> I added the following to Nova.conf on the computer1 node.(attached file >> 'compute_node_local_conf.txt') >> notification_topics = notifications,vitrage_notifications >> notification_driver = messagingv2 >> vif_plugging_timeout = 300 >> notify_on_state_change = vm_and_task_state >> instance_usage_audit_period = hour >> instance_usage_audit = True >> > > Hi, > > From the collector log RESP BODY messages I understand that in the > beginning there were the following servers: > compute1: deltesting > ubuntu: Apigateway, KubeMaster and others > > After ~20minutes, there was only Apigateway. Does it make sense? did you > delete the instances on ubuntu, in addition to deltesting? > In any case, I would expect to see the instances deleted from the graph at > this stage, since they were not returned by get_all. > Can you please send me the log of vitrage-graph at the same time (Nov 15, > 16:35-17:10)? > > There is still the question of why we don't see a notification from Nova, > but let's try to solve the issues one by one. > > Thanks, > Ifat > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

2 2