- openstack-discuss - lists.openstack.org

[manila][kolla-ansible] Fail to use manila with cephfs NFS share backend
by Boxiang Zhu 11 Nov '19

11 Nov '19

Hi everyone, I have deployed a OpenStack cluster with AIO mode by kolla-ansible. In my globals.yml, something is enabled as followed: ...... enable_ceph: "yes" enable_ceph_mds: "yes" enable_ceph_nfs: "yes" enable_manila: "yes" enable_manila_backend_cephfs_nfs: "yes" ....... And in my manila.conf, some configs are as followed: [DEFAULT] enabled_share_protocols = NFS,CIFS ...... [cephfsnfs1] driver_handles_share_servers = False share_backend_name = CEPHFSNFS1 share_driver = manila.share.drivers.cephfs.driver.CephFSDriver cephfs_protocol_helper_type = NFS cephfs_conf_path = /etc/ceph/ceph.conf cephfs_auth_id = manila cephfs_cluster_name = ceph cephfs_enable_snapshots = False cephfs_ganesha_server_is_remote = False cephfs_ganesha_server_ip = 172.16.60.84 ...... I use CLI to create the nfs, some commands as followed: -> manila type-create cephfsnfstype false -> manila type-key cephfsnfstype set vendor_name=Ceph storage_protocol=NFS -> manila create --share-type cephfsnfstype --name cephnfsshare1 nfs 1 -> manila share-export-location-list cephnfsshare1 +-----------------------------------------------+------------------------------------------------------------------------------------+-----------+ | ID | Path | Preferred | +-----------------------------------------------+------------------------------------------------------------------------------------+-----------+ | b101bf59-4cd7-4a09-a12e-b6dd48a5bb18 | 172.16.60.84:/volumes/_nogroup/93b1e23d-0166-41a4-a12a-51bf4c3654a5 | False | +-----------------------------------------------+------------------------------------------------------------------------------------+-----------+ -> manila access-allow cephnfsshare1 ip 172.16.60.119 But I have got some error messages from /var/lib/docker/volumes/kolla_logs/_data/manila/manila-share.log 2019-11-11 10:11:12.035 26 ERROR manila.share.drivers.ganesha.manager [req-673584f9-eaed-4464-819e-ace63dfa0f49 6f4bd93b18c546c880d5d4e526a0243e 70dba7786a8b4326a35e03d0ad8707f2 - - -] Error while executing management command on Ganesha node <no name>: dbus call exportmgr.AddExport.: ProcessExecutionError: Unexpected error while running command. 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server [req-673584f9-eaed-4464-819e-ace63dfa0f49 6f4bd93b18c546c880d5d4e526a0243e 70dba7786a8b4326a35e03d0ad8707f2 - - -] Exception during message handling: GaneshaCommandFailure: Ganesha management command failed. Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-93b1e23d-0166-41a4-a12a-51bf4c3654a5.conf string:EXPORT(Export_Id=105) Exit code: 1 Stdout: u'' Stderr: u'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n' 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/manager.py", line 187, in wrapped 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/utils.py", line 568, in wrapper 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/manager.py", line 3554, in update_access 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/access.py", line 283, in update_access_rules 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/access.py", line 322, in _update_access_rules 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/access.py", line 390, in _update_rules_through_share_driver 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/drivers/cephfs/driver.py", line 289, in update_access 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server share_server=share_server) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/drivers/ganesha/__init__.py", line 308, in update_access 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server self.ganesha.add_export(share['name'], confdict) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/manila/share/drivers/ganesha/manager.py", line 491, in add_export 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server cmd=e.cmd) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server GaneshaCommandFailure: Ganesha management command failed. 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/etc/ganesha/export.d/share-93b1e23d-0166-41a4-a12a-51bf4c3654a5.conf string:EXPORT(Export_Id=105) 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Exit code: 1 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Stdout: u'' 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server Stderr: u'Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.ganesha.nfsd was not provided by any .service files\n' 2019-11-11 10:11:14.954 26 ERROR oslo_messaging.rpc.server I found some possible solutions[0][1][2][3] to try to fix this, but failed. Any suggestions? [0] https://github.com/nfs-ganesha/nfs-ganesha/issues/483 [1] https://github.com/nfs-ganesha/nfs-ganesha/issues/219 [2] https://github.com/gluster/storhaug/issues/14 [3] https://sourceforge.net/p/nfs-ganesha/mailman/message/32227132/ Thanks, Boxiang

2 2

[Senlin] Splitting senlin-engine into three services
by Erik Olof Gunnar Andersson 10 Nov '19

10 Nov '19

We are looking into splitting the senlin-engine into three components (senlin-conductor, senlin-engine and senlin-health-manager) and wanted to get some feedback. The main goal here is to make the components more resilient and to reduce the number of threads per worker. Each one of the components already had it's own thread pool and in theory each worker could end up with thousands of thread. In the current version (Train) the engine process hosts these services. https://github.com/openstack/senlin/blob/stable/train/senlin/engine/dispatc… https://github.com/openstack/senlin/blob/stable/train/senlin/engine/health_… https://github.com/openstack/senlin/blob/stable/train/senlin/engine/service… In my patch we move two our of these out of the engine and into it's own service namespace. Split engine service into three services https://review.opendev.org/#/c/688784/ Please feel free to comment on the patch set, or let reply to this email with general feedback or concerns. Best Regards, Erik Olof Gunnar Andersson

1 1

[nova][neutron][ptg] Nova - Neutron cross project dicussions
by Balázs Gibizer 10 Nov '19

10 Nov '19

https://etherpad.openstack.org/p/ptg-ussuri-xproj-nova-neutron Nova-Neutron live-migration - ACTIVE/INACTIVE port bindings issue * nova needs vif plug event for inactive bindings * live migration with ml2-ovs only works by luck * agentless drivers does not support this https://bugs.launchpad.net/neutron/+bug/1834045 https://bugs.launchpad.net/neutron/+bug/1840147 Agreements: * document the nova-neutron live migration workflow to create a common base for discussion. Add ijw and sean-k-mooney to the review SR-IOV live migration utilizing kernel NET_FAILOVER feature * aim to support live migration with SRIOV nic without traffic interrupt by bonding a virtio interface to the SRIOV nic in the guest kernel with NET_FAILOVER. * https://www.kernel.org/doc/html/latest/networking/net_failover.html * There was multiple solution proposlas but no agreements about which one to pursue. * TODO: adrianc to propose a nova spec based on the discussion Correlation of Bandwidth RP with PCI RP when PCI will be tracked in Placement * Early heads up to Neutron team that Nova things about modelling PCI in Placement * Current best thinking on Nova side is to correlate BW RP with PCI RP in placement with a placement aggregate. * Nova will do the correlation * Neutron needs to prepare for possible BW RP generation conflict * Agreement: Neutron is OK with this plan. rubasov can help with the neutron impact when nova makes progress.

1 0

[nova][ptg] Ussuri cycle themes
by Balázs Gibizer 10 Nov '19

10 Nov '19

Ussuri themes proposal: * Policy work Note for the operators that this work only provides value after the whole work is ready and it is pretty possible that this will not finish in the current cycle. * Unified limits Discuss on the ML how can we help oslo to progress with the oslo-limits work. If we can help with that then, make unified limits as a cycle Goal. * Cyborg - Nova integration (bauzas) I'm OK with making Cyborg a priority for this cycle provided we don't hold any attempt to start thinking on fixing PCI tracking. We definitely needs to discuss this further with the whole team. Cheers, gibi

1 0

[nova][ironic][ptg] reconfigure the baremetal node through the resize API
by Balázs Gibizer 10 Nov '19

10 Nov '19

ironic spec https://review.opendev.org/#/c/672252/ TODOs: * alex_xu (?) to create a nova spec to list possible alternatives about which nova REST API could be used to trigger reconfigure (resize, reboot, new api) * A potential way forward would be Ironic supporting resize (confirm/revert) as well with some magical autodetection of an Ironic instance *or* some magical flavor-based solution Cheers, gibi

1 0

[nova][ptg] Other small discussions
by Balázs Gibizer 10 Nov '19

10 Nov '19

image precache support * aarents (OVH?) interested in the feauter * TODO: aarents to talk to dansmith about future improvements like rate limiting nova implications of a starlingx bug: https://bugs.launchpad.net/starlingx/+bug/1850834 * TODO: yawang to file a nova bug if reproducible * TODO: yawang to propose a patch in nova based on the starlingx fix https://gist.github.com/VictorRodriguez/e137a8cd87cf821f8076e9acc02ce195 vm scoped sriov numa affinity * spec: https://review.opendev.org/#/c/683174/4 * there was not enough knowledge in the room to really discuss * TODO: gibi to read and comment the spec midcylce * stephenfin will propose a midcycle disussion on the ML

1 0

Re: [nova][ptg] Flavor explosion
by Brin Zhang(张百林) 10 Nov '19

10 Nov '19

Gibi: The summary on the ML write in the nova ptg etherpad line 245 cannot open, raised a 404 error. Etherpad: https://etherpad.openstack.org/p/nova-shanghai-ptg Thanks. Brin Zhang items: [lists.openstack.org代发][nova][ptg] Flavor explosion original spec: https://review.opendev.org/#/c/663563 with -2s The first round of discussion was resulted in no agreement. Then on Friday we revisited the issue based on mdbooth's proposal about composability. Agreement in the room: * Do not try to change the model of the flavor in nova code and in the db. * Define a "ComposableFlavorBit" (bikeshed on the name please) REST API entity that can hold any kind of flavor bits (extra specs, normal flavor fields), propose some format in the spec for it. This entity can only be created by the admin by default * Extend the server create REST API to allow the end user to specify what "ComposableFlavorBit"s she wants to add to the "base" flavor she used in the create request. * The nova api then merges the "ComposableFalvorBit"s with the base flavor and embed the resulted flavor object into the instance. * Do a similar thing for resize TODOs: * brinzhang (with possible help from yawang) to re-write the spec Cheers, gibi

3 2

[nova][ptg] Flavor explosion
by Balázs Gibizer 10 Nov '19

10 Nov '19

original spec: https://review.opendev.org/#/c/663563 with -2s The first round of discussion was resulted in no agreement. Then on Friday we revisited the issue based on mdbooth's proposal about composability. Agreement in the room: * Do not try to change the model of the flavor in nova code and in the db. * Define a "ComposableFlavorBit" (bikeshed on the name please) REST API entity that can hold any kind of flavor bits (extra specs, normal flavor fields), propose some format in the spec for it. This entity can only be created by the admin by default * Extend the server create REST API to allow the end user to specify what "ComposableFlavorBit"s she wants to add to the "base" flavor she used in the create request. * The nova api then merges the "ComposableFalvorBit"s with the base flavor and embed the resulted flavor object into the instance. * Do a similar thing for resize TODOs: * brinzhang (with possible help from yawang) to re-write the spec Cheers, gibi

1 0

[kolla][ptg] Kolla Ussuri PVPTG (Partially Virtual PTG)
by Radosław Piliszek 08 Nov '19

08 Nov '19

Hello Everyone, As you may already know, Kolla core team is mostly not present on summit in Shanghai. Instead we are organizing a PTG next week, 7-8th Nov (Thu-Fri), in Białystok, Poland. Please let me know this week if you are interested in coming in person. We invite operators, contributors and contributors-to-be to join us for the virtual PTG online. The time schedule will be advertised later. Please fill yourself in on the whiteboard [1]. New ideas are welcome. [1] https://etherpad.openstack.org/p/kolla-ussuri-ptg Kind regards, Radek aka yoctozepto

6 9

[ops][nova][neutron] Proper way to migrate instances between nodes with different ML2 agents types
by Antoine Millet 08 Nov '19

08 Nov '19

Hi here, I'm trying to find a solution to migrate instances between hypervisors of an openstack cluster with nodes running different ML2 agents (OVS and bridges, I'm actually migrating the whole cluster to the latter). The cluster is running Rocky. I enabled both mechanisms in the neutron- server configuration and some nodes are running the neutron- openvswitch-agent and some other the neutron-linuxbridge-agent. My network nodes (running the l3 agent) are currently running the neutron- openvswitch-agent. I also noticed that when nova-compute is starting up, VIF plugins for OVS and Bridges are loaded ("INFO os_vif [-] Loaded VIF plugins: ovs, linux_bridge"). When I start a live migration for an instance running on an hypervisor using the OVS agent to an hypervisor using the bridge agent, it fails because the destination hypervisor try to execute 'ovs-*' commands to bind the VM to its network. I also tried cold migration and just restarting an hypervisor with the bridge agent instead of the OVS one, but it fails similarly when the instances startup. After some research, I discovered that the mechanism used to bind an instance port to a network is stored in the port binding configuration in the database and that the code that executes the 'ovs-*' commands is actually located in the os_vif library that is used by the nova-compute agent. So, I tried to remove the OVS plugin from the os_vif library. Ubuntu ship both plugins in the same package so I just deleted the plugin directory in /usr/lib/python2.7/dist-packages directory (don't judge me please, it's for science ;-)). And... it worked as expected (port bindings are converted to bridge mechanism), at least for the cold migration (hot migration is cancelled without any error message, I need to investigate more). How can I do those migration the proper way? Thank you for any help! Antoine

5 5

[nova][ptg] Virtual PTG
by Eric Fried 08 Nov '19

08 Nov '19

Hello nova contributors and other stakeholders. As you are aware, nova maintainers will be sparser than usual at the ussuri PTG. For that reason, and also because it promotes better inclusion anyway, I'd like us to do the majority of decision making via the mailing list. The PTG is still a useful place to talk through design ideas, but this will give those not attending a voice in the final direction. To that end, I call your attention to the etherpad [1]. As usual, list your topics there. And if your topic is something for which you only need (or wish to start with) in-person discussions (e.g. "I'd like to do $thing but could use some help figuring out $how"), you're done. But if what you're shooting for is discussion leading to some kind of decision, like... - My spec has been stalled because we can't decide among N different approaches; we need to reach a consensus. - My feature is really important; can we please prioritize it for ussuri? ...then in addition to putting your topic on the etherpad, please initiate a (separate) thread on this mailing list, including [nova][ptg] in your subject line. Some of these topics may be resolved before the PTG itself. Others may be discussed in Shanghai. However, even if a consensus is reached in person, expect that decision to be tentative pending closure of the ML thread. Thanks, efried [1] https://etherpad.openstack.org/p/nova-shanghai-ptg

3 3

[nova][ptg]Support volume local cache
by Balázs Gibizer 07 Nov '19

07 Nov '19

spec: https://review.opendev.org/#/c/689070/ TODOs: LiangFang to update the spec based on the discussion in the room[1]: * use traits to driver scheduling. The cache is not sliced per instance so it cannot be a resource class * document the alternative between doing a hard scheduling decision or only implement caching as a best effort optimization for the guest. * document the alternative to do the whole cache management on libvirt (or QEMU) level [1] https://etherpad.openstack.org/p/nova-shanghai-ptg

1 0

[nova][ptg] Community Goals
by Balázs Gibizer 07 Nov '19

07 Nov '19

python3 readiness: Stick with only considering py3 as default when writing code, not caring about refactoring existing code for the purpose of being Py3 pedantic. Write Py3-only code on master, and eventually make the backport py2-compatible on stable branches if needed. dropping paste: Need an operator ML post: is anybody using this? aarents (OVH?) said they are using paste.ini to insert middlewares. So we cannot simply drop paste. improved contributor documentation: we have PTL doc in tree [2]. Work on the CONTRIBUTING.rst [3]. [1] https://etherpad.openstack.org/p/nova-shanghai-ptg [2] https://docs.openstack.org/nova/latest/contributor/ptl-guide.html [3] https://review.opendev.org/#/c/640970/

1 0

[kuryr] Meeting at PTG - follow up
by Michał Dulko 07 Nov '19

07 Nov '19

Hi, Thank you all for a constructive discussion at the Shanghai PTG. First of all I want to clarify that the Neutron improvements in bulk port creation code I talked about were only merged in Stein and are not available in Queens. Sorry about that, my mistake. Below I want to list some highlights from the discussion. * There seemed no pushback on switching to independent release model, so I'll follow up on that soon. * We discussed using RPC instead of API polling to detect when port becomes ACTIVE in Neutron (https://review.opendev.org/#/c/669642/) I commented on the review with a follow up that came from discussion with Neutron team. Apparently there are 2 better ways of doing this. Let's continue that discussion on the review. * We know that we don't support running as a second CNI plugin on Multus because of eth0 being hardcoded as interface name and kuryr- controller handling pods that do not had CNI requests. This is something to solve. * Apparently we all suffer the same problems with Neutron performance when starting a bigger workload on Kuryr-configured K8s cluster (think big Helm chart, multiple operators or simultaneous test suite). This all comes to reducing the number of calls to Neutron. We still don't see a feasible solution to solve this problem, but it seems to be a priority for both Samsung and Red Hat at the moment. Let's see if we'll meet again in Vancouver! Thanks, Michał

1 1

[kolla] Repository setup in non-internet environment.
by Eddie Yen 07 Nov '19

07 Nov '19

Hi, I'm thinking about the deployment in non-internet environment. As we know Kolla has already prepared docker registry and kolla-build to let user can create local registry for deployment. But there's still have two problem about non-internet deployment. 1. Docker-ce repository. 2. Pip repository. (Also having others perhaps.) Does Kolla planning to support non-internet deployment? I would like to do this if possible. Looking forward to hearing from you, Eddie.

4 5

ephemeral dics storage
by Immo Wetzel 06 Nov '19

06 Nov '19

Gday mates, We do run a pike installation and do run into a problem with the ephemeral discs. As written these are on one side the common way for normal VMs to create a disk as long as the vm exists. On the other side these are stored on local discs. But these are usually not the fastest way and therefore the documentation said that in production environments, which we are going to be, these should be stored on shared spaces too. Like a SAN. I found some descriptions how to use ceph for locale storage via rbd backend but we don't use ceph. Each compute node has an FC connection which is used via cinder for the volumes. So what would be the recommendation to use ephemeral disc with FC SAN ? THX a lot Immo

2 1

CPU pinning blues
by Albert Braden 06 Nov '19

06 Nov '19

I'm following this document to setup CPU pinning on Rocky: https://www.redhat.com/en/blog/driving-fast-lane-cpu-pinning-and-numa-topol… I followed all of the steps except for modifying non-pinned flavors and I have one aggregate containing a single NUMA-capable host: root@us01odc-dev1-ctrl1:/var/log/nova# os aggregate list +----+-------+-------------------+ | ID | Name | Availability Zone | +----+-------+-------------------+ | 4 | perf3 | None | +----+-------+-------------------+ root@us01odc-dev1-ctrl1:/var/log/nova# os aggregate show 4 +-------------------+----------------------------+ | Field | Value | +-------------------+----------------------------+ | availability_zone | None | | created_at | 2019-10-30T23:05:41.000000 | | deleted | False | | deleted_at | None | | hosts | [u'us01odc-dev1-hv003'] | | id | 4 | | name | perf3 | | properties | pinned='true' | | updated_at | None | +-------------------+----------------------------+ I have a flavor with the NUMA properties: root@us01odc-dev1-ctrl1:/var/log/nova# os flavor show s1.perf3 +----------------------------+-------------------------------------------------------------------------+ | Field | Value | +----------------------------+-------------------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | disk | 35 | | id | be3d21c4-7e91-42a2-b832-47f42fdd3907 | | name | s1.perf3 | | os-flavor-access:is_public | True | | properties | aggregate_instance_extra_specs:pinned='true', hw:cpu_policy='dedicated' | | ram | 30720 | | rxtx_factor | 1.0 | | swap | 7168 | | vcpus | 4 | +----------------------------+-------------------------------------------------------------------------+ I create a VM with that flavor: openstack server create --flavor s1.perf3 --image NOT-QSC-CentOS6.10-19P1-v4 --network it-network alberttest4 but it goes to error status, and I see this in the logs: 2019-10-30 16:17:55.590 3248800 INFO nova.virt.hardware [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Computed NUMA topology CPU pinning: usable pCPUs: [[4], [5], [6], [7]], vCPUs mapping: [(0, 4), (1, 5), (2, 6), (3, 7)] 2019-10-30 16:17:55.595 3248800 INFO nova.virt.hardware [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Computed NUMA topology CPU pinning: usable pCPUs: [[0], [1], [2], [3], [4], [5], [6], [7]], vCPUs mapping: [(0, 0), (1, 1), (2, 2), (3, 3)] 2019-10-30 16:17:55.595 3248800 INFO nova.filters [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts 2019-10-30 16:17:55.596 3248800 INFO nova.filters [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filtering removed all hosts for the request with instance ID '73b1e584-0ce4-478c-a706-c5892609dc3f'. Filter results: ['RetryFilter: (start: 3, end: 3)', 'AvailabilityZoneFilter: (start: 3, end: 3)', 'CoreFilter: (start: 3, end: 2)', 'RamFilter: (start: 2, end: 2)', 'ComputeFilter: (start: 2, end: 2)', 'ComputeCapabilitiesFilter: (start: 2, end: 2)', 'ImagePropertiesFilter: (start: 2, end: 2)', 'ServerGroupAntiAffinityFilter: (start: 2, end: 2)', 'ServerGroupAffinityFilter: (start: 2, end: 2)', 'DifferentHostFilter: (start: 2, end: 2)', 'SameHostFilter: (start: 2, end: 2)', 'NUMATopologyFilter: (start: 2, end: 2)', 'AggregateInstanceExtraSpecsFilter: (start: 2, end: 0)'] It looks like my hypervisor is passing the hw:cpu_policy='dedicated' requirement but it is failing on "pinned=true" The interesting part of the problem is that if I add a second apparently identical hypervisor to the aggregate it starts working. I create s1.perf3 VMs and they land on us01odc-dev1-hv002 and the XML shows that they are correctly pinned. When us01odc-dev1-hv002 is full then they start failing again. What should I be looking for here? What could cause one apparently identical hypervisor to fail AggregateInstanceExtraSpecsFilter while another one passes? In the nova-compute log of the failing hypervisor I see this: 2019-10-31 10:43:01.147 1103 INFO nova.compute.resource_tracker [req-dda65a9c-9d0a-4888-b4cb-0bf4423dd2f3 - - - - -] Instance 4856d505-c220-4873-b881-836b5b75f7bb has allocations against this compute host but is not found in the database. 2019-10-31 10:43:01.148 1103 INFO nova.compute.resource_tracker [req-dda65a9c-9d0a-4888-b4cb-0bf4423dd2f3 - - - - -] Final resource view: name=us01odc-dev1-hv003.internal.synopsys.com phys_ram=128888MB used_ram=38912MB phys_disk=1208GB used_disk=297GB total_vcpus=8 used_vcpus=6 pci_stats=[] Openstack can't find a VM with UUID 4856d505-c220-4873-b881-836b5b75f7bb. There are no VMs on hv003 but I can create a non-pinned VM there and it works. Do I have a "phantom" VM that is consuming resources on hv003? How can I fix that?

2 5

[nova] Operator input on automatic heal behaviors
by Dan Smith 06 Nov '19

06 Nov '19

Hi all, If you're a nova operator, you probably know (and love) our increasing number of "heal $thing" commands in nova-manage. Despite appearances, we do not try to make these inconsistent, duplicative, and confusing. However, in reality, they pretty much are. Further, they require something being broken, an operator noticing, and then manual execution to (hopefully) fix things. While reviewing another such proposed nova-manage command today, I decided to propose a potential solution to make this better in the future. I've got a spec proposed to create new standalone command/service for nova that will consolidate all of these into a very consistent interface, with a daemon mode that can be run in the background to constantly (and slowly) periodically audit these things and heal them when issues are found. If you're an operator and have strong feelings on this topic, please review and opine here: https://review.opendev.org/#/c/693226 Thanks! --Dan

1 0

[neutron][tap-as-a-service] ERSPAN support
by Jean Bernard Beuque (jbeuque) 06 Nov '19

06 Nov '19

Hello, I'd like to add ERSPAN support to the Tap-as-a-Service project. I've currently implemented a prototype that can be used with networking-vpp: https://opendev.org/x/networking-vpp The modified version of tap as a service is available here (The API has been extended to support ERSPAN): https://github.com/jbeuque/tap-as-a-service I don't know who maintains the Taas project. But if you think adding this functionality could be useful, please contact me. (Please take the modified version of Taas as a proposal to be discussed). Regards, Jean-Bernard Beuque

2 2

[kuryr] Kuryr team at the PTG
by Michal Dulko 06 Nov '19

06 Nov '19

Hi, I had not reserved Kuryr space on the PTG as we weren't expecting many Kuryr team members here, but turns out there's some representation. We'll meet on Thursday at 2 PM Shanghai time to discuss anything related to Kuryr. We want meet in the Blue Room (where the tables are) and will try to find some space to run the discussion. Today you can find me at the K8s SIG table. Feel free to join! Thanks, Michał

3 4

[cinder] ussuri PTG schedule
by Brian Rosmaita 05 Nov '19

05 Nov '19

The Ussuri PTG schedule is live: https://etherpad.openstack.org/p/shanghai-ptg-cinder Please check the schedule and let me know right away if your session causes a conflict for you. Except for the few fixed-time topics, we will follow the cinder tradition of dynamic scheduling, giving each topic exactly as much time as it needs and adjusting as we go. cheers, brian

2 1

[nova][ptg] Team dinner
by Stephen Finucane 05 Nov '19

05 Nov '19

Hi all, Thanks to Alex Xu, we have organized a table for dinner this evening, Wed 6th November at 7pm. Xibo restaurant (Xinjiang food) 3/F 83 Changshu Rd, Jingan Qu, Shanghai, China (near "Changshu Road" or "Jing An Temple" subway stations) 中國上海市静安区常熟路83号 The address is on Google Maps - hopefully it's accurate :) Anyone working on nova is welcome, though I'd ask that you'd note your attendance on the PTG etherpad [1]. Looking forward to seeing everyone, Stephen (stephenfin) [1] https://etherpad.openstack.org/p/nova-shanghai-ptg

2 1

[barbican] PTG Schedule
by Douglas Mendizábal 05 Nov '19

05 Nov '19

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello Barbicaneers! I hope everyone made it to Shanghai safely. Our PTG session will be on Wednesday Nov 6 from 10:30am - 4:30p at the Kilo table. We've set up an etherpad to collect topics to talk about. Please feel free to add any topics you're interested in: https://etherpad.openstack.org/p/barbican-ussuri-ptg Additionally we've reserved a spot for a Team Photo on Thursday at 11am. Hope to see y'all soon! Cheers, - - Douglas Mendizabal -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEwcapj5oGTj2zd3XogB6WFOq/OrcFAl2/iBEACgkQgB6WFOq/ OrfFew/+IjhZe+qRCi/4EmaVEDf7QxJyZDIVUlLsPWHmF98wCdj+GsbzoWUuFfHM sJCpfpVAUjxrIFOEo5uF9WiZhU36G9pgoLd1Y8Kb0/QRIQEQQcKGnlYhCn+jQjbW J2tlDrkU0GBwEBzDt91gM5JCncviY8yT6nhlr/SSLqvZRQnPewerJNyJbYsVh6N2 moXQzfeRjg1SGqR0KVUcDVPe/pE+at8A5ARFCxDiJaOIUTP0qcfKtDXh714bevyi Sw2qgDZHbLHa1nEv3umuYGcrGpKz8Uuj5ju+7oGpPh4hX4pfPxbVDSzu8srfzTui ggvcxFrpZQvdff3Lec1eclxnB+c9Z1tBKYF7pPUVtN3NPfCATkVCSQACYORPZLdh GAnyxiiUXRwzIfOo0b6koa2pRi7ZWoz0DjVzpnl+D7qztUzyiguaj3KDnuTvlfQl iMQev1QHD6fAVvByHgDRj4dyUqUi2+V/DtNZ9w29AX7C+U/afSbNGvygc8yNCtHF vbkw68aPpj5zeB0OTjPQ6N5vsUc6bSXYGnECuGw24untnutvPKR+W9g9VQEUyN1h vhvn0IPHZ9QyBJ0ctpdfA6O9PNsjY/DQNyDeiNGljTIpBjepUmqMTXvycsn8VN/E yY0OL2QFGPhcsK7Q/yeUCzMm1sken2zMg8Bdxt10qbj4GsCMtyQ= =fdMR -----END PGP SIGNATURE-----

2 1

[Neutron] cancel the L3 meeting today
by LIU Yulong 05 Nov '19

05 Nov '19

Hi all, Due to the Summit and PTG next week, I will cancel the Neutron L3 meeting today. So see you there, and travel safe. Thanks. LIU Yulong

1 1

[cinder][osc][docs] openstackclient docs for cinder v2 vs v3
by Eric Fried 05 Nov '19

05 Nov '19

Howdy cinderinos. I've been on a mission to get all of the python-openstackclient command [1] and plugin [2] docs autogenerated rather than hardcoded [3] so you don't have to remember to update two places when you add/change a subcommand option. I'm almost done -- cinder is the last one -- but I want to confirm some odd observations before I dig in. - All of the v3 subcommands are implemented by code in the openstackclient.volume.v2 package. Where there's overlap, the command classes are identical from v2 to v3. However, it appears as though the v2 commands are a *superset* of the v3 commands. Specifically, the following appear in v2 but not v3 [4]: volume_backup_record_export volume_backup_record_import volume_backend_capability_show volume_backend_pool_list volume_host_failover Observations: * v3 has no other 'volume backup record' subcommands, but otherwise has the same 'volume backup' subcommands as v2. * v3 has no 'volume backend' subcommands. * v2 has both 'volume host failover' and 'volume host set', but v3 has only the latter. * It seems suspicious that the "missing" v3 commands comprise a contiguous block under the v2 entry point. So before I go creating a mess of v2-only and v2+v3 documents, I wanted to confirm that the above was actually intentional. - The existing hardcoded documents mention v1 and/or v2, but don't mention v3 at all (e.g. [5]). I want to confirm that it's okay for me to add mention of v3 where appropriate. Thanks, efried [1] https://docs.openstack.org/python-openstackclient/latest/cli/command-list.h… [2] https://docs.openstack.org/python-openstackclient/latest/cli/plugin-command… [3] https://review.opendev.org/#/q/topic:generate-docs+(status:open+OR+status:m… [4] https://opendev.org/openstack/python-openstackclient/src/tag/4.0.0/setup.cf… [5] https://docs.openstack.org/python-openstackclient/train/cli/command-objects…

2 1