- openstack-discuss - lists.openstack.org

[manila] PTG planning etherpad available and Forum topics reminder
by Tom Barron 01 Mar '19

01 Mar '19

Dear Zorillas, I set up an etherpad for PTG planning [1]. Please add topics with your nick or email alongside. As a reminder we also have a Forum topics brainstorming etherpad [2]. The deadline for submitting topics is next week so don't delay! -- Tom Barron [1] https://etherpad.openstack.org/p/manila-denver-train-ptg-planning [2] https://etherpad.openstack.org/p/DEN-train-forum-manila-brainstorming

1 0

[nova] Updates about Detaching/Attaching root volumes
by Zhenyu Zheng 01 Mar '19

01 Mar '19

Hi Nova, I'm working on a blueprint to support Detach/Attach root volumes. The blueprint has been proposed for quite a while since mitaka[1] in that version of proposal, we only talked about instances in shelved_offloaded status. And in Stein[2] the status of stopped was also added. But now we realized that support detach/attach root volume on a stopped instance could be problemastic since the underlying image could change which might invalidate the current host.[3] So Matt and Sean suggested maybe we could just do it for shelved_offloaded instances, and I have updated the patch according to this comment. And I will update the spec latter, so if anyone have thought on this, please let me know. Another thing I wanted to discuss is that in the proposal, we will reset some fields in the root_bdm instead of delete the whole record, among those fields, the tag field could be tricky. My idea was to reset it too. But there also could be cases that the users might think that it would not change[4]. Thoughts, BR, [1] http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/detac… [2] http://specs.openstack.org/openstack/nova-specs/specs/stein/approved/detach… [3] https://review.openstack.org/#/c/614750/34/nova/compute/manager.py@5467 [4] https://review.openstack.org/#/c/614750/37/nova/objects/block_device.py

5 18

[placement] update 19-08
by Chris Dent 01 Mar '19

01 Mar '19

HTML: https://anticdent.org/placement-update-19-08.html Welcome back to the placement update. If I've read the signs correctly, I should now be back to this as a regular thing. Apologies for the gap, I had to attend to some other responsibilities. # Most Important A lot has changed in the past few months, so it's hard to extract out a most important. It will depend on who is reading. Review what's changed for a summary of important stuff. # What's Changed * Placement is now its own official project. Until elections are held (it looks like nominations start this coming Tuesday), Mel is the PTL. * [Setting up storyboard](https://review.openstack.org/#/c/639445/) for placement-related projects is in progress. For the time being we are continuing to use launchpad for most tracking. See a [related email thread](http://lists.openstack.org/pipermail/openstack-discuss/2019-Februar…. * Deleting placement code from nova has been put on hold until Train to make it easier for certain types of upgrades to happen. New installs should prefer the extracted code, as the nova-side is frozen, but the placement side is not. * A large stack of code to remove oslo.versionedobjects from placement has merged. This has resulted in a significant change in performance on the `perfload` test that runs in the gate. While not a complete representation of the entire system, it's enough to say "yeah, that was worth it": A request for allocation candidates that used to take around 2.5 seconds now takes 1.2. That refactoring continues (see below), seeking additional simplifications. * Microversion 1.31 adds `in_tree` and `in_treeN` query parameters to GET /allocation_candidates. This is useful in a variety of nested resource provider scenarios, including the big bandwidth QoS changes that are in progress in nova and neutron. * Placement is now publishing [install docs](https://docs.openstack.org/placement/latest/install/) but it is important to note that those docs have not been validated (as far as I'm aware) by the packagers. That's a thing that needs to happen, presumably by the packagers. * os-resource-classes 0.3.0 has been [released](https://pypi.org/p/os-resource-classes) with a `normalize_name` function. * There are some pending specs from nova which are primarily placement feature specs. We'll continue with those as is (see below), but come the next cycle the plan is to manage specs in the placement repo, not have a separate repo, and not have separate spec cores. # Specs/Blueprints/Features ## Near to Done * [Filter Allocation Candidates by Provider Tree](http://specs.openstack.org/openstack/nova-specs/specs/stein/approved/… has been mostly completed by Tetsuro, but there's a [pending update to the spec](https://review.openstack.org/639033). ## Not yet Done * [Support filtering by forbidden aggregate membership](http://specs.openstack.org/openstack/nova-specs/specs/stein/app… * [Support any traits in allocation_candidates query](http://specs.openstack.org/openstack/nova-specs/specs/stein/approved… * [Support mixing required traits with any traits](http://specs.openstack.org/openstack/nova-specs/specs/stein/approve… ## Not yet Approved * [Update alloc-candidates-in-tree](https://review.openstack.org/#/c/639033/) updates the in-tree spec above to reflect what was learned while doing the actual implementation. Notably how numbered `in_tree` parameters impact results. * [Resource provider - request group mapping in allocation candidate](https://review.openstack.org/#/c/597601/) has had a recent resurgence in attention. # Bugs * Placement related [bugs not yet in progress](https://goo.gl/TgiPXb): 15. * [In progress placement bugs](https://goo.gl/vzGGDQ) 17. # osc-placement osc-placement is currently behind by 14 microversions. Code for 1.18 is [under review](https://review.openstack.org/#/c/639738/). # Main Themes This section now overlaps a bit with the Specs/Features bit above. This will settle out with a bit more clarity as we move along. ## Nested * Reshaper handing in nova keeps exposing additional things that need to be remembered on the nova-side, so there are a few patches remaining related to [vgpu reshaping](https://review.openstack.org/#/q/topic:bp/reshape-provider-tree+… but it is mostly ready. * The bandwidth-resource-provider topic has merged a vast amount of code but there is still [plenty left](https://review.openstack.org/#/q/topic:bp/bandwidth-resource-provider). Related to all this nested stuff: The complex hardware models that drove the development of the nested resource provider system are challenging to test. The cloud hardware provided to OpenStack infrastructure does not expose the hardware that would allow real integration tests. If anyone reading this is in a position to provide third party CI with fancy hardware for NUMA, NFV, FPGA, and GPU related integration testing with nova, there's a significant need for that. ## Refactoring (I think refactoring should be a constant theme. To reflect that, I'm going to have a section here. Editorial privilege or something.) There's a collection of patches in progress, currently under the topic [scrub-Lists](https://review.openstack.org/#/q/topic:scrub-Lists) that is a follow up to the patches that removed oslo versioned objects. That work pointed out some opportunities to DRY-up the List classes (e.g., UsageList) to remove some duplication and simplify. Then, after looking at that, it became clear that entirely removing the List classes, in favor of using python native lists, would further simplify the code. Apart from the previously mentioned performance and simplicity benefits of these changes, it's also managed to expose and fix a few bugs, simple because we were looking at things and moving them around. If you pick up rocks, you can see the bugs and squash them. If you don't, they breed. # Other Placement * <https://review.openstack.org/#/q/topic:improve-debug-log> A series of improvements leading to a better debug log when retrieving allocation candidates. * <https://review.openstack.org/#/c/639628/> Docs: extract testing info to own sub-page * <https://review.openstack.org/#/q/topic:cd/gabbi-tempest-job> Gabbi-based integration tests of placement. These recently found a bug that none of the functional, grenade, nor tempest tests did. * <https://review.openstack.org/#/c/619050/> Optionally migrate database at service startup (so you don't have to run `placement-manage db sync` if you don't want to). * <https://review.openstack.org/#/c/630216/> Add a vision-reflection (of the Technical Vision doc). # Other Service Users ## Nova See also the several links above for more nova changes. Also, I'm a bit behind on my tracking in this area, so there is likely plenty of other stuff too. This will improve over time. * <https://review.openstack.org/538498> Convert driver supported capabilities to compute node provider traits * <https://review.openstack.org/621494> Add descriptions of numbered resource classes and traits * <https://review.openstack.org/636412> Make move_allocations handle empty source allocations (Part of a series on cross-cell resize) * <https://review.openstack.org/#/q/topic:bp/count-quota-usage-from-placement> Using placement (from nova) for counting (some of) quota. ## Not Nova * <https://review.openstack.org/#/q/topic:tripleo-nova-placement-removal> * <https://review.openstack.org/#/q/topic:tripleo-placement-extraction> * <https://review.openstack.org/#/q/topic:minimum-bandwidth-allocation-placeme…> Neutron side of minimum bandwidth. * <https://review.openstack.org/#/q/topic:puppet-placement-extraction> * <https://review.openstack.org/#/q/bp/no-affinity-instance-reservation> Blazar reservation handling, including some manipulation of inventory in placement. * <https://review.openstack.org/633204> Blazar: Retry on inventory update conflict # End Though this is long, it doesn't really bring us fully up to date. If something is missing that you think is important please let me know. Once I'm back in the flow it should become increasingly complete. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent

2 1

Re: [scientific-sig] IRC meeting 1100 UTC: Edge computing and Scientific use cases
by Bogdan Dobrelya 01 Mar '19

01 Mar '19

On 27.02.2019 14:33, Adrien Lebre wrote: > Hi, > > Getting a look to this post could also make sense: > http://beyondtheclouds.github.io/blog/openstack/cockroachdb/2018/06/04/eval… Thank you. The latter links give a lot of insights and sources of information to be consumed by me in a bunch of years perhaps :) In case someone would love to collaborate for the aforementioned positioning paper to help to make it into some future events hopefully, please consider contributing your ideas directly to the draft [0]. By the way, I don't mind if anyone's willing to handover it and being a main presenter/driver of it for the future events (once/if the paper is ever finished ofc). [0] https://github.com/bogdando/papers-ieee/tree/master/ICFC-2019/LaTeX > > Best regards, > Adrien > PS: Thanks Pierre. > -- > Prof. IMT Atlantique / Inria / LS2N > Head of the STACK Research Group > www.emn.fr/x-info/alebre08 <http://www.emn.fr/x-info/alebre08> > >> On 27 Feb 2019, at 14:18, Pierre Riteau <pierre(a)stackhpc.com >> <mailto:pierre@stackhpc.com>> wrote: >> >> Hello Bogdan, >> >> In the context of data management in edge computing, you may be >> interested by research done by the Discovery Initiative: >> http://beyondtheclouds.github.io/publications.html >> I am adding Adrien Lèbre to the thread, he may be able to point you to >> the publications most relevant to your whitepaper draft. >> >> Best wishes, >> Pierre Riteau >> >> On Wed, 27 Feb 2019 at 10:21, Bogdan Dobrelya <bdobreli(a)redhat.com> wrote: >>> >>> On 27.02.2019 10:51, Stig Telfer wrote: >>>> Hi All - >>>> >>>> We have an IRC meeting at 1100 UTC (about an hour’s time) in channel >>>> #openstack-meeting. Everyone is welcome. >>>> >>>> Today’s agenda is here: >>>> >>>> https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_February_27th_20… >>>> >>>> Today’s main event is that we have Ildikó Vancsa joining us to talk >>>> about edge computing use cases and find overlap with Scientific SIG >>>> use cases. >>>> >>>> Cheers, >>>> Stig >>>> >>>> >>> >>> Edge computing is a *great* topic for lot of future research work indeed. >>> Thank you for this useful announcement, appreciated! >>> I'll attend at least the first half of it, and in case I'll miss the >>> agenda item that I placed at the end of the list, please be kind to >>> review that research request with my absence as well. Thanks! >>> >>> >>> -- >>> Best regards, >>> Bogdan Dobrelya, >>> Irc #bogdando >>> > -- Best regards, Bogdan Dobrelya, Irc #bogdando

1 0

[nova][operators] Upcoming change breaking out of tree virt drivers in Nova
by Jim Rollenhagen 01 Mar '19

01 Mar '19

Hi, There's a nasty race condition when using the ironic driver with Nova.[0] The patch to fix it[1] adds a use_cache argument to the virt driver's get_info method. This is caught and handled so that we can backport the fix, but a follow-up change on master[2] removes the error handling. You'll need to update your out-of-tree virt drivers, if you have one, to add this argument to work with the Stein release. And if your virt driver caches instance information, it would be worth skipping or refreshing that cache when use_cache=True is passed. Sorry for the inconvenience. :) [0] https://bugs.launchpad.net/nova/+bug/1815791 [1] https://review.openstack.org/#/c/636699 [2] https://review.openstack.org/#/c/640043 // jim

1 0

[nova] snapshots default to qcow2
by Eugen Block 01 Mar '19

01 Mar '19

Hi all, this still seems to be related to a thread I created last year [1]. There was a part left which I didn't understand and couldn't reproduce, now a colleague seems to have created an instance where I can reproduce the issue. I also found [2] and [3] referring to this, both were marked as invalid, but this is not resolved IMO. I'll try to explain: We have a volume backed instance, its base image was in raw format (all non-raw images were updated after I learned about the consequences): # nova show output | image | Attempt to boot from volume - no image supplied | | os-extended-volumes:volumes_attached | [{"id": "ccca4607-3296-472b-a521-fdc3e76451a2"}] | The respective volume has the following meta-data: # cinder show output | volume_image_metadata | {'container_format': 'bare', 'min_ram': '0', 'disk_format': 'raw', 'image_name': '###_SLES12-SP3-JeOS', 'image_id': '9ed2b16e-c939-42e7-9e09-4a732caf2449', 'checksum': 'e3282f363c62194c1e4dea1bacccdda6', 'min_disk': '0', 'size': '1084227584'} | I also double-checked the (exported) volume with qemu-img info that it's indeed a raw formatted volume. Now when I create a nova snapshot either with Horizon or the CLI it results in a qcow2 image: control:~ # nova image-create d1d4e694-3920-47e6-8338-060015c4eb71 test-image control1:~ # openstack image show 7de2fd37-1124-4382-9fb4-66fde8c92676 | grep disk_format | disk_format | qcow2 I changed 'snapshot_image_format' from '<None>' to 'raw' in nova.conf on both control and compute node, although nova states to use the source image's format: # If set, this decides what format is used when sending the snapshot to the # image service. # If not set, defaults to same type as source image. Obviously, this doesn't work as expected, so the question is which component is responsible for the qcow2 format? I couldn't find related cinder config options, could [2] be a valid bug report after all? Thanks for any thoughts on this! Regards, Eugen [1] https://www.mail-archive.com/openstack@lists.openstack.org/msg20935.html [2] https://bugs.launchpad.net/nova/+bug/1332558 [3] https://ask.openstack.org/en/question/65451/snapshot-defaults-to-qcow2/

1 1

[kolla] - Ceph bootstrap monitor is not properly configured
by Tyler Bishop 01 Mar '19

01 Mar '19

Trying to deploy a new cluster using the bootstrapping for ceph but running into issues with the admin keys being incorrectly deployed. Successful ansible deploy up until: TASK [ceph : Getting ceph mgr keyring] ****************************************************************************************************************************** failed: [osctlr.home.visualbits.net -> osctlr.home.visualbits.net] (item= osctlr.home.visualbits.net) => {"changed": false, "item": " osctlr.home.visualbits.net", "msg": "Failed to call command: ['docker', 'exec', 'ceph_mon', 'ceph', '--format', 'json', 'auth', 'get-or-create', ' mgr.osctlr.home.visualbits.net', 'mds', 'allow *', 'mon', 'allow profile mgr', 'osd', 'allow *'] returncode: 1 output: stdout: \"\", stderr: \"[errno 1] error connecting to the cluster\n\""} Errors in log from ceph: TASK [ceph : Getting ceph mgr keyring] ****************************************************************************************************************************** failed: [osctlr.home.visualbits.net -> osctlr.home.visualbits.net] (item= osctlr.home.visualbits.net) => {"changed": false, "item": " osctlr.home.visualbits.net", "msg": "Failed to call command: ['docker', 'exec', 'ceph_mon', 'ceph', '--format', 'json', 'auth', 'get-or-create', ' mgr.osctlr.home.visualbits.net', 'mds', 'allow *', 'mon', 'allow profile mgr', 'osd', 'allow *'] returncode: 1 output: stdout: \"\", stderr: \"[errno 1] error connecting to the cluster\n\""} keyrings look proper: (openstack) [root@osctlr ~]# md5sum /etc/kolla/ceph-mon/ceph.client.admin.keyring 4658c01282c791bce9c75678df9e21c9 /etc/kolla/ceph-mon/ceph.client.admin.keyring (openstack) [root@osctlr ~]# md5sum /var/lib/docker/volumes/ceph_mon_config/_data/ceph.client.admin.keyring 4658c01282c791bce9c75678df9e21c9 /var/lib/docker/volumes/ceph_mon_config/_data/ceph.client.admin.keyring I've removed the docker container, volume and kolla config directories multple times with the same error. I can't even run ceph status from the container bash itself. Any ideas?

2 2

RE: [nova] NUMA live migration - mostly how it's tested
by Zhong, Luyao 01 Mar '19

01 Mar '19

Hi all, There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details. (I remove the original mail message otherwise the email message body will be so big that it will be held)

1 0

nimble PyPI status
by Devin Shanahan 28 Feb '19

28 Feb '19

Dear maintainer of nimble: I am reaching out since you are listed as a maintainer of nimble <https://pypi.org/project/nimble/> on PyPI. I am currently working on a project and we are exploring releasing our distribution under the same name. Our project is a library of data science and machine learning tools. It has been under development and used internally at our company for 7 years, and we are now considering providing it as an open-source project. Nimble is a metaphor for our team's vision for the project, and we would very much prefer to use that exact name on PyPI over a similar, but more complicated name, that could cause confusion. Since PyPI shows that nimble has not had a new release or been updated since August 10, 2016, I would like to know if you would be willing to relinquish the name, nimble, so that we could provide our project in that namespace on PyPI. It's a simple process and if you are willing to do this, we would be extremely grateful. I also completely understand if this is not something that you are willing to do, or if I have misinterpreted the status of nimble and it is in fact in active development. I would really appreciate hearing your reply either way, whether or not you would be willing to consider transferring ownership of nimble over to us. Thank you very much for your time. Devin

3 2

Router not allowing traffic in reverse
by John Carew 28 Feb '19

28 Feb '19

I have setup OpenStack with OVS. I have a single Hyper-V server running the controller and three CentOS instances(10.0.0.x) on a private subnet. I created a router in OpenStack with SNAT disabled, as I only want it to route traffic between the private subnet(10.0.0.x) and the external subnet(172.16.1.x)/internet. All of the instances can ping each other along with the external network(172.16.1.x). From the external network, I can ping the interface of the ovs router on the external network. I can not though ping inside the private network. A trace route stops at the IP of the OVS router. With wireshark, I do not see anything coming from the external pc’s IP. If I trace route it, I see packets making all the way to the OVS router and then stop. Since I can ping one way, and not the other; I believe there is something in the router/OVS that is stopping the packets to route into the private subnet. What do I need to look at? (I have disabled all firewalls on all OSes involved.) Here is a sudo design of the setup. Does respond to ping VM OVS Router priv int OVS Router ext int External PC 10.0.0.14 -> 10.0.0.1 -> 172.16.1.1 -> 172.16.1.10 Does not respond to ping External PC OVS Router ext int OVS Router priv int VM 172.16.1.10 -> 172.16.1.1 -> 10.0.0.1 -> 10.0.0.14

1 0