From dms at danplanet.com Fri Mar 1 00:02:57 2019 From: dms at danplanet.com (Dan Smith) Date: Thu, 28 Feb 2019 16:02:57 -0800 Subject: [nova] Updates about Detaching/Attaching root volumes In-Reply-To: (Matt Riedemann's message of "Thu, 28 Feb 2019 17:29:20 -0600") References: <19d57159-69b3-0b4b-cec8-2018fb672d41@gmail.com> <59c11c15-a139-dd05-1f97-fe9315585dcd@gmail.com> <015dbbc1-f0da-34db-df9a-f2769e42a970@gmail.com> Message-ID: > We talked about this in the nova meeting today: > > http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-02-28-21.00.log.html#l-95 > > It sounds like we might be leaning to option 4 for a couple of reasons: > > a) It's getting late in stein to be lifting the restriction on > attaching a volume with a tag to a shelved offloaded server and > munging that in with the root attach/detach volume API change. > > b) As we got talking I noted that for the same reason as a tag, you > currently cannot attach a multi-attach volume to a shelved offloaded > server, because we don't make the "reserve_block_device_name" RPC call > to the compute service to determine if the driver supports it. > > c) Once we have code in the API to properly translate tags/multiattach > requests to the scheduler (based on [1]) so that we know when we > unshelve that we'll pick a host that supports device tags and > multiattach volumes, we could lift the restriction in the API with a > new microversion. > > If we go this route, it means the API reference for the root volume > detach/attach change needs to mention that the tag will be reset and > cannot be set again on attach of the root volume (at least until we > lift that restriction). So, first off, sorry I got pulled away from the meeting and am now showing up here with alternatives. My opinion is that it sucks to have to choose between "detaching my root volume" and "can have tags". I think I understand the reasoning for doing the RPC call when attaching to a server on a host to make sure we can honor the tag you want, but I think the shelved case is a bit weird and shouldn't really be held to the same standard. It's always a valid outcome when trying to unshelve a host to say "sorry, but there's no room for that anymore" or even "sorry, you've had this shelved since our last hardware refresh and your instance no longer fits with the group." Thus, attaching a volume with a tag to a shelved instance making it either unbootable or silently ignore-able seems like not the worst thing, especially if the user can just detach and re-attach without a tag to get out of that situation. Further, I also think that wiping the tag on detach is the wrong thing to do. The point of tags for both volumes and network devices was to indicate function, not content. Things like "this is my fast SSD-backed database store" or "this is my slow rotating-rust backup disk" or "this is the fast unmetered internal network." They don't really make sense to say "this is ubuntu 16.04" because ... the image (metadata) tells you that. Thus, I would expect someone with a root volume to use a tag like "root" or "boot" or "where the virus lives". Detaching the root volume doesn't likely change that function. So, IMHO, we should select a path that does not make the user choose between the "tags or detach" dichotomy above. Some options for that are: 1. Don't wipe the tag on detach. On re-attach, let them specify a tag if there is already one set on the BDM. Clearly that means tags were supported before, so a reasonable guess that they still are. 2. Specifically for the above case, either (a) require they pass in the tag they want on the new volume if they want to keep/change it or (b) assume it stays the same unless they specify a null tag to remove it. 3. Always let them specify a tag on attach, and if that means they're not un-shelve-able because no compute nodes support tags, then that's the same case as if you show up with tag-having boot request. Let them de-and-re-attach the volume to wipe the tag and then unshelve. 4. Always let them specify a tag on attach and just not freak out on unshelve if we end up picking a host with no tag support. Personally, I think 2a or 2b are totally fine, but they might not be as obvious as #3. Either will avoid the user having to make a hard life decision about whether they're going to lose their volume tag forever because they need to detach their root volume. Since someone doing such an operation on an instance is probably caring for a loved pet, that's a sucky position to be in, especially if you're reading the docs after having done so only to realize you've already lost it forever when you did your detach. --Dan From mriedemos at gmail.com Fri Mar 1 00:24:08 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Thu, 28 Feb 2019 18:24:08 -0600 Subject: [nova] Updates about Detaching/Attaching root volumes In-Reply-To: References: <19d57159-69b3-0b4b-cec8-2018fb672d41@gmail.com> <59c11c15-a139-dd05-1f97-fe9315585dcd@gmail.com> <015dbbc1-f0da-34db-df9a-f2769e42a970@gmail.com> Message-ID: On 2/28/2019 6:02 PM, Dan Smith wrote: > So, IMHO, we should select a path that does not make the user choose > between the "tags or detach" dichotomy above. Some options for that are: I've got a few questions for clarification below. > > 1. Don't wipe the tag on detach. On re-attach, let them specify a tag if > there is already one set on the BDM. Clearly that means tags were > supported before, so a reasonable guess that they still are. > > 2. Specifically for the above case, either (a) require they pass in the > tag they want on the new volume if they want to keep/change it or (b) > assume it stays the same unless they specify a null tag to remove it. Is this (a) *or* (b)? If I create a server with a device tag on the root bdm, then detach the root volume and am happy with the tag that was there, so on attach of a new root volume I don't specify a new tag - do I lose it because I didn't specify the same tag over again? I would expect that if I had a tag, I can keep it without specifying anything different, or change it by specifying some new (new tag or null it out). If I didn't have a tag on the root bdm when I detached, I can't specify a new one because we don't know if we can honor it (existing behavior for attaching volumes to shelved instances). Does what I just described equate to option 1 above? If so, then I'd vote for option 1 here. > > 3. Always let them specify a tag on attach, and if that means they're > not un-shelve-able because no compute nodes support tags, then that's > the same case as if you show up with tag-having boot request. Let them > de-and-re-attach the volume to wipe the tag and then unshelve. So this is option 4 but fail the unshelve if the compute driver they land on doesn't support tags, thus making it more restrictive than option 4. I'm not crazy about this one because if we just start checking device tags on unshelve always we will arguably change behavior - granted the behavior we have today is you get lucky and land on a host that supports tags (this is not an unreasonable assumption if you are using a cloud with homogeneous virt drivers which I expect most do) or you land on a host that doesn't and we don't honor the tags, so sorry. > > 4. Always let them specify a tag on attach and just not freak out on > unshelve if we end up picking a host with no tag support. This was essentially my first option: "1: lift the restriction in this same microversion so that you can attach volumes to a shelved offloaded server and specify a tag (despite bug 1817927 which we can fix later)" -- Thanks, Matt From dms at danplanet.com Fri Mar 1 00:32:54 2019 From: dms at danplanet.com (Dan Smith) Date: Thu, 28 Feb 2019 16:32:54 -0800 Subject: [nova] Updates about Detaching/Attaching root volumes In-Reply-To: (Matt Riedemann's message of "Thu, 28 Feb 2019 18:24:08 -0600") References: <19d57159-69b3-0b4b-cec8-2018fb672d41@gmail.com> <59c11c15-a139-dd05-1f97-fe9315585dcd@gmail.com> <015dbbc1-f0da-34db-df9a-f2769e42a970@gmail.com> Message-ID: >> 1. Don't wipe the tag on detach. On re-attach, let them specify a tag if >> there is already one set on the BDM. Clearly that means tags were >> supported before, so a reasonable guess that they still are. >> >> 2. Specifically for the above case, either (a) require they pass in the >> tag they want on the new volume if they want to keep/change it or (b) >> assume it stays the same unless they specify a null tag to remove it. > > Is this (a) *or* (b)? Yes because it describes complementary behaviors. > If I create a server with a device tag on the root bdm, then detach > the root volume and am happy with the tag that was there, so on attach > of a new root volume I don't specify a new tag - do I lose it because > I didn't specify the same tag over again? I would expect that if I had > a tag, I can keep it without specifying anything different, or change > it by specifying some new (new tag or null it out). If I didn't have a > tag on the root bdm when I detached, I can't specify a new one because > we don't know if we can honor it (existing behavior for attaching > volumes to shelved instances). Does what I just described equate to > option 1 above? If so, then I'd vote for option 1 here. No, what you describe is 2b and that's fine with me. Maybe I should check my outline style, but 1 is the general idea, 2a and 2b are each behaviors we'd choose from when implementing 1. >> 3. Always let them specify a tag on attach, and if that means they're >> not un-shelve-able because no compute nodes support tags, then that's >> the same case as if you show up with tag-having boot request. Let them >> de-and-re-attach the volume to wipe the tag and then unshelve. > > So this is option 4 but fail the unshelve if the compute driver they > land on doesn't support tags, thus making it more restrictive than > option 4. I'm not crazy about this one because if we just start > checking device tags on unshelve always we will arguably change > behavior - > granted the behavior we have today is you get lucky and land on a host > that supports tags (this is not an unreasonable assumption if you are > using a cloud with homogeneous virt drivers which I expect most do) or > you land on a host that doesn't and we don't honor the tags, so sorry. Sure, that's fine. it seems like the most obvious option to me, but if it means we'd be tweaking existing behavior, then it makes sense to avoid it. --Dan From zhengzhenyulixi at gmail.com Fri Mar 1 01:54:45 2019 From: zhengzhenyulixi at gmail.com (Zhenyu Zheng) Date: Fri, 1 Mar 2019 09:54:45 +0800 Subject: [nova] Updates about Detaching/Attaching root volumes In-Reply-To: References: <19d57159-69b3-0b4b-cec8-2018fb672d41@gmail.com> <59c11c15-a139-dd05-1f97-fe9315585dcd@gmail.com> <015dbbc1-f0da-34db-df9a-f2769e42a970@gmail.com> Message-ID: Thanks alot for all the suggestions, so as a summary: 1. do not wipe out the tag when detach 2. free the limit on do not allow attach volume with tag for shelved_offloaded instance if it is a root volume(we can check whether ``is_root`` is True) 2a. if user does not provide any tag, we keep the old tag 2b. if user provide a new tag, we update the new tag 2c. provide a way for user to indicate that I want to null out the previous tag is my understanding correct? BR, On Fri, Mar 1, 2019 at 8:32 AM Dan Smith wrote: > >> 1. Don't wipe the tag on detach. On re-attach, let them specify a tag if > >> there is already one set on the BDM. Clearly that means tags were > >> supported before, so a reasonable guess that they still are. > >> > >> 2. Specifically for the above case, either (a) require they pass in the > >> tag they want on the new volume if they want to keep/change it or (b) > >> assume it stays the same unless they specify a null tag to remove it. > > > > Is this (a) *or* (b)? > > Yes because it describes complementary behaviors. > > > If I create a server with a device tag on the root bdm, then detach > > the root volume and am happy with the tag that was there, so on attach > > of a new root volume I don't specify a new tag - do I lose it because > > I didn't specify the same tag over again? I would expect that if I had > > a tag, I can keep it without specifying anything different, or change > > it by specifying some new (new tag or null it out). If I didn't have a > > tag on the root bdm when I detached, I can't specify a new one because > > we don't know if we can honor it (existing behavior for attaching > > volumes to shelved instances). Does what I just described equate to > > option 1 above? If so, then I'd vote for option 1 here. > > No, what you describe is 2b and that's fine with me. Maybe I should > check my outline style, but 1 is the general idea, 2a and 2b are each > behaviors we'd choose from when implementing 1. > > >> 3. Always let them specify a tag on attach, and if that means they're > >> not un-shelve-able because no compute nodes support tags, then that's > >> the same case as if you show up with tag-having boot request. Let them > >> de-and-re-attach the volume to wipe the tag and then unshelve. > > > > So this is option 4 but fail the unshelve if the compute driver they > > land on doesn't support tags, thus making it more restrictive than > > option 4. I'm not crazy about this one because if we just start > > checking device tags on unshelve always we will arguably change > > behavior - > > granted the behavior we have today is you get lucky and land on a host > > that supports tags (this is not an unreasonable assumption if you are > > using a cloud with homogeneous virt drivers which I expect most do) or > > you land on a host that doesn't and we don't honor the tags, so sorry. > > Sure, that's fine. it seems like the most obvious option to me, but if > it means we'd be tweaking existing behavior, then it makes sense to > avoid it. > > --Dan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From manuel.sb at garvan.org.au Fri Mar 1 01:59:24 2019 From: manuel.sb at garvan.org.au (Manuel Sopena Ballesteros) Date: Fri, 1 Mar 2019 01:59:24 +0000 Subject: sriov bonding In-Reply-To: <553f272f8be80ef6c29b0cc47a548e52bd7a1c2a.camel@redhat.com> References: <9D8A2486E35F0941A60430473E29F15B017E860EDF@MXDB2.ad.garvan.unsw.edu.au> <553f272f8be80ef6c29b0cc47a548e52bd7a1c2a.camel@redhat.com> Message-ID: <9D8A2486E35F0941A60430473E29F15B017E863CAD@MXDB2.ad.garvan.unsw.edu.au> Ok, My nic is mellanox connect-4 lx so I was thinking this: Create bonding at the PF level use bond interface as br-ext Do ovs offload That would mean I don't need to change my switch configuration and keep my LACP. Does it sounds feasible? Second question: OVS offload needs Linux Kernel >= 4.13 can I use CentOS Linux release 7.5.1804 (Core) with kernel 3.10.0-862.el7.x86_64? Thank you Manuel -----Original Message----- From: Sean Mooney [mailto:smooney at redhat.com] Sent: Wednesday, February 27, 2019 9:36 PM To: Bence Romsics; openstack at lists.openstack.org Subject: Re: sriov bonding On Wed, 2019-02-27 at 08:38 +0100, Bence Romsics wrote: > Hi, > > On Wed, Feb 27, 2019 at 8:00 AM Manuel Sopena Ballesteros > wrote: > > Is there a documentation that explains how to setup bonding on SR-IOV neutron? > > Not right now to my knowledge, but I remember seeing effort to design > and introduce this feature. I think there may have been multiple > rounds of design already, this is maybe the last one that's still > ongoing: > > Neutron side: > https://bugs.launchpad.net/neutron/+bug/1809037 > > Nova side: > https://blueprints.launchpad.net/nova/+spec/schedule-vm-nics-to-different-pf > https://blueprints.launchpad.net/nova/+spec/sriov-bond most of the previous attempts have not proceeded as they have tried to hide the bonding from nova's and neutron's data models vai configs or opaque strings. for bonding to really be supported at the openstack level we will need to take a similar a approch to trunk ports. e.g. we create a logical bond port and a set of bond peer ports at the neutron api level then we attach the bond port to the vm. currently im not aware of any proposal that really has tracktion. you can manually create a bond in the guest but you cannot today guarentee that the vf will come from different pfs. one of the reason i think we need to go the logical bond port direction is that it will allow us to constuct resouce requests using the fucntionality that is being added for bandwidth based schduleing. that will make expressing affinity and anti affinty simpler useing the request groups syntax. i personally have stopped trying to add bond support untill that bandwith based schduling effort is finished. > > Hope that helps, > Bence > NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. From johnwcarew at gmail.com Fri Mar 1 00:45:46 2019 From: johnwcarew at gmail.com (John Carew) Date: Thu, 28 Feb 2019 18:45:46 -0600 Subject: Router not allowing traffic in reverse Message-ID: <42ADBCE5-1869-4C64-990A-5C3EBFEABD4F@gmail.com> I have setup OpenStack with OVS. I have a single Hyper-V server running the controller and three CentOS instances(10.0.0.x) on a private subnet. I created a router in OpenStack with SNAT disabled, as I only want it to route traffic between the private subnet(10.0.0.x) and the external subnet(172.16.1.x)/internet. All of the instances can ping each other along with the external network(172.16.1.x). From the external network, I can ping the interface of the ovs router on the external network. I can not though ping inside the private network. A trace route stops at the IP of the OVS router. With wireshark, I do not see anything coming from the external pc’s IP. If I trace route it, I see packets making all the way to the OVS router and then stop. Since I can ping one way, and not the other; I believe there is something in the router/OVS that is stopping the packets to route into the private subnet. What do I need to look at? (I have disabled all firewalls on all OSes involved.) Here is a sudo design of the setup. Does respond to ping VM OVS Router priv int OVS Router ext int External PC 10.0.0.14 -> 10.0.0.1 -> 172.16.1.1 -> 172.16.1.10 Does not respond to ping External PC OVS Router ext int OVS Router priv int VM 172.16.1.10 -> 172.16.1.1 -> 10.0.0.1 -> 10.0.0.14 -------------- next part -------------- An HTML attachment was scrubbed... URL: From niu.zglinux at gmail.com Fri Mar 1 01:24:54 2019 From: niu.zglinux at gmail.com (Zhenguo Niu) Date: Fri, 1 Mar 2019 09:24:54 +0800 Subject: nimble PyPI status In-Reply-To: References: Message-ID: The 'nimble' is not used anymore, we have renamed the project from nimble to mogan long time ago. Please feel free to do the swap, sorry for the inconvenience! On Fri, Mar 1, 2019 at 7:01 AM Clark Boylan wrote: > On Thu, Feb 28, 2019, at 2:27 PM, Devin Shanahan wrote: > > Dear maintainer of nimble: > > > > I am reaching out since you are listed as a maintainer of nimble > > on PyPI. I am currently working on a > > project and we are exploring releasing our distribution under the same > > name. > > > > Our project is a library of data science and machine learning tools. It > > has been under development and used internally at our company for 7 > > years, and we are now considering providing it as an open-source > > project. Nimble is a metaphor for our team's vision for the project, > > and we would very much prefer to use that exact name on PyPI over a > > similar, but more complicated name, that could cause confusion. > > > > Since PyPI shows that nimble has not had a new release or been updated > > since August 10, 2016, I would like to know if you would be willing to > > relinquish the name, nimble, so that we could provide our project in > > that namespace on PyPI. It's a simple process and if you are willing to > > do this, we would be extremely grateful. I also completely understand > > if this is not something that you are willing to do, or if I have > > misinterpreted the status of nimble and it is in fact in active > > development. > > > > I would really appreciate hearing your reply either way, whether or not > > you would be willing to consider transferring ownership of nimble over > > to us. > > > > Thank you very much for your time. > > > > Devin > > I've CC'd the original creator of this project on our end to do our best > to make sure they see this request. I think it would be best if Zhenguo > Niu can respond with an answer and if they don't have a problem with this I > can help facilitate the swap if necessary. > > If we aren't able to get a response within a reasonable amount of time > (say 7-10 days?) we can review it without their input and go from there. > > Hope this helps, > Clark > -- Best Regards, Zhenguo Niu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mriedemos at gmail.com Fri Mar 1 02:36:34 2019 From: mriedemos at gmail.com (Matt Riedemann) Date: Thu, 28 Feb 2019 20:36:34 -0600 Subject: [nova] Updates about Detaching/Attaching root volumes In-Reply-To: References: <19d57159-69b3-0b4b-cec8-2018fb672d41@gmail.com> <59c11c15-a139-dd05-1f97-fe9315585dcd@gmail.com> <015dbbc1-f0da-34db-df9a-f2769e42a970@gmail.com> Message-ID: <58b438db-fd9a-ad4e-6187-739e2952ba25@gmail.com> On 2/28/2019 7:54 PM, Zhenyu Zheng wrote: > Thanks alot for all the suggestions, so as a summary: > 1. do not wipe out the tag when detach > 2. free the limit on do not allow attach volume with tag for > shelved_offloaded instance if it is a root volume(we can check whether > ``is_root`` is True) >     2a. if user does not provide any tag, we keep the old tag >     2b. if user provide a new tag, we update the new tag >     2c. provide a way for user to indicate that I want to null out the > previous tag > > is my understanding correct? I think so. For 2c, that gets complicated because the schema for the tag is not nullable [1][2]. So then you're left with leave it or overwrite the tag you had, but you can't go from no tag to tag or vice-versa (without a schema change....). Also one final note here, but multiattach volumes are in the same conundrum, meaning they aren't allowed to attach to a shelved offloaded instance, but if we draw on Dan's use case, picking between detaching a root volume and not being able to attach another multiattach root volume, isn't a good option. So if we're going to do this for tags, I think we ought to do it for multiattach volumes as well. In other words, if the root volume was multiattach before it was detached, then the new root volume can be multiattach, but you can't go from non-multiattach to multiattach while shelved offloaded. The crappy thing about multiattach is it's not a clear attribute on the BDM object, it's buried in the BDM.connection_info [3], but we can still sort that out with a helper method (but it means you can't reset the connection_info when detaching the root volume). [1] https://github.com/openstack/nova/blob/a8c065dea946599a1b07d003cd21409c4cd58df0/nova/api/openstack/compute/schemas/volumes.py#L88 [2] https://github.com/openstack/nova/blob/a8c065dea946599a1b07d003cd21409c4cd58df0/nova/api/validation/parameter_types.py#L498 [3] https://github.com/openstack/nova/blob/a8c065dea946599a1b07d003cd21409c4cd58df0/nova/virt/block_device.py#L549 -- Thanks, Matt From cheng1.li at intel.com Fri Mar 1 05:29:00 2019 From: cheng1.li at intel.com (Li, Cheng1) Date: Fri, 1 Mar 2019 05:29:00 +0000 Subject: [openstack-helm] support per-host auto_bridge_add Message-ID: Hi Pete, Sorry for disturbing you, but I really would like to merge this patch ASAP. I have updated the patch https://github.com/stackanetes/kubernetes-entrypoint Could you please help review it again? Thanks. Thanks, Cheng -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at stackhpc.com Fri Mar 1 09:14:10 2019 From: mark at stackhpc.com (Mark Goddard) Date: Fri, 1 Mar 2019 09:14:10 +0000 Subject: [kolla] - Ceph bootstrap monitor is not properly configured In-Reply-To: References: Message-ID: Glad you worked it out Tyler, thanks in advance for pushing the fix. Mark On Thu, 28 Feb 2019 at 19:03, Tyler Bishop wrote: > This ended up being a cleanup issue in the management of the tmp file. I > will be submitting a PR for it. > > > > On Wed, Feb 27, 2019 at 2:30 PM Tyler Bishop < > tyler.bishop at beyondhosting.net> wrote: > >> Trying to deploy a new cluster using the bootstrapping for ceph but >> running into issues with the admin keys being incorrectly deployed. >> >> Successful ansible deploy up until: >> >> TASK [ceph : Getting ceph mgr keyring] >> ****************************************************************************************************************************** >> failed: [osctlr.home.visualbits.net -> osctlr.home.visualbits.net] (item= >> osctlr.home.visualbits.net) => {"changed": false, "item": " >> osctlr.home.visualbits.net", "msg": "Failed to call command: ['docker', >> 'exec', 'ceph_mon', 'ceph', '--format', 'json', 'auth', 'get-or-create', ' >> mgr.osctlr.home.visualbits.net', 'mds', 'allow *', 'mon', 'allow profile >> mgr', 'osd', 'allow *'] returncode: 1 output: stdout: \"\", stderr: >> \"[errno 1] error connecting to the cluster\n\""} >> >> >> Errors in log from ceph: >> TASK [ceph : Getting ceph mgr keyring] >> ****************************************************************************************************************************** >> failed: [osctlr.home.visualbits.net -> osctlr.home.visualbits.net] (item= >> osctlr.home.visualbits.net) => {"changed": false, "item": " >> osctlr.home.visualbits.net", "msg": "Failed to call command: ['docker', >> 'exec', 'ceph_mon', 'ceph', '--format', 'json', 'auth', 'get-or-create', ' >> mgr.osctlr.home.visualbits.net', 'mds', 'allow *', 'mon', 'allow profile >> mgr', 'osd', 'allow *'] returncode: 1 output: stdout: \"\", stderr: >> \"[errno 1] error connecting to the cluster\n\""} >> >> keyrings look proper: >> >> (openstack) [root at osctlr ~]# md5sum >> /etc/kolla/ceph-mon/ceph.client.admin.keyring >> 4658c01282c791bce9c75678df9e21c9 >> /etc/kolla/ceph-mon/ceph.client.admin.keyring >> (openstack) [root at osctlr ~]# md5sum >> /var/lib/docker/volumes/ceph_mon_config/_data/ceph.client.admin.keyring >> 4658c01282c791bce9c75678df9e21c9 >> /var/lib/docker/volumes/ceph_mon_config/_data/ceph.client.admin.keyring >> >> I've removed the docker container, volume and kolla config directories >> multple times with the same error. I can't even run ceph status from the >> container bash itself. >> >> Any ideas? >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luyao.zhong at intel.com Fri Mar 1 09:30:14 2019 From: luyao.zhong at intel.com (Luyao Zhong) Date: Fri, 1 Mar 2019 17:30:14 +0800 Subject: [nova] NUMA live migration - mostly how it's tested In-Reply-To: References: Message-ID: Hi all, There was something wrong with the live migration when using 'dedicated' cpu_policy in my test. Attached file contains the details. The message body will be so big that it will be held if I attach the debug info, so I will send another email. Regards, Luyao On 2019/2/28 下午9:28, Sean Mooney wrote: > On Wed, 2019-02-27 at 21:33 -0500, Artom Lifshitz wrote: >> >> >> On Wed, Feb 27, 2019, 21:27 Matt Riedemann, wrote: >>> On 2/27/2019 7:25 PM, Artom Lifshitz wrote: >>>> What I've been using for testing is this: [3]. It's a series of >>>> patches to whitebox_tempest_plugin, a Tempest plugin used by a bunch >>>> of us Nova Red Hatters to automate testing that's outside of Tempest's >>>> scope. >>> >>> And where is that pulling in your nova series of changes and posting >>> test results (like a 3rd party CI) so anyone can see it? Or do you mean >>> here are tests, but you need to provide your own environment if you want >>> to verify the code prior to merging it. >> >> Sorry, wasn't clear. It's the latter. The test code exists, and has run against my devstack environment with my >> patches checked out, but there's no CI or public posting of test results. Getting CI coverage for these NUMA things >> (like the old Intel one) is a whole other topic. > on the ci front i resolved the nested vert on the server i bought to set up a personal ci for numa testing. > that set me back a few weeks in setting up that ci but i hope to run artom whitebox test amoung other in that at some > point. vexhost also provided nested virt to the gate vms. im going to see if we can actully create a non voting job > using the ubuntu-bionic-vexxhost nodeset. if ovh or one of the other providers of ci resource renable nested virt > then we can maybe make that job voting and not need thridparty ci anymor. >>> Can we really not even have functional tests with the fake libvirt >>> driver and fake numa resources to ensure the flow doesn't blow up? >> >> That's something I have to look into. We have live migration functional tests, and we have NUMA functional tests, but >> I'm not sure how we can combine the two. > > jus as an addtional proof point im am planning to do a bunch of migration and live migration testing in the next 2-4 > weeks. > > my current backlog on no particalar order is > sriov migration > numa migration > vtpm migration > cross-cell migration > cross-neutron backend migration (ovs<->linuxbridge) > cross-firwall migraton (iptables<->contrack) (previously tested and worked at end of queens) > > narrowong in on the numa migration the current set of testcases i plan to manually verify are as follows: > > note assume all flavor will have 256mb of ram and 4 cores unless otherwise stated > > basic tests > pinned guests (hw:cpu_policy=dedicated) > pinned-isolated guests (hw:cpu_policy=dedicated hw:thread_policy=isolate) > pinned-prefer guests (hw:cpu_policy=dedicated hw:thread_policy=prefer) > unpinned-singel-numa guest (hw:numa_nodes=1) > unpinned-dual-numa guest (hw:numa_nodes=2) > unpinned-dual-numa-unblanced guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 > hw:numa_mem.0=64 hw:numa_mem.0=192) > unpinned-hugepage-implcit numa guest (hw:mem_page_size=large) > unpinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2) > pinned-hugepage-multi numa guest (hw:mem_page_size=large hw:numa_nodes=2 hw:cpu_policy=dedicated) > realtime guest (hw:cpu_policy=dedicated hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1) > emulator-thread-iosolated guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=isolate) > > advanced tests (require extra nova.conf changes) > emulator-thread-shared guest (hw:cpu_policy=dedicated hw:emulator_threads_policy=shared) note cpu_share_set configrued > unpinned-singel-numa-hetorgious-host guest (hw:numa_nodes=1) note vcpu_pin_set adjusted so that > host 1 only has cpus on > numa 1 and host 2 only has cpus on numa node 2. > supper-optimiesd-guest (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 > hw:numa_mem.0=64 hw:numa_mem.0=192 hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=isolate) > supper-optimiesd-guest-2 (hw:numa_nodes=2 hw:numa_cpu.0=1 hw:numa_cpu.1=1-3 hw:numa_mem.0=64 hw:numa_mem.0=192 > hw:cpu_realtime=yes hw:cpu_realtime_mask=^0-1 hw:emulator_threads_policy=share) > > > for each of these test ill provide a test-command file with the command i used to run the tests and reustlts file > with a summary at the top plus the xmls before and after the migration showing that intially the resouces > would conflict on migration and then the updated xmls after the migration. > i will also provide the local.conf for the devstack deployment and some details about the env like distor/qemu/libvirt > versions. > > eventurally i hope all those test cases can be added to the whitebox plugin and verifed in a ci. > we could also try and valideate them in functional tests. > > i have attached the xml for the pinned guest as an example of what to expect but i will be compileing this slowly as i > go and zip everying up in an email to the list. > this will take some time to complete and hosestly i had planned to do most of this testing after feature freeze when we > can focus on testing more. > > regards > sean > > -------------- next part -------------- --------------------------VM cpu pining info------------------------------ # VM server_on_host1 cpu pinning info 4096 # VM server_on_host2 cpu pinning info (before migration) 4096 # VM server_on_host2 cpu pinning info (after this VM was migrated to host1) 4096 --------------------flavor's extra spec setting---------------------- hw:cpu_policy='dedicated' hw:numa_cpus.0='0-3' hw:numa_mem.0='512' hw:numa_nodes='1' ---------------------server_on_host1 domain xml----------------------- $ sudo virsh dumpxml instance-0000003d instance-0000003d 0f84973c-62de-4eb0-ac3e-ba3f6f8a869b server_on_host1 2019-03-01 05:14:59 512 1 0 0 4 admin admin 524288 524288 4 4096 /machine OpenStack Foundation OpenStack Nova 18.1.0 0f84973c-62de-4eb0-ac3e-ba3f6f8a869b 0f84973c-62de-4eb0-ac3e-ba3f6f8a869b Virtual Machine hvm destroy restart destroy /usr/bin/kvm-spice