[watcher][qa] Thoughts on performance testing for Watcher
Hi, I'm new to Watcher and would like to do some performance and scale testing in a simulated environment and wondering if anyone can give some pointers on what I could be testing or looking for. If possible, I'd like to be able to just setup a single-node devstack with the nova fake virt driver which allows me to create dozens of fake compute nodes. I could also create multiple cells with devstack, but there gets to be a limit with how much you can cram into a single node 8GB RAM 8VCPU VM (I could maybe split 20 nodes across 2 cells). I could then create dozens of VMs to fill into those compute nodes. I'm mostly trying to figure out what could be an interesting set of tests. The biggest problem I'm trying to solve with Watcher is optimizing resource utilization, i.e. once the computes hit the Tetris problem and there is some room on some nodes but none of the nodes are fully packed. I was thinking I could simulate this by configuring nova so it spreads rather than packs VMs onto hosts (or just use the chance scheduler which randomly picks a host), using VMs of varying sizes, and then run some audit / action plan (I'm still learning the terminology here) to live migrate the VMs such that they get packed onto as few hosts as possible and see how long that takes. Naturally with devstack using fake nodes and no networking on the VMs, that live migration is basically a noop, but I'm more interested in profiling how long it takes Watcher itself to execute the actions. Once I get to know a bit more about how Watcher works, I could help with optimizing some of the nova-specific stuff using placement [1]. Any advice or guidance here would be appreciated. [1] https://review.opendev.org/#/c/656448/ -- Thanks, Matt
Hi Matt, I'm glad that you are interesting to Watcher. Though we never do such a simulated test, I wish you can get what you want. Some notes: 1, Watcher updates its data model based nova versioned notifications, so you should enable nova notification in your simulated environment. 2, Watcher needs node name getting from CONF.host or socket.gethostname, If you have two or more controller nodes they don't have same host name. 3, Watcher doesn't consider nova cell, now watcher filter nodes through host aggregate and zone. You can get more info by CLI cmd: watcher help audittemplate create 4, Watcher needs metric data source such as Ceilometer, so your fake nodes and VMs should have metric data. 5, For optimizing resource utilization, I think you could use strategy [1] 6, There are two audit type:ONESHOT and CONTINUOUS in Watcher, you can get more help by CLI cmd: watcher help audit create If any questions, let us know Thanks, licanwei [1] https://docs.openstack.org/watcher/latest/strategies/vm_workload_consolidati... 发件人：MattRiedemann <mriedemos@gmail.com> 收件人：openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>; 日 期 ：2019年05月08日 04:57 主 题 ：[watcher][qa] Thoughts on performance testing for Watcher Hi, I'm new to Watcher and would like to do some performance and scale testing in a simulated environment and wondering if anyone can give some pointers on what I could be testing or looking for. If possible, I'd like to be able to just setup a single-node devstack with the nova fake virt driver which allows me to create dozens of fake compute nodes. I could also create multiple cells with devstack, but there gets to be a limit with how much you can cram into a single node 8GB RAM 8VCPU VM (I could maybe split 20 nodes across 2 cells). I could then create dozens of VMs to fill into those compute nodes. I'm mostly trying to figure out what could be an interesting set of tests. The biggest problem I'm trying to solve with Watcher is optimizing resource utilization, i.e. once the computes hit the Tetris problem and there is some room on some nodes but none of the nodes are fully packed. I was thinking I could simulate this by configuring nova so it spreads rather than packs VMs onto hosts (or just use the chance scheduler which randomly picks a host), using VMs of varying sizes, and then run some audit / action plan (I'm still learning the terminology here) to live migrate the VMs such that they get packed onto as few hosts as possible and see how long that takes. Naturally with devstack using fake nodes and no networking on the VMs, that live migration is basically a noop, but I'm more interested in profiling how long it takes Watcher itself to execute the actions. Once I get to know a bit more about how Watcher works, I could help with optimizing some of the nova-specific stuff using placement [1]. Any advice or guidance here would be appreciated. [1] https://review.opendev.org/#/c/656448/ -- Thanks, Matt
On 5/7/2019 10:26 PM, li.canwei2@zte.com.cn wrote:
Some notes:
1, Watcher updates its data model based nova versioned notifications, so you
should enable nova notification in your simulated environment.
Sure, this shouldn't be a problem in a devstack environment.
2, Watcher needs node name getting from CONF.host or socket.gethostname,
If you have two or more controller nodes they don't have same host name.
When you say "node name" do you mean the name of a nova-compute hypervisor node, i.e. what is returned from "openstack hypervisor list"? Or do you mean something specific to Watcher itself? The former is already supported in devstack using fake computes [1].
3, Watcher doesn't consider nova cell, now watcher filter nodes through host
aggregate and zone. You can get more info by CLI cmd: watcher help audittemplate create
Yup, that's fine and is part of what I want to test. I wouldn't expect Watcher to know anything about cells anyway since there is no REST API or notification about those in nova.
4, Watcher needs metric data source such as Ceilometer, so your fake nodes and VMs
should have metric data.
Hmm, is this where the ceilometer agent pulls metric data from the hypervisor (libvirt) directly? If so, that could be a problem for me if I'm using the fake virt driver in devstack to do some scale testing since that's not a real hypervisor. There would be notifications from nova for servers getting created and deleted and such, but there wouldn't be a real hypervisor to pull data from for the ceilometer agent. Honestly I'd like to avoid using ceilometer at all if possible but I'm also unfamiliar with gnocchi and monasca, but I'll probably try to get one of those working first. Looking at the Watcher configuration I shouldn't depend on ceilometer anyway since it looks like the datasource is deprecated? [2] I am a little confused about that because the configuration docs talk about using ceilometer (maybe those are just out of date?) [3].
5, For optimizing resource utilization, I think you could use strategy [1]
Thanks I'll try it.
6, There are two audit type:ONESHOT and CONTINUOUS in Watcher, you can get
more help by CLI cmd: watcher help audit create
Thanks for all of the help here. [1] https://docs.openstack.org/devstack/latest/guides/nova.html#scaling [2] https://docs.openstack.org/watcher/latest/configuration/watcher.html#ceilome... [3] https://docs.openstack.org/watcher/latest/configuration/configuring.html#con... -- Thanks, Matt
another note, Watcher provides a WORKLOAD optimization(balancing or consolidation). If you want to maximize the node resource (such as vCPU, Ram...) usage through VM migration, Watcher doesn't have such a strategy now. Thanks! licanwei 原始邮件 发件人：MattRiedemann <mriedemos@gmail.com> 收件人：openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>; 日 期 ：2019年05月08日 04:57 主 题 ：[watcher][qa] Thoughts on performance testing for Watcher Hi, I'm new to Watcher and would like to do some performance and scale testing in a simulated environment and wondering if anyone can give some pointers on what I could be testing or looking for. If possible, I'd like to be able to just setup a single-node devstack with the nova fake virt driver which allows me to create dozens of fake compute nodes. I could also create multiple cells with devstack, but there gets to be a limit with how much you can cram into a single node 8GB RAM 8VCPU VM (I could maybe split 20 nodes across 2 cells). I could then create dozens of VMs to fill into those compute nodes. I'm mostly trying to figure out what could be an interesting set of tests. The biggest problem I'm trying to solve with Watcher is optimizing resource utilization, i.e. once the computes hit the Tetris problem and there is some room on some nodes but none of the nodes are fully packed. I was thinking I could simulate this by configuring nova so it spreads rather than packs VMs onto hosts (or just use the chance scheduler which randomly picks a host), using VMs of varying sizes, and then run some audit / action plan (I'm still learning the terminology here) to live migrate the VMs such that they get packed onto as few hosts as possible and see how long that takes. Naturally with devstack using fake nodes and no networking on the VMs, that live migration is basically a noop, but I'm more interested in profiling how long it takes Watcher itself to execute the actions. Once I get to know a bit more about how Watcher works, I could help with optimizing some of the nova-specific stuff using placement [1]. Any advice or guidance here would be appreciated. [1] https://review.opendev.org/#/c/656448/ -- Thanks, Matt
Hi, Actually, we are working on providing such feature in combination with aardvark. The idea is to create a strategy that fills up with preemptible resources, that later on could be reclaimed by aardvark if a normal instance is deployed. https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22248/to... https://www.openstack.org/summit/denver-2019/summit-schedule/events/23187/im... Cheers Jose Castro Leon CERN Cloud Team On Wed, 2019-05-08 at 14:19 +0800, li.canwei2@zte.com.cn wrote: another note, Watcher provides a WORKLOAD optimization(balancing or consolidation). If you want to maximize the node resource (such as vCPU, Ram...) usage through VM migration, Watcher doesn't have such a strategy now. Thanks! licanwei 原始邮件 发件人：MattRiedemann <mriedemos@gmail.com> 收件人：openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>; 日 期 ：2019年05月08日 04:57 主 题 ：[watcher][qa] Thoughts on performance testing for Watcher Hi, I'm new to Watcher and would like to do some performance and scale testing in a simulated environment and wondering if anyone can give some pointers on what I could be testing or looking for. If possible, I'd like to be able to just setup a single-node devstack with the nova fake virt driver which allows me to create dozens of fake compute nodes. I could also create multiple cells with devstack, but there gets to be a limit with how much you can cram into a single node 8GB RAM 8VCPU VM (I could maybe split 20 nodes across 2 cells). I could then create dozens of VMs to fill into those compute nodes. I'm mostly trying to figure out what could be an interesting set of tests. The biggest problem I'm trying to solve with Watcher is optimizing resource utilization, i.e. once the computes hit the Tetris problem and there is some room on some nodes but none of the nodes are fully packed. I was thinking I could simulate this by configuring nova so it spreads rather than packs VMs onto hosts (or just use the chance scheduler which randomly picks a host), using VMs of varying sizes, and then run some audit / action plan (I'm still learning the terminology here) to live migrate the VMs such that they get packed onto as few hosts as possible and see how long that takes. Naturally with devstack using fake nodes and no networking on the VMs, that live migration is basically a noop, but I'm more interested in profiling how long it takes Watcher itself to execute the actions. Once I get to know a bit more about how Watcher works, I could help with optimizing some of the nova-specific stuff using placement [1]. Any advice or guidance here would be appreciated. [1] https://review.opendev.org/#/c/656448/ -- Thanks, Matt
participants (3)
-
Jose Castro Leon
-
li.canwei2＠zte.com.cn
-
Matt Riedemann