<br><br><div class="gmail_quote">On Mon, Feb 25, 2013 at 6:14 PM, Sam Morrison <span dir="ltr"><<a href="mailto:sorrison@gmail.com" target="_blank">sorrison@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hi Joe,<div><br><div><div class="im"><div>On 26/02/2013, at 11:19 AM, Joe Gordon <<a href="mailto:jogo@cloudscaling.com" target="_blank">jogo@cloudscaling.com</a>> wrote:</div><br>
<blockquote type="cite">
<div class="gmail_quote">On Sun, Feb 24, 2013 at 3:31 PM, Sam Morrison <span dir="ltr"><<a href="mailto:sorrison@gmail.com" target="_blank">sorrison@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have been playing with the AggregateInstanceExtraSpecs filter and can't get it to work.<br>
<br>
In our staging environment it works fine with 4 compute nodes, I have 2 aggregates to split them into 2.<br>
<br>
When I try to do the same in our production environment which has 80 compute nodes (splitting them again into 2 aggregates) it doesn't work.<br>
<br>
nova-scheduler starts to go very slow, I scheduled an instance and gave up after 5 minutes, it seemed to be taking ages and the host was at 100% cpu. Also got about 500 messages in rabbit that were unacknowledged.<br>
<br></blockquote><div><br></div><div>what does the nova-scheduler log say? Where is the unacknowledged rabbitmq messages sent from?</div></div></blockquote><div><br></div></div><div>Logs are below. Note the large time gap between selecting a host, this is pretty much instantaneous without this filter.</div>
<div><br></div><div>Can't figure out how to see an unacknowledged message in rabbit but my guess is it is the compute service updates from all the compute nodes. These aren't happening and I think this is the reason that the attempts to schedule further down are rejected with "is disabled or has not been heard from in a while"</div>
<div><br></div><div>Do you see anything that could be an issue? Flags we use for scheduler are below also:</div><div><br></div><div>Thanks for your help,</div><div>Sam</div><div><br></div><div><br></div><div><div># Scheduler Flags</div>
<div>compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler</div><div>ram_allocation_ratio=1.0</div><div>cpu_allocation_ratio=0.92</div><div>reserved_host_memory_mb=1024</div><div>reserved_host_disk_mb=0</div>
<div>scheduler_default_filters=RetryFilter,AggregateInstanceExtraSpecsFilter,RamFilter,CoreFilter,ComputeFilter</div><div>compute_fill_first_cost_fn_weight=1.0</div></div><div><br></div><div><br></div><div><br></div><div>
2013-02-25 10:01:35 DEBUG nova.scheduler.filter_scheduler [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Attempting to build 1 instance(s) schedule_run_instance /usr/lib/python2.7/dist-packages/nova/sc</div><div>heduler/filter_scheduler.py:66</div>
<div>2013-02-25 10:01:35 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc27) host_passes /usr/lib/python2.7/dist-packages/n</div><div>ova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:02:13 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter passes for qh2-rcc27 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:178</div>
<div>2013-02-25 10:02:13 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc26) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:02:51 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method CoreFilter.host_passes of <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed for qh2-rcc26 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:02:51 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc25) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:03:28 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc25': free_ram_mb:71086 free_disk_mb:3035136 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37</div>
<div>2013-02-25 10:03:28 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc25 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:03:28 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc24) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:04:05 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc24': free_ram_mb:99758 free_disk_mb:3296256 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37</div>
<div>2013-02-25 10:04:05 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc24 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:04:05 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc23) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:04:43 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc23': free_ram_mb:71086 free_disk_mb:3012608 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37</div>
<div>2013-02-25 10:04:43 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc23 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:04:43 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc22) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:05:22 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc22': free_ram_mb:91566 free_disk_mb:3196928 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37</div>
<div>2013-02-25 10:05:22 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc22 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:05:22 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc21) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:06:01 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc21': free_ram_mb:91566 free_disk_mb:3216384 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37</div>
<div>2013-02-25 10:06:01 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc21 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:06:01 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc20) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:06:37 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method CoreFilter.host_passes of <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed for qh2-rcc20 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:06:37 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc77) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:07:14 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc77': free_ram_mb:75182 free_disk_mb:3059712 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37</div>
</div></div></div></blockquote><div><br></div><div><br></div><div>It looks like the scheduler issues are related to the rabbitmq issues. "host 'qh2-rcc77' ... is disabled or has not been heard from in a while"</div>
<div><br></div><div>What does 'nova host-list' say? the clocks must all be synced up?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">
<div><div><div>2013-02-25 10:07:14 DEBUG nova.scheduler.host_manager [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function <bound method ComputeFilter.host_passes of <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> failed for qh2-rcc77 passes_filters /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175</div>
<div>2013-02-25 10:07:14 DEBUG nova.scheduler.filters.retry_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: []. (host=qh2-rcc73) host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39</div>
<div>2013-02-25 10:07:52 DEBUG nova.scheduler.filters.compute_filter [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc73': free_ram_mb:87470 free_disk_mb:3148800 is disabled or has not been heard from in a while host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37</div>
<div><br></div><div><br></div></div></div></div></blockquote></div><br>