Open Stack

Thanks John,

What I'm trying to do is to run an asynchronous task that pre-organizes the target hosts for an image. Then scheduler only need to read the top of the list or priority queue. We have a paper proposed for the summit that will explain the approach, hopefully it gets accepted so we can have a conversation on this at the summit. I suspect the DB overhead will go away if we try our approach. Still theory though, that is why I want to get a significant test environment to appreciate the performance better.

Regards,
David Peraza

-----Original Message-----
From: John Garbutt [mailto:john at johngarbutt.com] 
Sent: Tuesday, February 25, 2014 5:45 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute nodes for scheduler testing

On 24 February 2014 20:13, David Peraza <david_peraza at persistentsys.com> wrote:
> Thanks John,
>
> I also think it is a good idea to test the algorithm at unit test level, but I will like to try out over amqp as well, that is, we process and threads talking to each other over rabbit or qpid. I'm trying to test out performance as well.
>

Nothing beats testing the thing for real, of course.

As a heads up, the overheads of DB calls turned out to dwarf any algorithmic improvements I managed. There will clearly be some RPC overhead, but it didn't stand out as much as the DB issue.

The move to conductor work should certainly stop the scheduler making those pesky DB calls to update the nova instance. And then, improvements like no-db-scheduler and improvements to scheduling algorithms should shine through much more.

Thanks,
John


> -----Original Message-----
> From: John Garbutt [mailto:john at johngarbutt.com]
> Sent: Monday, February 24, 2014 11:51 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [nova] Simulating many fake nova compute 
> nodes for scheduler testing
>
> On 24 February 2014 16:24, David Peraza <david_peraza at persistentsys.com> wrote:
>> Hello all,
>>
>> I have been trying some new ideas on scheduler and I think I'm 
>> reaching a resource issue. I'm running 6 compute service right on my 
>> 4 CPU 4 Gig VM, and I started to get some memory allocation issues.
>> Keystone and Nova are already complaining there is not enough memory.
>> The obvious solution to add more candidates is to get another VM and set another 6 Fake compute service.
>> I could do that but I think I need to be able to scale more without 
>> the need to use this much resources. I will like to simulate a cloud 
>> of 100 maybe
>> 1000 compute nodes that do nothing (Fake driver) this should not take 
>> this much memory. Anyone knows of a more efficient way to  simulate 
>> many computes? I was thinking changing the Fake driver to report many 
>> compute services in different threads instead of having to spawn a 
>> process per compute service. Any other ideas?
>
> It depends what you want to test, but I was able to look at tuning the filters and weights using the test at the end of this file:
> https://review.openstack.org/#/c/67855/33/nova/tests/scheduler/test_ca
> ching_scheduler.py
>
> Cheers,
> John
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.