Open Stack

Exposing the detailed info in private cloud, sure makes sense. For
public clouds, not so sure. Would be nice to find something that works
for both.

We let the user express their intent through the instance groups api.
The scheduler will then do a best effort to meet that criteria, using
its private information. At a courser grain, we have availability
zones, that you could use to express "closeness", and probably often
give you a good measure of closeness anyway.

So a Hadoop user could request a several small groups of VMs defined
in instance groups to be close, and maybe spread across different
availability zones.

Would that do the trick? Or does Hadoop/HDFS need a bit more
granularity than that? Could it look to auto-detect "closeness" in
some auto-setup phase, given rough user hints?

John

On 13 September 2013 07:40, Alex Glikson <GLIKSON at il.ibm.com> wrote:
> If I understand correctly, what really matters at least in case of Hadoop is
> network proximity between instances.
> Hence, maybe Neutron would be a better fit to provide such information. In
> particular, depending on virtual network configuration, having 2 instances
> on the same node does not guarantee that the network traffic between them
> will be routed within the node.
> Physical layout could be useful for availability-related purposes. But even
> then, it should be abstracted in such a way that it will not reveal details
> that a cloud provider will typically prefer not to expose. Maybe this can be
> done by Ironic -- or a separate/new project (Tuskar sounds related).
>
> Regards,
> Alex
>
>
>
>
> From:        Mike Spreitzer <mspreitz at us.ibm.com>
> To:        OpenStack Development Mailing List
> <openstack-dev at lists.openstack.org>,
> Date:        13/09/2013 08:54 AM
> Subject:        Re: [openstack-dev] [nova] [savanna] Host information for
> non        admin        users
> ________________________________
>
>
>
>> From: Nirmal Ranganathan <rnirmal at gmail.com>
>> ...
>> Well that's left upto the specific block placement policies in hdfs,
>> all we are providing with the topology information is a hint on
>> node/rack placement.
>
> Oh, you are looking at the placement of HDFS blocks within the fixed storage
> volumes, not choosing where to put the storage volumes.  In that case I
> understand and agree that simply providing identifiers from the
> infrastructure to the middleware (HDFS) will suffice.  Coincidentally my
> group is working on this very example right now in our own environment.  We
> have a holistic scheduler that is given a whole template to place, and it
> returns placement information.  We imagine, as does Hadoop, a general
> hierarchy in the physical layout, and the holistic scheduler returns, for
> each VM, the path from the root to the VM's host.
>
> Regards,
>
> Mike_______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>