[Openstack] Allowing clients to pass capability requests through tags?
Brian Schott
bfschott at gmail.com
Fri Feb 11 21:02:19 UTC 2011
In addition to zone ids being opaque, they have to be anonymized. On AWS, they randomize the assignment of availability zones per account because "us-east-1a" datacenter was melting the floor tiles due to load while "us-east-1c" datacenter was idling. This causes problems for those of us who manage development and deployment accounts.
Unexpected consequences of lazy cloud application developers in a hurry.
As one of those lazy developers, I don't really want visibility into or have to understand the provider's physical infrastructure or even logical organization for that matter. The only value I see in zones is that spreading across them gives me fault tolerance and selecting one gives me somewhat better network performance (or saves bandwidth charges).
Exposing your topology can be a bad thing.
Brian Schott
bfschott at gmail.com
On Feb 11, 2011, at 3:01 PM, Vishvananda Ishaya wrote:
> Yes, i think that cross-project should be requests across an admin api. I don't think high level zone info is enough information for the scheduler
> to decide where the instance should go. Especially with things like swift, where there are three copies that may be in different zones altogether.
>
> Vish
> On Feb 11, 2011, at 11:55 AM, Eric Day wrote:
>
>> Hey Vish!
>>
>> On Fri, Feb 11, 2011 at 12:10:14PM -0600, Vishvananda Ishaya wrote:
>>> I think that providers may not wish to expose the internal structure of
>>> their network to the degree that you are suggesting. I prefer the idea of
>>> near other object with an internal lookup into zone.
>>
>> Providers can always obscure the naming to hide topology, or
>> services can have options to only expose a certain depth of
>> the hierarchy (ie, truncate at the 'datacenter level', hide all
>> room/rack/switch zone level details). So even though volume-001
>> may be in rack12.room2.dc1.dfw.rackspace.com, the API only returns
>> dc1.dfw.rackspace.com. We can of course allow for both too when the
>> granularity is needed within a project:
>>
>> Within nova requests, allow:
>>
>> near-volume:volume-001
>>
>> But across projects:
>> in-zone: dc1.dfw.rackspace.com
>>
>> If we don't expose object zones at all, how would you support
>> cross-project lookups? If nova gets a request 'near:swift-12345',
>> would it be configured to query swift to get the zone via an admin API?
>>
>> We could also expose detailed object zones but keep them completely
>> opaque, and instead require each service to allow pluggable
>> scheduler/zone mapping. For example, a volume may return zone
>> '87af1b7c', which internally all services could lookup and translate
>> to where that maps in the deployment hierarchy. I was hoping DNS names
>> were abstract enough to reduce complexity, but perhaps they are not. :)
>>
>> -Eric
>>
>>> On Feb 11, 2011 9:38 AM, "Eric Day" <eday at oddments.org> wrote:
>>>> Hi Sandy,
>>>>
>>>> I agree with using tags for full scheduler selection, it's something
>>>> I've been pushing for from the start. The request contains any number
>>>> of k/v pairs, the services provide any number of k/v pairs, and the
>>>> scheduler performs a match (some required, some optional, ...). I see
>>>> the URI/zone as one of those tags, not something we need to overload
>>>> to contain all of the capabilities. It should only be a hierarchical
>>>> "location", which may be geographic location, organizational location
>>>> (dept, ...), or some other type (however you decide to construct
>>>> your zones).
>>>>
>>>> For example, imagine a dynamic del.icio.us tag that allowed for domain
>>>> name filtering on bookmarks (give me all bookmarks with tags [book
>>>> review] [domain:slashdot.org]). For Nova, this means issuing requests
>>>> like "create instance with [GPU] [Fast disk] [zone:dc1.example.com]".
>>>>
>>>> The important thing is that this is not a tag specific to a particular
>>>> service. For example, Swift would never care or need to understand a
>>>> 'GPU' tag, but it can share and understand zone tags.
>>>>
>>>> -Eric
>>>>
>>>> On Fri, Feb 11, 2011 at 12:40:44PM +0000, Sandy Walsh wrote:
>>>>> Heh, hate to be the one to bust up the URI love-fest :)
>>>>>
>>>>> The issue I have with a single URI being used as the heuristic for node
>>> selection is that it is very rigid.
>>>>>
>>>>> Different business units have different views on the network:
>>>>> * Operations may view it as geography/data centers.
>>>>> * Consumers may view it as technical ability (gpu's, fast disk, good
>>> inter-server speed, etc)
>>>>> * Sales/marketing may view it as the number of martinis they can buy ;)
>>>>>
>>>>> Trees become unmanageable/hard to visualize for users beyond a couple
>>> hundred nodes. We are lucky that our geographical/DC-based hierarchy is
>>> relatively flat. This is why I was initially pushing for a tag-based
>>> system for selection (aka Zone/Host Capabilities).
>>>>>
>>>>> Consider the way delicio.us works. They manage many millions of URL's
>>> and tags are an effective way to slice & dice your way through the data:
>>>>> "Show me all the URL's on [OpenStack] [Python] [Zones] [Scheduler]" ...
>>> blam.
>>>>>
>>>>> This is also the way the old Trader services worked:
>>>>> "I want a [wax transfer] [color] printer that can has [30ppm] and
>>> [300dpi] on [Floor 2]"
>>>>>
>>>>> "Near" simply has to mean the distance in zones from the most-optimal
>>> zones, based on the tags.
>>>>>
>>>>> "I want a new instance with [GPU] and [Fast Disk] [Good inter-instance
>>> network speed] [near] [DRW] [DC1]"
>>>>> * where "[near]" implies "as close as possible to" in zone distance.
>>>>>
>>>>> Personally I don't like overloading the zone name to have a
>>> "meaningful" URI when we can get the same functionality with
>>> Capabilities/Tags already. And we already know we need Capability support
>>> anyway. Especially if it means enforcing a rigid hierarchy.
>>>>>
>>>>> $0.02
>>>>>
>>>>> -S
>>>>>
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: openstack-bounces+sandy.walsh=rackspace.com at lists.launchpad.net
>>> [openstack-bounces+sandy.walsh=rackspace.com at lists.launchpad.net] on
>>> behalf of Eric Day [eday at oddments.org]
>>>>> Sent: Friday, February 11, 2011 4:30 AM
>>>>> To: Justin Santa Barbara
>>>>> Cc: openstack at lists.launchpad.net; Devin Carlen
>>>>> Subject: Re: [Openstack] Allowing clients to pass capability requests
>>> through tags?
>>>>>
>>>>> The main reason I was proposing full location/zone of objects is to
>>>>> allow this type of 'near' scheduling to happen without understanding
>>>>> what the actual object is. For example, imagine we want to start an
>>>>> instance near a particular swift object. We could query the swift
>>>>> object and in the metadata there could be a 'zone' tag (well, three,
>>>>> one for each copy). For example:
>>>>>
>>>>> get swift-12345: zone=rack12.room2.dc1.dfw.rackspace.com
>>>>>
>>>>> I can now use that zone name to:
>>>>>
>>>>> create_instance: openstack:near=rack12.room2.dc1.dfw.rackspace.com
>>>>>
>>>>> The deployment can decide what 'near' is (perhaps a measure of link
>>>>> speed or latency). This way a particular deployment that uses the
>>>>> same URI/zone names across projects can account for locality without
>>>>> knowing what objects from different services are. If it were just
>>>>> 'near=swift-12345', it would need to understand what a swift object
>>>>> was and perform that lookup to find out where it is.
>>>>>
>>>>> So you can still grab a zone tag from a volume you created:
>>>>>
>>>>> get vol-000001: rack4.room2.dc1.dfw.rackspace.com
>>>>>
>>>>> and use the zone to launch an instance with:
>>>>>
>>>>> create_instance: openstack:near=rack4.room2.dc1.dfw.rackspace.com
>>>>>
>>>>> We can also write schedulers/tools for a particular deployment
>>>>> that understands the zones to just say 'always prefer in
>>>>> dc1.dfw.rackspace.com', because power is cheaper there right now, or
>>>>> 'test.dc1.dfw.rackspace.com' because that is my test zone (perhaps
>>>>> only enabled for certain accounts in the scheduler too).
>>>>>
>>>>> -Eric
>>>>>
>>>>> On Thu, Feb 10, 2011 at 03:38:42PM -0800, Justin Santa Barbara wrote:
>>>>>> I think the blueprint was largely complementary to the multi-zone
>>> stuff;
>>>>>> this is more about how the client _requests_ a particular
>>>>>> location/capability through the API. The multi-zone blueprint seems
>>> to be
>>>>>> more about how nova would satisfy those requests (in a non-trivial
>>> zone
>>>>>> structure.)
>>>>>> The root motivator is indeed getting a 'good' connection to a storage
>>>>>> volume. I'm thinking of iSCSI SAN storage here, so in my case this
>>>>>> probably means the SAN device with the least number of switches in
>>>>>> between. There could well be SAN devices in each rack (e.g. Solaris
>>>>>> volume nodes), or the devices could even be running on the host
>>> nodes, and
>>>>>> I don't believe that zones in the EC2 sense are sufficient here.
>>>>>> But I guess that if the zone hierarchy went all the way down to the
>>> rack
>>>>>> (or machine), that would work. So I could create a volume and it
>>> would
>>>>>> come back with a location of "rack4.room2.dc1.dfw.rackspace.com" and
>>> I
>>>>>> could then request allocation of machines in that same rack? Is that
>>> the
>>>>>> vision of the nested zones?
>>>>>> I do have a concern that long-term if we _only_ use zones, that's
>>> trying
>>>>>> to multiplex a lot of information into the zone hierarchy, and we can
>>>>>> really only put one attribute in there. I also like the flexibility
>>> of
>>>>>> the 'openstack:near=vol-000001' request, because then the cloud can
>>> decide
>>>>>> how near to place the instance based on its knowledge of the
>>> topology, and
>>>>>> the clients can be oblivious to the storage system and arrangement.
>>> But,
>>>>>> my immediate requirement would indeed be satisfied if the zones went
>>> down
>>>>>> to the rack/machine level.
>>>>>> An alternative way to look at zones and instance-types is that
>>> they're
>>>>>> actually just fail-if-not-satisfiable tags of the creation request
>>>>>> (openstack:+zone=us-east-1a and openstack:+instancetype=m1.large)
>>> They're
>>>>>> only distinguished attributes because AWS doesn't have an
>>>>>> extensibility mechanism, which this blueprint would give us.
>>>>>> Justin
>>>>>>
>>>>>> On Thu, Feb 10, 2011 at 3:12 PM, Devin Carlen <devcamcar at me.com>
>>> wrote:
>>>>>>
>>>>>> I haven't totally digested this blueprint yet but it seems like there
>>> is
>>>>>> some overlap with what is being discussed with the multi zone
>>> metadata
>>>>>> stuff. One approach might be to handle this awt the scheduler level
>>>>>> though and try to ensure things are always in the same zone when
>>>>>> appropriate.
>>>>>> I think the bigger question you raise is how to request local volumes
>>>>>> when possible, yes?
>>>>>>
>>>>>> Devin
>>>>>> On Feb 10, 2011, at 3:37 PM, Justin Santa Barbara
>>> <justin at fathomdb.com>
>>>>>> wrote:
>>>>>>
>>>>>> Does anyone have any thoughts/objections on the blueprint I posted
>>> for
>>>>>> allowing clients to pass capability-requests through tags? I'm
>>>>>> planning on starting implementation soon, so if people think this is
>>> a
>>>>>> bad idea I'd rather know before I start coding!
>>>>>> Blueprint:
>>> https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
>>>>>> Wiki:
>>> https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities
>>>>>> And a quick TLDR:
>>>>>> API clients need a way to request e.g. placement of machines near
>>> each
>>>>>> other / near volumes, or that a volume be created with a particular
>>>>>> RAID level, or that a machine be created in a HIPAA compliant
>>>>>> environment. (This is complementary to the work on hierarchical zones
>>>>>> & URL naming, I believe)
>>>>>> I propose using the instance tags for this, e.g. specifying
>>>>>> openstack:near=vol-000001 when creating an instance to request
>>>>>> locating the instance 'close to' that volume.
>>>>>> By default these requests would be best-effort and
>>> ignored-if-unknown;
>>>>>> if the client wants to specify that something is required and should
>>>>>> fail if not understood or not satisfiable, they could use a "+" e.g.
>>>>>> openstack:+location=*.dc1.north.rackspace.com
>>>>>> Controversially (?), this would not be supported for clients using
>>> the
>>>>>> AWS API, because tags can only be specified once the instance has
>>>>>> already been created.
>>>>>> Feedback appreciated!
>>>>>> Justin
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to : openstack at lists.launchpad.net
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to : openstack at lists.launchpad.net
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to : openstack at lists.launchpad.net
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>> Confidentiality Notice: This e-mail message (including any attached or
>>>>> embedded documents) is intended for the exclusive and confidential use
>>> of the
>>>>> individual or entity to which this message is addressed, and unless
>>> otherwise
>>>>> expressly indicated, is confidential and privileged information of
>>> Rackspace.
>>>>> Any dissemination, distribution or copying of the enclosed material is
>>> prohibited.
>>>>> If you receive this transmission in error, please notify us immediately
>>> by e-mail
>>>>> at abuse at rackspace.com, and delete the original message.
>>>>> Your cooperation is appreciated.
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to : openstack at lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help : https://help.launchpad.net/ListHelp
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
More information about the Openstack
mailing list