<p>I think that providers may not wish to expose the internal structure of their network to the degree that you are suggesting. I prefer the idea of near other object with an internal lookup into zone.</p>
<div class="gmail_quote">On Feb 11, 2011 9:38 AM, "Eric Day" <<a href="mailto:eday@oddments.org">eday@oddments.org</a>> wrote:<br type="attribution">> Hi Sandy,<br>> <br>> I agree with using tags for full scheduler selection, it's something<br>
> I've been pushing for from the start. The request contains any number<br>> of k/v pairs, the services provide any number of k/v pairs, and the<br>> scheduler performs a match (some required, some optional, ...). I see<br>
> the URI/zone as one of those tags, not something we need to overload<br>> to contain all of the capabilities. It should only be a hierarchical<br>> "location", which may be geographic location, organizational location<br>
> (dept, ...), or some other type (however you decide to construct<br>> your zones).<br>> <br>> For example, imagine a dynamic <a href="http://del.icio.us">del.icio.us</a> tag that allowed for domain<br>> name filtering on bookmarks (give me all bookmarks with tags [book<br>
> review] [domain:<a href="http://slashdot.org">slashdot.org</a>]). For Nova, this means issuing requests<br>> like "create instance with [GPU] [Fast disk] [zone:<a href="http://dc1.example.com">dc1.example.com</a>]".<br>
> <br>> The important thing is that this is not a tag specific to a particular<br>> service. For example, Swift would never care or need to understand a<br>> 'GPU' tag, but it can share and understand zone tags.<br>
> <br>> -Eric<br>> <br>> On Fri, Feb 11, 2011 at 12:40:44PM +0000, Sandy Walsh wrote:<br>>> Heh, hate to be the one to bust up the URI love-fest :)<br>>> <br>>> The issue I have with a single URI being used as the heuristic for node selection is that it is very rigid.<br>
>> <br>>> Different business units have different views on the network:<br>>> * Operations may view it as geography/data centers.<br>>> * Consumers may view it as technical ability (gpu's, fast disk, good inter-server speed, etc)<br>
>> * Sales/marketing may view it as the number of martinis they can buy ;)<br>>> <br>>> Trees become unmanageable/hard to visualize for users beyond a couple hundred nodes. We are lucky that our geographical/DC-based hierarchy is relatively flat. This is why I was initially pushing for a tag-based system for selection (aka Zone/Host Capabilities).<br>
>> <br>>> Consider the way <a href="http://delicio.us">delicio.us</a> works. They manage many millions of URL's and tags are an effective way to slice & dice your way through the data: <br>>> "Show me all the URL's on [OpenStack] [Python] [Zones] [Scheduler]" ... blam.<br>
>> <br>>> This is also the way the old Trader services worked:<br>>> "I want a [wax transfer] [color] printer that can has [30ppm] and [300dpi] on [Floor 2]"<br>>> <br>>> "Near" simply has to mean the distance in zones from the most-optimal zones, based on the tags.<br>
>> <br>>> "I want a new instance with [GPU] and [Fast Disk] [Good inter-instance network speed] [near] [DRW] [DC1]"<br>>> * where "[near]" implies "as close as possible to" in zone distance.<br>
>> <br>>> Personally I don't like overloading the zone name to have a "meaningful" URI when we can get the same functionality with Capabilities/Tags already. And we already know we need Capability support anyway. Especially if it means enforcing a rigid hierarchy.<br>
>> <br>>> $0.02<br>>> <br>>> -S<br>>> <br>>> <br>>> <br>>> ________________________________________<br>>> From: openstack-bounces+sandy.walsh=<a href="http://rackspace.com">rackspace.com</a>@<a href="http://lists.launchpad.net">lists.launchpad.net</a> [openstack-bounces+sandy.walsh=<a href="http://rackspace.com">rackspace.com</a>@<a href="http://lists.launchpad.net">lists.launchpad.net</a>] on behalf of Eric Day [<a href="mailto:eday@oddments.org">eday@oddments.org</a>]<br>
>> Sent: Friday, February 11, 2011 4:30 AM<br>>> To: Justin Santa Barbara<br>>> Cc: <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a>; Devin Carlen<br>>> Subject: Re: [Openstack] Allowing clients to pass capability requests through tags?<br>
>> <br>>> The main reason I was proposing full location/zone of objects is to<br>>> allow this type of 'near' scheduling to happen without understanding<br>>> what the actual object is. For example, imagine we want to start an<br>
>> instance near a particular swift object. We could query the swift<br>>> object and in the metadata there could be a 'zone' tag (well, three,<br>>> one for each copy). For example:<br>>> <br>
>> get swift-12345: zone=<a href="http://rack12.room2.dc1.dfw.rackspace.com">rack12.room2.dc1.dfw.rackspace.com</a><br>>> <br>>> I can now use that zone name to:<br>>> <br>>> create_instance: openstack:near=<a href="http://rack12.room2.dc1.dfw.rackspace.com">rack12.room2.dc1.dfw.rackspace.com</a><br>
>> <br>>> The deployment can decide what 'near' is (perhaps a measure of link<br>>> speed or latency). This way a particular deployment that uses the<br>>> same URI/zone names across projects can account for locality without<br>
>> knowing what objects from different services are. If it were just<br>>> 'near=swift-12345', it would need to understand what a swift object<br>>> was and perform that lookup to find out where it is.<br>
>> <br>>> So you can still grab a zone tag from a volume you created:<br>>> <br>>> get vol-000001: <a href="http://rack4.room2.dc1.dfw.rackspace.com">rack4.room2.dc1.dfw.rackspace.com</a><br>>> <br>
>> and use the zone to launch an instance with:<br>>> <br>>> create_instance: openstack:near=<a href="http://rack4.room2.dc1.dfw.rackspace.com">rack4.room2.dc1.dfw.rackspace.com</a><br>>> <br>>> We can also write schedulers/tools for a particular deployment<br>
>> that understands the zones to just say 'always prefer in<br>>> <a href="http://dc1.dfw.rackspace.com">dc1.dfw.rackspace.com</a>', because power is cheaper there right now, or<br>>> '<a href="http://test.dc1.dfw.rackspace.com">test.dc1.dfw.rackspace.com</a>' because that is my test zone (perhaps<br>
>> only enabled for certain accounts in the scheduler too).<br>>> <br>>> -Eric<br>>> <br>>> On Thu, Feb 10, 2011 at 03:38:42PM -0800, Justin Santa Barbara wrote:<br>>> > I think the blueprint was largely complementary to the multi-zone stuff;<br>
>> > this is more about how the client _requests_ a particular<br>>> > location/capability through the API. The multi-zone blueprint seems to be<br>>> > more about how nova would satisfy those requests (in a non-trivial zone<br>
>> > structure.)<br>>> > The root motivator is indeed getting a 'good' connection to a storage<br>>> > volume. I'm thinking of iSCSI SAN storage here, so in my case this<br>
>> > probably means the SAN device with the least number of switches in<br>>> > between. There could well be SAN devices in each rack (e.g. Solaris<br>>> > volume nodes), or the devices could even be running on the host nodes, and<br>
>> > I don't believe that zones in the EC2 sense are sufficient here.<br>>> > But I guess that if the zone hierarchy went all the way down to the rack<br>>> > (or machine), that would work. So I could create a volume and it would<br>
>> > come back with a location of "<a href="http://rack4.room2.dc1.dfw.rackspace.com">rack4.room2.dc1.dfw.rackspace.com</a>" and I<br>>> > could then request allocation of machines in that same rack? Is that the<br>
>> > vision of the nested zones?<br>>> > I do have a concern that long-term if we _only_ use zones, that's trying<br>>> > to multiplex a lot of information into the zone hierarchy, and we can<br>
>> > really only put one attribute in there. I also like the flexibility of<br>>> > the 'openstack:near=vol-000001' request, because then the cloud can decide<br>>> > how near to place the instance based on its knowledge of the topology, and<br>
>> > the clients can be oblivious to the storage system and arrangement. But,<br>>> > my immediate requirement would indeed be satisfied if the zones went down<br>>> > to the rack/machine level.<br>
>> > An alternative way to look at zones and instance-types is that they're<br>>> > actually just fail-if-not-satisfiable tags of the creation request<br>>> > (openstack:+zone=us-east-1a and openstack:+instancetype=m1.large) They're<br>
>> > only distinguished attributes because AWS doesn't have an<br>>> > extensibility mechanism, which this blueprint would give us.<br>>> > Justin<br>>> ><br>>> > On Thu, Feb 10, 2011 at 3:12 PM, Devin Carlen <<a href="mailto:devcamcar@me.com">devcamcar@me.com</a>> wrote:<br>
>> ><br>>> > I haven't totally digested this blueprint yet but it seems like there is<br>>> > some overlap with what is being discussed with the multi zone metadata<br>>> > stuff. One approach might be to handle this awt the scheduler level<br>
>> > though and try to ensure things are always in the same zone when<br>>> > appropriate.<br>>> > I think the bigger question you raise is how to request local volumes<br>>> > when possible, yes?<br>
>> ><br>>> > Devin<br>>> > On Feb 10, 2011, at 3:37 PM, Justin Santa Barbara <<a href="mailto:justin@fathomdb.com">justin@fathomdb.com</a>><br>>> > wrote:<br>>> ><br>
>> > Does anyone have any thoughts/objections on the blueprint I posted for<br>>> > allowing clients to pass capability-requests through tags? I'm<br>>> > planning on starting implementation soon, so if people think this is a<br>
>> > bad idea I'd rather know before I start coding!<br>>> > Blueprint: <a href="https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities">https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities</a><br>
>> > Wiki: <a href="https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities">https://blueprints.launchpad.net/nova/+spec/use-metadata-tags-for-capabilities</a><br>>> > And a quick TLDR:<br>
>> > API clients need a way to request e.g. placement of machines near each<br>>> > other / near volumes, or that a volume be created with a particular<br>>> > RAID level, or that a machine be created in a HIPAA compliant<br>
>> > environment. (This is complementary to the work on hierarchical zones<br>>> > & URL naming, I believe)<br>>> > I propose using the instance tags for this, e.g. specifying<br>
>> > openstack:near=vol-000001 when creating an instance to request<br>>> > locating the instance 'close to' that volume.<br>>> > By default these requests would be best-effort and ignored-if-unknown;<br>
>> > if the client wants to specify that something is required and should<br>>> > fail if not understood or not satisfiable, they could use a "+" e.g.<br>>> > openstack:+location=*.<a href="http://dc1.north.rackspace.com">dc1.north.rackspace.com</a><br>
>> > Controversially (?), this would not be supported for clients using the<br>>> > AWS API, because tags can only be specified once the instance has<br>>> > already been created.<br>
>> > Feedback appreciated!<br>>> > Justin<br>>> ><br>>> > _______________________________________________<br>>> > Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
>> > Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>>> > Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
>> > More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br>>> <br>>> > _______________________________________________<br>>> > Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
>> > Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>>> > Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
>> > More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br>>> <br>>> <br>>> _______________________________________________<br>>> Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
>> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>>> Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>>> More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br>
>> <br>>> <br>>> Confidentiality Notice: This e-mail message (including any attached or<br>>> embedded documents) is intended for the exclusive and confidential use of the<br>>> individual or entity to which this message is addressed, and unless otherwise<br>
>> expressly indicated, is confidential and privileged information of Rackspace.<br>>> Any dissemination, distribution or copying of the enclosed material is prohibited.<br>>> If you receive this transmission in error, please notify us immediately by e-mail<br>
>> at <a href="mailto:abuse@rackspace.com">abuse@rackspace.com</a>, and delete the original message.<br>>> Your cooperation is appreciated.<br>> <br>> _______________________________________________<br>> Mailing list: <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>> Unsubscribe : <a href="https://launchpad.net/~openstack">https://launchpad.net/~openstack</a><br>> More help : <a href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a><br>
</div>