[Openstack] Pondering multi-tenant needs in nova.
Salvatore Orlando
Salvatore.Orlando at eu.citrix.com
Fri Feb 4 10:02:39 UTC 2011
Hi,
I agree with Aimon and Patrick that a VLAN per customer would be a good fit only for small deployments.
Moreover, if I'm not wrong, datacentres are typically organized in small layer-2 PoD in the Access layer interconnected by L3 routers (I'm referring to the Cisco Data Center Infrastructure design guide); to the best of my knowledge L3 routers do not forward VLAN tags. In the access layer, PoDs are typically small in order to limit the impact of L2 broadcast traffic, and, I believe, to make spanning tree working more efficiently.
One thing we might consider in order to overcome this limit is to employ L3 tunnelling protocols for VM traffic; in my opinion this might have several benefits:
- No need to perform any configuration, such as setting ports on trunking mode, on physical network switches: they would not see VLAN-tagged frames, but only IP traffic;
- MAC forwarding tables on physical network switches will hardly be saturated, even when using 'cheap' switches, as the only MACs they would see are the endpoints of the tunnels;
- L2 broadcast domain will still be contained in size, even if distributed over several PoD in the same data centre and potentially over several data centres.
One (potentially big) issue with tunnelling protocols is the topology of the network created by the tunnel themselves; it might define traffic bottlenecks, or even contain loops.
I have been using Open vSwitch for a few months now, which has been mentioned on several blueprints in the nova project. It supports GRE tunnels and VLAN tagging (though AFAIK only 802.1Q); it could potentially be a good fit for implementing multi-tenant support for large deployments.
Open vSwitch currently runs on both Xen and KVM hypervisors; however it is not supported in ESX, and (If I'm not wrong) in Hyper-V.
Regards,
Salvatore
-----Original Message-----
From: openstack-bounces+salvatore.orlando=eu.citrix.com at lists.launchpad.net [mailto:openstack-bounces+salvatore.orlando=eu.citrix.com at lists.launchpad.net] On Behalf Of Aimon Bustardo
Sent: 04 February 2011 07:14
To: Patrick Ancillotti
Cc: <openstack at lists.launchpad.net>
Subject: Re: [Openstack] Pondering multi-tenant needs in nova.
Hi Patrick,
that would be great if you would go into details. I am most interested in this as it directly effects our cloud platforms adoption of OpenStack and the subsequent networking blueprint we have designed (trying to get a hold of the networking blueprint Ewan is leading also).
Thank you,
Aimon
On 2/3/11 10:18 PM, Patrick Ancillotti wrote:
> Aimon,
>
> You're correct of course for simply defining a customer per VLAN as realistically we wouldn't hit 16M+ customers in any regional area as it stands today ;) but there are other issues with QinQ at large scale, especially with Layer 2 domains of the size that we're envisaging in the long run... many of which we can go into detail if you want?
>
> For those interested in QinQ : http://en.wikipedia.org/wiki/802.1QinQ
>
> Thanks,
> Patrick
>
> On 3 Feb 2011, at 23:45, Aimon Bustardo wrote:
>
>> Hi Patrick, by using 802.1QiQ aren't we increasing the total VLAN
>> count to 4096*4096 (16277216) possible VLANS, each with the
>> possibility of using the same /8 network? In which case he in table
>> storage of MACs is the limiting factor.
>>
>>
>> Aimon
>>
>> On 2/3/11 8:03 PM, Patrick Ancillotti wrote:
>>> Hey Guys,
>>>
>>> I think Paul may have gotten a bit mixed up between VLAN and CAM tables on switches. The VLAN part of an ethernet frame is 12 bits (0 - 4095) which limits it accordingly. CAM tables however are a limit within switching gear that lists the MAC addresses and their respective source and destination ports. The lower end Cisco switches like the 2960 class switches have a limit of around 8000 MAC addresses, whereas others higher in the stack such as the 4948E class have limits upwards of 55000 MAC addresses, although some vendors have CAM tables in the hundreds of thousands for even their mid range switches.
>>>
>>> That said, VLAN's are extremely limited, even with QinQ (VLANs within VLANs) when you're talking about 50 000 customers, or even 500 000 customers in the same layer 2 domain, and in many cases using smaller layer 2 domains creates unfortunately small service areas for capacity.
>>>
>>> For more info:
>>> http://en.wikipedia.org/wiki/Virtual_LAN
>>> http://en.wikipedia.org/wiki/CAM_Table
>>>
>>> Thanks,
>>> Patrick
>>>
>>> On 3 Feb 2011, at 07:47, Paul Voccio wrote:
>>>
>>>> Diego,
>>>>
>>>> Due to our networking topology, having a vlan per customer isn't
>>>> really feasible. Most switches are limited at 4k or 8k or even 32k.
>>>> With more customers than these switches can reasonably accommodate,
>>>> having a single vlan per customer either limits the portability
>>>> within a cloud or limits the scale at which you can build your
>>>> cloud. Open vSwitch will alleviate some of this pain, but until we
>>>> get it in XenServer, we're somewhat stuck on flat networking.
>>>>
>>>> Paul
>>>>
>>>> On 2/3/11 4:20 AM, "Diego Parrilla Santamaría"
>>>> <diego.parrilla.santamaria at gmail.com> wrote:
>>>>
>>>>> Hi Monsyne,
>>>>>
>>>>> it's a very interesting topic and I'm curious about the reason why
>>>>> you are using the Flat Networking set up. From the conversations
>>>>> in other threads it seems the Service Providers prefer different
>>>>> networking
>>>>> approaches: VLAN oriented basically.
>>>>>
>>>>> Regards
>>>>> Diego
>>>>>
>>>>> -
>>>>> Diego Parrilla
>>>>> nubeblog.com | nubeblog at nubeblog.com | twitter.com/nubeblog
>>>>> +34 649 94 43 29
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Feb 3, 2011 at 2:37 AM, Monsyne Dragon
>>>>> <mdragon at rackspace.com>
>>>>> wrote:
>>>>>> I am sorting out some possible implementations for the
>>>>>> multi-tenant-accounting blueprint, and the related
>>>>>> system-usage-records bp, and I just wanted to run this by anyone
>>>>>> interested in such matters.
>>>>>>
>>>>>> Basically, for multitenant purposes we need to introduce the
>>>>>> concept of an 'account' in nova, representing a customer, that
>>>>>> basically acts as a label for a group of resources (instances,
>>>>>> etc), and for access control (i.e customer a cannot mess w/
>>>>>> customer b's stuff)
>>>>>>
>>>>>> There was some confusion on how best to implement this, in
>>>>>> relation to nova's project concept. Projects are kind of like
>>>>>> what we want an account to be, but there are some associations
>>>>>> (like one project per network) which are not valid for our flat
>>>>>> networking setup. I am kind of straw-polling on which is better
>>>>>> here:
>>>>>>
>>>>>> The options are:
>>>>>> 1) Create a new 'account' concept in nova, with an account
>>>>>> basically being a subgroup of a project (providers would use a
>>>>>> single, default project, with additional projects added if needed
>>>>>> for separate brands, or resellers, etc), add in access control
>>>>>> per account as well as project, and make sure apis/auth specify
>>>>>> account appropriately, have some way for a default account to
>>>>>> used (per project) so account doesn't get in the way for
>>>>>> non-multitenant users.
>>>>>>
>>>>>> 2) having account == nova's "project", and changing the network
>>>>>> associations, etc so projects can support our model (as well as
>>>>>> current models). Support for associating accounts (projects)
>>>>>> together for resellers, etc would either be delegated outside of
>>>>>> nova or added later (it's not a current requirement).
>>>>>>
>>>>>> In either case, accounts would be identified by name, which would
>>>>>> be an opaque string an outside system/person would assign, and
>>>>>> could structure to their needs (ie. for associating accounts with
>>>>>> common prefixes, etc)
>>>>>>
>>>>>> --
>>>>>>
>>>>>> --
>>>>>> -Monsyne Dragon
>>>>>> work: 210-312-4190
>>>>>> mobile 210-441-0965
>>>>>> google voice: 210-338-0336
>>>>>>
>>>>>>
>>>>>>
>>>>>> Confidentiality Notice: This e-mail message (including any
>>>>>> attached or embedded documents) is intended for the exclusive and
>>>>>> confidential use of the individual or entity to which this
>>>>>> message is addressed, and unless otherwise expressly indicated,
>>>>>> is confidential and privileged information of Rackspace.
>>>>>> Any dissemination, distribution or copying of the enclosed
>>>>>> material is prohibited.
>>>>>> If you receive this transmission in error, please notify us
>>>>>> immediately by e-mail at abuse at rackspace.com, and delete the
>>>>>> original message.
>>>>>> Your cooperation is appreciated.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to : openstack at lists.launchpad.net
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to : openstack at lists.launchpad.net
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>> Confidentiality Notice: This e-mail message (including any attached
>>>> or embedded documents) is intended for the exclusive and
>>>> confidential use of the individual or entity to which this message
>>>> is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace.
>>>> Any dissemination, distribution or copying of the enclosed material is prohibited.
>>>> If you receive this transmission in error, please notify us
>>>> immediately by e-mail at abuse at rackspace.com, and delete the original message.
>>>> Your cooperation is appreciated.
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to : openstack at lists.launchpad.net
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help : https://help.launchpad.net/ListHelp
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help : https://help.launchpad.net/ListHelp
>> --
>> Aimon Bustardo | Morph Labs | +1.310.625.0608 (mobile) |
>> +1.310.437.4898 (office) | www.morphlabs.com | aimon at mor.ph
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>
>
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use
> of the individual or entity to which this message is addressed, and
> unless otherwise expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is prohibited.
> If you receive this transmission in error, please notify us
> immediately by e-mail at abuse at rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
--
Aimon Bustardo | Morph Labs | +1.310.625.0608 (mobile) | +1.310.437.4898 (office) | www.morphlabs.com | aimon at mor.ph
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack at lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
More information about the Openstack
mailing list