Open Stack

Hi, Phil,

I am sorry that no enough information for you to understand the cascading in the document.  If we can talk f2f, I can explain much more in detail. But in short, I will give a simplified picture how a virtual machine will be booted:

The general process to boot a VM is like this:
Nova API -> Nova Scheduler -> Nova Compute( manager ) -> Nova Compute( Libvirt driver ) -> Nova Compute( KVM )

After OpenStack cascading is introduced, the process is a little difference and can be divided into two parts:
1. inside cascading OpenStack: Nova API -> Nova Scheduler -> Nova Proxy ->
2, inside cascaded OpenStack: Nova API -> Nova Scheduler -> Nova Compute( manager ) -> Nova Compute( Libvirt driver ) -> Nova Compute( KVM )
 
After schedule a Nova-Proxy, the instance object will be persisted in the DB in the cascading layer. And VM query to the cloud will be answered by the  cascading Nova API from cascading layer DB. No need to touch cascaded Nova. (it's not bad thing to persist the data in the cascading layer, quota control, system healing and consistency correction, fast user experience, etc...)

All VM generation in the cascaded OpenStack has nothing different with the process of general VM boot process, and is a asynchronous process from the cascading layer.

How the scheduler in the cascading layer to select proper Nova Proxy. The answer is that if hosts the cascaded Nova was added to AZ1 (AZ: availability zone in short), then the Nova proxy (it's a host in the cascading layer) will also be added to AZ1 in the cascading layer, and this nova proxy will be configured to send all request to the endpoint of the regarding cascaded Nova. And scheduler will be configured to use availability zone filter only, we know all VM boot  request has AZ parameter, that's the key for scheduling in the cascading layer.  Host Aggregate could be done in the same way.

After Nova-proxy receive the RPC message from the Nova-scheduler, it will not work like libvirt driver to boot a VM in the local host, it'll pickup all request parameter and call python client, to send the restful nova-boot request to the cascaded Nova.

How the flavor will be synchronized to the cascaded Nova? The flavor will be synchronized to the cascaded Nova only if the flavor does not exist in the cascaded Nova, or the flavor is recently updated but not synchronized to the cascaded Nova. Because the VM boot request has been answered after scheduling, so all the things done in the nova-proxy is asynchronous operation just like a VM booted in a host, it'll take seconds to minutes in general host, but in the cascading, some API calling will be done by nova-proxy to cascaded Nova, or cascaded Cinder & Neutron. 

I wrote a few blogs to explain something in detail, but I am too busy, and not able to write all things we have done in the PoC. [ 1 ]

[1] blog about cascading:  http://www.linkedin.com/today/author/23841540

Best Regards

Chaoyi Huang

________________________________________
From: Day, Phil [philip.day at hp.com]
Sent: 23 October 2014 19:24
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all] [tc] Multi-clouds integration by OpenStack cascading

Hi,

> -----Original Message-----
> From: joehuang [mailto:joehuang at huawei.com]
> Sent: 23 October 2014 09:59
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [all] [tc] Multi-clouds integration by OpenStack
> cascading
>
> Hi,
>
> Because I am not able to find a meeting room to have deep diving OpenStack
> cascading before design summit. You are welcome to have a f2f conversation
> about the cascading before design summit. I planned to stay at Paris from
> Oct.30 to Nov.8, if you have any doubt or question, please feel free to
> contact me. All the conversation is for clarification / idea exchange purpose,
> not for any secret agreement purpose. It is necessary before design summit,
> for design summit session, it's only 40 minutes, if all 40 minutes are spent on
> basic question and clarification, then no valuable conclusion can be drawn in
> the meeting. So I want to work as client-server mode, anyone who is
> interested in talking cascading with me, just tell me when he will come to the
> hotel where I stay at Paris, then a chat could be made to reduce
> misunderstanding, get more clear picture, and focus on what need to be
> discussed and consensuses during the design summit session.
>
Sure, I'll certainly try and find some time to meet and talk.


> >>>"It kind of feels to me that if we just concentrated on the part of this that
> is working out how to distribute/federate Neutron then we'd have a solution
> that could be mapped as easily cells and/or regions - and I wonder if then
> why really need yet another aggregation concept ?"
>
> My answer is that it seems to be feasible but can not meet the muti-site
> cloud demand (that's the drive force for cascading):
> 1) large cloud operator ask multi-vendor to build the distributed but unified
> multi-site cloud together and each vendor has his own OpenStack based
> solution. If shared Nova/Cinder with federated Neutron used, the cross data
> center integration through RPC message for multi-vendor infrastrcuture is
> very difficult, and no clear responsibility boundry, it leads to difficulty for
> trouble shooting, upgrade, etc.

So if the scope of what you're doing to is to provide a single API across multiple clouds that are being built and operated independently then I'm not sure how you can impose enough consistency to guarantee any operations.    What if one of those clouds has Nova AZs configured, and your using (from what I understand AZs to try and route to a specific cloud) ?   How do you get image and flavor consistency across the clouds ?

I picked up on the Network aspect because that seems to be something you've covered in some depth here https://docs.google.com/presentation/d/1wIqWgbZBS_EotaERV18xYYA99CXeAa4tv6v_3VlD2ik/edit?pli=1#slide=id.g390a1cf23_2_149 so I'd assumed it was an intrinsic part of your proposal.  Now I'm even less clear on the scope of what you're trying to achieve ;-(

If this is a federation layer for in effect arbitrary Openstack clouds then it kind of feels like it can't be anything other than an aggregator of queries (list the VMs in all of the clouds you know about, and show the results in one output).   If you have to make API calls into many clouds (when only one of them may have any results) then that feels like it would be a performance issue.  If you're going to cache the results somehow then in effect you needs the Cells approach for propogating up results, which means the sub-clouds have to be co-operating.

Maybe I missed it somewhere, but is there a clear write-up of the restrictions / expectations of sub-clouds to work in this model ?

Kind Regards
Phil

> 2) restful API /CLI is required for each site to make the cloud always workable
> and manageable. If shared Nova/Cinder with federated Neutron, then some
> data center is not able to expose restful API/CLI for management purpose.
> 3) the unified cloud need to expose open and standard api. If shared Nova /
> Cinder with federated Neutron, this point can be arhieved.
>
> Best Regards
>
> Chaoyi Huang ( joehuang )
>
> -----Original Message-----
> From: henry hly [mailto:henry4hly at gmail.com]
> Sent: Thursday, October 23, 2014 3:13 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [all] [tc] Multi-clouds integration by OpenStack
> cascading
>
> Hi Phil,
>
> Thanks for your feedback, and patience of this long history reading :) See
> comments inline.
>
> On Wed, Oct 22, 2014 at 5:59 PM, Day, Phil <philip.day at hp.com> wrote:
> >> -----Original Message-----
> >> From: henry hly [mailto:henry4hly at gmail.com]
> >> Sent: 08 October 2014 09:16
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> Subject: Re: [openstack-dev] [all] [tc] Multi-clouds integration by
> >> OpenStack cascading
> >>
> >> Hi,
> >>
> >> Good questions: why not just keeping multiple endpoints, and leaving
> >> orchestration effort in the client side?
> >>
> >> From feedback of some large data center operators, they want the
> >> cloud exposed to tenant as a single region with multiple AZs, while
> >> each AZ may be distributed in different/same locations, very similar with
> AZ concept of AWS.
> >> And the OpenStack API is indispensable for the cloud for eco-system
> >> friendly.
> >>
> >> The cascading is mainly doing one thing: map each standalone child
> >> Openstack to AZs in the parent Openstack, hide separated child
> >> endpoints, thus converge them into a single standard OS-API endpoint.
> >>
> >> One of the obvious benefit doing so is the networking: we can create
> >> a single Router/LB, with subnet/port member from different child,
> >> just like in a single OpenStack instance. Without the parent
> >> OpenStack working as the aggregation layer, it is not so easy to do
> >> so. Explicit VPN endpoint may be required in each child.
> >>
> > I've read through the thread and the various links, and to me this still
> sounds an awful lot like having multiple regions in Keystone.
> >
> > First of all I think we're in danger of getting badly mixed up in terminology
> here around AZs which is an awfully overloaded term - esp when we make
> comparisons to AWS AZs.  Whether we think the current Openstack usage of
> these terms or not, lets at least stick to how they are currently defined and
> used in Openstack:
> >
> > AZs - A scheduling concept in Nova and Cinder.    Simply provides some
> isolation schemantic about a compute host or storage server.  Nothing to do
> with explicit physical or geographical location, although some degree of that
> (separate racks, power, etc) is usually implied.
> >
> > Regions - A keystone concept for a collection of Openstack Endpoints.
> They may be distinct (a completely isolated set of Openstack service) or
> overlap (some shared services).  Openstack clients support explicit user
> selection of a region.
> >
> > Cells - A scalability / fault-isolation concept within Nova.  Because Cells
> aspires to provide all Nova features transparently across cells this kind or acts
> like multiple regions where only the Nova service is distinct (Networking has
> to be common, Glance has to be common or at least federated in a
> transparent way, etc).   The difference from regions is that the user doesn’t
> have to make an explicit region choice - they get a single Nova URL for all
> cells.   From what I remember Cells originally started out also using the
> existing APIs as the way to connect the Cells together, but had to move away
> from that because of the performance overhead of going through multiple
> layers.
> >
> >
>
> Agree, it's very clear now. However isolation is not all about hardware and
> facility fault, REST API is preferred in terms of system level isolation despite
> the theoretical protocol serialization overhead.
>
> >
> > Now with Cascading it seems that we're pretty much building on the
> > Regions concept, wrapping it behind a single set of endpoints for user
> > convenience, overloading the term AZ
>
> Sorry not very certain of the meaning "overloading". It's just a configuration
> choice by admin in the wrapper Openstack. As you mentioned, there is no
> explicit definition of what a AZ should be, so Cascading select to map it to a
> child Openstack. Surely we could use another concept or invent new concept
> instead of AZ, but AZ is the most appropriate one because it share the same
> semantic of "isolation"
> with those child.
>
> > to re-expose those sets of services to allow the user to choose between
> them (doesn't this kind of negate the advantage of not having to specify the
> region in the client- is that really such a bit deal for users ?) , and doing
> something to provide a sort of federated Neutron service - because as we all
> know the hard part in all of this is how you handle the Networking.
> >
> > It kind of feels to me that if we just concentrated on the part of this that is
> working out how to distribute/federate Neutron then we'd have a solution
> that could be mapped as easily cells and/or regions - and I wonder if then
> why really need yet another aggregation concept ?
> >
>
> I agree that it's not so huge a gap between cascading AZ and standalone
> endpoints for Nova and Cinder. However, wrapping is strongly needed by
> customer feedback for Neutron, especially for those who operate multiple
> internally connected DC. They don't like to force tenants to create multiple
> route domain, connected with explicit vpnaas. Instead they prefer a simple
> L3 router connecting subnets and ports from different DC, just as today
> within the single endpoint.
>
> It's certainly a reasonable advice with hybrid solution: standalone Nova
> endpoints, and single wrapped tree of distributed Neutron. In fact tt has
> been an option in the history of our Cascading project, but we found it is
> complex for the customer to manage, and not so clear in the architecture
> model. Detail f2f discuss is appreciated if there is a chance later.
>
> Best Regards
> Wu
>
>
>
> > Phil
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev