Open Stack

James,

Great stuff! Please see @PCM in-line...



On Tue, Aug 25, 2015 at 6:26 PM James Dempsey <jamesd at catalyst.net.nz>
wrote:

> Oops, sorry about the blank email.  Answers/Questions in-line.
>
> On 26/08/15 07:46, Paul Michali wrote:
> > Previous post only went to dev list. Ensuring both and adding a bit
> more...
> >
> >
> >
> > On Tue, Aug 25, 2015 at 8:37 AM Paul Michali <pc at michali.net> wrote:
> >
> >> Xav,
> >>
> >> The discussion is very important, and hence why both Kyle and I have
> been
> >> posting these questions on the operator (and dev) lists. Unfortunately,
> I
> >> wasn't subscribed to the operator's list and missed some responses to
> >> Kyle's message, which were posted only to that list.
> >>
> >> As a result, I had an incomplete picture and posted this thread to see
> if
> >> it was OK to do this without backward compatibility, based on the
> >> (incorrect) assumption that there was no production use. That is
> corrected
> >> now, and I'm getting all the messages and thanks to everyone, have
> input on
> >> messages I missed.
> >>
> >> So given that, let's try a reset on the discussion, so that I can better
> >> understand the issues...
> >>
>
> Great!  Thanks very much for expanding the scope.  We really appreciate it.
>
> >> Do you feel that not having backward compatibility (but having a
> migration
> >> path) would seriously affect you or would it be manageable?
>
> Currently, this feels like it would seriously affect us.  I don't feel
> confident that the following concerns won't cause us big problems.
>
>
> As Xav mentioned previously, we have a few major concerns:
>

@PCM Thanks! It's good to know what things we need to be aware of.



>
> 1) Horizon compatibility
>
> We run a newer version of horizon than we do neutron.  If Horizon
> version X doesn't work with Neutron version X-1, this is a very big
> problem for us.
>

@PCM Interesting. I always thought that Horizon updates lagged Neutron
changes, and this wouldn't be a concern.



>
> 2) Service interruption
>
> How much of a service interruption would the 'migration path' cause?


@PCM The expectation of the proposal is that the migration would occur as
part of the normal OpenStack upgrade process (new services installed,
current services stopped, database migration occurs, new services are
started).

It would have the same impact as what would happen today, if you update
from one release to another. I'm sure you folks have a much better handle
on that impact and how to handle it (maintenance windows, scheduled
updates, etc).


We
> all know that IPsec VPNs can be fragile...  How much of a guarantee will
> we have that migration doesn't break a bunch of VPNs all at the same
> time because of some slight difference in the way configurations are
> generated?
>

@PCM I see the risk as extremely low. With the migration, the end result is
really just moving/copying fields from one table to another. The underlying
configuration done to *Swan would be the same.

For example, the subnet ID, which is specified in the VPN service API and
stored in the vpnservices table, would be stored in a new vpn_endpoints
table, and the ipsec_site_connections table would reference that entry
(rather than looking up the subnet in the vpnservices table).



> 3) Heat compatibility
>
> We don't always run the same version of Heat and Neutron.
>

@PCM I must admit, I've never used Heat, and am woefully ignorant about it.
Can you elaborate on Heat concerns as may be related to VPN API differences?

Is Heat being used to setup VPN connections, as part of orchestration?



>
> >>
> >> Is there pain for the customers beyond learning about the new API
> changes
> >> and capabilities (something that would apply whether there is backward
> >> compatibility or not)?
> >>
>
> See points 1,2, and 3 above.
>

> >
> > Another implication of not having backwards compatibility would be that
> > end-users would need to immediately switch to using the new API, once the
> > migration occurs, versus doing so on their own time frame.  Would this
> be a
> > concern for you (customers not having the convenience of delaying their
> > switch to the new API)?
> >
> >
> >> I was thinking that backward incompatible changes would adversely affect
> >> people who were using client scripts/apps to configure (a large number
> of)
> >> IPsec connections, where they'd have to have client scripts/apps
> in-place
> >> to support the new API.
> >>
>
> This is actually less of a concern.  We have found that VPN creation is
> mostly done manually and anyone who is clever enough to make IPsec go is
> clever enough to learn a new API/horizon interface.
>

@PCM Do you see much reliance on tooling to setup VPN (such that having to
update the tooling would be a concern for end-users), or is this something
that could be managed through process/preparation?



>
> >
> > Which is more of a logistics issue, and could be managed, IMHO.
> >
> >
> >
> >>
> >> Would there be customers that would fall into that category, or are
> >> customers manually configuring IPSec connections in that they could just
> >> use the new API?
> >>
>
> Most customers could easily adapt to a new API.
>
> >>
> > Are there other adverse effects of not having backward compatibility that
> >> need to be considered?
> >>
>
> As with the dashboard, heat also needs a bit of consideration.  How
> would Heat deal with the API changes?
>

@PCM I don't know and will try to learn more. Is Heat used to configure VPN
connections? I need to understand more about the relationship there.



> >
> > So far, I'm identifying one effect that is more of a convenience
> (although
> > nice one at that), and one effect that can be avoided by planning for the
> > upgrade.  I'd like to know if I'm missing something more important to
> > operators.
> >
> > I'd also like to know if we thing there is a user base large enough (and
> > how large is large?0 that would warrant going through the complexity and
> > risk to support both API versions simultaneously?
>
> This is a bit frustrating...  It implies that only large clouds matter.
>

@PCM Sorry. That's not the intent of the statement that I was trying to get
across.

At the time I made the statement, my only knowledge of a need for
supporting multiple simultaneous versions was to accommodate client tools
(which could be handled via proper planning,IMHO) and end-users who would
like to switch to the newer API on their own schedule, versus when the
cloud is upgraded.

If there aren't many people who need that flexibility, then I'm asking
whether it makes sense to assume the effort and risk in getting this right,
as it is not easy.

I was "assuming" that if there is a large number of end-users wanting this
flexibility, that an operator would feel compelled to insist on this
capability.

>From your case, it doesn't sound like that aspect is a concern at all (end
users would just adapt to the new version immediately).  It sounds like
Horizon and Heat are more of a concern for you.



 There is a further tacit implication the API is not really a contract
> that can be relied upon.
>

@PCM Not sure I follow that statement. Can you elaborate on what you mean?

I'm not talking about not supporting an API version in some cases. I'm
talking about not supporting two API versions *simultaneously.*



> We are operating a multi-region cloud with many clients who depend upon
> VPNaaS for business-critical production workloads.
>
> Of course, this is a two-way road!  We all want what is best for
> OpenStack, so we should talk about the complexity and risk on your end.
>  Can you tell us more about that?


@PCM Sure. I updated the developer reference document on this design, which
is under review. Check out https://review.openstack.org/#/c/191944/,
especially towards the bottom, where there is a section added with my
initial thoughts on how to handle backward compatibility. I'd appreciate
feedback in the review for any comments, questions, concerns you may have.



> I really have no interest in being an
> operator who demands the world from developers, but I am worried about
> what all this means for my cloud.
>

@PCM Understood completely, and is why I appreciate the discussion on this.
As a developer, I'm trying to provide a (great? :) feature, without
over-engineering things. I'm trying to strike a balance between risk,
effort, and capability.

Regards,

Paul Michali (pc_m)




>
> Cheers,
> James Dempsey
>
> --
> James Dempsey
> Senior Cloud Engineer
> Catalyst IT Limited
> +64 4 803 2264
> --
>
>
> >
> > Regards,
> >
> > Paul Michali (pc_m)
> >
> >
> >> Specifically, we're talking about the VPN service create API no longer
> >> taking a subnet ID (instead an endpoint group is create that contains
> the
> >> subnet ID), and the IPSec site-to-site connection create API would no
> >> longer take a list of peer CIDRs, but instead would take a pair of
> endpoint
> >> group IDs (one for the local subnet(s) formally specified by the service
> >> API, and one for peer CIDRs).
> >>
> >
> >
> >>
> >>
> >> Regards,
> >>
> >> Paul Michali (pc_m)
> >>
> >>
> >> On Mon, Aug 24, 2015 at 5:32 PM Xav Paice <xavpaice at gmail.com> wrote:
> >>
> >>> I'm sure I'm not the only one that finds the vast amount of traffic in
> >>> the dev list to be completely unmanageable to catch the important
> messages
> >>> - the ops list is much lower traffic, and as an operator I pay a bunch
> more
> >>> attention to it.  The discussion of deprecating an API is something
> that
> >>> HAS to be discussed with operators, on the operators list or
> highlighted
> >>> somehow so that people get attention drawn to the message.
> >>>
> >>> Let's be clear - I fully appreciate the extra effort that would be
> >>> required in supporting both the new and the old APIs, and also would
> >>> absolutely love to see the new feature.  I do think we need to be able
> to
> >>> support our customers in the transition, and extra pain for them
> results in
> >>> lower uptake of the services we provide.
> >>>
> >>> On 25 August 2015 at 09:27, Xav Paice <xavpaice at gmail.com> wrote:
> >>>
> >>>> Also:
> >>>>
> >>>>
> >>>>
> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007928.html
> >>>>
> >>>>
> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007891.html
> >>>>
> >>>>
> >>>> On 25 August 2015 at 09:09, Kevin Benton <blak111 at gmail.com> wrote:
> >>>>
> >>>>> It sounds like you might have missed a couple responses:
> >>>>>
> >>>>>
> >>>>>
> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007903.html
> >>>>>
> >>>>>
> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007910.html
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 24, 2015 at 1:53 PM, Paul Michali <pc at michali.net>
> wrote:
> >>>>>
> >>>>>> Xav,
> >>>>>>
> >>>>>> In the email, there were no responses of anyone using VPNaaS *in a
> >>>>>> production environment*. Summary from responders:
> >>>>>>
> >>>>>> Erik M - Tried in Juno with no success. Will retry.
> >>>>>> Edgar M - said no reports from operators about VPNaaS code
> >>>>>> Sam S - Using VPN in VMs and not VPNaaS
> >>>>>> Kevin B - Not used. Use VMs instead
> >>>>>> Sriram S - Indicating not used.
> >>>>>>
> >>>>>> If I misread the responses, or if someone has not spoken up, right
> now
> >>>>>> is the time to let us know of your situation and the impact this
> proposal
> >>>>>> would have on your use of VPNaaS IPSec site-to-site connections.
> >>>>>>
> >>>>>> The request here, is that if operators are not using this in a
> >>>>>> production deployment where they need backward compatibility, then
> we'd
> >>>>>> like to avoid having to provide the complexity needed to support
> backward
> >>>>>> compatibility.
> >>>>>>
> >>>>>> In the proposal, there are two APIs that would be changed with this
> >>>>>> enhancement. It's detailed in reference [1], and I can elaborate,
> if needed.
> >>>>>>
> >>>>>> Please keep in mind, that users/operators using previous versions,
> can
> >>>>>> upgrade to the new version, and any existing VPNaaS configuration
> will be
> >>>>>> automatically migrated to the new table structures, so that
> existing IPSec
> >>>>>> connections would continue to operate with the new release.
> >>>>>>
> >>>>>> The proposal would not support using the older APIs, once the new
> APIs
> >>>>>> are available. Client apps/scripts would need to be updated to use
> the new
> >>>>>> API (neutron client and the Horizon dashboard will be updated as
> part of
> >>>>>> the overall effort).
> >>>>>>
> >>>>>> I hope that clarifies the discussion.
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Paul Michali (pc_m)
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Aug 24, 2015 at 3:50 PM Xav Paice <xavpaice at gmail.com>
> wrote:
> >>>>>>
> >>>>>>> On 25/08/15 06:32, Paul Michali wrote:
> >>>>>>>
> >>>>>>> As part of the effort to provide support for multiple local subnets
> >>>>>>> for VPNaaS IPSec connections, there are three API changes planned
> [1].
> >>>>>>>
> >>>>>>> One is to add a new "endpoint groups" API that will describe what
> is
> >>>>>>> being connected. This is the place were local and peer subnets
> will be
> >>>>>>> specified. It will allow future VPN flavors to reuse this API for
> other
> >>>>>>> types of VPN endpoints.
> >>>>>>>
> >>>>>>> The second is to modify the IPSec site connection API to make use
> of
> >>>>>>> this new endpoint groups API and no longer use the peer CIDRs
> arguments.
> >>>>>>>
> >>>>>>> The third is to remove the subnet ID from the VPN service API, and
> >>>>>>> again, use the new endpoint groups API for this information (and
> to allow
> >>>>>>>> 1).
> >>>>>>>
> >>>>>>> A previous email send out from Kyle Mestery, asking about the
> >>>>>>> production usage of VPNaaS, did not indicate that this service was
> used in
> >>>>>>> production by operators.
> >>>>>>>
> >>>>>>>
> >>>>>>> Not true.  There were replies indicating that it is indeed in use,
> >>>>>>> but perhaps not from operators of installations deemed significant
> enough -
> >>>>>>> what's the criteria here?
> >>>>>>>
> >>>>>>>
> >>>>>>> As a result, we'd like to propose making this change as a
> >>>>>>> non-backward compatible change, which greatly reduces the effort.
> >>>>>>>
> >>>>>>> The change WILL support migration, so older databases can be
> migrated
> >>>>>>> to the new schema and continue to operate, with future changes
> using the
> >>>>>>> new API. This gives a smooth upgrade path to the new capability.
> >>>>>>>
> >>>>>>> The change would NOT support the older API in parallel with the new
> >>>>>>> API, so users upgrading would need to also update any client
> scripts/tools
> >>>>>>> to use the revised/new VPN API.
> >>>>>>>
> >>>>>>>
> >>>>>>> How will this integrate with Horizon?  The majority of our users
> >>>>>>> don't use the API directly, but use Horizon for the config, does
> this mean
> >>>>>>> we'll be tied in to using a particular version of Horizon to match
> Neutron?
> >>>>>>>
> >>>>>>>
> >>>>>>> If this is acceptable to operators, we'd like to go forward with
> >>>>>>> these plans. Please respond within 7 days of this email, if you
> have
> >>>>>>> concerns about the proposal.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Paul Michali (IRC pc_m)
> >>>>>>> VPNaaS Core
> >>>>>>>
> >>>>>>> Ref: https://review.openstack.org/#/c/191944/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> __________________________________________________________________________
> >>>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>>> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> __________________________________________________________________________
> >>>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>>> Unsubscribe:
> >>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> __________________________________________________________________________
> >>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>> Unsubscribe:
> >>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kevin Benton
> >>>>>
> >>>>>
> >>>>>
> __________________________________________________________________________
> >>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>> Unsubscribe:
> >>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150826/6d458d60/attachment.html>