Open Stack

On 26/08/15 23:43, Paul Michali wrote:
> James,
> 
> Great stuff! Please see @PCM in-line...
> 
> On Tue, Aug 25, 2015 at 6:26 PM James Dempsey <jamesd at catalyst.net.nz>

<SNIP>

>> 1) Horizon compatibility
>>
>> We run a newer version of horizon than we do neutron.  If Horizon
>> version X doesn't work with Neutron version X-1, this is a very big
>> problem for us.
>>
> 
> @PCM Interesting. I always thought that Horizon updates lagged Neutron
> changes, and this wouldn't be a concern.
> 

@JPD
Our Installed Neutron typically lags Horizon by zero or one release.  My
concern is how will Horizon version X cope with a point-in-time API
change?  Worded slightly differently: We rarely update Horizon and
Neutron at the same time so there would need to be a version(or
versions) of Horizon that could detect a Neutron upgrade and start using
the new API.  (I'm fine if there is a Horizon config option to select
old/new VPN API usage.)

> 
> 
>>
>> 2) Service interruption
>>
>> How much of a service interruption would the 'migration path' cause?
> 
> 
> @PCM The expectation of the proposal is that the migration would occur as
> part of the normal OpenStack upgrade process (new services installed,
> current services stopped, database migration occurs, new services are
> started).
> 
> It would have the same impact as what would happen today, if you update
> from one release to another. I'm sure you folks have a much better handle
> on that impact and how to handle it (maintenance windows, scheduled
> updates, etc).
> 

@JPD This seems fine.

> 
> We
>> all know that IPsec VPNs can be fragile...  How much of a guarantee will
>> we have that migration doesn't break a bunch of VPNs all at the same
>> time because of some slight difference in the way configurations are
>> generated?
>>
> 
> @PCM I see the risk as extremely low. With the migration, the end result is
> really just moving/copying fields from one table to another. The underlying
> configuration done to *Swan would be the same.
> 
> For example, the subnet ID, which is specified in the VPN service API and
> stored in the vpnservices table, would be stored in a new vpn_endpoints
> table, and the ipsec_site_connections table would reference that entry
> (rather than looking up the subnet in the vpnservices table).
> 

@JPD This makes me feel more comfortable; thanks for explaining.

> 
> 
>> 3) Heat compatibility
>>
>> We don't always run the same version of Heat and Neutron.
>>
> 
> @PCM I must admit, I've never used Heat, and am woefully ignorant about it.
> Can you elaborate on Heat concerns as may be related to VPN API differences?
> 
> Is Heat being used to setup VPN connections, as part of orchestration?
> 

@JPD
My concerns are two-fold:

1) Because Heat makes use of the VPNaaS API, it seems like the same
situation exists as with Horizon.  Some version or versions of Heat will
need to be able to make use of both old and new VPNaaS APIs in order to
cope with a Neutron upgrade.

2) Because we use Heat resource types like
OS::Neutron::IPsecSiteConnection [1], we may lose the ability to
orchestrate VPNs if endpoint groups are not added to Heat at the same time.


Number 1 seems like a real problem that needs a fix.  Number 2 is a fact
of life that I am not excited about, but am prepared to deal with.

Yes, Heat is being used to build VPNs, but I am prepared to make the
decision on behalf of my users... VPN creation via Heat is probably less
important than the new VPNaaS features, but it would be really great if
we could work on the updated heat resource types in parallel.

[1]
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Neutron::IPsecSiteConnection

> 
> 
>>
>>>>
>>>> Is there pain for the customers beyond learning about the new API
>> changes
>>>> and capabilities (something that would apply whether there is backward
>>>> compatibility or not)?
>>>>
>>
>> See points 1,2, and 3 above.
>>
> 
>>>
>>> Another implication of not having backwards compatibility would be that
>>> end-users would need to immediately switch to using the new API, once the
>>> migration occurs, versus doing so on their own time frame.  Would this
>> be a
>>> concern for you (customers not having the convenience of delaying their
>>> switch to the new API)?
>>>
>>>
>>>> I was thinking that backward incompatible changes would adversely affect
>>>> people who were using client scripts/apps to configure (a large number
>> of)
>>>> IPsec connections, where they'd have to have client scripts/apps
>> in-place
>>>> to support the new API.
>>>>
>>
>> This is actually less of a concern.  We have found that VPN creation is
>> mostly done manually and anyone who is clever enough to make IPsec go is
>> clever enough to learn a new API/horizon interface.
>>
> 
> @PCM Do you see much reliance on tooling to setup VPN (such that having to
> update the tooling would be a concern for end-users), or is this something
> that could be managed through process/preparation?
> 

@JPD  I see very little reliance on tooling to setup VPNs.  We could
manage this through preparation.

> 
> 
>>
>>>
>>> Which is more of a logistics issue, and could be managed, IMHO.
>>>
>>>
>>>
>>>>
>>>> Would there be customers that would fall into that category, or are
>>>> customers manually configuring IPSec connections in that they could just
>>>> use the new API?
>>>>
>>
>> Most customers could easily adapt to a new API.
>>
>>>>
>>> Are there other adverse effects of not having backward compatibility that
>>>> need to be considered?
>>>>
>>
>> As with the dashboard, heat also needs a bit of consideration.  How
>> would Heat deal with the API changes?
>>
> 
> @PCM I don't know and will try to learn more. Is Heat used to configure VPN
> connections? I need to understand more about the relationship there.
> 

@JPD I've emailed Heat/Horizon devs to ask about the impact.

The issue I see is that Heat contains information about the structure of
Neutron Resources (see: [2]).  This would force a coupling of neutron
and heat versions in a way that previously was not required, if we want
to maintain support for VPNs in Heat.

[2]
https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/neutron/vpnservice.py


> 
> 
>>>
>>> So far, I'm identifying one effect that is more of a convenience
>> (although
>>> nice one at that), and one effect that can be avoided by planning for the
>>> upgrade.  I'd like to know if I'm missing something more important to
>>> operators.
>>>
>>> I'd also like to know if we thing there is a user base large enough (and
>>> how large is large?0 that would warrant going through the complexity and
>>> risk to support both API versions simultaneously?
>>
>> This is a bit frustrating...  It implies that only large clouds matter.
>>
> 
> @PCM Sorry. That's not the intent of the statement that I was trying to get
> across.
> 
> At the time I made the statement, my only knowledge of a need for
> supporting multiple simultaneous versions was to accommodate client tools
> (which could be handled via proper planning,IMHO) and end-users who would
> like to switch to the newer API on their own schedule, versus when the
> cloud is upgraded.
> 
> If there aren't many people who need that flexibility, then I'm asking
> whether it makes sense to assume the effort and risk in getting this right,
> as it is not easy.
> 
> I was "assuming" that if there is a large number of end-users wanting this
> flexibility, that an operator would feel compelled to insist on this
> capability.
> 
> From your case, it doesn't sound like that aspect is a concern at all (end
> users would just adapt to the new version immediately).  It sounds like
> Horizon and Heat are more of a concern for you.
> 

@JPD Correct.  Essentially, we don't really mind if we have to change
our tooling, but we would *really* like to avoid being forced to upgrade
Heat, Horizon, and Neutron all at the same time and to the same
versions.  This would be a step backwards on the path to painless
rolling upgrades.

> 
> 
>  There is a further tacit implication the API is not really a contract
>> that can be relied upon.
>>
> 
> @PCM Not sure I follow that statement. Can you elaborate on what you mean?

@JPD I suppose it was an expression of frustration at feeling like
backwards-incompatible API changes are being made lightly.  Possibly not
the most productive thing to say.  I hope it wasn't offensive.

> 
> I'm not talking about not supporting an API version in some cases. I'm
> talking about not supporting two API versions *simultaneously.*
> 

@JPD Yeah, I'm just worried about that point where support switches
over.  I suspect that there be dragons.

> 
> 
>> We are operating a multi-region cloud with many clients who depend upon
>> VPNaaS for business-critical production workloads.
>>
>> Of course, this is a two-way road!  We all want what is best for
>> OpenStack, so we should talk about the complexity and risk on your end.
>>  Can you tell us more about that?
> 
> 
> @PCM Sure. I updated the developer reference document on this design, which
> is under review. Check out https://review.openstack.org/#/c/191944/,
> especially towards the bottom, where there is a section added with my
> initial thoughts on how to handle backward compatibility. I'd appreciate
> feedback in the review for any comments, questions, concerns you may have.
> 

@JPD Thanks.  Looks good after a quick first pass(just added a couple of
small questions.)  I'll take a more critical look later(hopefully today?)


> 
> 
>> I really have no interest in being an
>> operator who demands the world from developers, but I am worried about
>> what all this means for my cloud.
>>
> 
> @PCM Understood completely, and is why I appreciate the discussion on this.
> As a developer, I'm trying to provide a (great? :) feature, without
> over-engineering things. I'm trying to strike a balance between risk,
> effort, and capability.
> 

@JPD Let me be clear: my users are *VERY* excited about this.  Thanks
and keep up the great work!

> Regards,
> 
> Paul Michali (pc_m)
> 
> 
> 
> 
>>
>> Cheers,
>> James Dempsey
>>
>> --
>> James Dempsey
>> Senior Cloud Engineer
>> Catalyst IT Limited
>> +64 4 803 2264
>> --
>>
>>
>>>
>>> Regards,
>>>
>>> Paul Michali (pc_m)
>>>
>>>
>>>> Specifically, we're talking about the VPN service create API no longer
>>>> taking a subnet ID (instead an endpoint group is create that contains
>> the
>>>> subnet ID), and the IPSec site-to-site connection create API would no
>>>> longer take a list of peer CIDRs, but instead would take a pair of
>> endpoint
>>>> group IDs (one for the local subnet(s) formally specified by the service
>>>> API, and one for peer CIDRs).
>>>>
>>>
>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Paul Michali (pc_m)
>>>>
>>>>
>>>> On Mon, Aug 24, 2015 at 5:32 PM Xav Paice <xavpaice at gmail.com> wrote:
>>>>
>>>>> I'm sure I'm not the only one that finds the vast amount of traffic in
>>>>> the dev list to be completely unmanageable to catch the important
>> messages
>>>>> - the ops list is much lower traffic, and as an operator I pay a bunch
>> more
>>>>> attention to it.  The discussion of deprecating an API is something
>> that
>>>>> HAS to be discussed with operators, on the operators list or
>> highlighted
>>>>> somehow so that people get attention drawn to the message.
>>>>>
>>>>> Let's be clear - I fully appreciate the extra effort that would be
>>>>> required in supporting both the new and the old APIs, and also would
>>>>> absolutely love to see the new feature.  I do think we need to be able
>> to
>>>>> support our customers in the transition, and extra pain for them
>> results in
>>>>> lower uptake of the services we provide.
>>>>>
>>>>> On 25 August 2015 at 09:27, Xav Paice <xavpaice at gmail.com> wrote:
>>>>>
>>>>>> Also:
>>>>>>
>>>>>>
>>>>>>
>> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007928.html
>>>>>>
>>>>>>
>> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007891.html
>>>>>>
>>>>>>
>>>>>> On 25 August 2015 at 09:09, Kevin Benton <blak111 at gmail.com> wrote:
>>>>>>
>>>>>>> It sounds like you might have missed a couple responses:
>>>>>>>
>>>>>>>
>>>>>>>
>> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007903.html
>>>>>>>
>>>>>>>
>> http://lists.openstack.org/pipermail/openstack-operators/2015-August/007910.html
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 24, 2015 at 1:53 PM, Paul Michali <pc at michali.net>
>> wrote:
>>>>>>>
>>>>>>>> Xav,
>>>>>>>>
>>>>>>>> In the email, there were no responses of anyone using VPNaaS *in a
>>>>>>>> production environment*. Summary from responders:
>>>>>>>>
>>>>>>>> Erik M - Tried in Juno with no success. Will retry.
>>>>>>>> Edgar M - said no reports from operators about VPNaaS code
>>>>>>>> Sam S - Using VPN in VMs and not VPNaaS
>>>>>>>> Kevin B - Not used. Use VMs instead
>>>>>>>> Sriram S - Indicating not used.
>>>>>>>>
>>>>>>>> If I misread the responses, or if someone has not spoken up, right
>> now
>>>>>>>> is the time to let us know of your situation and the impact this
>> proposal
>>>>>>>> would have on your use of VPNaaS IPSec site-to-site connections.
>>>>>>>>
>>>>>>>> The request here, is that if operators are not using this in a
>>>>>>>> production deployment where they need backward compatibility, then
>> we'd
>>>>>>>> like to avoid having to provide the complexity needed to support
>> backward
>>>>>>>> compatibility.
>>>>>>>>
>>>>>>>> In the proposal, there are two APIs that would be changed with this
>>>>>>>> enhancement. It's detailed in reference [1], and I can elaborate,
>> if needed.
>>>>>>>>
>>>>>>>> Please keep in mind, that users/operators using previous versions,
>> can
>>>>>>>> upgrade to the new version, and any existing VPNaaS configuration
>> will be
>>>>>>>> automatically migrated to the new table structures, so that
>> existing IPSec
>>>>>>>> connections would continue to operate with the new release.
>>>>>>>>
>>>>>>>> The proposal would not support using the older APIs, once the new
>> APIs
>>>>>>>> are available. Client apps/scripts would need to be updated to use
>> the new
>>>>>>>> API (neutron client and the Horizon dashboard will be updated as
>> part of
>>>>>>>> the overall effort).
>>>>>>>>
>>>>>>>> I hope that clarifies the discussion.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Paul Michali (pc_m)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 24, 2015 at 3:50 PM Xav Paice <xavpaice at gmail.com>
>> wrote:
>>>>>>>>
>>>>>>>>> On 25/08/15 06:32, Paul Michali wrote:
>>>>>>>>>
>>>>>>>>> As part of the effort to provide support for multiple local subnets
>>>>>>>>> for VPNaaS IPSec connections, there are three API changes planned
>> [1].
>>>>>>>>>
>>>>>>>>> One is to add a new "endpoint groups" API that will describe what
>> is
>>>>>>>>> being connected. This is the place were local and peer subnets
>> will be
>>>>>>>>> specified. It will allow future VPN flavors to reuse this API for
>> other
>>>>>>>>> types of VPN endpoints.
>>>>>>>>>
>>>>>>>>> The second is to modify the IPSec site connection API to make use
>> of
>>>>>>>>> this new endpoint groups API and no longer use the peer CIDRs
>> arguments.
>>>>>>>>>
>>>>>>>>> The third is to remove the subnet ID from the VPN service API, and
>>>>>>>>> again, use the new endpoint groups API for this information (and
>> to allow
>>>>>>>>>> 1).
>>>>>>>>>
>>>>>>>>> A previous email send out from Kyle Mestery, asking about the
>>>>>>>>> production usage of VPNaaS, did not indicate that this service was
>> used in
>>>>>>>>> production by operators.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Not true.  There were replies indicating that it is indeed in use,
>>>>>>>>> but perhaps not from operators of installations deemed significant
>> enough -
>>>>>>>>> what's the criteria here?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As a result, we'd like to propose making this change as a
>>>>>>>>> non-backward compatible change, which greatly reduces the effort.
>>>>>>>>>
>>>>>>>>> The change WILL support migration, so older databases can be
>> migrated
>>>>>>>>> to the new schema and continue to operate, with future changes
>> using the
>>>>>>>>> new API. This gives a smooth upgrade path to the new capability.
>>>>>>>>>
>>>>>>>>> The change would NOT support the older API in parallel with the new
>>>>>>>>> API, so users upgrading would need to also update any client
>> scripts/tools
>>>>>>>>> to use the revised/new VPN API.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> How will this integrate with Horizon?  The majority of our users
>>>>>>>>> don't use the API directly, but use Horizon for the config, does
>> this mean
>>>>>>>>> we'll be tied in to using a particular version of Horizon to match
>> Neutron?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If this is acceptable to operators, we'd like to go forward with
>>>>>>>>> these plans. Please respond within 7 days of this email, if you
>> have
>>>>>>>>> concerns about the proposal.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Paul Michali (IRC pc_m)
>>>>>>>>> VPNaaS Core
>>>>>>>>>
>>>>>>>>> Ref: https://review.openstack.org/#/c/191944/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>