Open Stack

On 15.05.17 19:01, Zane Bitter wrote:
> On 15/05/17 12:10, Steven Hardy wrote:
>> On Mon, May 15, 2017 at 04:46:28PM +0200, Lance Haig wrote:
>>> Hi Steve,
>>>
>>> I am happy to assist in any way to be honest.
>
> It was great to meet you in Boston, and thanks very much for 
> volunteering to help out.
>
> BTW one issue I'm aware of is that the autoscaling template examples 
> we have all use OS::Ceilometer::* resources for alarms. We have a 
> global environment thingy that maps those to OS::Aodh::*, so at least 
> in theory those templates should continue to work, but there are 
> actually no examples that I can find of autoscaling templates doing 
> things the way we want everyone to do them.
I think we can perhaps come up with some standard scenarios that we want 
to showcase and then we can work on getting this setup.

I might suggest that you look at the repo that my colleague Florin and I 
setup for our library and training material.
https://github.com/heat-extras

In the lib repo we have a test directory that tests each library 
template it might be an idea as to how to achieve test coverage of the 
different resources.
We currently just run yamllint testing with the script in there but I am 
sure we can add other tests as needed.
>
>>> The backwards compatibility is not always correct as I have seen when
>>> developing our library of templates on Liberty and then trying to 
>>> deploy it
>>> on Mitaka for example.
>>
>> Yeah, I guess it's true that there are sometimes deprecated resource
>> interfaces that get removed on upgrade to a new OpenStack version, 
>> and that
>> is independent of the HOT version.
>
> What if instead of a directory per release, we just had a 'deprecated' 
> directory where we move stuff that is going away (e.g. anything 
> relying on OS::Glance::Image), and then deleted them when it 
> disappeared from any supported release (e.g. LBaaSv1 must be close if 
> it isn't gone already).
>
I agree in general this would be good. How would we deal with users who 
are running older versions of openstack?
Most of the customers I support have Liberty and newer so I would 
perhaps like to have these available as tested.
The challenge for us is that the newer the OStack version the more 
features are available e.g. conditionals etc..
To support that in a backwards compatible fashion is going to be tough I 
think. Unless I am missing something.
>> As we've proven, maintaining these templates has been a challenge 
>> given the
>> available resources, so I guess I'm still in favor of not duplicating 
>> a bunch
>> of templates, e.g perhaps we could focus on a target of CI testing
>> templates on the current stable release as a first step?
>
> I'd rather do CI against Heat master, I think, but yeah that sounds 
> like the first step. Note that if we're doing CI on old stuff then 
> we'd need to do heat-templates stable branches rather than 
> directory-per-release.
>
> With my suggestion above, we could just not check anything in the 
> 'deprecated' directory maybe?
I agree in part.
If we are using the heat examples to test the functionality of the 
master branch then that would be a good idea.
If we want to provide useable templates for users to reference and use 
then I would suggest we test against stable.

I am sure we could find a way to do both.
I would suggets that we first get reliable CICD running on the current 
templates and fix what we can in there.
Then we can look at what would be a good way forward.

I am just brain dumping so any other ideas would also be good.
>
>>> As you guys mentioned in our discussions the Networking example I 
>>> quoted is
>>> not something you guys can deal with as the source project affects 
>>> this.
>>>
>>> Unless we can use this exercise to test these and fix them then I am
>>> happier.
>>>
>>> My vision would be to have a set of templates and examples that are 
>>> tested
>>> regularly against a running OS deployment so that we can make sure the
>>> combinations still run. I am sure we can agree on a way to do this 
>>> with CICD
>>> so that we test the fetureset.
>>
>> Agreed, getting the approach to testing agreed seems like the first 
>> step -
>> FYI we do already have automated scenario tests in the main heat tree 
>> that
>> consume templates similar to many of the examples:
>>
>> https://github.com/openstack/heat/tree/master/heat_integrationtests/scenario 
>>
>>
>> So, in theory, getting a similar test running on heat_templates 
>> should be
>> fairly simple, but getting all the existing templates working is 
>> likely to
>> be a bigger challenge.
>
> Even if we just ran the 'template validate' command on them to check 
> that all of the resource types & properties still exist, that would be 
> pretty helpful. It'd catch of of the times when we break backwards 
> compatibility so we can decide to either fix it or deprecate/remove 
> the obsolete template. (Note that you still need all of the services 
> installed, or at least endpoints in the catalog, for the validation to 
> work.)
>
> Actually creating all of the stuff would be nice, but it'll likely be 
> difficult (just keeping up-to-date OS images to boot from is a giant 
> pain). And even then that isn't sufficient to test that it actually 
> _works_. Let's keep that out of scope for now?
I was thinking if it was possible to get a moch API that we can validate 
against so that the tests don't have to run against a full OStack 
install. I am not sure if that is possible.
It would make it easier to test things locally when developing new 
templates.

Regards

Lance