Open Stack

Thanks for the detailed reply Monty. Your description of the IP situation cleared things up for me. I had been thinking of interop strictly from the perspective of compatible APIs and not as much from the perspective of a consumer trying to build infrastructure on top of OpenStack resources.

We bump into some of these issues in testing, but hand wave it away by providing configurations that provide context as to determine public networks and other relevant information. Given the proper capabilities, existing Tempest tests could be modified to use those capabilities rather than configuration, but I've seen that picking up on networking issues when trying to verify something like a server's hostname can be confusing to someone triaging test failures.

If this is the crux of the problem trying to be solved by measuring interop is to verify operator scenarios like the ones you described, targeting these scenarios directly with tests whose results directly speak to those requirements could provide clearer feedback than trying to infer these capabilities as side effects of other existing tests (You can assume that if you could run test_verify_created_server_vcpus that you were somehow able to figure out the public IP of a server, but that fact isn't the primary purpose of the test). Is it possible that we're missing a classification of tests here?

Daryl
________________________________________
From: Monty Taylor <mordred at inaugust.com>
Sent: Thursday, June 18, 2015 7:39 AM
To: defcore-committee at lists.openstack.org
Subject: Re: [OpenStack-DefCore] Identifying and defining missing/new tests for capabilities :Was Image APIs in Glance and Nova

On 06/18/2015 04:39 AM, Daryl Walleck wrote:
> I think what Rocky is speaking to here is similar to the line of thinking I had earlier, which I may not have communicated as well.
>
> As a tester and a test developer, I want to identify high level use cases or scenarios, and exercise all possible paths so that I have a thorough picture of the behaviors of the system under test. This can conflict with the notion of a DefCore capability, which speaks to having a single, guaranteed paths to perform an operation. This is where I see the DefCore test selection process kicks in and the workflows that define interop can be selected, given that the tests are designed in an atomic fashion such that individual tests are tightly scoped and do not creep into steps that may not relate directly to the capability under test. Tests for distinct functionality can still exist to satisfy test coverage, which then can then be used as data about what functionality deployers are exposing since one of the aspects of the process is to report all Tempest test results and not just the selected ones.
>
> While I've seen the Glance example bounced around a bit, it sounds like there must be other similar areas of concern, which understanding would help fill in some of the knowledge gaps around these concerns. Is there an existing list of more examples of inconsistencies that are specifically a concern for interop?

Yes - there are. Glance upload is a big problem (at LEAST 3 different ways)

Finding a server's externally routable IP requires this logic:

http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/meta.py#n75

The inverse of that is "get me a server with an externally routable IP"
which has a related logic:
- nova boot --with-externally-routable-ip
  - do I have an IP already? (see function above)
  - if not, does my cloud have floating ips from neutron, get one
  - if not, does my cloud have floating ips from nova, get one
  - if not - why do I not have an external IP - was there a network I
should have attached to?
    - does my cloud have neutron, if so, do any of the networks have the
router:external attribute
    - STAB STAB STAB

Anything that is a neutron function that nova also provides and may
provide a proxy method for: security groups and floating ips spring to mind.

BTW - here are my use cases:

As an application developer,
I want to deploy and run an application on the internet
so that my customers all over the world can consume it.

As an application developer,
I want to deploy the application across multiple clouds
so that my service survives issues in any one of them.

As an application developer,
I want to upload the exact same image to each cloud
so that my base operating environment is the same.

As an application developer,
I want to boot VMs based on my image on each cloud that can route off
their cloud
so that I can run my workloads on them.

Today this is possible, but it takes a large amount of logic, and also
requires working around vendor "value add" choices on each cloud.

I want to point out that some of these things aren't REST APIs. The
mechanism to get an IP on your instance on Rackspace is speicific to
Rackspace. It's neither supported by cloud-init nor is it a normal thing
like DHCP. The "normal" way to do that is to run nova-agent, which is
open source and in python so is ostensibly not an issue ... except that
running nova-agent in an image I'm going to upload to HP and Dreamhost
is not really a useful thing.

This means we had to sort out a mechanism for getting the IP on the box
that worked across all clouds that we could bake in to our images. Now -
the rax team was super helpful in that they rolled out a patch to nova
that is still in review upstream to put the information into
config-drive so that we could write a custom startup script that looked
for it and used it if present.

So, I agree that having a clear set of use cases helps us to identify
where there are divergent.

I strongly believe that we need to be working towards a place where we
minimize or elminate user-visible divergence in mechanisms to achieve
those use cases.

> Thanks,
>
> Daryl
>
> Sent from my Windows Phone
> ________________________________
> From: Mark Voelker<mailto:mvoelker at vmware.com>
> Sent: ‎6/‎17/‎2015 10:08 PM
> To: Rochelle Grober<mailto:rochelle.grober at huawei.com>
> Cc: defcore-committee at lists.openstack.org<mailto:defcore-committee at lists.openstack.org>
> Subject: Re: [OpenStack-DefCore] Identifying and defining missing/new tests for capabilities :Was Image APIs in Glance and Nova
>
>> On Jun 17, 2015, at 8:51 PM, Rochelle Grober <rochelle.grober at huawei.com> wrote:
>>
>> I'm replying to this version of John's mail to expand on his "use case" point.
>>
>> What John calls a use case is, in this instance, the same as a high level "Test Case".  Essentially, this is the documentation/comments that define what the test(s) for this case are doing.  DefCore (or others) could create these high level test cases and use them as the framework into which new tests, specific for the capability, could be written.  This, I think, would be an ideal way to start documenting missing capabilities tests.  For some capabilities, there could be multiple use cases (or test cases) that define the capability.
>>
>> This could also be used for existing tests, but instead of including code for each case or step in a case, the git repo link to the existing test could be included.  This process could also be used with new tests being linked into the test case document as they are merged into the trunk.
>>
>> This process provides the higher level test "specs" that developers implement into tests.  It provides both a way to track what tests exist and what don't, and a description of the capability that is understandable by end users and other interested parties not concerned with the "guts," but just the behavior of the capability.
>>
>> For the case of multiple ways of performing an action, the tests would need to implement all methods that meet DefCore criteria.  If any of the methods result in the appropriate end result, then for DefCore purposes, the test has passed.
>
> Maybe I’m misunderstanding, but this seems hugely problematic for interoperability.  If we say there’s more than one way to do a thing but you pass if you implement any one of those ways, then app developers still don’t know what methods they can depend on being available.  Instead of interoperable clouds we have clouds that can do the same things but for which I still have to write a bunch of this**:
>
> def list_images():
>    “”” A function to list images. Because all OpenStack Powered Platforms can do that…somehow.””"
>    if $cloud == ‘vendorA’:
>       # TODO: this also works for vendorX
>       list_images_via_nova_image_api()
>    elif $cloud == ‘vendorB’:
>       # TODO: this also worked for vendorY last week but now, um?
>       list_images_via_glance_v1()
>    elif $cloud == ‘vendorC’:
>       list_images_via_glance_v2()
>    else:
>       # I dunno what cloud this is, but it’s OpenStack Powered! So something must work.
>       try:
>          list_images_via_nova_image_api()
>       except NopeError:
>           # D’oh, guess that wasn’t it…
>           try:
>              list_images_via_glance_v1()
>           except StillNopeError:
>              # Aww…well third time’s the charm?
>              try:
>                 list_images_via_glance_v2()
>              except NopeNopeNopeError:
>                 rage_quit()
>
> into any app that I want to be able to run on OpenStack Powered Platforms.  Which is a Bad Thing and basically where we’re at now.  How you expose a capability matters tremendously.  IMHO, interoperability by if/else loop isn’t interoperability at all.
>
> ** I’m strawmanning slightly here in that I’m pretending these three methods all somehow met DefCore criteria, but you get the picture.
>
> At Your Service,
>
> Mark T. Voelker
>
>>
>> Does this make sense for DefCore?  As OpenStack matures and acts more like a collection of products rather than a collection of code bases, the processes and collateral of software process that exist to turn code into product come more into play. Hence, the QA as not just verification gate, but a measurement and metrics repository.
>>
>> Sorry!  Had to get this thought out.  You don't need the QA that most SW companies have until you have product.  We are getting close to one and so are now in need of expanding OpenStack QA to encompass the other.
>>
>> --Rocky
>>
>>
>>
>> -----Original Message-----
>> From: John Garbutt [mailto:john at johngarbutt.com]
>> Sent: Wednesday, June 17, 2015 09:57
>> To: defcore-commit.
>> Cc: Nikhil Komawar
>> Subject: [OpenStack-DefCore] Image APIs in Glance and Nova
>>
>> Hi,
>>
>> So this was raised in the cross project meeting, and thank you for that.
>>
>> I keep mentioning use cases, so here we go...
>> * create from cloud base image for ubuntu 12.04
>> * above but with boot from volume
>> * create a snapshot
>> * download snapshot
>> * upload image into another openstack cloud
>> * boot server from uploaded image
>>
>> Now, at a higher level:
>> * user does above using custom script for Cloud A and Cloud B
>> * user keeps to just the APIs that are defcore tested
>> * user gets access to Cloud C and Cloud D
>> * user wants to point script at new clouds, and everything should just work
>>
>> So I think thats where we want to be. Now lets dig in...
>>
>> To list images, the user could:
>> * use nova to list images (stable, but project wants to delete it)
>> * use glance v1 (should never be exposed to end users, was designed as
>> internal only API)
>> * use glance v2 (only just released, not really deployed anywhere)
>>
>> For the
>>
>> _______________________________________________
>> Defcore-committee mailing list
>> Defcore-committee at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
>>
>> _______________________________________________
>> Defcore-committee mailing list
>> Defcore-committee at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
>
> _______________________________________________
> Defcore-committee mailing list
> Defcore-committee at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
>
>
>
> _______________________________________________
> Defcore-committee mailing list
> Defcore-committee at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
>


_______________________________________________
Defcore-committee mailing list
Defcore-committee at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee