Open Stack

----- Original Message -----
> From: "Jay Pipes" <jaypipes at gmail.com>
> To: openstack-qa at lists.openstack.org
> Sent: Sunday, June 9, 2013 12:14:50 AM
> Subject: Re: [openstack-qa] Tempest Architecture/Design brainstorming
> 
> On 06/07/2013 03:42 AM, Attila Fazekas wrote:
> > I have several questions and concerns.
> >
> > 1. role based user creation
> >   * As we discussed long time ago the user creation should be based on the
> >   roles.
> 
> That is precisely what is demonstrated below. 'non-admin' and 'admin'
> are roles.
> 
> >   * As I remember you wanted to distinguish the compute admin from the
> >   identity admin,
> >     we could use even richer model, the rule based negative test are
> >     missing from tempest now.
> 
> I don't understand what this means. Could you elaborate please?
> 
Long time ago I removed the compute_admin credentials from the config file,
and you was not happy about this.
Do you need it nowadays ?


Extending into the roles can have two goals:
- Covering deployment scenarios when the administrative roles spitted to multiple teams.
 For example a network admin does not have admin permission on the Image system.

- Covering policy decisions
 The tempest in this is very weak, if any of the component forgets to
 validate the  user-role-teant-policy combination in a particular step, we would not see it.
 In this area an user without any capability would be the simplest and most beneficial "tool".
 But the reality is little bit more complex, if we want it to test fully.

> >   * The identity admin credentials does not necessarily be known (not my
> >   use case), AFAIK several user
> >    wants to use tempest just with the predefined users.
> >    (If this use case does not really exists, it can simplify lot of things)
> 
> Are you asking whether Tempest should not create users for use in its
> tests? As I've said previously, this was done for two reasons:
> 
> 1) To get past many issues with quotas for a tenant

Those very bad leakages are fixed since long, the other will be fixed soon,
and leakage  will be prevented.
https://blueprints.launchpad.net/tempest/+spec/stop-leaking

> 2) To provide the basis for parallel execution -- if you are running X
> compute tests in parallel with the exact same user, you are needlessly
> going to run into issues where calls to things like GET
> /<TENANT>/servers is going to return non-deterministic results between
> processes
> 
> What are you thinking will be simplified by removing Tempest's test
> tenant/user isolation?

Yes.
Our list tests are written in way which tolerate this situation.
If you find it works in different way, it is a HIGH priority issue.

If we really want to test something in clean tenant, we can create tenants.
These test will not be part of the service provider OS API validation test case set.
Bit if we need it for something what worth to test, we will test it, and create new tenant.
 
> > 2. keystone stress
> >   * if we have all test cases to request multiple tokens
> >     and create users and tenants and set the quota set,  it will be a lot
> >     of overhead.
> 
> The volume is the same on Keystone tokens regardless of number of
> Keystone users you use to run tests. In the default Keystone, every
> request to any endpoint is going to involve a token, and that token will
> need to be validated on each REST call. Makes no difference whether it's
> 1 user or >1 user running the tests.
> 
> >   * token request if the token request math /test case is
> >     number_of_used_users * number_of_used_clients *
> >     number_of_impersonate_switch,
> >     is not too efficient.
> 
> That is incorrect. Number of token requests is number of requests made
> to any endpoint. Has nothing to do with number of users.

I am speaking about the token-get requests: http://paste.openstack.org/show/38294/

The token-get response contains all endpoints and a token (id) which usable with all services.

The tempest current model does not takes it into consideration.

You need to do a new token get only once per user in 24h.
You juts need a new token when the old one is expired.

> With PKI, the auth_token middlewares can cache a user's token and reduce
> the number of requests to the Keystone API server, true, and the number
> of users would affect the number of cached tokens the auth_token
> middleware would cache.  But, I believe that focusing on these
> micro-performance issues/optimizations is missing the point of using
> multiple users: to better isolate the test cases from the actions of
> other users.

Yes, I am speaking about the token-get, not about the token validation.

> >     The RSA signing is not cheap operation, the request/response latency is
> >     measurable side
> >     effect anyway.
> >     Does it scales well with multi-thread ?.
> >     Keystone uses cPython and we just have 1 keystone process on the gate.
> 
> This is a deployment issue, frankly. We can change the gate to put
> Keystone in an Apache wsgi servlet container and multiplex Keystone API
> that way.
> 
> > 3. multiple client tool chain usage
> >   * how will this model will be usable for multiple client usage /per test
> >   case ?
> 
> Not sure what you mean by this. Could you elaborate a bit more, perhaps
> with an example piece of code?

See above. 
> > 4. resource sharing
> >   * several resource creation does not scales well for example the server
> >   creation.
> >     The owner tenant members has more capabilities on given server
> >     resource,
> >     if you want to combine the servers with another resources, the owner
> >     tenant is a more important fact.
> 
> You lost me on this one, Attila :) I don't understand what you mean by this.

If you create a server in tenant A and you try do anything with an user who 
 just have roles on tenant B. 
- You cannot see the server
- You cannot attache volume/.. created in tenant B..

--> If you create a new dynamic user in dynamic tenant,
 you cannot touch the existing resources. 

(These negative test are mostly missing)
 
> Best,
> -jay
> 
> > ----- Original Message -----
> >> From: "Jay Pipes" <jaypipes at gmail.com>
> >> To: openstack-qa at lists.openstack.org
> >> Sent: Thursday, June 6, 2013 4:04:35 PM
> >> Subject: Re: [openstack-qa] Tempest Architecture/Design brainstorming
> >>
> >> On 06/06/2013 01:57 AM, Attila Fazekas wrote:
> >>> Hi All,
> >>>
> >>> I am looking for ideas for unifying the tempest "Object Model" and to
> >>> enhance it's capabilities.
> >>>
> >>> Several interesting topic:
> >>> - Plug-ability
> >>> - Identity management ("impersonate the right person efficiently")
> >>
> >> I put together a proof of concept of this exact functionality using
> >> fixtures.Fixture and a context manager called acting_as() [1] that was
> >> extended by subclasses' do_as() method [2] to make it easy to write
> >> clean impersonation code [3]:
> >>
> >> class TestFlavorsNonAdmin(base.ComputeApiTest):
> >>
> >>       def test_list_flavors(self):
> >>           with self.do_as('non_admin') as client:
> >>               flavors = client.flavors.list()
> >>               self.assertTrue(len(flavors) > 0)
> >>
> >>       def test_unauthorized(self):
> >>           with self.do_as('non_admin') as client:
> >>               self.assertRaises(Exception, client.flavors.create,
> >> 'flavorname')
> >>
> >>
> >> class TestFlavorsAdmin(base.ComputeApiTest):
> >>
> >>       def test_list_flavors(self):
> >>           with self.do_as('admin') as client:
> >>               flavors = client.flavors.list()
> >>               self.assertTrue(len(flavors) > 0)
> >>
> >>       def test_crud(self):
> >>           with self.do_as('admin') as client:
> >>               flavor_name = self.getUniqueString("flavor")
> >>               flavor = client.flavors.create(flavor_name, 64, 1, 10)
> >>               self.assertEqual(flavor_name, flavor.name)
> >>               self.assertEqual(64, flavor.ram)
> >>               self.assertEqual(1, flavor.vcpus)
> >>               self.assertEqual(10, flavor.disk)
> >>               flavors = client.flavors.list()
> >>               flavor_ids = [x.id for x in flavors]
> >>               self.assertIn(flavor.id, flavor_ids)
> >>               flavor = client.flavors.find(name=flavor_name)
> >>               self.assertEqual(flavor_name, flavor.name)
> >>               self.assertEqual(64, flavor.ram)
> >>               self.assertEqual(1, flavor.vcpus)
> >>               self.assertEqual(10, flavor.disk)
> >>               client.flavors.delete(flavor.id)
> >>
> >> Best,
> >> -jay
> >>
> >> [1]
> >> https://github.com/jaypipes/tempest/blob/api/tempest/tests/api/base.py#L89
> >> [2]
> >> https://github.com/jaypipes/tempest/blob/api/tempest/tests/api/compute/base.py#L27
> >> [3]
> >> https://github.com/jaypipes/tempest/blob/api/tempest/tests/api/compute/test_flavors.py
> >>
> >>
> >>> - test case selection (based on supported feature, installed services..)
> >>>
> >>> An ideal proposal should answer the
> >>> - What you would like to achieve.
> >>> - Why do you want/need this.
> >>> - How do you want to do it.
> >>>
> >>> Feel free to add incomplete idea as well.
> >>>
> >>> Best Regards,
> >>> Attila
> >>>
> >>> PS.: python is not juts OOP language,
> >>> we should use the imperative and functional programming styles as well.
> >>>
> >>> _______________________________________________
> >>> openstack-qa mailing list
> >>> openstack-qa at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >>>
> >>
> >>
> >> _______________________________________________
> >> openstack-qa mailing list
> >> openstack-qa at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >>
> >
> > _______________________________________________
> > openstack-qa mailing list
> > openstack-qa at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >
> 
> 
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>