Open Stack

On 4/6/15 2:52 PM, Morgan Fainberg wrote:
>
>
> On Apr 6, 2015, at 11:41, Mike Bayer <mbayer at redhat.com 
> <mailto:mbayer at redhat.com>> wrote:
>
>>
>>
>> On 4/6/15 12:49 PM, David Stanek wrote:
>>>
>>> Exactly. This is the direction I have been going. Functional tests 
>>> are written using the public APIs using the client.
>>>
>>> I would also add that I don't like that the Keystone unit tests are 
>>> so database heavy. I would not want MySQL or ant RDBMS to be a 
>>> requirement for running the tests.
>>>
>> is that because you'd prefer that the unit tests were more isolated, 
>> or just that an external service is being used?
>>
>> I've done some work with extensive mocking of SQL databases; 
>> specifically mocking at the ORM level.  It is nice when you get it to 
>> run, but it's also a much bigger job to write fine-grained mocks like 
>> that, the mocks can be brittle in relation to the code they're 
>> targeting, and you also need to come up with some solution to 
>> actually functional test your database access code.
>>
>> I tend to think that having a mysqld service running is the lesser of 
>> two evils and you get a lot more code coverage by going all the way 
>> out to the DB.
>>
>
> What David is specifically referencing is that we want our functional 
> tests to only require direct API access. There is almost no reason to 
> need access to the DB backend. We have many ways to perform isolation 
> where needed (tempest does a lot of this today).
>
> The goal is to allow the functional test suite to run against any 
> keystone deployment (be agnostic to db, non-db, etc driver used). This 
> makes environment setup a separate concern the tests don't need to be 
> involved in/aware of. It makes our functional tests more useful for 
> validating a driver or configuration passes muster.

ah.  for sure.  I would think functional tests could run against a 
backend that was entirely mock / file-based.

>
> If there are legitimately cases we need to test a specific db function 
> in isolation we will make specific efforts to support it. Those are 
> apt to be the exception to the rule.
>
> --Morgan
> Sent via mobile
>
>>
>>
>>>
>>> On Mon, Apr 6, 2015, 12:42 Morgan Fainberg 
>>> <morgan.fainberg at gmail.com <mailto:morgan.fainberg at gmail.com>> wrote:
>>>
>>>
>>>
>>>     > On Apr 6, 2015, at 09:20, Mike Bayer <mbayer at redhat.com> wrote:
>>>     >
>>>     >
>>>     >
>>>     >> On 4/6/15 12:06 PM, Clint Byrum wrote:
>>>     >> Excerpts from Boris Bobrov's message of 2015-04-03 18:29:08
>>>     -0700:
>>>     >>>> On Saturday 04 April 2015 03:55:59 Morgan Fainberg wrote:
>>>     >>>> I am looking forward to the Liberty cycle and seeing the
>>>     special casing we
>>>     >>>> do for SQLite in our migrations (and elsewhere). My
>>>     inclination is that we
>>>     >>>> should (similar to the deprecation of eventlet) deprecate
>>>     support for
>>>     >>>> SQLite in Keystone. In Liberty we will have a full
>>>     functional test suite
>>>     >>>> that can (and will) be used to validate everything against
>>>     much more real
>>>     >>>> environments instead of in-process “eventlet-like”
>>>     test-keystone-services;
>>>     >>>> the “Restful test cases” will no longer be part of the
>>>     standard unit tests
>>>     >>>> (as they are functional testing). With this change I’m
>>>     inclined to say
>>>     >>>> SQLite (being the non-production usable DB) what it is we
>>>     should look at
>>>     >>>> dropping migration support for SQLite and the custom
>>>     work-arounds.
>>>     >>>>
>>>     >>>> Most deployers and developers (as far as I know) use
>>>     devstack and MySQL or
>>>     >>>> Postgres to really suss out DB interactions.
>>>     >>>>
>>>     >>>> I am looking for feedback from the community on the general
>>>     stance for
>>>     >>>> SQLite, and more specifically the benefit (if any) of
>>>     supporting it in
>>>     >>>> Keystone.
>>>     >>> +1. Drop it and clean up tons of code used for support of
>>>     sqlite only.
>>>     >>>
>>>     >>> Doing tests with mysql is as easy, as with sqlite
>>>     ("mysqladmin drop -f;
>>>     >>> mysqladmin create" for "reset"), and using it by default
>>>     will finally make
>>>     >>> people test their code on real rdbmses.
>>>     >> Please please please be careful with that and make sure the
>>>     database
>>>     >> name is _always_ random in tests... or even better, write a
>>>     fixture to
>>>     >> spin up a mysqld inside a private tempdir. That would be a
>>>     really cool
>>>     >> thing for oslo.db to provide actually.
>>>     >>
>>>     >> I'm just thinking some poor sap runs the test suite with the
>>>     wrong
>>>     >> .my.cnf in the wrong place and <poof> there went keystone's db.
>>>     > The standard approach here is that tests based on the oslo.db
>>>     approach by default connect using a username openstack_citest on
>>>     localhost, which is then used to create databases of random
>>>     names. If you base your database tests on oslo.db, you get that
>>>     right now.  This username/host/etc is also configurable through
>>>     environment variables.  I don't see how my.cnf is involved in
>>>     this nor how it would impact someone's keystone database, unless
>>>     we're talking about tests that happen to load down and/or crash
>>>     the whole database server.
>>>     >
>>>     > spinning up a whole mysqld seems really heavy-handed and
>>>     unnecessary.  Not to mention the tests run on other backends as
>>>     well such as Postgresql.
>>>     >
>>>
>>>     The reasons outlined by both Clint and Mike are why we won't be
>>>     attempting this until we are happy with our functional test
>>>     suite. But once we are happy dropping SQLite is on the table.
>>>     The way I see it the functional tests should be performed
>>>     against a real keystone server, even if it is one spun up for
>>>     testing specifically.
>>>
>>>     Per test db creation / other such stuff will be part of that
>>>     discussion.
>>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150406/1f27a9e8/attachment.html>