Open Stack

Thanks for the update.

On 02/27/2014 05:18 PM, Eric Windisch wrote:
> We have a Jenkins server and slave configuration that has been tested
> and integrated into upstream OpenStack CI.  We do not yet trigger on
> rechecks due to limitations of the Gerrit Jenkins trigger plugin.
>  However,  Arista has published a patch for this that we may be able to
> test.  Reporting into OpenStack Gerrit has been tested, but is currently
> disabled as we know that tests are failing. Re-enabling the reporting is
> as simple as clicking a checkbox in Jenkins, however.
> 
> The test itself where we bring Nova up with the Docker plugin and run
> tempest against it is working fairly well. The process of building the
> VM image and running it is fully automated and running smoothly. Nova is
> installed, started, and tempest runs against it.
> 
> Tempest is working without failures on the majority of tests. To speed
> development I've been concentrating on the "tempest.api.compute" tests.
> To date, I've only disabled neutron, cinder, and the v3 api. I expect
> that I'll need to disable the config_drive and migration extensions as
> we do not support these features in our driver. I haven't yet identified
> any other extensions that do not work.

The number of things that don't work with this driver is a big issue, I
think.  However, we haven't really set rules on a baseline for what we
expect every driver to support.  This is something I'd like to tackle in
the Juno cycle, including another deadline.

> Tuesday's pass/fail for Tempest was 32 failures to 937 tests. The total
> number of tests is as low as 937 because this only includes the compute
> api tests, knowing that we're passing or skipping all other test suites.
> 
> Since Tuesday, I've made a number of changes including bugfixes for the
> Docker driver and disabling of the config_drive and migration
> extensions. I'm still running tempest against these changes, but expect
> to see fewer than 20 failing tests today.
> 
> Here is a list of the tests that failed as of Tuesday:
>  http://paste.openstack.org/show/69566/
> 
> Related changes in review:
> * Nova
>   - https://review.openstack.org/#/c/76382/
>   - https://review.openstack.org/#/c/76373/
> * Tempest
>   - https://review.openstack.org/#/c/75267/
>   # following have -1's for me to review, may be rolled into a single patch
>   - https://review.openstack.org/#/c/75249/
>   - https://review.openstack.org/#/c/75254/
>   - https://review.openstack.org/#/c/75274/
> 
> A fair number of the remaining failures are timeout errors creating
> tenants in Keystone and uploading images into Glance. It isn't clear why
> I'm seeing these errors, but I'm going to attempt increasing the
> timeout. There may be some more subtle problem with my environment, or
> it may simply be a matter of performance, but I doubt these issues are
> specific to the Docker hypervisor.

These have got to be specific to your environment given the many
thousands of times these tests are run in other environments every week.

> Because we don't support Neutron and the v3 api doesn't work with
> nova-network, I haven't yet concentrated effort into v3. Having done
> some limited testing of the v3 API, however, I've seen relatively few
> failures and most or all overlapped with the existing v2 failures. I'm
> not sure how Russell or the community feels about skipping Tempest tests
> for v3, and I would like to try making these pass, but I presently see
> it as lower priority versus making v2 work and pass.

It's certainly not ideal.  On the one hand, we haven't enforced a
baseline for drivers yet.  On the other hand, we did communicate that we
expect a "full tempest run" in CI, so that's up to some interpretation.
 We also said "Note that hypervisors missing specific bits of feature
support may exclude those tests from their published Tempest
configuration, and the Nova team will validate the effectiveness of the
given config on a per-case basis to ensure reasonable coverage."

The deprecation schedule [1] says that the removal won't be committed
until just before the RC, so you still have a couple of weeks.  I would
sprint toward getting everything passing, even if it means applying
fixes to your env that haven't merged yet to demonstrate it working sooner.

[1] https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan

-- 
Russell Bryant