Open Stack

Hi,

As of last week, we have organized a team to deal with the Quantum in full
gate issues. There are 3 people working on this: Jordan Pittier,
Ala.Rezmerita and myself. Below you will find a document describing all the
identified issues and who is assigned to fix them. Of cource, we can use
more hands / heads. If anyone wants to help or has identified other issues
not listed below, please get in touch with us.

Regards

Fix Jenkins gate-tempest-devstack-vm-quantum-full

This document lists all the Tempest tests that are failing on a Devstack +
Quantum Trunk setup. The goal is to make these tests pass for the Havana
Milestone 3. Alla, Miguel and Jordan are working on this.

For each test :

   -

   Provide the full path of the test. If the test has a double interface
   (JSON and XML), it’s enough to provide only the JSON path.
   -

   Provide a small stacktrace or a link to a Paste service such as
   http://paste.openstack.org/
   -

   Would be great to provide the CURL URL that triggers the bug
   -

   Provide  a small analysis of what the bug may be and the components
   involved (Quantum, Quantum API on Nova’s side, Nova-network etc.)
   -

   As far as possible, provide the file and the line number where the
   Python exception is raised or translated (“casted”)
   -

   If already filed, an URL to the bug report in Launchpad

Tests related to the Fixed IPs Compute API extension (Miguel)

1)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_list_fixed_ip_details

   -

   Trace : http://paste.openstack.org/show/38363/
   -

   curl -H "X-Auth-Token:$TOKEN" -X GET http://
   $IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3
   -

   API : http://api.openstack.org/api-ref.html#ext-os-fixed-ips
   -

   Explanation : Call to
   nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show() This
   file doesn’t use the Quantum API nor the Nova-Network API. It interacts
   directly with the DB, which is bad.
   -

   A possible fix would be to :
   1.

      Change
      nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show() to
      use either nova.network.api.API::get_fixed_ip() (for Nova Network) or
      nova.network.quantumv2.api.API:get_fixed_ip() (for Nova Network)
      2.

      Implement nova.network.quantumv2.api.API:get_fixed_ip() which
      currently raises a NotImplementedError exception

2)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_reserve

   -

   Trace : http://paste.openstack.org/show/38372/
   -

   curl  -H "X-Auth-Token:$TOKEN" -X POST http://
   $IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3/action
   -

   API : http://api.openstack.org/api-ref.html#ext-os-fixed-ips
   -

   Possible Fix : Should call
   nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::_set_reserved()
   (once the result of db.fixed_ip_get_by_address() is made through the API).
   Here the problem is that neither Nova-Network nor Quatum API implement an
   equivalent of db.fixed_ip_update()

3)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_unreserve

   -

   Same as 2)



Tests related to Quotas Admin(Miguel)

4)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_exceed_limit

5)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_rules_exceed_limit

Tests related to Floating IPs

6)
tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_associate_ip_to_server_without_passing_floating_ip

   -

   Trace Tempest http://paste.openstack.org/show/38431/
   -

   Bug : https://bugs.launchpad.net/quantum/+bug/1190242


   -

   curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
    POST http://$IP:8774/v2/$TENANT_ID/servers/$SERVER_ID/action<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action>-d
‘{"addFloatingIp": {"address": ""}}’
   -

   Notice that the address of the FloatingIP is empty.
   -

   API : http://api.openstack.org/api-ref.html#ext-floating-ips
   -

   Explanation : Mismatch in the 2 API. If there is only one IP in the
   pool, Quantum allows the floating IP to be empty and returns the one and
   only IP in the pool. Nova-network doesn’t allow this and returns a 404
   -

   Review https://review.openstack.org/#/c/32740/


7)
tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_floating_ip

   -

   Trace Tempest : http://paste.openstack.org/show/38505/
   -

   Bug : https://bugs.launchpad.net/tempest/+bug/1160309 (see comment #2)
   -

   This is related to bug 9)


8)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_nonexistant_floating_ip


   -

   Related to 9)


9)
tempest.api.compute.floating_ips.test_list_floating_ips:FloatingIPDetailsTestJSON.test_get_nonexistant_floating_ip_details

   -

   Bug :https://bugs.launchpad.net/tempest/+bug/1160309


   -

   Trace tempest:http://paste.openstack.org/show/38430/
   -

   Trace nova: http://paste.openstack.org/show/38433/
   -

   Curl:curl -H "X-Auth-Token:$TOKEN" -X GET http://
   $IP:8774/v2/$PROJECT_ID/os-floating-ips/99987878787878

Proposed Fix: https://review.openstack.org/33024

Tests Related to Security Groups (Jordan)

10)t
empest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_create_with_invalid_id

   -

   TRACE :  http://paste.openstack.org/show/38373/
   -

   curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
    POST http://$IP:8774/v2/$TENANT_ID/os-security-group-rules -d
   ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp", "to_port":
   22, "parent_group_id": "9991393497170", "cidr": null, "group_id": null}}’


   -

   Explanation : Notice that the parent_goup_id is  a numerical ID and not
   a UUID. Quantum has an additional check to validate that the ID is an UUID
   (see nova/network/security_group/quantum_driver.py::validate_id())


   -

   API : http://api.openstack.org/api-ref.html#ext-os-security-groups
   -

   Bug : https://bugs.launchpad.net/tempest/+bug/1182384
   -

   Review : https://review.openstack.org/#/c/29899/


11)
tempest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_delete_with_invalid_id

   -

   http://paste.openstack.org/show/38424/
   -

   curl  -H "X-Auth-Token:$TOKEN" -X  DELETE http://
   $IP:8774/v2$TENANT_ID/os-security-group-rules/9991407551273
   -

   Explanation : Same bug as 10
   -

   API : http://api.openstack.org/api-ref.html#ext-os-security-groups
   -

   Review : https://review.openstack.org/#/c/29899/

12)
tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_delete_nonexistant_security_group

   -

   Same as bug 10
   -

   Review : https://review.openstack.org/#/c/29899/


13)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_get_nonexistant_group

   -

   Same as bug 10
   -

   Review : https://review.openstack.org/#/c/29899/


14)
tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_duplicate_name

Security Group with duplicate name should not be created, but two groups
with the same name can be created in quantum. We have here the same problem
as in 15 and in 16. With Quantum, there is no validation that a  group with
given name exists already or if the given SG name is empty or is composed
of white spaces or is more than 255 chars.

In the description of bug https://bugs.launchpad.net/nova/+bug/1161411 this
issue is generally discussed. SecurityGroup API  are based on the ID and
not the names, except for adding an instance to a security group.  In order
to solve the last problem the bug
https://bugs.launchpad.net/nova/+bug/1161473 was added.

The major question is if these 3 tests (14, 15, 16) : does the name of a
security group is really that important? If so, we must add some validation
methods. If not the test suit concerning this part must be disable in
tempest. What do you think Miguel?


15)
tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_description

16)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_name

BUG:https://bugs.launchpad.net/nova/+bug/1161411 +
https://bugs.launchpad.net/nova/+bug/1161473

Traceback(tempest) :http://paste.openstack.org/show/38423/

The Security Group should not be created with group name an empty string or
with white spaces/chars more than 255

CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
POST http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
 '{"security_group": {"name": " ", "description":
"description-1554950088"}}'

curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X POST
http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
'{"security_group": {"name":
"  ", "description": "description-1554950088"}}'

17)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_server_security_groups

   -

   Tempest Trace : http://paste.openstack.org/show/38427/
   -

   Fixed by https://review.openstack.org/#/c/32288/ (merged on June,12th)



Tests related to servers(Ala)

18)
tempest.api.compute.servers.test_list_server_filters:ListServerFiltersTestXML.test_list_servers_filtered_by_ip_regex

   -

   BUG: https://bugs.launchpad.net/quantum/+bug/1182883
   -

   BP: https://blueprints.launchpad.net/quantum/+spec/like-op-list
   -

   CURL : GET http://$IP:8774/v2/$PROJECT_ID/servers?ip=10.
   -

   Explanation : The  regex search is not supported by Quantum. Thus
   Quantum returns a 404 Not Found (0 server match) where Tempest expects one
   server to be found.
   -

   Possible Fix : For "search port by IP with regex" feature, I think the
   best place to hack it would be in file
   quantum/db/db_base_plugin_v2.py::_get_ports_query()


19)tempest.api.compute.servers.test_servers_negative:ServersNegativeTest.test_create_with_nonexistent_security_group

FIXED : https://review.openstack.org/#/c/30271/

20)tempest.api.compute.servers.test_virtual_interfaces:VirtualInterfacesTestXML.test_list_virtual_interfaces

   -

   TRACE (NOVA): http://paste.openstack.org/show/38371/
   -

   BUG: https://bugs.launchpad.net/tempest/+bug/1183436
   -

   CURL: GET http://
   $IP:8774/v2/$PROJECT_ID/servers/$SERVER/os-virtual-interfaces
   -

   Explanation: This HTTP request calls the Quantum API
   (nova/nova/network/quantumv2/api.py) and specifically the get_vifs_by_*
   methods which are not implemented (raise NotImplementedError())
   -

   Possible Fix:
   -

      skip this test if Quantum is enabled as set in Tempest configuration.
      Or
      -

      Implement the get_vifs_by_* methods

PATCH: (still not approved)  https://review.openstack.org/#/c/31755/


On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <sean at dague.net> wrote:

> On 06/18/2013 03:32 PM, Monty Taylor wrote:
>
>>
>>
>> On 06/18/2013 03:14 PM, David Ripton wrote:
>>
>>> On 06/18/2013 12:43 PM, Martina Kollarova wrote:
>>>
>>>> Jenkins keeps running all the tests, even if the basic pep8 test fails,
>>>> and runs all of the (very slow) Tempest Quantum tests, even though
>>>> almost all of them are failing.
>>>>
>>>> I propose that it should fail and stop all of the other tests once there
>>>> is a failure in a voting test. For non-voting tests, it should stop only
>>>> itself, not the others.
>>>>
>>>> This would decrease the feedback loop and we wouldn't have to wait for
>>>> the non-voting Quantum tests to see that they failed as always.
>>>>
>>>
>>> -1
>>>
>>> In addition to the other objections, we currently get a lot of false
>>> positives (fail, retry, fail, retry, succeed), and it would be harder to
>>> debug these problems if the output was truncated differently each time.
>>>
>>> Is anyone working on fixing the perma-failing Quantum test?  When the
>>> Postgres test was perma-failing, one of the infrastructure folks gave us
>>> an ultimatum that if nobody fixed it soon, it would be disabled.  (Happy
>>> ending: Mauro fixed it before it got disabled.)
>>>
>>
>> That was brought up a little while ago, but we had already spent so much
>> effort to get it working in the first place, none of us had the heart to
>> put in such an ultimatum. But seriously- it might be time for an
>> all-hands-on-deck dogpile to figure out what's up the the quantum gate.
>>
>
> The biggest cause of the Quantum vs. Full Tempest runs is that a lot of
> the network api's in nova currently don't do translation of errors. So
> under nova-network certain data validation and error codes are returned,
> when quantum is used others are returned.
>
> This is a nova-api, so it needs to be consistent regardless of backend
> (i.e. we don't return different API responses on different databases).
>
> Issues like this one - https://bugs.launchpad.net/**nova/+bug/1160309<https://bugs.launchpad.net/nova/+bug/1160309>
>
> Jordan Pittier has been working on some of these issues (he's the only one
> I've seen working them from a Tempest / nova side), and got to the crux of
> the problem. It could use more hands though to organize the rest of those
> and get them banged out.
>
> I'm sure there are other issues once we get past this class. But that
> would go a long way.
>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
> ______________________________**_________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130619/35cdbbca/attachment.html>