Open Stack

Hi,

During the last Quantum IRC meeting, I mentioned that in working with
Tempest tests we have identified a good number of situations where the nova
and quantum api's behaviors are not consistent. Below you will find a
document that describes the identified issues and who is assigned to fix
them. If anyone knows of other issues not listed below or wants to help,
please let us know.

Fix Jenkins gate-tempest-devstack-vm-quantum-full

This document lists all the Tempest tests that are failing on a Devstack +
Quantum Trunk setup. The goal is to make these tests pass for the Havana
Milestone 3. Alla, Miguel and Jordan are working on this.

For each test :

   -

   Provide the full path of the test. If the test has a double interface
   (JSON and XML), it’s enough to provide only the JSON path.
   -

   Provide a small stacktrace or a link to a Paste service such as
   http://paste.openstack.org/
   -

   Would be great to provide the CURL URL that triggers the bug
   -

   Provide  a small analysis of what the bug may be and the components
   involved (Quantum, Quantum API on Nova’s side, Nova-network etc.)
   -

   As far as possible, provide the file and the line number where the
   Python exception is raised or translated (“casted”)
   -

   If already filed, an URL to the bug report in Launchpad

Tests related to the Fixed IPs Compute API extension (Miguel)

1)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_list_fixed_ip_details

   -

   Trace : http://paste.openstack.org/show/38363/
   -

   curl -H "X-Auth-Token:$TOKEN" -X GET http://
   $IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3
   -

   API : http://api.openstack.org/api-ref.html#ext-os-fixed-ips
   -

   Explanation : Call to
   nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show() This
   file doesn’t use the Quantum API nor the Nova-Network API. It interacts
   directly with the DB, which is bad.
   -

   A possible fix would be to :
   1.

      Change
      nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show() to
      use either nova.network.api.API::get_fixed_ip() (for Nova Network) or
      nova.network.quantumv2.api.API:get_fixed_ip() (for Nova Network)
      2.

      Implement nova.network.quantumv2.api.API:get_fixed_ip() which
      currently raises a NotImplementedError exception

2)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_reserve

   -

   Trace : http://paste.openstack.org/show/38372/
   -

   curl  -H "X-Auth-Token:$TOKEN" -X POST http://
   $IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3/action
   -

   API : http://api.openstack.org/api-ref.html#ext-os-fixed-ips
   -

   Possible Fix : Should call
   nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::_set_reserved()
   (once the result of db.fixed_ip_get_by_address() is made through the API).
   Here the problem is that neither Nova-Network nor Quatum API implement an
   equivalent of db.fixed_ip_update()

3)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_unreserve

   -

   Same as 2)



Tests related to Quotas Admin(Miguel)

4)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_exceed_limit

5)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_rules_exceed_limit

Tests related to Floating IPs

6)
tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_associate_ip_to_server_without_passing_floating_ip

   -

   Trace Tempest http://paste.openstack.org/show/38431/
   -

   Bug : https://bugs.launchpad.net/quantum/+bug/1190242


   -

   curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
    POST http://$IP:8774/v2/$TENANT_ID/servers/$SERVER_ID/action<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action>-d
‘{"addFloatingIp": {"address": ""}}’
   -

   Notice that the address of the FloatingIP is empty.
   -

   API : http://api.openstack.org/api-ref.html#ext-floating-ips
   -

   Explanation : Mismatch in the 2 API. If there is only one IP in the
   pool, Quantum allows the floating IP to be empty and returns the one and
   only IP in the pool. Nova-network doesn’t allow this and returns a 404
   -

   Review https://review.openstack.org/#/c/32740/


7)
tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_floating_ip

   -

   Trace Tempest : http://paste.openstack.org/show/38505/
   -

   Bug : https://bugs.launchpad.net/tempest/+bug/1160309 (see comment #2)
   -

   This is related to bug 9)


8)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_nonexistant_floating_ip


   -

   Related to 9)


9)
tempest.api.compute.floating_ips.test_list_floating_ips:FloatingIPDetailsTestJSON.test_get_nonexistant_floating_ip_details

   -

   Bug :https://bugs.launchpad.net/tempest/+bug/1160309


   -

   Trace tempest:http://paste.openstack.org/show/38430/
   -

   Trace nova: http://paste.openstack.org/show/38433/
   -

   Curl:curl -H "X-Auth-Token:$TOKEN" -X GET http://
   $IP:8774/v2/$PROJECT_ID/os-floating-ips/99987878787878

Proposed Fix: https://review.openstack.org/33024

Tests Related to Security Groups (Jordan)

10)t
empest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_create_with_invalid_id

   -

   TRACE :  http://paste.openstack.org/show/38373/
   -

   curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
    POST http://$IP:8774/v2/$TENANT_ID/os-security-group-rules -d
   ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp", "to_port":
   22, "parent_group_id": "9991393497170", "cidr": null, "group_id": null}}’


   -

   Explanation : Notice that the parent_goup_id is  a numerical ID and not
   a UUID. Quantum has an additional check to validate that the ID is an UUID
   (see nova/network/security_group/quantum_driver.py::validate_id())


   -

   API : http://api.openstack.org/api-ref.html#ext-os-security-groups
   -

   Bug : https://bugs.launchpad.net/tempest/+bug/1182384
   -

   Review : https://review.openstack.org/#/c/29899/


11)
tempest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_delete_with_invalid_id

   -

   http://paste.openstack.org/show/38424/
   -

   curl  -H "X-Auth-Token:$TOKEN" -X  DELETE http://
   $IP:8774/v2$TENANT_ID/os-security-group-rules/9991407551273
   -

   Explanation : Same bug as 10
   -

   API : http://api.openstack.org/api-ref.html#ext-os-security-groups
   -

   Review : https://review.openstack.org/#/c/29899/

12)
tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_delete_nonexistant_security_group

   -

   Same as bug 10
   -

   Review : https://review.openstack.org/#/c/29899/


13)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_get_nonexistant_group

   -

   Same as bug 10
   -

   Review : https://review.openstack.org/#/c/29899/


14)
tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_duplicate_name

Security Group with duplicate name should not be created, but two groups
with the same name can be created in quantum. We have here the same problem
as in 15 and in 16. With Quantum, there is no validation that a  group with
given name exists already or if the given SG name is empty or is composed
of white spaces or is more than 255 chars.

In the description of bug https://bugs.launchpad.net/nova/+bug/1161411 this
issue is generally discussed. SecurityGroup API  are based on the ID and
not the names, except for adding an instance to a security group.  In order
to solve the last problem the bug
https://bugs.launchpad.net/nova/+bug/1161473 was added.

The major question is if these 3 tests (14, 15, 16) : does the name of a
security group is really that important? If so, we must add some validation
methods. If not the test suit concerning this part must be disable in
tempest. What do you think Miguel?


 15)
tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_description

16)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_name

BUG:https://bugs.launchpad.net/nova/+bug/1161411 +
https://bugs.launchpad.net/nova/+bug/1161473

Traceback(tempest) :http://paste.openstack.org/show/38423/

The Security Group should not be created with group name an empty string or
with white spaces/chars more than 255

CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
POST http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
 '{"security_group": {"name": " ", "description":
"description-1554950088"}}'

curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X POST
http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
'{"security_group": {"name":
"  ", "description": "description-1554950088"}}'

17)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_server_security_groups

   -

   Tempest Trace : http://paste.openstack.org/show/38427/
   -

   Fixed by https://review.openstack.org/#/c/32288/ (merged on June,12th)



Tests related to servers(Ala)

18)
tempest.api.compute.servers.test_list_server_filters:ListServerFiltersTestXML.test_list_servers_filtered_by_ip_regex

   -

   BUG: https://bugs.launchpad.net/quantum/+bug/1182883
   -

   BP: https://blueprints.launchpad.net/quantum/+spec/like-op-list
   -

   CURL : GET http://$IP:8774/v2/$PROJECT_ID/servers?ip=10.
   -

   Explanation : The  regex search is not supported by Quantum. Thus
   Quantum returns a 404 Not Found (0 server match) where Tempest expects one
   server to be found.
   -

   Possible Fix : For "search port by IP with regex" feature, I think the
   best place to hack it would be in file
   quantum/db/db_base_plugin_v2.py::_get_ports_query()


19)tempest.api.compute.servers.test_servers_negative:ServersNegativeTest.test_create_with_nonexistent_security_group

FIXED : https://review.openstack.org/#/c/30271/

20)tempest.api.compute.servers.test_virtual_interfaces:VirtualInterfacesTestXML.test_list_virtual_interfaces

   -

   TRACE (NOVA): http://paste.openstack.org/show/38371/
   -

   BUG: https://bugs.launchpad.net/tempest/+bug/1183436
   -

   CURL: GET http://
   $IP:8774/v2/$PROJECT_ID/servers/$SERVER/os-virtual-interfaces
   -

   Explanation: This HTTP request calls the Quantum API
   (nova/nova/network/quantumv2/api.py) and specifically the get_vifs_by_*
   methods which are not implemented (raise NotImplementedError())
   -

   Possible Fix:
   -

      skip this test if Quantum is enabled as set in Tempest configuration.
      Or
      -

      Implement the get_vifs_by_* methods

PATCH: (still not approved)  https://review.openstack.org/#/c/31755/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130619/8f3fdd7f/attachment.html>