[openstack-dev] Quantum in Full Gate Issues (was Re: Gerrit's Jenkins should stop running tests after first failure)
Ala Rezmerita
ala.rezmerita at cloudwatt.com
Wed Jun 19 15:38:13 UTC 2013
I put it here
https://etherpad.openstack.org/gate-tempest-devstack-vm-quantum-full
Ala
2013/6/19 Miguel Lavalle <miguel at mlavalle.com>
> No, we haven't put this document in etherpad. But it's a great idea. This
> way other folks can jump in with ideas. I'll put it up in eteherpad over
> the next few days
>
>
> On Wed, Jun 19, 2013 at 10:10 AM, Sean Dague <sean at dague.net> wrote:
>
>> This is great stuff! You guys have this up in etherpad somewhere?
>>
>> Really great to see people organizing around this. I'd reviewed a bunch
>> of Jordan's code, so I knew he was in the mix, sorry for not realizing
>> there were other folks working on this as well.
>>
>> -Sean
>>
>>
>> On 06/19/2013 10:27 AM, Miguel Lavalle wrote:
>>
>>> Hi,
>>>
>>> As of last week, we have organized a team to deal with the Quantum in
>>> full gate issues. There are 3 people working on this: Jordan Pittier,
>>> Ala.Rezmerita and myself. Below you will find a document describing all
>>> the identified issues and who is assigned to fix them. Of cource, we can
>>> use more hands / heads. If anyone wants to help or has identified other
>>> issues not listed below, please get in touch with us.
>>>
>>> Regards
>>>
>>> Fix Jenkins gate-tempest-devstack-vm-**quantum-full
>>>
>>>
>>> This document lists all the Tempest tests that are failing on a Devstack
>>> + Quantum Trunk setup. The goal is to make these tests pass for the
>>> Havana Milestone 3. Alla, Miguel and Jordan are working on this.
>>>
>>>
>>> For each test :
>>>
>>> *
>>>
>>> Provide the full path of the test. If the test has a double
>>> interface (JSON and XML), it’s enough to provide only the JSON path.
>>>
>>> *
>>>
>>> Provide a small stacktrace or a link to a Paste service such as
>>> http://paste.openstack.org/
>>>
>>> *
>>>
>>> Would be great to provide the CURL URL that triggers the bug
>>>
>>> *
>>>
>>> Provide a small analysis of what the bug may be and the components
>>> involved (Quantum, Quantum API on Nova’s side, Nova-network etc.)
>>>
>>> *
>>>
>>> As far as possible, provide the file and the line number where the
>>> Python exception is raised or translated (“casted”)
>>>
>>> *
>>>
>>> If already filed, an URL to the bug report in Launchpad
>>>
>>>
>>> Tests related to the Fixed IPs Compute API extension (Miguel)
>>>
>>>
>>> 1)
>>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_list_
>>> **fixed_ip_details
>>>
>>> *
>>>
>>> Trace: http://paste.openstack.org/**show/38363/<http://paste.openstack.org/show/38363/>
>>>
>>> *
>>>
>>> curl-H "X-Auth-Token:$TOKEN" -X GET
>>> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3 <
>>> http://10.0.0.3>
>>>
>>> *
>>>
>>> API: http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips>
>>>
>>>
>>> *
>>>
>>> Explanation: Call to
>>> nova.api.openstack.compute.**contrib.fixed_ips.**
>>> FixedIPController::show()
>>> This file doesn’t use the Quantum API nor the Nova-Network API. It
>>> interacts directly with the DB, which is bad.
>>>
>>> *
>>>
>>> A possible fix would be to :
>>>
>>> 1.
>>>
>>>
>>> Change
>>> nova.api.openstack.compute.**contrib.fixed_ips.**
>>> FixedIPController::show()
>>> to use either nova.network.api.API::get_**fixed_ip() (for Nova
>>> Network) or nova.network.quantumv2.api.**API:get_fixed_ip() (for
>>> Nova Network)
>>>
>>> 2.
>>>
>>>
>>> Implement nova.network.quantumv2.api.**API:get_fixed_ip() which
>>> currently raises a NotImplementedError exception
>>>
>>> 2)
>>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_*
>>> *reserve
>>>
>>> *
>>>
>>> Trace: http://paste.openstack.org/**show/38372/<http://paste.openstack.org/show/38372/>
>>>
>>> *
>>>
>>> curl-H "X-Auth-Token:$TOKEN" -X POST
>>> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3/action
>>> <http://10.0.0.3/action>
>>>
>>> *
>>>
>>> API: http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips>
>>>
>>>
>>> *
>>>
>>> Possible Fix: Should call
>>> nova.api.openstack.compute.**contrib.fixed_ips.**
>>> FixedIPController::_set_**reserved()
>>> (once the result of db.fixed_ip_get_by_address() is made through the
>>> API). Here the problem is that neither Nova-Network nor Quatum API
>>> implement an equivalent of db.fixed_ip_update()
>>>
>>> 3)
>>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_*
>>> *unreserve
>>>
>>> *
>>>
>>> Same as 2)
>>>
>>>
>>>
>>> Tests related to Quotas Admin(Miguel)
>>>
>>>
>>> 4)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_**
>>> security_groups_exceed_limit
>>>
>>> 5)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_**
>>> security_groups_rules_exceed_**limit
>>>
>>>
>>> Tests related to Floating IPs
>>>
>>>
>>> 6)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>>> FloatingIPsTestJSON.test_**associate_ip_to_server_**
>>> without_passing_floating_ip
>>>
>>> *
>>>
>>> Trace Tempest http://paste.openstack.org/**show/38431/<http://paste.openstack.org/show/38431/>
>>>
>>> *
>>>
>>> Bug : https://bugs.launchpad.net/**quantum/+bug/1190242<https://bugs.launchpad.net/quantum/+bug/1190242>
>>>
>>> *
>>>
>>> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>> POST http://$IP:8774/v2/$TENANT_ID/**servers/$SERVER_ID/action
>>> <http://10.1.59.157:8774/v2/**de6c5fbc55a34cbcaa3d79eb6b21a7**
>>> 84/servers/0b2ad3b6-c14a-4d89-**b2a0-c015f0a88a1f/action<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action>
>>> >-d
>>> ‘{"addFloatingIp": {"address": ""}}’
>>>
>>>
>>> *
>>>
>>> Notice that the address of the FloatingIP is empty.
>>>
>>> *
>>>
>>> API : http://api.openstack.org/api-**ref.html#ext-floating-ips<http://api.openstack.org/api-ref.html#ext-floating-ips>
>>>
>>>
>>> *
>>>
>>> Explanation : Mismatch in the 2 API. If there is only one IP in the
>>> pool, Quantum allows the floating IP to be empty and returns the one
>>> and only IP in the pool. Nova-network doesn’t allow this and returns
>>> a 404
>>>
>>> *
>>>
>>> Review https://review.openstack.org/#**/c/32740/<https://review.openstack.org/#/c/32740/>
>>>
>>>
>>> 7)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>>> FloatingIPsTestJSON.test_**delete_floating_ip
>>>
>>> *
>>>
>>> Trace Tempest : http://paste.openstack.org/**show/38505/<http://paste.openstack.org/show/38505/>
>>>
>>> *
>>>
>>> Bug : https://bugs.launchpad.net/**tempest/+bug/1160309(see<https://bugs.launchpad.net/tempest/+bug/1160309(see>comment #2)
>>>
>>>
>>> *
>>>
>>> This is related to bug 9)
>>>
>>>
>>> 8)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>>> FloatingIPsTestJSON.test_**delete_nonexistant_floating_ip
>>>
>>>
>>> *
>>>
>>> Related to 9)
>>>
>>>
>>> 9)tempest.api.compute.**floating_ips.test_list_**floating_ips:**
>>> FloatingIPDetailsTestJSON.**test_get_nonexistant_floating_**ip_details
>>>
>>> *
>>>
>>> Bug :https://bugs.launchpad.net/**tempest/+bug/1160309<https://bugs.launchpad.net/tempest/+bug/1160309>
>>>
>>> *
>>>
>>> Trace tempest:http://paste.**openstack.org/show/38430/<http://paste.openstack.org/show/38430/>
>>>
>>> *
>>>
>>> Trace nova: http://paste.openstack.org/**show/38433/<http://paste.openstack.org/show/38433/>
>>>
>>>
>>> *
>>>
>>> Curl:curl -H "X-Auth-Token:$TOKEN" -X GET
>>> http://$IP:8774/v2/$PROJECT_**ID/os-floating-ips/**99987878787878
>>>
>>> Proposed Fix: https://review.openstack.org/**33024<https://review.openstack.org/33024>
>>>
>>>
>>> Tests Related to Security Groups (Jordan)
>>>
>>>
>>> 10)tempest.api.compute.**security_groups.test_security_**group_rules:**
>>> SecurityGroupRulesTestJSON.**test_security_group_rules_**
>>> create_with_invalid_id
>>>
>>> *
>>>
>>> TRACE: http://paste.openstack.org/**show/38373/<http://paste.openstack.org/show/38373/>
>>>
>>> *
>>>
>>> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>>
>>> POST http://$IP:8774/v2/$TENANT_ID/**os-security-group-rules -d
>>> ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp",
>>> "to_port": 22, "parent_group_id": "9991393497170", "cidr": null,
>>> "group_id": null}}’
>>>
>>> *
>>>
>>> Explanation: Notice that the parent_goup_id is a numerical ID and
>>> not a UUID. Quantum has an additional check to validate that the ID
>>> is an UUID (see
>>> nova/network/security_group/**quantum_driver.py::validate_**id())
>>>
>>> *
>>>
>>> API: http://api.openstack.org/api-**ref.html#ext-os-security-**
>>> groups <http://api.openstack.org/api-ref.html#ext-os-security-groups>
>>>
>>> *
>>>
>>> Bug: https://bugs.launchpad.net/**tempest/+bug/1182384<https://bugs.launchpad.net/tempest/+bug/1182384>
>>>
>>>
>>> *
>>>
>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>
>>>
>>> 11)tempest.api.compute.**security_groups.test_security_**group_rules:**
>>> SecurityGroupRulesTestJSON.**test_security_group_rules_**
>>> delete_with_invalid_id
>>>
>>> *
>>>
>>> http://paste.openstack.org/**show/38424/<http://paste.openstack.org/show/38424/>
>>>
>>>
>>> *
>>>
>>> curl -H "X-Auth-Token:$TOKEN" -X DELETE
>>> http://$IP:8774/v2$TENANT_ID/**os-security-group-rules/**
>>> 9991407551273
>>>
>>> *
>>>
>>> Explanation : Same bug as 10
>>>
>>> *
>>>
>>> API: http://api.openstack.org/api-**ref.html#ext-os-security-**
>>> groups <http://api.openstack.org/api-ref.html#ext-os-security-groups>
>>>
>>>
>>> *
>>>
>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>
>>> 12)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_delete_nonexistant_**security_group
>>>
>>> *
>>>
>>> Same as bug 10
>>>
>>>
>>> *
>>>
>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>
>>>
>>> 13)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_get_**
>>> nonexistant_group
>>>
>>> *
>>>
>>> Same as bug 10
>>>
>>>
>>> *
>>>
>>> Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>>
>>>
>>> 14)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>>> with_duplicate_name
>>>
>>> Security Group with duplicate name should not be created, but two groups
>>> with the same name can be created in quantum. We have here the same
>>> problem as in 15 and in 16. With Quantum, there is no validation that a
>>> group with given name exists already or if the given SG name is empty
>>> or is composed of white spaces or is more than 255 chars.
>>>
>>>
>>> In the description of bug
>>> https://bugs.launchpad.net/**nova/+bug/1161411this<https://bugs.launchpad.net/nova/+bug/1161411this>issue is generally
>>>
>>> discussed. SecurityGroup API are based on the ID and not the names,
>>> except for adding an instance to a security group. In order to solve
>>> the last problem the bug https://bugs.launchpad.net/**
>>> nova/+bug/1161473was <https://bugs.launchpad.net/nova/+bug/1161473was>
>>> added.
>>>
>>>
>>> The major question is if these 3 tests (14, 15, 16) : does the name of a
>>> security group is really that important? If so, we must add some
>>> validation methods. If not the test suit concerning this part must be
>>> disable in tempest. What do you think Miguel?
>>>
>>>
>>>
>>> 15)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>>> with_invalid_group_description
>>>
>>> 16)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>>> with_invalid_group_name
>>>
>>> BUG:https://bugs.launchpad.**net/nova/+bug/1161411+<https://bugs.launchpad.net/nova/+bug/1161411+>
>>> https://bugs.launchpad.net/**nova/+bug/1161473<https://bugs.launchpad.net/nova/+bug/1161473>
>>>
>>> Traceback(tempest) :http://paste.openstack.org/**show/38423/<http://paste.openstack.org/show/38423/>
>>>
>>> The Security Group should not be created with group name an empty string
>>> or with white spaces/chars more than 255
>>>
>>> CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN"
>>> -X POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d
>>> '{"security_group": {"name": " ", "description":
>>> "description-1554950088"}}'
>>>
>>> curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>> POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d
>>> '{"security_group": {"name": " ", "description":
>>> "description-1554950088"}}'
>>>
>>>
>>> 17)tempest.api.compute.**security_groups.test_security_**
>>> groups:SecurityGroupsTestJSON.**test_server_security_groups
>>>
>>> *
>>>
>>> Tempest Trace : http://paste.openstack.org/**show/38427/<http://paste.openstack.org/show/38427/>
>>>
>>> *
>>>
>>> Fixed by https://review.openstack.org/#**/c/32288/(merged<https://review.openstack.org/#/c/32288/(merged>on June,12th)
>>>
>>>
>>>
>>>
>>> Tests related to servers(Ala)
>>>
>>>
>>> 18)tempest.api.compute.**servers.test_list_server_**filters:**
>>> ListServerFiltersTestXML.test_**list_servers_filtered_by_ip_**regex
>>>
>>> *
>>>
>>> BUG: https://bugs.launchpad.net/**quantum/+bug/1182883<https://bugs.launchpad.net/quantum/+bug/1182883>
>>>
>>> *
>>>
>>> BP: https://blueprints.launchpad.**net/quantum/+spec/like-op-list<https://blueprints.launchpad.net/quantum/+spec/like-op-list>
>>>
>>>
>>> *
>>>
>>> CURL : GET http://$IP:8774/v2/$PROJECT_**ID/servers?ip=10.
>>>
>>> *
>>>
>>> Explanation: The regex search is not supported by Quantum. Thus
>>> Quantum returns a 404 Not Found (0 server match) where Tempest
>>> expects one server to be found.
>>>
>>> *
>>>
>>> Possible Fix : For "search port by IP with regex" feature, I think
>>> the best place to hack it would be in file
>>> quantum/db/db_base_plugin_v2.**py::_get_ports_query()
>>>
>>>
>>> 19)tempest.api.compute.**servers.test_servers_negative:**
>>> ServersNegativeTest.test_**create_with_nonexistent_**security_group
>>>
>>> FIXED : https://review.openstack.org/#**/c/30271/<https://review.openstack.org/#/c/30271/>
>>>
>>>
>>> 20)tempest.api.compute.**servers.test_virtual_**interfaces:**
>>> VirtualInterfacesTestXML.test_**list_virtual_interfaces
>>>
>>> *
>>>
>>> TRACE (NOVA): http://paste.openstack.org/**show/38371/<http://paste.openstack.org/show/38371/>
>>>
>>> *
>>>
>>> BUG: https://bugs.launchpad.net/**tempest/+bug/1183436<https://bugs.launchpad.net/tempest/+bug/1183436>
>>>
>>>
>>> *
>>>
>>> CURL: GET
>>> http://$IP:8774/v2/$PROJECT_**ID/servers/$SERVER/os-virtual-**
>>> interfaces
>>>
>>> *
>>>
>>> Explanation: This HTTP request calls the Quantum API
>>> (nova/nova/network/quantumv2/**api.py) and specifically the
>>> get_vifs_by_* methods which are not implemented (raise
>>> NotImplementedError())
>>>
>>> *
>>>
>>> Possible Fix:
>>>
>>>
>>> o
>>>
>>> skip this test if Quantum is enabled as set in Tempest
>>> configuration. Or
>>>
>>> o
>>>
>>> Implement the get_vifs_by_* methods
>>>
>>> PATCH: (still not approved) https://review.openstack.org/#**/c/31755/<https://review.openstack.org/#/c/31755/>
>>>
>>>
>>> On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <sean at dague.net
>>> <mailto:sean at dague.net>> wrote:
>>>
>>> On 06/18/2013 03:32 PM, Monty Taylor wrote:
>>>
>>>
>>>
>>> On 06/18/2013 03:14 PM, David Ripton wrote:
>>>
>>> On 06/18/2013 12:43 PM, Martina Kollarova wrote:
>>>
>>> Jenkins keeps running all the tests, even if the basic
>>> pep8 test fails,
>>> and runs all of the (very slow) Tempest Quantum tests,
>>> even though
>>> almost all of them are failing.
>>>
>>> I propose that it should fail and stop all of the other
>>> tests once there
>>> is a failure in a voting test. For non-voting tests, it
>>> should stop only
>>> itself, not the others.
>>>
>>> This would decrease the feedback loop and we wouldn't
>>> have to wait for
>>> the non-voting Quantum tests to see that they failed as
>>> always.
>>>
>>>
>>> -1
>>>
>>> In addition to the other objections, we currently get a lot
>>> of false
>>> positives (fail, retry, fail, retry, succeed), and it would
>>> be harder to
>>> debug these problems if the output was truncated differently
>>> each time.
>>>
>>> Is anyone working on fixing the perma-failing Quantum test?
>>> When the
>>> Postgres test was perma-failing, one of the infrastructure
>>> folks gave us
>>> an ultimatum that if nobody fixed it soon, it would be
>>> disabled. (Happy
>>> ending: Mauro fixed it before it got disabled.)
>>>
>>>
>>> That was brought up a little while ago, but we had already spent
>>> so much
>>> effort to get it working in the first place, none of us had the
>>> heart to
>>> put in such an ultimatum. But seriously- it might be time for an
>>> all-hands-on-deck dogpile to figure out what's up the the
>>> quantum gate.
>>>
>>>
>>> The biggest cause of the Quantum vs. Full Tempest runs is that a lot
>>> of the network api's in nova currently don't do translation of
>>> errors. So under nova-network certain data validation and error
>>> codes are returned, when quantum is used others are returned.
>>>
>>> This is a nova-api, so it needs to be consistent regardless of
>>> backend (i.e. we don't return different API responses on different
>>> databases).
>>>
>>> Issues like this one -
>>> https://bugs.launchpad.net/__**nova/+bug/1160309<https://bugs.launchpad.net/__nova/+bug/1160309>
>>>
>>> <https://bugs.launchpad.net/**nova/+bug/1160309<https://bugs.launchpad.net/nova/+bug/1160309>
>>> >
>>>
>>> Jordan Pittier has been working on some of these issues (he's the
>>> only one I've seen working them from a Tempest / nova side), and got
>>> to the crux of the problem. It could use more hands though to
>>> organize the rest of those and get them banged out.
>>>
>>> I'm sure there are other issues once we get past this class. But
>>> that would go a long way.
>>>
>>> -Sean
>>>
>>> --
>>> Sean Dague
>>> http://dague.net
>>>
>>> ______________________________**___________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.**__org
>>> <mailto:OpenStack-dev at lists.**openstack.org<OpenStack-dev at lists.openstack.org>
>>> >
>>> http://lists.openstack.org/__**cgi-bin/mailman/listinfo/__**
>>> openstack-dev<http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev><
>>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>> >
>>>
>>>
>>>
>>>
>>>
>>> ______________________________**_________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>>
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> ______________________________**_________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
Ala Rezmerita
Ingénieur
CloudWatt
Tel : 06 77 43 23 91
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130619/3438a25a/attachment.html>
More information about the OpenStack-dev
mailing list