[openstack-dev] Quantum in Full Gate Issues (was Re: Gerrit's Jenkins should stop running tests after first failure)
Sean Dague
sean at dague.net
Wed Jun 19 15:10:37 UTC 2013
This is great stuff! You guys have this up in etherpad somewhere?
Really great to see people organizing around this. I'd reviewed a bunch
of Jordan's code, so I knew he was in the mix, sorry for not realizing
there were other folks working on this as well.
-Sean
On 06/19/2013 10:27 AM, Miguel Lavalle wrote:
> Hi,
>
> As of last week, we have organized a team to deal with the Quantum in
> full gate issues. There are 3 people working on this: Jordan Pittier,
> Ala.Rezmerita and myself. Below you will find a document describing all
> the identified issues and who is assigned to fix them. Of cource, we can
> use more hands / heads. If anyone wants to help or has identified other
> issues not listed below, please get in touch with us.
>
> Regards
>
> Fix Jenkins gate-tempest-devstack-vm-quantum-full
>
>
> This document lists all the Tempest tests that are failing on a Devstack
> + Quantum Trunk setup. The goal is to make these tests pass for the
> Havana Milestone 3. Alla, Miguel and Jordan are working on this.
>
>
> For each test :
>
> *
>
> Provide the full path of the test. If the test has a double
> interface (JSON and XML), it’s enough to provide only the JSON path.
>
> *
>
> Provide a small stacktrace or a link to a Paste service such as
> http://paste.openstack.org/
>
> *
>
> Would be great to provide the CURL URL that triggers the bug
>
> *
>
> Provide a small analysis of what the bug may be and the components
> involved (Quantum, Quantum API on Nova’s side, Nova-network etc.)
>
> *
>
> As far as possible, provide the file and the line number where the
> Python exception is raised or translated (“casted”)
>
> *
>
> If already filed, an URL to the bug report in Launchpad
>
>
> Tests related to the Fixed IPs Compute API extension (Miguel)
>
>
> 1)
> tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_list_fixed_ip_details
>
> *
>
> Trace: http://paste.openstack.org/show/38363/
>
> *
>
> curl-H "X-Auth-Token:$TOKEN" -X GET
> http://$IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3 <http://10.0.0.3>
>
> *
>
> API: http://api.openstack.org/api-ref.html#ext-os-fixed-ips
>
> *
>
> Explanation: Call to
> nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show()
> This file doesn’t use the Quantum API nor the Nova-Network API. It
> interacts directly with the DB, which is bad.
>
> *
>
> A possible fix would be to :
>
> 1.
>
> Change
> nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show()
> to use either nova.network.api.API::get_fixed_ip() (for Nova
> Network) or nova.network.quantumv2.api.API:get_fixed_ip() (for
> Nova Network)
>
> 2.
>
> Implement nova.network.quantumv2.api.API:get_fixed_ip() which
> currently raises a NotImplementedError exception
>
> 2)
> tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_reserve
>
> *
>
> Trace: http://paste.openstack.org/show/38372/
>
> *
>
> curl-H "X-Auth-Token:$TOKEN" -X POST
> http://$IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3/action
> <http://10.0.0.3/action>
>
> *
>
> API: http://api.openstack.org/api-ref.html#ext-os-fixed-ips
>
> *
>
> Possible Fix: Should call
> nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::_set_reserved()
> (once the result of db.fixed_ip_get_by_address() is made through the
> API). Here the problem is that neither Nova-Network nor Quatum API
> implement an equivalent of db.fixed_ip_update()
>
> 3)
> tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_unreserve
>
> *
>
> Same as 2)
>
>
>
> Tests related to Quotas Admin(Miguel)
>
>
> 4)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_exceed_limit
>
> 5)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_rules_exceed_limit
>
>
> Tests related to Floating IPs
>
>
> 6)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_associate_ip_to_server_without_passing_floating_ip
>
> *
>
> Trace Tempest http://paste.openstack.org/show/38431/
>
> *
>
> Bug : https://bugs.launchpad.net/quantum/+bug/1190242
>
> *
>
> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
> POST http://$IP:8774/v2/$TENANT_ID/servers/$SERVER_ID/action
> <http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action>-d
> ‘{"addFloatingIp": {"address": ""}}’
>
> *
>
> Notice that the address of the FloatingIP is empty.
>
> *
>
> API : http://api.openstack.org/api-ref.html#ext-floating-ips
>
> *
>
> Explanation : Mismatch in the 2 API. If there is only one IP in the
> pool, Quantum allows the floating IP to be empty and returns the one
> and only IP in the pool. Nova-network doesn’t allow this and returns
> a 404
>
> *
>
> Review https://review.openstack.org/#/c/32740/
>
>
> 7)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_floating_ip
>
> *
>
> Trace Tempest : http://paste.openstack.org/show/38505/
>
> *
>
> Bug : https://bugs.launchpad.net/tempest/+bug/1160309(see comment #2)
>
> *
>
> This is related to bug 9)
>
>
> 8)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_nonexistant_floating_ip
>
>
> *
>
> Related to 9)
>
>
> 9)tempest.api.compute.floating_ips.test_list_floating_ips:FloatingIPDetailsTestJSON.test_get_nonexistant_floating_ip_details
>
> *
>
> Bug :https://bugs.launchpad.net/tempest/+bug/1160309
>
> *
>
> Trace tempest:http://paste.openstack.org/show/38430/
>
> *
>
> Trace nova: http://paste.openstack.org/show/38433/
>
> *
>
> Curl:curl -H "X-Auth-Token:$TOKEN" -X GET
> http://$IP:8774/v2/$PROJECT_ID/os-floating-ips/99987878787878
>
> Proposed Fix: https://review.openstack.org/33024
>
>
> Tests Related to Security Groups (Jordan)
>
>
> 10)tempest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_create_with_invalid_id
>
> *
>
> TRACE: http://paste.openstack.org/show/38373/
>
> *
>
> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
> POST http://$IP:8774/v2/$TENANT_ID/os-security-group-rules -d
> ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp",
> "to_port": 22, "parent_group_id": "9991393497170", "cidr": null,
> "group_id": null}}’
>
> *
>
> Explanation: Notice that the parent_goup_id is a numerical ID and
> not a UUID. Quantum has an additional check to validate that the ID
> is an UUID (see
> nova/network/security_group/quantum_driver.py::validate_id())
>
> *
>
> API: http://api.openstack.org/api-ref.html#ext-os-security-groups
>
> *
>
> Bug: https://bugs.launchpad.net/tempest/+bug/1182384
>
> *
>
> Review: https://review.openstack.org/#/c/29899/
>
>
> 11)tempest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_delete_with_invalid_id
>
> *
>
> http://paste.openstack.org/show/38424/
>
> *
>
> curl -H "X-Auth-Token:$TOKEN" -X DELETE
> http://$IP:8774/v2$TENANT_ID/os-security-group-rules/9991407551273
>
> *
>
> Explanation : Same bug as 10
>
> *
>
> API: http://api.openstack.org/api-ref.html#ext-os-security-groups
>
> *
>
> Review: https://review.openstack.org/#/c/29899/
>
> 12)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_delete_nonexistant_security_group
>
> *
>
> Same as bug 10
>
> *
>
> Review: https://review.openstack.org/#/c/29899/
>
>
> 13)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_get_nonexistant_group
>
> *
>
> Same as bug 10
>
> *
>
> Review: https://review.openstack.org/#/c/29899/
>
>
> 14)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_duplicate_name
>
> Security Group with duplicate name should not be created, but two groups
> with the same name can be created in quantum. We have here the same
> problem as in 15 and in 16. With Quantum, there is no validation that a
> group with given name exists already or if the given SG name is empty
> or is composed of white spaces or is more than 255 chars.
>
>
> In the description of bug
> https://bugs.launchpad.net/nova/+bug/1161411this issue is generally
> discussed. SecurityGroup API are based on the ID and not the names,
> except for adding an instance to a security group. In order to solve
> the last problem the bug https://bugs.launchpad.net/nova/+bug/1161473was
> added.
>
>
> The major question is if these 3 tests (14, 15, 16) : does the name of a
> security group is really that important? If so, we must add some
> validation methods. If not the test suit concerning this part must be
> disable in tempest. What do you think Miguel?
>
>
>
> 15)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_description
>
> 16)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_name
>
> BUG:https://bugs.launchpad.net/nova/+bug/1161411+
> https://bugs.launchpad.net/nova/+bug/1161473
>
> Traceback(tempest) :http://paste.openstack.org/show/38423/
>
> The Security Group should not be created with group name an empty string
> or with white spaces/chars more than 255
>
> CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN"
> -X POST http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
> '{"security_group": {"name": " ", "description":
> "description-1554950088"}}'
>
> curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
> POST http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
> '{"security_group": {"name": " ", "description":
> "description-1554950088"}}'
>
>
> 17)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_server_security_groups
>
> *
>
> Tempest Trace : http://paste.openstack.org/show/38427/
>
> *
>
> Fixed by https://review.openstack.org/#/c/32288/(merged on June,12th)
>
>
>
> Tests related to servers(Ala)
>
>
> 18)tempest.api.compute.servers.test_list_server_filters:ListServerFiltersTestXML.test_list_servers_filtered_by_ip_regex
>
> *
>
> BUG: https://bugs.launchpad.net/quantum/+bug/1182883
>
> *
>
> BP: https://blueprints.launchpad.net/quantum/+spec/like-op-list
>
> *
>
> CURL : GET http://$IP:8774/v2/$PROJECT_ID/servers?ip=10.
>
> *
>
> Explanation: The regex search is not supported by Quantum. Thus
> Quantum returns a 404 Not Found (0 server match) where Tempest
> expects one server to be found.
>
> *
>
> Possible Fix : For "search port by IP with regex" feature, I think
> the best place to hack it would be in file
> quantum/db/db_base_plugin_v2.py::_get_ports_query()
>
>
> 19)tempest.api.compute.servers.test_servers_negative:ServersNegativeTest.test_create_with_nonexistent_security_group
>
> FIXED : https://review.openstack.org/#/c/30271/
>
>
> 20)tempest.api.compute.servers.test_virtual_interfaces:VirtualInterfacesTestXML.test_list_virtual_interfaces
>
> *
>
> TRACE (NOVA): http://paste.openstack.org/show/38371/
>
> *
>
> BUG: https://bugs.launchpad.net/tempest/+bug/1183436
>
> *
>
> CURL: GET
> http://$IP:8774/v2/$PROJECT_ID/servers/$SERVER/os-virtual-interfaces
>
> *
>
> Explanation: This HTTP request calls the Quantum API
> (nova/nova/network/quantumv2/api.py) and specifically the
> get_vifs_by_* methods which are not implemented (raise
> NotImplementedError())
>
> *
>
> Possible Fix:
>
> o
>
> skip this test if Quantum is enabled as set in Tempest
> configuration. Or
>
> o
>
> Implement the get_vifs_by_* methods
>
> PATCH: (still not approved) https://review.openstack.org/#/c/31755/
>
>
> On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
>
> On 06/18/2013 03:32 PM, Monty Taylor wrote:
>
>
>
> On 06/18/2013 03:14 PM, David Ripton wrote:
>
> On 06/18/2013 12:43 PM, Martina Kollarova wrote:
>
> Jenkins keeps running all the tests, even if the basic
> pep8 test fails,
> and runs all of the (very slow) Tempest Quantum tests,
> even though
> almost all of them are failing.
>
> I propose that it should fail and stop all of the other
> tests once there
> is a failure in a voting test. For non-voting tests, it
> should stop only
> itself, not the others.
>
> This would decrease the feedback loop and we wouldn't
> have to wait for
> the non-voting Quantum tests to see that they failed as
> always.
>
>
> -1
>
> In addition to the other objections, we currently get a lot
> of false
> positives (fail, retry, fail, retry, succeed), and it would
> be harder to
> debug these problems if the output was truncated differently
> each time.
>
> Is anyone working on fixing the perma-failing Quantum test?
> When the
> Postgres test was perma-failing, one of the infrastructure
> folks gave us
> an ultimatum that if nobody fixed it soon, it would be
> disabled. (Happy
> ending: Mauro fixed it before it got disabled.)
>
>
> That was brought up a little while ago, but we had already spent
> so much
> effort to get it working in the first place, none of us had the
> heart to
> put in such an ultimatum. But seriously- it might be time for an
> all-hands-on-deck dogpile to figure out what's up the the
> quantum gate.
>
>
> The biggest cause of the Quantum vs. Full Tempest runs is that a lot
> of the network api's in nova currently don't do translation of
> errors. So under nova-network certain data validation and error
> codes are returned, when quantum is used others are returned.
>
> This is a nova-api, so it needs to be consistent regardless of
> backend (i.e. we don't return different API responses on different
> databases).
>
> Issues like this one -
> https://bugs.launchpad.net/__nova/+bug/1160309
> <https://bugs.launchpad.net/nova/+bug/1160309>
>
> Jordan Pittier has been working on some of these issues (he's the
> only one I've seen working them from a Tempest / nova side), and got
> to the crux of the problem. It could use more hands though to
> organize the rest of those and get them banged out.
>
> I'm sure there are other issues once we get past this class. But
> that would go a long way.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> _________________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.__org
> <mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Sean Dague
http://dague.net
More information about the OpenStack-dev
mailing list