Open Stack

No, we haven't put this document in etherpad. But it's a great idea. This
way other folks can jump in with ideas. I'll put it up in eteherpad over
the next few days


On Wed, Jun 19, 2013 at 10:10 AM, Sean Dague <sean at dague.net> wrote:

> This is great stuff! You guys have this up in etherpad somewhere?
>
> Really great to see people organizing around this. I'd reviewed a bunch of
> Jordan's code, so I knew he was in the mix, sorry for not realizing there
> were other folks working on this as well.
>
>         -Sean
>
>
> On 06/19/2013 10:27 AM, Miguel Lavalle wrote:
>
>> Hi,
>>
>> As of last week, we have organized a team to deal with the Quantum in
>> full gate issues. There are 3 people working on this: Jordan Pittier,
>> Ala.Rezmerita and myself. Below you will find a document describing all
>> the identified issues and who is assigned to fix them. Of cource, we can
>> use more hands / heads. If anyone wants to help or has identified other
>> issues not listed below, please get in touch with us.
>>
>> Regards
>>
>> Fix Jenkins gate-tempest-devstack-vm-**quantum-full
>>
>>
>> This document lists all the Tempest tests that are failing on a Devstack
>> + Quantum Trunk setup. The goal is to make these tests pass for the
>> Havana Milestone 3. Alla, Miguel and Jordan are working on this.
>>
>>
>> For each test :
>>
>>   *
>>
>>     Provide the full path of the test. If the test has a double
>>     interface (JSON and XML), it’s enough to provide only the JSON path.
>>
>>   *
>>
>>     Provide a small stacktrace or a link to a Paste service such as
>>     http://paste.openstack.org/
>>
>>   *
>>
>>     Would be great to provide the CURL URL that triggers the bug
>>
>>   *
>>
>>     Provide  a small analysis of what the bug may be and the components
>>     involved (Quantum, Quantum API on Nova’s side, Nova-network etc.)
>>
>>   *
>>
>>     As far as possible, provide the file and the line number where the
>>     Python exception is raised or translated (“casted”)
>>
>>   *
>>
>>     If already filed, an URL to the bug report in Launchpad
>>
>>
>>   Tests related to the Fixed IPs Compute API extension (Miguel)
>>
>>
>> 1)
>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_list_*
>> *fixed_ip_details
>>
>>   *
>>
>>     Trace: http://paste.openstack.org/**show/38363/<http://paste.openstack.org/show/38363/>
>>
>>   *
>>
>>     curl-H "X-Auth-Token:$TOKEN" -X GET
>>     http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3 <
>> http://10.0.0.3>
>>
>>   *
>>
>>     API: http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips>
>>
>>
>>   *
>>
>>     Explanation: Call to
>>     nova.api.openstack.compute.**contrib.fixed_ips.**
>> FixedIPController::show()
>>     This file doesn’t use the Quantum API nor the Nova-Network API. It
>>     interacts directly with the DB, which is bad.
>>
>>   *
>>
>>     A possible fix would be to :
>>
>>      1.
>>
>>
>>         Change
>>         nova.api.openstack.compute.**contrib.fixed_ips.**
>> FixedIPController::show()
>>         to use either nova.network.api.API::get_**fixed_ip() (for Nova
>>         Network) or nova.network.quantumv2.api.**API:get_fixed_ip() (for
>>         Nova Network)
>>
>>      2.
>>
>>
>>         Implement nova.network.quantumv2.api.**API:get_fixed_ip() which
>>         currently raises a NotImplementedError exception
>>
>> 2)
>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_**
>> reserve
>>
>>   *
>>
>>     Trace: http://paste.openstack.org/**show/38372/<http://paste.openstack.org/show/38372/>
>>
>>   *
>>
>>     curl-H "X-Auth-Token:$TOKEN" -X POST
>>     http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3/action
>>     <http://10.0.0.3/action>
>>
>>   *
>>
>>     API: http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips>
>>
>>
>>   *
>>
>>     Possible Fix: Should call
>>     nova.api.openstack.compute.**contrib.fixed_ips.**
>> FixedIPController::_set_**reserved()
>>     (once the result of db.fixed_ip_get_by_address() is made through the
>>     API). Here the problem is that neither Nova-Network nor Quatum API
>>     implement an equivalent of db.fixed_ip_update()
>>
>> 3)
>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_**
>> unreserve
>>
>>   *
>>
>>     Same as 2)
>>
>>
>>
>>   Tests related to Quotas Admin(Miguel)
>>
>>
>> 4)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_**
>> security_groups_exceed_limit
>>
>> 5)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_**
>> security_groups_rules_exceed_**limit
>>
>>
>>   Tests related to Floating IPs
>>
>>
>> 6)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>> FloatingIPsTestJSON.test_**associate_ip_to_server_**
>> without_passing_floating_ip
>>
>>   *
>>
>>     Trace Tempest http://paste.openstack.org/**show/38431/<http://paste.openstack.org/show/38431/>
>>
>>   *
>>
>>     Bug : https://bugs.launchpad.net/**quantum/+bug/1190242<https://bugs.launchpad.net/quantum/+bug/1190242>
>>
>>   *
>>
>>     curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>       POST http://$IP:8774/v2/$TENANT_ID/**servers/$SERVER_ID/action
>>     <http://10.1.59.157:8774/v2/**de6c5fbc55a34cbcaa3d79eb6b21a7**
>> 84/servers/0b2ad3b6-c14a-4d89-**b2a0-c015f0a88a1f/action<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action>
>> >-d
>>     ‘{"addFloatingIp": {"address": ""}}’
>>
>>
>>   *
>>
>>     Notice that the address of the FloatingIP is empty.
>>
>>   *
>>
>>     API : http://api.openstack.org/api-**ref.html#ext-floating-ips<http://api.openstack.org/api-ref.html#ext-floating-ips>
>>
>>
>>   *
>>
>>     Explanation : Mismatch in the 2 API. If there is only one IP in the
>>     pool, Quantum allows the floating IP to be empty and returns the one
>>     and only IP in the pool. Nova-network doesn’t allow this and returns
>>     a 404
>>
>>   *
>>
>>     Review https://review.openstack.org/#**/c/32740/<https://review.openstack.org/#/c/32740/>
>>
>>
>> 7)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>> FloatingIPsTestJSON.test_**delete_floating_ip
>>
>>   *
>>
>>     Trace Tempest : http://paste.openstack.org/**show/38505/<http://paste.openstack.org/show/38505/>
>>
>>   *
>>
>>     Bug : https://bugs.launchpad.net/**tempest/+bug/1160309(see<https://bugs.launchpad.net/tempest/+bug/1160309(see>comment #2)
>>
>>
>>   *
>>
>>     This is related to bug 9)
>>
>>
>> 8)tempest.api.compute.**floating_ips.test_floating_**ips_actions:**
>> FloatingIPsTestJSON.test_**delete_nonexistant_floating_ip
>>
>>
>>   *
>>
>>     Related to 9)
>>
>>
>> 9)tempest.api.compute.**floating_ips.test_list_**floating_ips:**
>> FloatingIPDetailsTestJSON.**test_get_nonexistant_floating_**ip_details
>>
>>   *
>>
>>     Bug :https://bugs.launchpad.net/**tempest/+bug/1160309<https://bugs.launchpad.net/tempest/+bug/1160309>
>>
>>   *
>>
>>     Trace tempest:http://paste.**openstack.org/show/38430/<http://paste.openstack.org/show/38430/>
>>
>>   *
>>
>>     Trace nova: http://paste.openstack.org/**show/38433/<http://paste.openstack.org/show/38433/>
>>
>>
>>   *
>>
>>     Curl:curl -H "X-Auth-Token:$TOKEN" -X GET
>>     http://$IP:8774/v2/$PROJECT_**ID/os-floating-ips/**99987878787878
>>
>> Proposed Fix: https://review.openstack.org/**33024<https://review.openstack.org/33024>
>>
>>
>>   Tests Related to Security Groups (Jordan)
>>
>>
>> 10)tempest.api.compute.**security_groups.test_security_**group_rules:**
>> SecurityGroupRulesTestJSON.**test_security_group_rules_**
>> create_with_invalid_id
>>
>>   *
>>
>>     TRACE: http://paste.openstack.org/**show/38373/<http://paste.openstack.org/show/38373/>
>>
>>   *
>>
>>     curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>>
>>       POST http://$IP:8774/v2/$TENANT_ID/**os-security-group-rules -d
>>     ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp",
>>     "to_port": 22, "parent_group_id": "9991393497170", "cidr": null,
>>     "group_id": null}}’
>>
>>   *
>>
>>     Explanation: Notice that the parent_goup_id is  a numerical ID and
>>     not a UUID. Quantum has an additional check to validate that the ID
>>     is an UUID (see
>>     nova/network/security_group/**quantum_driver.py::validate_**id())
>>
>>   *
>>
>>     API: http://api.openstack.org/api-**ref.html#ext-os-security-**groups<http://api.openstack.org/api-ref.html#ext-os-security-groups>
>>
>>   *
>>
>>     Bug: https://bugs.launchpad.net/**tempest/+bug/1182384<https://bugs.launchpad.net/tempest/+bug/1182384>
>>
>>
>>   *
>>
>>     Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>
>>
>> 11)tempest.api.compute.**security_groups.test_security_**group_rules:**
>> SecurityGroupRulesTestJSON.**test_security_group_rules_**
>> delete_with_invalid_id
>>
>>   *
>>
>>     http://paste.openstack.org/**show/38424/<http://paste.openstack.org/show/38424/>
>>
>>
>>   *
>>
>>     curl  -H "X-Auth-Token:$TOKEN" -X  DELETE
>>     http://$IP:8774/v2$TENANT_ID/**os-security-group-rules/**
>> 9991407551273
>>
>>   *
>>
>>     Explanation : Same bug as 10
>>
>>   *
>>
>>     API: http://api.openstack.org/api-**ref.html#ext-os-security-**groups<http://api.openstack.org/api-ref.html#ext-os-security-groups>
>>
>>
>>   *
>>
>>     Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>
>> 12)tempest.api.compute.**security_groups.test_security_**
>> groups:SecurityGroupsTestJSON.**test_delete_nonexistant_**security_group
>>
>>   *
>>
>>     Same as bug 10
>>
>>
>>   *
>>
>>     Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>
>>
>> 13)tempest.api.compute.**security_groups.test_security_**
>> groups:SecurityGroupsTestJSON.**test_security_group_get_**
>> nonexistant_group
>>
>>   *
>>
>>     Same as bug 10
>>
>>
>>   *
>>
>>     Review: https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/>
>>
>>
>> 14)tempest.api.compute.**security_groups.test_security_**
>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>> with_duplicate_name
>>
>> Security Group with duplicate name should not be created, but two groups
>> with the same name can be created in quantum. We have here the same
>> problem as in 15 and in 16. With Quantum, there is no validation that a
>>   group with given name exists already or if the given SG name is empty
>> or is composed of white spaces or is more than 255 chars.
>>
>>
>> In the description of bug
>> https://bugs.launchpad.net/**nova/+bug/1161411this<https://bugs.launchpad.net/nova/+bug/1161411this>issue is generally
>>
>> discussed. SecurityGroup API  are based on the ID and not the names,
>> except for adding an instance to a security group.  In order to solve
>> the last problem the bug https://bugs.launchpad.net/**
>> nova/+bug/1161473was <https://bugs.launchpad.net/nova/+bug/1161473was>
>> added.
>>
>>
>> The major question is if these 3 tests (14, 15, 16) : does the name of a
>> security group is really that important? If so, we must add some
>> validation methods. If not the test suit concerning this part must be
>> disable in tempest. What do you think Miguel?
>>
>>
>>
>> 15)tempest.api.compute.**security_groups.test_security_**
>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>> with_invalid_group_description
>>
>> 16)tempest.api.compute.**security_groups.test_security_**
>> groups:SecurityGroupsTestJSON.**test_security_group_create_**
>> with_invalid_group_name
>>
>> BUG:https://bugs.launchpad.**net/nova/+bug/1161411+<https://bugs.launchpad.net/nova/+bug/1161411+>
>> https://bugs.launchpad.net/**nova/+bug/1161473<https://bugs.launchpad.net/nova/+bug/1161473>
>>
>> Traceback(tempest) :http://paste.openstack.org/**show/38423/<http://paste.openstack.org/show/38423/>
>>
>> The Security Group should not be created with group name an empty string
>> or with white spaces/chars more than 255
>>
>> CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN"
>> -X POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d
>>   '{"security_group": {"name": " ", "description":
>> "description-1554950088"}}'
>>
>> curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
>> POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d
>>   '{"security_group": {"name": "  ", "description":
>> "description-1554950088"}}'
>>
>>
>> 17)tempest.api.compute.**security_groups.test_security_**
>> groups:SecurityGroupsTestJSON.**test_server_security_groups
>>
>>   *
>>
>>     Tempest Trace : http://paste.openstack.org/**show/38427/<http://paste.openstack.org/show/38427/>
>>
>>   *
>>
>>     Fixed by https://review.openstack.org/#**/c/32288/(merged<https://review.openstack.org/#/c/32288/(merged>on June,12th)
>>
>>
>>
>>
>>   Tests related to servers(Ala)
>>
>>
>> 18)tempest.api.compute.**servers.test_list_server_**filters:**
>> ListServerFiltersTestXML.test_**list_servers_filtered_by_ip_**regex
>>
>>   *
>>
>>     BUG: https://bugs.launchpad.net/**quantum/+bug/1182883<https://bugs.launchpad.net/quantum/+bug/1182883>
>>
>>   *
>>
>>     BP: https://blueprints.launchpad.**net/quantum/+spec/like-op-list<https://blueprints.launchpad.net/quantum/+spec/like-op-list>
>>
>>
>>   *
>>
>>     CURL : GET http://$IP:8774/v2/$PROJECT_**ID/servers?ip=10.
>>
>>   *
>>
>>     Explanation: The  regex search is not supported by Quantum. Thus
>>     Quantum returns a 404 Not Found (0 server match) where Tempest
>>     expects one server to be found.
>>
>>   *
>>
>>     Possible Fix : For "search port by IP with regex" feature, I think
>>     the best place to hack it would be in file
>>     quantum/db/db_base_plugin_v2.**py::_get_ports_query()
>>
>>
>> 19)tempest.api.compute.**servers.test_servers_negative:**
>> ServersNegativeTest.test_**create_with_nonexistent_**security_group
>>
>> FIXED : https://review.openstack.org/#**/c/30271/<https://review.openstack.org/#/c/30271/>
>>
>>
>> 20)tempest.api.compute.**servers.test_virtual_**interfaces:**
>> VirtualInterfacesTestXML.test_**list_virtual_interfaces
>>
>>   *
>>
>>     TRACE (NOVA): http://paste.openstack.org/**show/38371/<http://paste.openstack.org/show/38371/>
>>
>>   *
>>
>>     BUG: https://bugs.launchpad.net/**tempest/+bug/1183436<https://bugs.launchpad.net/tempest/+bug/1183436>
>>
>>
>>   *
>>
>>     CURL: GET
>>     http://$IP:8774/v2/$PROJECT_**ID/servers/$SERVER/os-virtual-**
>> interfaces
>>
>>   *
>>
>>     Explanation: This HTTP request calls the Quantum API
>>     (nova/nova/network/quantumv2/**api.py) and specifically the
>>     get_vifs_by_* methods which are not implemented (raise
>>     NotImplementedError())
>>
>>   *
>>
>>     Possible Fix:
>>
>>
>>       o
>>
>>         skip this test if Quantum is enabled as set in Tempest
>>         configuration. Or
>>
>>       o
>>
>>         Implement the get_vifs_by_* methods
>>
>> PATCH: (still not approved) https://review.openstack.org/#**/c/31755/<https://review.openstack.org/#/c/31755/>
>>
>>
>> On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <sean at dague.net
>> <mailto:sean at dague.net>> wrote:
>>
>>     On 06/18/2013 03:32 PM, Monty Taylor wrote:
>>
>>
>>
>>         On 06/18/2013 03:14 PM, David Ripton wrote:
>>
>>             On 06/18/2013 12:43 PM, Martina Kollarova wrote:
>>
>>                 Jenkins keeps running all the tests, even if the basic
>>                 pep8 test fails,
>>                 and runs all of the (very slow) Tempest Quantum tests,
>>                 even though
>>                 almost all of them are failing.
>>
>>                 I propose that it should fail and stop all of the other
>>                 tests once there
>>                 is a failure in a voting test. For non-voting tests, it
>>                 should stop only
>>                 itself, not the others.
>>
>>                 This would decrease the feedback loop and we wouldn't
>>                 have to wait for
>>                 the non-voting Quantum tests to see that they failed as
>>                 always.
>>
>>
>>             -1
>>
>>             In addition to the other objections, we currently get a lot
>>             of false
>>             positives (fail, retry, fail, retry, succeed), and it would
>>             be harder to
>>             debug these problems if the output was truncated differently
>>             each time.
>>
>>             Is anyone working on fixing the perma-failing Quantum test?
>>               When the
>>             Postgres test was perma-failing, one of the infrastructure
>>             folks gave us
>>             an ultimatum that if nobody fixed it soon, it would be
>>             disabled.  (Happy
>>             ending: Mauro fixed it before it got disabled.)
>>
>>
>>         That was brought up a little while ago, but we had already spent
>>         so much
>>         effort to get it working in the first place, none of us had the
>>         heart to
>>         put in such an ultimatum. But seriously- it might be time for an
>>         all-hands-on-deck dogpile to figure out what's up the the
>>         quantum gate.
>>
>>
>>     The biggest cause of the Quantum vs. Full Tempest runs is that a lot
>>     of the network api's in nova currently don't do translation of
>>     errors. So under nova-network certain data validation and error
>>     codes are returned, when quantum is used others are returned.
>>
>>     This is a nova-api, so it needs to be consistent regardless of
>>     backend (i.e. we don't return different API responses on different
>>     databases).
>>
>>     Issues like this one -
>>     https://bugs.launchpad.net/__**nova/+bug/1160309<https://bugs.launchpad.net/__nova/+bug/1160309>
>>
>>     <https://bugs.launchpad.net/**nova/+bug/1160309<https://bugs.launchpad.net/nova/+bug/1160309>
>> >
>>
>>     Jordan Pittier has been working on some of these issues (he's the
>>     only one I've seen working them from a Tempest / nova side), and got
>>     to the crux of the problem. It could use more hands though to
>>     organize the rest of those and get them banged out.
>>
>>     I'm sure there are other issues once we get past this class. But
>>     that would go a long way.
>>
>>              -Sean
>>
>>     --
>>     Sean Dague
>>     http://dague.net
>>
>>     ______________________________**___________________
>>     OpenStack-dev mailing list
>>     OpenStack-dev at lists.openstack.**__org
>>     <mailto:OpenStack-dev at lists.**openstack.org<OpenStack-dev at lists.openstack.org>
>> >
>>     http://lists.openstack.org/__**cgi-bin/mailman/listinfo/__**
>> openstack-dev<http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev><
>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>> >
>>
>>
>>
>>
>>
>> ______________________________**_________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>
> --
> Sean Dague
> http://dague.net
>
> ______________________________**_________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130619/a230877f/attachment-0001.html>