[Openstack] [Netstack] Fwd: [devstack/quantum] Configuration issue
Gary Kotton
gkotton at redhat.com
Thu May 17 07:24:34 UTC 2012
Hi,
Running devstack this morning the following happens:
+ /opt/stack/nova/bin/nova-manage network create private 10.0.0.0/24 1 256
2012-05-17 03:22:05 WARNING nova.network.quantum.manager
[req-5458d926-54d1-4877-9c46-d04035742eaf None None] QuantumManager does
not use 'bridge' parameter.
2012-05-17 03:22:05 WARNING nova.network.quantum.manager
[req-5458d926-54d1-4877-9c46-d04035742eaf None None] QuantumManager does
not use 'bridge_interface' parameter.
2012-05-17 03:22:05 DEBUG nova.network.quantum.quantum_connection
[req-5458d926-54d1-4877-9c46-d04035742eaf None None] Quantum Client
Request: POST /v1.1/tenants/default/networks.json from (pid=3826)
do_request /opt/stack/nova/nova/network/quantum/client.py:178
2012-05-17 03:22:05 DEBUG nova.network.quantum.quantum_connection
[req-5458d926-54d1-4877-9c46-d04035742eaf None None] Quantum Client
Reply (code = 400) :
{"Error": {"message": "The server could not comply with the request
since it is either malformed or otherwise incorrect.", "code": 400}}
from (pid=3826) do_request
/opt/stack/nova/nova/network/quantum/client.py:189
Command failed, please check log for more info
2012-05-17 03:22:05 CRITICAL nova
[req-5458d926-54d1-4877-9c46-d04035742eaf None None] Server 400 error:
{"Error": {"message": "The server could not comply with the request
since it is either malformed or otherwise incorrect.", "code": 400}}
2012-05-17 03:22:05 TRACE nova Traceback (most recent call last):
2012-05-17 03:22:05 TRACE nova File "/opt/stack/nova/bin/nova-manage",
line 1755, in <module>
2012-05-17 03:22:05 TRACE nova main()
2012-05-17 03:22:05 TRACE nova File "/opt/stack/nova/bin/nova-manage",
line 1742, in main
2012-05-17 03:22:05 TRACE nova fn(*fn_args, **fn_kwargs)
2012-05-17 03:22:05 TRACE nova File "/opt/stack/nova/bin/nova-manage",
line 817, in create
2012-05-17 03:22:05 TRACE nova fixed_cidr=fixed_cidr)
2012-05-17 03:22:05 TRACE nova File
"/opt/stack/nova/nova/network/quantum/manager.py", line 232, in
create_networks
2012-05-17 03:22:05 TRACE nova nova_id=nova_id)
2012-05-17 03:22:05 TRACE nova File
"/opt/stack/nova/nova/network/quantum/quantum_connection.py", line 67,
in create_network
2012-05-17 03:22:05 TRACE nova resdict =
self.client.create_network(data, tenant=tenant_id)
2012-05-17 03:22:05 TRACE nova File
"/opt/stack/nova/nova/network/quantum/client.py", line 82, in with_params
2012-05-17 03:22:05 TRACE nova ret = self.func(instance, *args,
**kwargs)
2012-05-17 03:22:05 TRACE nova File
"/opt/stack/nova/nova/network/quantum/client.py", line 250, in
create_network
2012-05-17 03:22:05 TRACE nova return self.do_request("POST",
self.networks_path, body=body)
2012-05-17 03:22:05 TRACE nova File
"/opt/stack/nova/nova/network/quantum/client.py", line 204, in do_request
2012-05-17 03:22:05 TRACE nova % locals())
2012-05-17 03:22:05 TRACE nova QuantumServerException: Server 400 error:
{"Error": {"message": "The server could not comply with the request
since it is either malformed or otherwise incorrect.", "code": 400}}
2012-05-17 03:22:05 TRACE nova
++ failed
++ local r=1
++ set +o xtrace
Thanks
Gary
On 05/17/2012 12:07 AM, Dan Wendlandt wrote:
>
>
> On Wed, May 16, 2012 at 2:01 PM, Sumit Naiksatam (snaiksat)
> <snaiksat at cisco.com <mailto:snaiksat at cisco.com>> wrote:
>
> The interesting data point here is that this only happens to the
> default network created when the installation is done via
> DevStack. For all network created subsequently I am not seeing
> this issue. I earlier thought that this might have something to do
> with the exercises script getting executed and possibly leaving a
> residue. However, in spite of not running the exercises script I
> am still seeing this issue. It seems that for some reason, the
> initialize_gateway() was getting called only when a new VM was
> being created. But here it seems to be getting called before that
> (thus leading to the creation of the bridge and the gateway
> devices), not sure why. I am just trying to figure out how I can
> test the fix with respect to the devstack setup (since this
> happens only on installation)…
>
>
> Your explanation makes sense. I think there was a recent comment to
> automatically initialize all networks when nova-network is started, in
> case nova-network was started on a new host (e.g., in an HA
> configuration) or after a host reboot. Since devstack creates the
> network using nova-manage before starting nova-network, this would
> trigger the behavior your describe. For more info, see:
> https://github.com/openstack/nova/commit/30c8293ea7d4c9464e0139fb488e202fd0947f52
>
> I suspect you should hit this every time you re-run stack.sh, no?
>
>
>
> *From:*Dan Wendlandt [mailto:dan at nicira.com <mailto:dan at nicira.com>]
> *Sent:* Wednesday, May 16, 2012 7:55 AM
> *To:* gkotton at redhat.com <mailto:gkotton at redhat.com>
> *Cc:* Sumit Naiksatam (snaiksat); netstack at lists.launchpad.net
> <mailto:netstack at lists.launchpad.net>
> *Subject:* Re: [Netstack] Fwd: [Openstack] [devstack/quantum]
> Configuration issue
>
> On Wed, May 16, 2012 at 1:43 AM, Gary Kotton <gkotton at redhat.com
> <mailto:gkotton at redhat.com>> wrote:
>
> Thanks!
> As far as I recall, and I may certain be wrong here, but in the
> past I did not see the GW created until the first VM was deployed.
> Now I am seeing the GW created when the various services are started.
>
> I have noticed this as well. I'm not aware of any changes to
> QuantumManager that had this affect, though its possible someone
> pushed a change that I didn't notice. I primarily use the OVS
> interface-driver and didn't notice any issues, so I suspect Sumit
> is correct that we can at least work around this with a change to
> the LB interface-driver (even though the change that caused the
> issue was likely somewhere else).
>
> This is a good example of why we need Quantum integrated into
> devstack commit-gating, as it will catch this type of complex
> integration issues. Anyone have cycles to help push on this? I'm
> unlikely to make much progress on it in the next week or two due
> to the F-1 release.
>
> Dan
>
> Thanks
> Gary
>
>
>
> On 05/16/2012 10:54 AM, Sumit Naiksatam (snaiksat) wrote:
>
> This is probably not a devstack issue. The problem is with the
> LB gateway driver wherein the IP address is trying to be set
> on a bridge device which already has an IP address. The check
> for an existing bridge is being performed, but the IP address
> is being set outside that check. Ideally, this code should not
> have been invoked if the gateway was already set. Something
> seems to have changed in the QuantumManager as result of which
> this code is being invoked again. At any rate, I will fix the
> LB gateway driver, and we will not see this.
>
> Thanks,
>
> ~Sumit.
>
> *From:*netstack-bounces+snaiksat=cisco.com at lists.launchpad.net
> <mailto:netstack-bounces+snaiksat=cisco.com at lists.launchpad.net>
> [mailto:netstack-bounces+snaiksat=cisco.com at lists.launchpad.net]
> *On Behalf Of *Gary Kotton
> *Sent:* Monday, May 14, 2012 11:19 PM
> *To:* <netstack at lists.launchpad.net>
> <mailto:netstack at lists.launchpad.net>
> *Subject:* [Netstack] Fwd: [Openstack] [devstack/quantum]
> Configuration issue
>
>
>
> -------- Original Message --------
>
> *Subject: *
>
>
>
> [Openstack] [devstack/quantum] Configuration issue
>
> *Date: *
>
>
>
> Tue, 15 May 2012 09:00:52 +0300
>
> *From: *
>
>
>
> Gary Kotton <gkotton at redhat.com> <mailto:gkotton at redhat.com>
>
> *Reply-To: *
>
>
>
> gkotton at redhat.com <mailto:gkotton at redhat.com>
>
> *Organization: *
>
>
>
> Red Hat
>
> *To: *
>
>
>
> openstack at lists.launchpad.net
> <mailto:openstack at lists.launchpad.net>
>
> Hi,
>
> This morning I encountered a problem (which did not happen a few days
>
> ago :)). When devstack is launched, with quantum configured, the gateway
>
> and bridge devices are created. This causes problems with quantum.
>
>
>
> For example when devstack is up and running prior to deploying an
>
> instance we have:
>
>
>
> brq744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55
>
> inet addr:10.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.0
>
> UP BROADCAST MULTICAST MTU:1500 Metric:1
>
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>
> collisions:0 txqueuelen:0
>
> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
>
>
>
> gw-744ec2f4-c0 Link encap:Ethernet HWaddr fa:16:3e:03:a6:55
>
> inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
>
> UP BROADCAST MULTICAST MTU:1500 Metric:1
>
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>
> collisions:0 txqueuelen:500
>
> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
>
>
>
> When an instance is deployed the following happens:
>
>
>
> 2012-05-15 01:59:18 DEBUG nova.utils
>
> [req-4d50ed10-46e1-406c-9074-dc45da860365
>
> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e]
>
> Running cmd (subprocess): sudo /usr/local/bin/nova-rootwrap ip address
>
> add10.0.0.1/24 <http://10.0.0.1/24> dev brq744ec2f4-c0 from (pid=4234) execute
>
> /opt/stack/nova/nova/utils.py:178
>
> 2012-05-15 01:59:18 DEBUG nova.utils
>
> [req-4d50ed10-46e1-406c-9074-dc45da860365
>
> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e]
>
> Result was 254 from (pid=4234) execute /opt/stack/nova/nova/utils.py:194
>
> 2012-05-15 01:59:18 ERROR nova.rpc.amqp
>
> [req-4d50ed10-46e1-406c-9074-dc45da860365
>
> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e]
>
> Exception during message handling
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Traceback (most recent call last):
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp rval =
>
> node_func(context=ctxt, **node_args)
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in
>
> allocate_for_instance
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp network, vif_rec,
>
> network['net_tenant_id'])
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/utils.py", line 880, in inner
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in enable_dhcp
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp
>
> self.l3driver.initialize_gateway(network_ref)
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/network/l3.py", line 98, in initialize_gateway
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp
>
> gateway=(network_ref['gateway'] is not None))
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/network/linux_net.py", line 900, in plug
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp return
>
> _get_interface_driver().plug(network, mac_address, gateway)
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp run_as_root=True)
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp File
>
> "/opt/stack/nova/nova/utils.py", line 201, in execute
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp cmd=' '.join(cmd))
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp ProcessExecutionError:
>
> Unexpected error while running command.
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Command: sudo
>
> /usr/local/bin/nova-rootwrap ip address add10.0.0.1/24 <http://10.0.0.1/24> dev brq744ec2f4-c0
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Exit code: 254
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stdout: ''
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp Stderr: 'RTNETLINK answers: File
>
> exists\n'
>
> 2012-05-15 01:59:18 TRACE nova.rpc.amqp
>
> 2012-05-15 01:59:18 ERROR nova.rpc.common
>
> [req-4d50ed10-46e1-406c-9074-dc45da860365
>
> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e]
>
> Returning exception Unexpected error while running command.
>
> Command: sudo /usr/local/bin/nova-rootwrap ip address add10.0.0.1/24 <http://10.0.0.1/24>
>
> dev brq744ec2f4-c0
>
> Exit code: 254
>
> Stdout: ''
>
> Stderr: 'RTNETLINK answers: File exists\n' to caller
>
> 2012-05-15 01:59:18 ERROR nova.rpc.common
>
> [req-4d50ed10-46e1-406c-9074-dc45da860365
>
> df07eec326434b25800f3ebc17202fb3 2cafe0be4d7740098a89be39ffd1b72e]
>
> ['Traceback (most recent call last):\n', ' File
>
> "/opt/stack/nova/nova/rpc/amqp.py", line 263, in _process_data\n rval
>
> = node_func(context=ctxt, **node_args)\n', ' File
>
> "/opt/stack/nova/nova/network/quantum/manager.py", line 390, in
>
> allocate_for_instance\n network, vif_rec,
>
> network[\'net_tenant_id\'])\n', ' File "/opt/stack/nova/nova/utils.py",
>
> line 880, in inner\n retval = f(*args, **kwargs)\n', ' File
>
> "/opt/stack/nova/nova/network/quantum/manager.py", line 501, in
>
> enable_dhcp\n self.l3driver.initialize_gateway(network_ref)\n', '
>
> File "/opt/stack/nova/nova/network/l3.py", line 98, in
>
> initialize_gateway\n gateway=(network_ref[\'gateway\'] is not
>
> None))\n', ' File "/opt/stack/nova/nova/network/linux_net.py", line
>
> 900, in plug\n return _get_interface_driver().plug(network,
>
> mac_address, gateway)\n', ' File
>
> "/opt/stack/nova/nova/network/linux_net.py", line 1160, in plug\n
>
> run_as_root=True)\n', ' File "/opt/stack/nova/nova/utils.py", line 201,
>
> in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError:
>
> Unexpected error while running command.\nCommand: sudo
>
> /usr/local/bin/nova-rootwrap ip address add10.0.0.1/24 <http://10.0.0.1/24> dev
>
> brq744ec2f4-c0\nExit code: 254\nStdout: ''\nStderr: 'RTNETLINK answers:
>
> File exists\\n'\n"]
>
>
>
>
>
> My guess is that this can be solved with the nova configuration. Sadly I
>
> am not too familiar with these. Any help would be great.
>
> Thanks
>
> Gary
>
>
>
> _______________________________________________
>
> Mailing list:https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
>
> Post to :openstack at lists.launchpad.net <mailto:openstack at lists.launchpad.net>
>
> Unsubscribe :https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
>
> More help :https://help.launchpad.net/ListHelp
>
>
> --
> Mailing list: https://launchpad.net/~netstack
> <https://launchpad.net/%7Enetstack>
> Post to : netstack at lists.launchpad.net
> <mailto:netstack at lists.launchpad.net>
> Unsubscribe : https://launchpad.net/~netstack
> <https://launchpad.net/%7Enetstack>
> More help : https://help.launchpad.net/ListHelp
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
>
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20120517/c49818da/attachment.html>
More information about the Openstack
mailing list