Open Stack

I've seen issues where the quota_use table in the nova database gets
out of sync with the resources actually used.

try this query and see if the fixed_ips matches what is in use:

mysql nova -e 'select * from quota_usages where
project_id="60d776fe573f44a4810cb294b95e09d6" and
resource="fixed_ips";

if it doesn't you'll need to manually update it and check the other
resources for this tenant (if one is wrong usually they all are)

I was just ranting privately after running into this issue today (not
for the first time) about how having this in its own table is bad for
these reasons & that I'd rather have the right answer slowly than the
wrong answer quickly. Though this may not be your problem & I haven't
yet reproduced the issue to see if I can file a proper bug (a previous
bug related to this was fixed a long time ago)

-Jon

On Wed, Dec 11, 2013 at 2:11 PM, JR <botemout at gmail.com> wrote:
> Hi all,
>
> I'm having an odd issue where I can't spin up new VMs; the error is below.
> I've confirmed that the tenant starting the instance has sufficient
> capacity.
> I have another tenant who is running a similar number of VMs and it's not
> having the problem.
>
> This is ubuntu 12.04 running grizzly.
>
> Any thoughts?  I assume that something in the DB is wedged ...
>
> Thanks
> JR
>
> root at nebula01:/mnt/gdm_source# su - dd_test -c 'nova quota-show'
> +-----------------------------+--------+
> | Property                    | Value  |
> +-----------------------------+--------+
> | metadata_items              | 128    |
> | injected_file_content_bytes | 10240  |
> | ram                         | 256000 |
> | floating_ips                | 128    |
> | key_pairs                   | 100    |
> | instances                   | 128    |
> | security_group_rules        | 20     |
> | injected_files              | 5      |
> | cores                       | 128    |
> | fixed_ips                   | 128    |
> | injected_file_path_bytes    | 255    |
> | security_groups             | 50     |
> +-----------------------------+--------+
>
> root at nebula01:/mnt/gdm_source# su - dd_test -c 'nova show gdm-worker17-t'
> +-----------------------------+---------------------------------------------------------------------------------+
> | Property                    | Value
> |
> +-----------------------------+---------------------------------------------------------------------------------+
> | status                      | ERROR
> |
> | updated                     | 2013-12-11T19:05:15Z
> |
> | OS-EXT-STS:task_state       | None
> |
> | key_name                    | dd_test
> |
> | image                       | gdm-worker
> (fe0147b8-d355-4d7e-826f-195ba97d7594)                               |
> | hostId                      |
> 5f1a4bd11ee027f10f8fc37b0e2ce091904b94befdf5abeb196c7f67
> |
> | OS-EXT-STS:vm_state         | error
> |
> | public network              |
> |
> | flavor                      | i3.small_d10g
> (7d1037f1-dfd0-4a0e-8da7-097258c9934d)                            |
> | id                          | d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85
> |
> | security_groups             | [{u'name': u'dd_test'}]
> |
> | user_id                     | dcc3374a58984a92981d09f27c76a869
> |
> | name                        | gdm-worker17-t
> |
> | created                     | 2013-12-11T19:05:10Z
> |
> | fault                       | {u'message': u'NoValidHost', u'code': 500,
> u'created': u'2013-12-11T19:05:15Z'} |
> | OS-DCF:diskConfig           | MANUAL
> |
> | metadata                    | {}
> |
> | accessIPv4                  |
> |
> | accessIPv6                  |
> |
> | tenant_id                   | 60d776fe573f44a4810cb294b95e09d6
> |
> | OS-EXT-STS:power_state      | 0
> |
> | OS-EXT-AZ:availability_zone | nova
> |
> | config_drive                |
> |
> +-----------------------------+---------------------------------------------------------------------------------+
> root at nebula01:/mnt/gdm_source# keystone tenant-list|grep 60d
> | 60d776fe573f44a4810cb294b95e09d6 | dd_test |   True  |
>
>
> Error:
>
> 2013-12-11 12:05:15.153 ERROR nova.compute.manager
> [req-2363d68e-a947-41af-9359-2a6e6720a0ee dcc3374a58984a92981d09f27c76a869
> 60d776fe573f44a4810cb294b95e09d6] [instance:
> d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85] Error: ['Traceback (most recent call
> last):\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 847, in
> _run_instance\n    requested_networks, macs, security_groups)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1091, in
> _allocate_network\n    instance=instance)\n', '  File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
> self.gen.next()\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1087, in
> _allocate_network\n    security_groups=security_groups)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 102, in
> wrapped\n    return func(self, context, *args, **kwargs)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 47, in
> wrapper\n    res = f(self, context, *args, **kwargs)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 283, in
> allocate_for_instance\n    nw_info =
> self.network_rpcapi.allocate_for_instance(context, **args)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 171, in
> allocate_for_instance\n    topic=topic, version=\'1.9\')\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line
> 80, in call\n    return rpc.call(context, self._get_topic(topic), msg,
> timeout)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py",
> line 140, in call\n    return _get_impl().call(CONF, context, topic, msg,
> timeout)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py",
> line 798, in call\n    rpc_amqp.get_connection_pool(conf, Connection))\n', '
> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
> line 612, in call\n    rv = list(rv)\n', '  File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
> 561, in __iter__\n    raise result\n', 'FixedIpLimitExceeded_Remote: Maximum
> number of fixed ips exceeded\nTraceback (most recent call last):\n\n  File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
> 430, in _process_data\n    rval = self.proxy.dispatch(ctxt, version, method,
> **args)\n\n  File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
> line 133, in dispatch\n    return getattr(proxyobj, method)(ctxt,
> **kwargs)\n\n  File
> "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 116,
> in allocate_for_instance\n    **kwargs)\n\n  File
> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 513, in
> allocate_for_instance\n    requested_networks=requested_networks)\n\n  File
> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 217, in
> _allocate_fixed_ips\n    vpn=vpn, address=address)\n\n  File
> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 836, in
> allocate_fixed_ip\n    raise
> exception.FixedIpLimitExceeded()\n\nFixedIpLimitExceeded: Maximum number of
> fixed ips exceeded\n\n']
>
> --
> Your electronic communications are being monitored; strong encryption is an
> answer. My public key
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>