[Openstack] out of fixed-ips!
Pranesh Pandurangan
praneshp at yahoo-inc.com
Wed Dec 11 23:05:55 UTC 2013
To those who might be interested, some of us are working on a framework
called
Entropy that would enable openstack users to plugin audit scripts that
detect such issues,
and repair scripts that would provide a fix if a known issue is seen. For
example, everytime
your audit script sees an inconsistency between nova db and dnsdb, it
would raise an alert, and
the repair script that is listening for the alert would take some action
on the nova db.
We are just getting started on https://github.com/stackforge/entropy/ ,
you're welcome to contribute.
Wiki:http://wiki.openstack.org/Entropy
Pranesh
On 12/11/13 11:51 AM, "JR" <botemout at gmail.com> wrote:
>Jon,
>
>The value in quota_usages was wrong. Fixed. Thanks for the help.
>
>JR
>
>On 12/11/2013 2:36 PM, Jonathan Proulx wrote:
>> I've seen issues where the quota_use table in the nova database gets
>> out of sync with the resources actually used.
>>
>> try this query and see if the fixed_ips matches what is in use:
>>
>> mysql nova -e 'select * from quota_usages where
>> project_id="60d776fe573f44a4810cb294b95e09d6" and
>> resource="fixed_ips";
>>
>> if it doesn't you'll need to manually update it and check the other
>> resources for this tenant (if one is wrong usually they all are)
>>
>> I was just ranting privately after running into this issue today (not
>> for the first time) about how having this in its own table is bad for
>> these reasons & that I'd rather have the right answer slowly than the
>> wrong answer quickly. Though this may not be your problem & I haven't
>> yet reproduced the issue to see if I can file a proper bug (a previous
>> bug related to this was fixed a long time ago)
>>
>> -Jon
>>
>> On Wed, Dec 11, 2013 at 2:11 PM, JR <botemout at gmail.com> wrote:
>>> Hi all,
>>>
>>> I'm having an odd issue where I can't spin up new VMs; the error is
>>>below.
>>> I've confirmed that the tenant starting the instance has sufficient
>>> capacity.
>>> I have another tenant who is running a similar number of VMs and it's
>>>not
>>> having the problem.
>>>
>>> This is ubuntu 12.04 running grizzly.
>>>
>>> Any thoughts? I assume that something in the DB is wedged ...
>>>
>>> Thanks
>>> JR
>>>
>>> root at nebula01:/mnt/gdm_source# su - dd_test -c 'nova quota-show'
>>> +-----------------------------+--------+
>>> | Property | Value |
>>> +-----------------------------+--------+
>>> | metadata_items | 128 |
>>> | injected_file_content_bytes | 10240 |
>>> | ram | 256000 |
>>> | floating_ips | 128 |
>>> | key_pairs | 100 |
>>> | instances | 128 |
>>> | security_group_rules | 20 |
>>> | injected_files | 5 |
>>> | cores | 128 |
>>> | fixed_ips | 128 |
>>> | injected_file_path_bytes | 255 |
>>> | security_groups | 50 |
>>> +-----------------------------+--------+
>>>
>>> root at nebula01:/mnt/gdm_source# su - dd_test -c 'nova show
>>>gdm-worker17-t'
>>>
>>>+-----------------------------+-----------------------------------------
>>>----------------------------------------+
>>> | Property | Value
>>> |
>>>
>>>+-----------------------------+-----------------------------------------
>>>----------------------------------------+
>>> | status | ERROR
>>> |
>>> | updated | 2013-12-11T19:05:15Z
>>> |
>>> | OS-EXT-STS:task_state | None
>>> |
>>> | key_name | dd_test
>>> |
>>> | image | gdm-worker
>>> (fe0147b8-d355-4d7e-826f-195ba97d7594) |
>>> | hostId |
>>> 5f1a4bd11ee027f10f8fc37b0e2ce091904b94befdf5abeb196c7f67
>>> |
>>> | OS-EXT-STS:vm_state | error
>>> |
>>> | public network |
>>> |
>>> | flavor | i3.small_d10g
>>> (7d1037f1-dfd0-4a0e-8da7-097258c9934d) |
>>> | id | d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85
>>> |
>>> | security_groups | [{u'name': u'dd_test'}]
>>> |
>>> | user_id | dcc3374a58984a92981d09f27c76a869
>>> |
>>> | name | gdm-worker17-t
>>> |
>>> | created | 2013-12-11T19:05:10Z
>>> |
>>> | fault | {u'message': u'NoValidHost', u'code':
>>>500,
>>> u'created': u'2013-12-11T19:05:15Z'} |
>>> | OS-DCF:diskConfig | MANUAL
>>> |
>>> | metadata | {}
>>> |
>>> | accessIPv4 |
>>> |
>>> | accessIPv6 |
>>> |
>>> | tenant_id | 60d776fe573f44a4810cb294b95e09d6
>>> |
>>> | OS-EXT-STS:power_state | 0
>>> |
>>> | OS-EXT-AZ:availability_zone | nova
>>> |
>>> | config_drive |
>>> |
>>>
>>>+-----------------------------+-----------------------------------------
>>>----------------------------------------+
>>> root at nebula01:/mnt/gdm_source# keystone tenant-list|grep 60d
>>> | 60d776fe573f44a4810cb294b95e09d6 | dd_test | True |
>>>
>>>
>>> Error:
>>>
>>> 2013-12-11 12:05:15.153 ERROR nova.compute.manager
>>> [req-2363d68e-a947-41af-9359-2a6e6720a0ee
>>>dcc3374a58984a92981d09f27c76a869
>>> 60d776fe573f44a4810cb294b95e09d6] [instance:
>>> d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85] Error: ['Traceback (most recent
>>>call
>>> last):\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 847,
>>>in
>>> _run_instance\n requested_networks, macs, security_groups)\n', '
>>>File
>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1091,
>>>in
>>> _allocate_network\n instance=instance)\n', ' File
>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>> self.gen.next()\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1087,
>>>in
>>> _allocate_network\n security_groups=security_groups)\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 102, in
>>> wrapped\n return func(self, context, *args, **kwargs)\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 47, in
>>> wrapper\n res = f(self, context, *args, **kwargs)\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 283, in
>>> allocate_for_instance\n nw_info =
>>> self.network_rpcapi.allocate_for_instance(context, **args)\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 171, in
>>> allocate_for_instance\n topic=topic, version=\'1.9\')\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py",
>>>line
>>> 80, in call\n return rpc.call(context, self._get_topic(topic), msg,
>>> timeout)\n', ' File
>>>
>>>"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py"
>>>,
>>> line 140, in call\n return _get_impl().call(CONF, context, topic,
>>>msg,
>>> timeout)\n', ' File
>>>
>>>"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.p
>>>y",
>>> line 798, in call\n rpc_amqp.get_connection_pool(conf,
>>>Connection))\n', '
>>> File
>>>"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>> line 612, in call\n rv = list(rv)\n', ' File
>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>line
>>> 561, in __iter__\n raise result\n', 'FixedIpLimitExceeded_Remote:
>>>Maximum
>>> number of fixed ips exceeded\nTraceback (most recent call last):\n\n
>>>File
>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>line
>>> 430, in _process_data\n rval = self.proxy.dispatch(ctxt, version,
>>>method,
>>> **args)\n\n File
>>>
>>>"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.p
>>>y",
>>> line 133, in dispatch\n return getattr(proxyobj, method)(ctxt,
>>> **kwargs)\n\n File
>>> "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line
>>>116,
>>> in allocate_for_instance\n **kwargs)\n\n File
>>> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 513,
>>>in
>>> allocate_for_instance\n requested_networks=requested_networks)\n\n
>>>File
>>> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 217,
>>>in
>>> _allocate_fixed_ips\n vpn=vpn, address=address)\n\n File
>>> "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 836,
>>>in
>>> allocate_fixed_ip\n raise
>>> exception.FixedIpLimitExceeded()\n\nFixedIpLimitExceeded: Maximum
>>>number of
>>> fixed ips exceeded\n\n']
>>>
>>> --
>>> Your electronic communications are being monitored; strong encryption
>>>is an
>>> answer. My public key
>>>
>>> _______________________________________________
>>> Mailing list:
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to : openstack at lists.openstack.org
>>> Unsubscribe :
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>
>--
>Your electronic communications are being monitored; strong encryption is
>an answer. My public key
><http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x4F08C504BD634953>
>
>_______________________________________________
>Mailing list:
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>Post to : openstack at lists.openstack.org
>Unsubscribe :
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
More information about the Openstack
mailing list