[Openstack] Max open files limit for nova-api

Prashant Shetty prashantshetty1985 at gmail.com
Tue Dec 20 16:37:08 UTC 2016


Hi Remo,

Its using ext4 file system on controller.

root at devstackvm:/home/stack# df -T | awk '{print $1,$2,$NF}' | grep "^/dev"
/dev/sda1 ext4 /
root at devstackvm:/home/stack# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
root at devstackvm:/home/stack#

Thanks,
Prashant

On Tue, Dec 20, 2016 at 8:58 PM, Remo Mattei <remo at italy1.com> wrote:

> What file system do you have configured on this box?
>
> Inviato da iPhone
>
> Il giorno 20 dic 2016, alle ore 04:18, Prashant Shetty <
> prashantshetty1985 at gmail.com> ha scritto:
>
> Hi Jose,
>
> As mentioned in previous email it doesn't work on ubuntu 14.04 machine.
>
> stack at devstackvm:~$ cat /proc/3070/limits  | grep "Max open files"
> Max open files            1024                 4096
> files
> stack at devstackvm:~$
> stack at devstackvm:~$ echo -n "Max open files=8192:unlimited"  >
> /proc/3070/limits
> -bash: /proc/3070/limits: Permission denied
> stack at devstackvm:~$ sudo echo -n "Max open files=8192:unlimited"  >
> /proc/3070/limits
> -bash: /proc/3070/limits: Permission denied
> stack at devstackvm:~$
>
> root at devstackvm:/home/stack# echo -n "Max open files=8192:unlimited" >
> /proc/3070/limits
> bash: echo: write error: Invalid argument
> root at devstackvm:/home/stack#
>
> Thanks,
> Prashant
>
> On Tue, Dec 20, 2016 at 4:42 PM, Jose Manuel Ferrer Mosteiro <jmferrer.
> paradigmatecnologico at gmail.com> wrote:
>
>> You cannot modify /proc/3070/limits without root permissions.
>>
>> Try:
>>
>> ###
>> sudo su -
>> echo -n "Max open files=8192:unlimited"  > /proc/3070/limits
>> ###
>>
>> El 2016-12-19 19:21, John Petrini escribió:
>>
>> Hi Prashant,
>>
>> On second thought that trick might only work on CentOS.  You might have
>> success using prlimit instead.
>>
>> ___
>>
>> John Petrini
>>
>> NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
>>    //   <blocked.gif> <https://twitter.com/coredial>   <blocked.gif>
>> <http://www.linkedin.com/company/99631>   <blocked.gif>
>> <https://plus.google.com/104062177220750809525/posts>   <blocked.gif>
>> <http://success.coredial.com/blog>
>> Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
>> *P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
>> jpetrini at coredial.com
>>
>> <blocked.gif>
>> <http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>
>>
>> The information transmitted is intended only for the person or entity to
>> which it is addressed and may contain confidential and/or privileged
>> material. Any review, retransmission,  dissemination or other use of, or
>> taking of any action in reliance upon, this information by persons or
>> entities other than the intended recipient is prohibited. If you received
>> this in error, please contact the sender and delete the material from any
>> computer.
>>
>> On Mon, Dec 19, 2016 at 1:13 PM, Prashant Shetty <
>> prashantshetty1985 at gmail.com> wrote:
>>
>>> Hi John,
>>>
>>> Echo option doesnt seems to work on below controller.
>>>
>>> stack at devstackvm:~$ cat /etc/lsb-release
>>> DISTRIB_ID=Ubuntu
>>> DISTRIB_RELEASE=14.04
>>> DISTRIB_CODENAME=trusty
>>> DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
>>> stack at devstackvm:~$
>>>
>>> stack at devstackvm:~$ ps aux | grep nova-api
>>> stack     3070  1.1  0.1 271576 121092 pts/9   S+   Dec16  50:29
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3079  0.3  0.8 1045308 881676 pts/9  S+   Dec16  14:32
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3080  0.2  0.1 324808 161096 pts/9   S+   Dec16   9:25
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3081  0.2  0.7 980172 816468 pts/9   S+   Dec16  10:37
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3082  0.3  0.1 336824 173288 pts/9   S+   Dec16  16:11
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3083  0.4  0.1 338788 175264 pts/9   S+   Dec16  19:05
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3084  0.4  0.1 336616 172912 pts/9   S+   Dec16  17:41
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3085  0.2  0.8 1054900 891272 pts/9  S+   Dec16  10:09
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3086  0.2  0.1 325088 161228 pts/9   S+   Dec16   9:59
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3097  0.2  0.1 308088 151828 pts/9   S+   Dec16  11:10
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3098  0.2  0.1 308296 152360 pts/9   S+   Dec16  11:15
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3099  0.2  0.1 308708 152544 pts/9   S+   Dec16  11:42
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3100  0.2  0.1 309588 153624 pts/9   S+   Dec16  11:06
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3101  0.2  0.1 308372 152396 pts/9   S+   Dec16  11:14
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3102  0.2  0.1 308084 152052 pts/9   S+   Dec16  11:10
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3103  0.2  0.1 308380 152416 pts/9   S+   Dec16  11:09
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     3104  0.2  0.1 307652 151560 pts/9   S+   Dec16  10:29
>>> /usr/bin/python /usr/local/bin/nova-api
>>> stack     8087  0.0  0.0  11752  2044 pts/21   S+   10:07   0:00 grep
>>> --color=auto nova-api
>>> stack at devstackvm:~$
>>>
>>> stack at devstackvm:~$ cat /proc/3070/limits  | grep "Max open files"
>>> Max open files            1024                 4096
>>> files
>>> stack at devstackvm:~$
>>> stack at devstackvm:~$ echo -n "Max open files=8192:unlimited"  >
>>> /proc/3070/limits
>>> -bash: /proc/3070/limits: Permission denied
>>> stack at devstackvm:~$ sudo echo -n "Max open files=8192:unlimited"  >
>>> /proc/3070/limits
>>> -bash: /proc/3070/limits: Permission denied
>>> stack at devstackvm:~$
>>>
>>> root at devstackvm:/home/stack# echo -n "Max open files=8192:unlimited" >
>>> /proc/3070/limits
>>> bash: echo: write error: Invalid argument
>>> root at devstackvm:/home/stack#
>>>
>>> On Mon, Dec 19, 2016 at 11:27 PM, John Petrini <jpetrini at coredial.com>
>>> wrote:
>>>
>>>> Hi Preshant,
>>>>
>>>> You can change the open file limit of the running process by echoing
>>>> the value to it. For example...
>>>>
>>>>
>>>> echo -n "Max open files=8192:unlimited"  > /proc/<PID>/limits
>>>>
>>>>
>>>> ___
>>>>
>>>> John Petrini
>>>>
>>>>
>>>>
>>>> On Mon, Dec 19, 2016 at 12:21 PM, Prashant Shetty <
>>>> prashantshetty1985 at gmail.com> wrote:
>>>>
>>>>> Hi Arne,
>>>>> Thanks for your reply. Currently all these services are running on
>>>>> ubuntu controller under screen.
>>>>> Do we have any option to set the file limit option for n-api service
>>>>> in this case?. I am not using systemd in my setup to run these services.
>>>>>
>>>>> Thanks,
>>>>> Prashant
>>>>>
>>>>> On Mon, Dec 19, 2016 at 10:19 PM, Arne Wiebalck <Arne.Wiebalck at cern.ch
>>>>> > wrote:
>>>>>
>>>>>> Prashant,
>>>>>>
>>>>>> If this is for systemd, how about changing the nova-api unit file?
>>>>>>
>>>>>> Something like
>>>>>>
>>>>>> —>
>>>>>> [Service]
>>>>>> ...
>>>>>> LimitNOFILE=65536
>>>>>> <—
>>>>>>
>>>>>> should do it.
>>>>>>
>>>>>> Cheers,
>>>>>>  Arne
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 19 Dec 2016, at 17:23, Prashant Shetty <
>>>>>> prashantshetty1985 at gmail.com> wrote:
>>>>>> Team,
>>>>>>
>>>>>> I have scale setup and metadata requests are seems to fail from
>>>>>> instance. Main reason for failure is "Max open files" limit(1024) set on
>>>>>> nova-api service.
>>>>>> Though on controller we have set max open file limit of
>>>>>> 65k(limit.conf), nova-api always comes up with 1024 limit causing failure.
>>>>>>
>>>>>> Could someone let me know how can we change the max open files limit
>>>>>> of nova-api service?
>>>>>>
>>>>>> Setup Details:
>>>>>>
>>>>>> ·         Single controller
>>>>>> ·         500 KVM computes
>>>>>> ·         Devstack branch: stable/newton
>>>>>> ·         We have native metadata and dhcp running on platform
>>>>>> ·         3750 instances
>>>>>>
>>>>>>
>>>>>> stack at controller:/opt/stack/logs$ ps aux | grep nova-api
>>>>>> stack 14998 2.2 0.3 272104 121648 pts/8 S+ 09:53 0:14 /usr/bin/python
>>>>>> /usr/local/bin/nova-api
>>>>>> stack at controller:/opt/stack/logs$
>>>>>> stack at controller:/opt/stack/logs$
>>>>>> stack at controller:/opt/stack/logs$ cat /proc/14998/limits
>>>>>> Limit Soft Limit Hard Limit Units
>>>>>> Max cpu time unlimited unlimited seconds
>>>>>> Max file size unlimited unlimited bytes
>>>>>> Max data size unlimited unlimited bytes
>>>>>> Max stack size 8388608 unlimited bytes
>>>>>> Max core file size unlimited unlimited bytes
>>>>>> Max resident set unlimited unlimited bytes
>>>>>> Max processes 128611 128611 processes
>>>>>> Max open files 1024 4096 files
>>>>>> Max locked memory 65536 65536 bytes
>>>>>> Max address space unlimited unlimited bytes
>>>>>> Max file locks unlimited unlimited locks
>>>>>> Max pending signals 128611 128611 signals
>>>>>> Max msgqueue size 819200 819200 bytes
>>>>>> Max nice priority 0 0
>>>>>> Max realtime priority 0 0
>>>>>> Max realtime timeout unlimited unlimited us
>>>>>> stack at controller:/opt/stack/logs$
>>>>>>
>>>>>> n-api:
>>>>>>
>>>>>> 2016-11-08 18:44:26.168 30069 INFO nova.metadata.wsgi.server
>>>>>> [req-fb4d729b-a1cd-4df1-aaf8-3f854a739cce - -] (30069) wsgi exited,
>>>>>> is_accepting=True
>>>>>> Traceback (most recent call last):
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
>>>>>> line 457, in fire_timers
>>>>>>     timer()
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
>>>>>> line 58, in __call__
>>>>>>     cb(*args, **kw)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py",
>>>>>> line 168, in _do_send
>>>>>>     waiter.switch(result)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
>>>>>> line 214, in main
>>>>>>     result = function(*args, **kwargs)
>>>>>>   File "/opt/stack/nova/nova/utils.py", line 1066, in context_wrapper
>>>>>>     return func(*args, **kwargs)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py",
>>>>>> line 865, in server
>>>>>>     client_socket = sock.accept()
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py",
>>>>>> line 214, in accept
>>>>>>     res = socket_accept(fd)
>>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py",
>>>>>> line 56, in socket_accept
>>>>>>     return descriptor.accept()
>>>>>>   File "/usr/lib/python2.7/socket.py", line 206, in accept
>>>>>>     sock, addr = self._sock.accept()
>>>>>> error: [Errno 24] Too many open files
>>>>>>
>>>>>> Thanks,
>>>>>> Prashant
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>> -bin/mailman/listinfo/openstack
>>>>>> Post to     : openstack at lists.openstack.org
>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>> -bin/mailman/listinfo/openstack
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Arne Wiebalck
>>>>>> CERN IT
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>> -bin/mailman/listinfo/openstack
>>>>> Post to     : openstack at lists.openstack.org
>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>> -bin/mailman/listinfo/openstack
>>>>>
>>>>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
> !DSPAM:1,5859251090622543419488!
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
> !DSPAM:1,5859251090622543419488!
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20161220/78f3d0cf/attachment.html>


More information about the Openstack mailing list