[Openstack] Max open files limit for nova-api
Prashant Shetty
prashantshetty1985 at gmail.com
Tue Dec 20 12:18:00 UTC 2016
Hi Jose,
As mentioned in previous email it doesn't work on ubuntu 14.04 machine.
stack at devstackvm:~$ cat /proc/3070/limits | grep "Max open files"
Max open files 1024 4096
files
stack at devstackvm:~$
stack at devstackvm:~$ echo -n "Max open files=8192:unlimited" >
/proc/3070/limits
-bash: /proc/3070/limits: Permission denied
stack at devstackvm:~$ sudo echo -n "Max open files=8192:unlimited" >
/proc/3070/limits
-bash: /proc/3070/limits: Permission denied
stack at devstackvm:~$
root at devstackvm:/home/stack# echo -n "Max open files=8192:unlimited" >
/proc/3070/limits
bash: echo: write error: Invalid argument
root at devstackvm:/home/stack#
Thanks,
Prashant
On Tue, Dec 20, 2016 at 4:42 PM, Jose Manuel Ferrer Mosteiro <
jmferrer.paradigmatecnologico at gmail.com> wrote:
> You cannot modify /proc/3070/limits without root permissions.
>
> Try:
>
> ###
> sudo su -
> echo -n "Max open files=8192:unlimited" > /proc/3070/limits
> ###
>
> El 2016-12-19 19:21, John Petrini escribió:
>
> Hi Prashant,
>
> On second thought that trick might only work on CentOS. You might have
> success using prlimit instead.
>
> ___
>
> John Petrini
>
> NOC Systems Administrator // *CoreDial, LLC* // coredial.com
> // [image: Twitter] <https://twitter.com/coredial> [image:
> LinkedIn] <http://www.linkedin.com/company/99631> [image: Google Plus]
> <https://plus.google.com/104062177220750809525/posts> [image: Blog]
> <http://success.coredial.com/blog>
> Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
> *P: *215.297.4400 x232 // *F: *215.297.4401 // *E: *
> jpetrini at coredial.com
>
> [image: Exceptional people. Proven Processes. Innovative Technology.
> Discover CoreDial - watch our video]
> <http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
> On Mon, Dec 19, 2016 at 1:13 PM, Prashant Shetty <
> prashantshetty1985 at gmail.com> wrote:
>
>> Hi John,
>>
>> Echo option doesnt seems to work on below controller.
>>
>> stack at devstackvm:~$ cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=14.04
>> DISTRIB_CODENAME=trusty
>> DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
>> stack at devstackvm:~$
>>
>> stack at devstackvm:~$ ps aux | grep nova-api
>> stack 3070 1.1 0.1 271576 121092 pts/9 S+ Dec16 50:29
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3079 0.3 0.8 1045308 881676 pts/9 S+ Dec16 14:32
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3080 0.2 0.1 324808 161096 pts/9 S+ Dec16 9:25
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3081 0.2 0.7 980172 816468 pts/9 S+ Dec16 10:37
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3082 0.3 0.1 336824 173288 pts/9 S+ Dec16 16:11
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3083 0.4 0.1 338788 175264 pts/9 S+ Dec16 19:05
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3084 0.4 0.1 336616 172912 pts/9 S+ Dec16 17:41
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3085 0.2 0.8 1054900 891272 pts/9 S+ Dec16 10:09
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3086 0.2 0.1 325088 161228 pts/9 S+ Dec16 9:59
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3097 0.2 0.1 308088 151828 pts/9 S+ Dec16 11:10
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3098 0.2 0.1 308296 152360 pts/9 S+ Dec16 11:15
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3099 0.2 0.1 308708 152544 pts/9 S+ Dec16 11:42
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3100 0.2 0.1 309588 153624 pts/9 S+ Dec16 11:06
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3101 0.2 0.1 308372 152396 pts/9 S+ Dec16 11:14
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3102 0.2 0.1 308084 152052 pts/9 S+ Dec16 11:10
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3103 0.2 0.1 308380 152416 pts/9 S+ Dec16 11:09
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 3104 0.2 0.1 307652 151560 pts/9 S+ Dec16 10:29
>> /usr/bin/python /usr/local/bin/nova-api
>> stack 8087 0.0 0.0 11752 2044 pts/21 S+ 10:07 0:00 grep
>> --color=auto nova-api
>> stack at devstackvm:~$
>>
>> stack at devstackvm:~$ cat /proc/3070/limits | grep "Max open files"
>> Max open files 1024 4096
>> files
>> stack at devstackvm:~$
>> stack at devstackvm:~$ echo -n "Max open files=8192:unlimited" >
>> /proc/3070/limits
>> -bash: /proc/3070/limits: Permission denied
>> stack at devstackvm:~$ sudo echo -n "Max open files=8192:unlimited" >
>> /proc/3070/limits
>> -bash: /proc/3070/limits: Permission denied
>> stack at devstackvm:~$
>>
>> root at devstackvm:/home/stack# echo -n "Max open files=8192:unlimited" >
>> /proc/3070/limits
>> bash: echo: write error: Invalid argument
>> root at devstackvm:/home/stack#
>>
>> On Mon, Dec 19, 2016 at 11:27 PM, John Petrini <jpetrini at coredial.com>
>> wrote:
>>
>>> Hi Preshant,
>>>
>>> You can change the open file limit of the running process by echoing the
>>> value to it. For example...
>>>
>>>
>>> echo -n "Max open files=8192:unlimited" > /proc/<PID>/limits
>>>
>>>
>>> ___
>>>
>>> John Petrini
>>>
>>>
>>>
>>> On Mon, Dec 19, 2016 at 12:21 PM, Prashant Shetty <
>>> prashantshetty1985 at gmail.com> wrote:
>>>
>>>> Hi Arne,
>>>> Thanks for your reply. Currently all these services are running on
>>>> ubuntu controller under screen.
>>>> Do we have any option to set the file limit option for n-api service in
>>>> this case?. I am not using systemd in my setup to run these services.
>>>>
>>>> Thanks,
>>>> Prashant
>>>>
>>>> On Mon, Dec 19, 2016 at 10:19 PM, Arne Wiebalck <Arne.Wiebalck at cern.ch>
>>>> wrote:
>>>>
>>>>> Prashant,
>>>>>
>>>>> If this is for systemd, how about changing the nova-api unit file?
>>>>>
>>>>> Something like
>>>>>
>>>>> —>
>>>>> [Service]
>>>>> ...
>>>>> LimitNOFILE=65536
>>>>> <—
>>>>>
>>>>> should do it.
>>>>>
>>>>> Cheers,
>>>>> Arne
>>>>>
>>>>>
>>>>>
>>>>> On 19 Dec 2016, at 17:23, Prashant Shetty <
>>>>> prashantshetty1985 at gmail.com> wrote:
>>>>> Team,
>>>>>
>>>>> I have scale setup and metadata requests are seems to fail from
>>>>> instance. Main reason for failure is "Max open files" limit(1024) set on
>>>>> nova-api service.
>>>>> Though on controller we have set max open file limit of
>>>>> 65k(limit.conf), nova-api always comes up with 1024 limit causing failure.
>>>>>
>>>>> Could someone let me know how can we change the max open files limit
>>>>> of nova-api service?
>>>>>
>>>>> Setup Details:
>>>>>
>>>>> · Single controller
>>>>> · 500 KVM computes
>>>>> · Devstack branch: stable/newton
>>>>> · We have native metadata and dhcp running on platform
>>>>> · 3750 instances
>>>>>
>>>>>
>>>>> stack at controller:/opt/stack/logs$ ps aux | grep nova-api
>>>>> stack 14998 2.2 0.3 272104 121648 pts/8 S+ 09:53 0:14 /usr/bin/python
>>>>> /usr/local/bin/nova-api
>>>>> stack at controller:/opt/stack/logs$
>>>>> stack at controller:/opt/stack/logs$
>>>>> stack at controller:/opt/stack/logs$ cat /proc/14998/limits
>>>>> Limit Soft Limit Hard Limit Units
>>>>> Max cpu time unlimited unlimited seconds
>>>>> Max file size unlimited unlimited bytes
>>>>> Max data size unlimited unlimited bytes
>>>>> Max stack size 8388608 unlimited bytes
>>>>> Max core file size unlimited unlimited bytes
>>>>> Max resident set unlimited unlimited bytes
>>>>> Max processes 128611 128611 processes
>>>>> Max open files 1024 4096 files
>>>>> Max locked memory 65536 65536 bytes
>>>>> Max address space unlimited unlimited bytes
>>>>> Max file locks unlimited unlimited locks
>>>>> Max pending signals 128611 128611 signals
>>>>> Max msgqueue size 819200 819200 bytes
>>>>> Max nice priority 0 0
>>>>> Max realtime priority 0 0
>>>>> Max realtime timeout unlimited unlimited us
>>>>> stack at controller:/opt/stack/logs$
>>>>>
>>>>> n-api:
>>>>>
>>>>> 2016-11-08 18:44:26.168 30069 INFO nova.metadata.wsgi.server
>>>>> [req-fb4d729b-a1cd-4df1-aaf8-3f854a739cce - -] (30069) wsgi exited,
>>>>> is_accepting=True
>>>>> Traceback (most recent call last):
>>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
>>>>> line 457, in fire_timers
>>>>> timer()
>>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
>>>>> line 58, in __call__
>>>>> cb(*args, **kw)
>>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py",
>>>>> line 168, in _do_send
>>>>> waiter.switch(result)
>>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
>>>>> line 214, in main
>>>>> result = function(*args, **kwargs)
>>>>> File "/opt/stack/nova/nova/utils.py", line 1066, in context_wrapper
>>>>> return func(*args, **kwargs)
>>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py",
>>>>> line 865, in server
>>>>> client_socket = sock.accept()
>>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py",
>>>>> line 214, in accept
>>>>> res = socket_accept(fd)
>>>>> File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py",
>>>>> line 56, in socket_accept
>>>>> return descriptor.accept()
>>>>> File "/usr/lib/python2.7/socket.py", line 206, in accept
>>>>> sock, addr = self._sock.accept()
>>>>> error: [Errno 24] Too many open files
>>>>>
>>>>> Thanks,
>>>>> Prashant
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>> -bin/mailman/listinfo/openstack
>>>>> Post to : openstack at lists.openstack.org
>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>> -bin/mailman/listinfo/openstack
>>>>>
>>>>>
>>>>> --
>>>>> Arne Wiebalck
>>>>> CERN IT
>>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: http://lists.openstack.org/cgi
>>>> -bin/mailman/listinfo/openstack
>>>> Post to : openstack at lists.openstack.org
>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>> -bin/mailman/listinfo/openstack
>>>>
>>>>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20161220/7059eda6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: blocked.gif
Type: image/gif
Size: 118 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20161220/7059eda6/attachment.gif>
More information about the Openstack
mailing list
