Open Stack

Hey Sergii,

On Thu, Oct 20, 2016 at 3:34 AM, Sergii Golovatiuk
<sgolovatiuk at mirantis.com> wrote:
> Hi,
>
>
> On Thu, Sep 29, 2016 at 11:57 PM, Alex Schultz <aschultz at redhat.com> wrote:
>>
>> Hello all,
>>
>> So for many years we've been using either the service defaults
>> (usually python determined processor count) or the $processorcount
>> fact from facter in puppet for worker configuration options for the
>> OpenStack services.  If you are currently using the default values
>> provided by the puppet modules, you will be affected by this upcoming
>> change. After much discussion and feedback from deployers, we've
>> decided to change this to a default value that has a cap on it.  This
>> is primarily from the feedback when deploying on physical hardware
>> where processor counts can be 32, 48 or even 512.  These values can
>> lead to excessive memory consumption or errors due to connection
>> limits (mysql/rabbit).  As such we've come up with a new fact to that
>> will be used instead of $processorcount.
>>
>> The new fact is called $os_workers[0]. This fact uses the
>> $processorcount to help weigh in on the number of workers to configure
>> and won't be less than 2 but is capped at 8.  The $os_workers fact
>> will use the larger value of either '2' or '# of processors / 4' but
>> will not exceed 8.  The primary goal of this is to improve the user
>> experience when people install services using the puppet modules and
>> without having to tune all of these worker values.  We plan on
>> implementing this for all modules as part of the Ocata cycle.  This
>> work will can be tracked using the os_workers-fact[1] gerrit topic.
>> It should be noted that we have implemented this fact in such a way
>> that operators are free to override it using an external fact to
>> provide their own values as well.  If you are currently specifying
>> your own values for the worker configurations in your manifests then
>> this change will not affect you.  If you have been relying on the
>> defaults and wish to continue to use the $processorcount logic, we
>> would recommend either implementing your own external fact[2] for this
>> or updating your manifests to provide $::processorcount to the workers
>> configuration.
>
>
> This doesn't help a lot. I saw the case where 8 neutron-servers allocated
> 6GB of RAM. From OOM perspective, the biggest process was MySQL (or Rabbit)
> as it doesn't calculate the sum of processes. Instead of killing
> neutron-server it killed MySQL to release some RAM to node. IMO, I would
> focus on cgroup limitation for OpenStack services as that will allow the
> operator to specify some the upper limit of CPU and RAM usage for every
> service.
>

Yea this isn't a fix for excessive ram utilization of the services
themselves. This primarily reduces the default consumption. I recently
went to implement using this fact in tripleo[0] and it reduced the
default memory utilization of the undercloud by a little over 1gig[1]
in our CI runs.  This was on a 4 cpu vhost so we basically halfed the
number of running python processes by switching for the 17 different
services that may run.  So yea this isn't the end solution for dealing
with excessive memory consumption but rather an improvement over the
existing processor count usage.  The primary target for this is really
to prevent accidental blowing up of mysql connections when a host has
large number of cpus but also reducing the overall number of processes
by default.

Thanks,
-Alex

[0] https://review.openstack.org/#/c/386696/
[1] http://people.redhat.com/~aschultz/os_workers.html

>>
>> As always we'd love to hear feedback on this and any other issues
>> people might be facing. We're always available in #puppet-openstack on
>> freenode or via the mailing lists.
>>
>> Thanks,
>> -Alex
>>
>>
>> [0] https://review.openstack.org/#/c/375146/
>> [1] https://review.openstack.org/#/q/topic:os_workers-fact
>> [2] https://docs.puppet.com/facter/3.4/custom_facts.html#external-facts
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>