Open Stack

On 7/7/21 2:01 PM, Sean Mooney wrote:
> On Wed, 2021-07-07 at 11:34 +0200, Javier Pena wrote:
>> Hello Piotr,
>>
>> On Fri, Jul 2, 2021 at 6:53 PM Piotr Mossakowski
>> <piotr.mossakowski at tietoevry.com> wrote:
>>> Hello,
>>>
>>> There is a logging override in almost all openstack packages, for
>>> example in nova:
>>>
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Frpm-packaging%2Fsrc%2Fbranch%2Fstable%2Fvictoria%2Fopenstack%2Fnova%2Fopenstack-nova.defaultconf&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7%2FHlBNmjgHGOsaLH022Tp3ewzlt1y9FpekdnwMdoYfU%3D&reserved=0
>>>
>>> It lands in /etc/nova/nova.conf.d/010-nova.conf as stated in the
>>> spec:
>>>
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Frpm-packaging%2Fsrc%2Fbranch%2Fstable%2Fvictoria%2Fopenstack%2Fnova%2Fnova.spec.j2%23L407&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=X0PJWa38ozPRybinbHJRS2PmzsPuZk41fYlMkr9fFfw%3D&reserved=0
>>>
>>> Let's assume following scenario: in kolla-ansible based deployment I
>>> want to disable logging globally using /etc/kolla/config/global.conf:
>>>
>>> [DEFAULT]
>>> log_dir =
>>> log_file =
>>>
>>> For kolla images built from RPMs, the file
>>> '/etc/nova/nova.conf.d/010-nova.conf' will override what I want to
>>> achieve by global.conf.
>>>
>>> We have a situation when kolla images log inside the container into
>>> '/var/log/nova' where there is no log rotation.
> actully kolla has log rotation for those files
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fcommon%2Ftemplates%2Fcron-logrotate-nova.conf.j2&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VwjAg3jKrsxL9QCWbs7REyAS%2Bt7JSwd2KUuF98xAU%2Bc%3D&reserved=0
>
> ot does not use oslo logs in built log rotation it does log ratation
> via cron.
>
> kolla ansible also does not use the configs from the rpms it generates
> its own nova.conf that is used instead using
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova-cell%2Ftemplates%2Fnova.conf.j2&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tSmahTPy1C7PW2l3UPf8z8CoT80BGQWYFoF0NYF1Bz8%3D&reserved=0
> as the template which is renderd an copied on container start
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova-cell%2Ftemplates%2Fnova-compute.json.j2%23L3-L9&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VNQlRxFIEjSpp3WSCrPR8b08nuW6bfQJa1DYmBOdOU0%3D&reserved=0
> and kolla provides a config override mechaniume to future customise
> this if needed.
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Fkolla-ansible%2Flatest%2Fadmin%2Fdeployment-philosophy.html%23kolla-s-solution-to-customization&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0w%2Fm%2B9ZbRowgUbW5sf5Yuoac2eHAr%2BjeAw22HY7R57E%3D&reserved=0
>
> if you look at the nova and nova-cell role config tasks
>
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova%2Ftasks%2Fconfig.yml%23L51-L69&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BpA6lqzPqFe6HSk4BE8MmxrY418HXoRLzIfaEkX5xLo%3D&reserved=0
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova-cell%2Ftasks%2Fconfig.yml%23L56-L75&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306734942%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1YuSbgJVsTPjWYiFnKATAR7atCVC70wGpoX9OM46mO8%3D&reserved=0
>
> you wil see the list of filets that will be merged to generate the
> final nova.conf
>
>   sources:
>        - "{{ role_path }}/templates/nova.conf.j2"
>        - "{{ node_custom_config }}/global.conf"
>        - "{{ node_custom_config }}/nova.conf"
>        - "{{ node_custom_config }}/nova/{{ item.key }}.conf"
>        - "{{ node_custom_config }}/nova/{{ inventory_hostname }}/nova.conf"
>
>
> kolla renders its template then merges any seconts form  the other
> files if they exist
>
> {{ node_custom_config }} defaults to /etc/kolla/config
>
> so if you want to make global changes to novas log rotation using the
> oslo-log log rotation capablitys you would add those options to
>
> /etc/kolla/config/nova.conf
>
> you can do it speratly per serive by creating
> /etc/kolla/config/nova/nova-compute.conf
> /etc/kolla/config/nova/nova-api.conf
> ...
>
> and even per host if you want too.
>
> packaging project should ideally avoid deviating from upstream
> configuration where possible but in a kolla enve the package cofnigs
> should be unused.
>
> its possible you are using a very old version of kolla before the log
> rotate configs where intoduced but correct way to apply config changes
> form a "deployment-philosophy" point of view is to use the config
> override mechanium  not to do it via packaging changes.
>
>>>   There is no easy way to change that in kolla-ansible based
>>> deployments.
>>>
>>> Is there any specific reason why default logging has been hardcoded
>>> like that? Can we agree to change that situation to be able to
>>> override logging as expected?
>>>
>> In general, packages (both for rpm-packaging and RDO) try to set some
>> sensible minimum defaults for a few parameters, which are set
>> according to the base OS requirements. This is why we get those
>> defaults in the rpm-packaging Nova spec.
>>
>> I think the easiest way to ensure the override in
>> /etc/kolla/config/global.conf is enforced would be to parse it last in
>> the nova-api command-line (with a --config-file directive). Would that
>> be possible in your environment?
>>
>> Regards,
>> Javier
>>
>>> --
>>>
>>> Pozdrawiam serdecznie / Best regards,
>>>
>>> Piotr Mossakowski
>>>
>>> Lead System Engineer, Product Development Services, Cloud Infra &
>>> Applications
>>>
>>> Email piotr.mossakowski at tietoevry.com, +48795515407
>>> Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com
>>
>
Thanks for your replies!

@Sean: everything you said is true but the situation I described is 
happening inside the container so:

1) log rotation is applied on /var/log/kolla on the host but not on 
/var/log/nova inside the container - and this is the biggest pain as it 
will finally fill out docker partition on the host.

2) kolla-ansible generates /etc/nova/nova.conf inside the container but 
openstack-nova RPM injects /etc/nova/nova.conf.d/010-nova.conf and this 
overrides my customizations from /etc/kolla/config/global.conf

@Javier: your idea sounds a lot better than updating a ton of 
'extend_start.sh' scripts, thanks!


-- 

Pozdrawiam serdecznie / Best regards,

*Piotr Mossakowski*

Lead System Engineer, Product Development Services, Cloud Infra & 
Applications

Email piotr.mossakowski at tietoevry.com 
<mailto:piotr.mossakowski at tietoevry.com>, +48795515407 <tel:%2B48795515407>
Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com 
<http://www.tieto.com/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210720/990ac722/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4195 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210720/990ac722/attachment-0001.bin>