On Wed, 2021-07-07 at 11:34 +0200, Javier Pena wrote:

Hello Piotr,

On Fri, Jul 2, 2021 at 6:53 PM Piotr Mossakowski
<piotr.mossakowski@tietoevry.com> wrote:

Hello,

There is a logging override in almost all openstack packages, for
example in nova:

https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Frpm-packaging%2Fsrc%2Fbranch%2Fstable%2Fvictoria%2Fopenstack%2Fnova%2Fopenstack-nova.defaultconf&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7%2FHlBNmjgHGOsaLH022Tp3ewzlt1y9FpekdnwMdoYfU%3D&reserved=0

It lands in /etc/nova/nova.conf.d/010-nova.conf as stated in the
spec:

https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.org%2Fopenstack%2Frpm-packaging%2Fsrc%2Fbranch%2Fstable%2Fvictoria%2Fopenstack%2Fnova%2Fnova.spec.j2%23L407&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=X0PJWa38ozPRybinbHJRS2PmzsPuZk41fYlMkr9fFfw%3D&reserved=0

Let's assume following scenario: in kolla-ansible based deployment I
want to disable logging globally using /etc/kolla/config/global.conf:

[DEFAULT]
log_dir =
log_file =

For kolla images built from RPMs, the file
'/etc/nova/nova.conf.d/010-nova.conf' will override what I want to
achieve by global.conf.

We have a situation when kolla images log inside the container into
'/var/log/nova' where there is no log rotation.

actully kolla has log rotation for those files
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fcommon%2Ftemplates%2Fcron-logrotate-nova.conf.j2&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VwjAg3jKrsxL9QCWbs7REyAS%2Bt7JSwd2KUuF98xAU%2Bc%3D&reserved=0

ot does not use oslo logs in built log rotation it does log ratation
via cron.

kolla ansible also does not use the configs from the rpms it generates
its own nova.conf that is used instead using
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova-cell%2Ftemplates%2Fnova.conf.j2&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tSmahTPy1C7PW2l3UPf8z8CoT80BGQWYFoF0NYF1Bz8%3D&reserved=0
as the template which is renderd an copied on container start
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova-cell%2Ftemplates%2Fnova-compute.json.j2%23L3-L9&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VNQlRxFIEjSpp3WSCrPR8b08nuW6bfQJa1DYmBOdOU0%3D&reserved=0
and kolla provides a config override mechaniume to future customise
this if needed.
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Fkolla-ansible%2Flatest%2Fadmin%2Fdeployment-philosophy.html%23kolla-s-solution-to-customization&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0w%2Fm%2B9ZbRowgUbW5sf5Yuoac2eHAr%2BjeAw22HY7R57E%3D&reserved=0

if you look at the nova and nova-cell role config tasks

https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova%2Ftasks%2Fconfig.yml%23L51-L69&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306724949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BpA6lqzPqFe6HSk4BE8MmxrY418HXoRLzIfaEkX5xLo%3D&reserved=0
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenstack%2Fkolla-ansible%2Fblob%2Fmaster%2Fansible%2Froles%2Fnova-cell%2Ftasks%2Fconfig.yml%23L56-L75&data=04%7C01%7CPiotr.Mossakowski%40tietoevry.com%7C88f20f58f630468f716108d9413f0b8a%7Ccbede638a3d9459f8f4e24ced73b4e5e%7C1%7C0%7C637612561306734942%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1YuSbgJVsTPjWYiFnKATAR7atCVC70wGpoX9OM46mO8%3D&reserved=0

you wil see the list of filets that will be merged to generate the
final nova.conf

 sources:
      - "{{ role_path }}/templates/nova.conf.j2"
      - "{{ node_custom_config }}/global.conf"
      - "{{ node_custom_config }}/nova.conf"
      - "{{ node_custom_config }}/nova/{{ item.key }}.conf"
      - "{{ node_custom_config }}/nova/{{ inventory_hostname }}/nova.conf"


kolla renders its template then merges any seconts form  the other
files if they exist

{{ node_custom_config }} defaults to /etc/kolla/config

so if you want to make global changes to novas log rotation using the
oslo-log log rotation capablitys you would add those options to

/etc/kolla/config/nova.conf

you can do it speratly per serive by creating
/etc/kolla/config/nova/nova-compute.conf
/etc/kolla/config/nova/nova-api.conf
...

and even per host if you want too.

packaging project should ideally avoid deviating from upstream
configuration where possible but in a kolla enve the package cofnigs
should be unused.

its possible you are using a very old version of kolla before the log
rotate configs where intoduced but correct way to apply config changes
form a "deployment-philosophy" point of view is to use the config
override mechanium  not to do it via packaging changes.

 There is no easy way to change that in kolla-ansible based
deployments.

Is there any specific reason why default logging has been hardcoded
like that? Can we agree to change that situation to be able to
override logging as expected?

In general, packages (both for rpm-packaging and RDO) try to set some
sensible minimum defaults for a few parameters, which are set
according to the base OS requirements. This is why we get those
defaults in the rpm-packaging Nova spec.

I think the easiest way to ensure the override in
/etc/kolla/config/global.conf is enforced would be to parse it last in
the nova-api command-line (with a --config-file directive). Would that
be possible in your environment?

Regards,
Javier

--

Pozdrawiam serdecznie / Best regards,

Piotr Mossakowski

Lead System Engineer, Product Development Services, Cloud Infra &
Applications

Email piotr.mossakowski@tietoevry.com, +48795515407
Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com

Thanks for your replies!

@Sean: everything you said is true but the situation I described is happening inside the container so:

1) log rotation is applied on /var/log/kolla on the host but not on /var/log/nova inside the container - and this is the biggest pain as it will finally fill out docker partition on the host.

2) kolla-ansible generates /etc/nova/nova.conf inside the container but openstack-nova RPM injects /etc/nova/nova.conf.d/010-nova.conf and this overrides my customizations from /etc/kolla/config/global.conf

@Javier: your idea sounds a lot better than updating a ton of 'extend_start.sh' scripts, thanks!