[rpm-packaging] logging override in openstack packages
Hello, There is a logging override in almost all openstack packages, for example in nova: https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/opens... <https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/openstack/nova/openstack-nova.defaultconf> It lands in /etc/nova/nova.conf.d/010-nova.conf as stated in the spec: https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/opens... <https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/openstack/nova/nova.spec.j2#L407> Let's assume following scenario: in kolla-ansible based deployment I want to disable logging globally using /etc/kolla/config/global.conf: [DEFAULT] log_dir = log_file = For kolla images built from RPMs, the file '/etc/nova/nova.conf.d/010-nova.conf' will override what I want to achieve by global.conf. We have a situation when kolla images log inside the container into '/var/log/nova' where there is no log rotation. There is no easy way to change that in kolla-ansible based deployments. Is there any specific reason why default logging has been hardcoded like that? Can we agree to change that situation to be able to override logging as expected? -- Pozdrawiam serdecznie / Best regards, *Piotr Mossakowski* Lead System Engineer, Product Development Services, Cloud Infra & Applications Email piotr.mossakowski@tietoevry.com <mailto:piotr.mossakowski@tietoevry.com>, +48795515407 <tel:%2B48795515407> Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com <http://www.tieto.com/>
Hello Piotr, On Fri, Jul 2, 2021 at 6:53 PM Piotr Mossakowski <piotr.mossakowski@tietoevry.com> wrote:
Hello,
There is a logging override in almost all openstack packages, for example in nova:
https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/opens...
It lands in /etc/nova/nova.conf.d/010-nova.conf as stated in the spec:
https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/opens...
Let's assume following scenario: in kolla-ansible based deployment I want to disable logging globally using /etc/kolla/config/global.conf:
[DEFAULT] log_dir = log_file =
For kolla images built from RPMs, the file '/etc/nova/nova.conf.d/010-nova.conf' will override what I want to achieve by global.conf.
We have a situation when kolla images log inside the container into '/var/log/nova' where there is no log rotation. There is no easy way to change that in kolla-ansible based deployments.
Is there any specific reason why default logging has been hardcoded like that? Can we agree to change that situation to be able to override logging as expected?
In general, packages (both for rpm-packaging and RDO) try to set some sensible minimum defaults for a few parameters, which are set according to the base OS requirements. This is why we get those defaults in the rpm-packaging Nova spec. I think the easiest way to ensure the override in /etc/kolla/config/global.conf is enforced would be to parse it last in the nova-api command-line (with a --config-file directive). Would that be possible in your environment? Regards, Javier
--
Pozdrawiam serdecznie / Best regards,
Piotr Mossakowski
Lead System Engineer, Product Development Services, Cloud Infra & Applications
Email piotr.mossakowski@tietoevry.com, +48795515407 Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com
On Wed, 2021-07-07 at 11:34 +0200, Javier Pena wrote:
Hello Piotr,
On Fri, Jul 2, 2021 at 6:53 PM Piotr Mossakowski <piotr.mossakowski@tietoevry.com> wrote:
Hello,
There is a logging override in almost all openstack packages, for example in nova:
https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/opens...
It lands in /etc/nova/nova.conf.d/010-nova.conf as stated in the spec:
https://opendev.org/openstack/rpm-packaging/src/branch/stable/victoria/opens...
Let's assume following scenario: in kolla-ansible based deployment I want to disable logging globally using /etc/kolla/config/global.conf:
[DEFAULT] log_dir = log_file =
For kolla images built from RPMs, the file '/etc/nova/nova.conf.d/010-nova.conf' will override what I want to achieve by global.conf.
We have a situation when kolla images log inside the container into '/var/log/nova' where there is no log rotation.
actully kolla has log rotation for those files https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/common/... ot does not use oslo logs in built log rotation it does log ratation via cron. kolla ansible also does not use the configs from the rpms it generates its own nova.conf that is used instead using https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-ce... as the template which is renderd an copied on container start https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-ce... and kolla provides a config override mechaniume to future customise this if needed. https://docs.openstack.org/kolla-ansible/latest/admin/deployment-philosophy.... if you look at the nova and nova-cell role config tasks https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/ta... https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-ce... you wil see the list of filets that will be merged to generate the final nova.conf sources: - "{{ role_path }}/templates/nova.conf.j2" - "{{ node_custom_config }}/global.conf" - "{{ node_custom_config }}/nova.conf" - "{{ node_custom_config }}/nova/{{ item.key }}.conf" - "{{ node_custom_config }}/nova/{{ inventory_hostname }}/nova.conf" kolla renders its template then merges any seconts form the other files if they exist {{ node_custom_config }} defaults to /etc/kolla/config so if you want to make global changes to novas log rotation using the oslo-log log rotation capablitys you would add those options to /etc/kolla/config/nova.conf you can do it speratly per serive by creating /etc/kolla/config/nova/nova-compute.conf /etc/kolla/config/nova/nova-api.conf ... and even per host if you want too. packaging project should ideally avoid deviating from upstream configuration where possible but in a kolla enve the package cofnigs should be unused. its possible you are using a very old version of kolla before the log rotate configs where intoduced but correct way to apply config changes form a "deployment-philosophy" point of view is to use the config override mechanium not to do it via packaging changes.
There is no easy way to change that in kolla-ansible based deployments.
Is there any specific reason why default logging has been hardcoded like that? Can we agree to change that situation to be able to override logging as expected?
In general, packages (both for rpm-packaging and RDO) try to set some sensible minimum defaults for a few parameters, which are set according to the base OS requirements. This is why we get those defaults in the rpm-packaging Nova spec.
I think the easiest way to ensure the override in /etc/kolla/config/global.conf is enforced would be to parse it last in the nova-api command-line (with a --config-file directive). Would that be possible in your environment?
Regards, Javier
--
Pozdrawiam serdecznie / Best regards,
Piotr Mossakowski
Lead System Engineer, Product Development Services, Cloud Infra & Applications
Email piotr.mossakowski@tietoevry.com, +48795515407 Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com
On 7/7/21 2:01 PM, Sean Mooney wrote:
On Wed, 2021-07-07 at 11:34 +0200, Javier Pena wrote:
Hello Piotr,
On Fri, Jul 2, 2021 at 6:53 PM Piotr Mossakowski <piotr.mossakowski@tietoevry.com> wrote:
Hello,
There is a logging override in almost all openstack packages, for example in nova:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.or...
It lands in /etc/nova/nova.conf.d/010-nova.conf as stated in the spec:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopendev.or...
Let's assume following scenario: in kolla-ansible based deployment I want to disable logging globally using /etc/kolla/config/global.conf:
[DEFAULT] log_dir = log_file =
For kolla images built from RPMs, the file '/etc/nova/nova.conf.d/010-nova.conf' will override what I want to achieve by global.conf.
We have a situation when kolla images log inside the container into '/var/log/nova' where there is no log rotation. actully kolla has log rotation for those files https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com...
ot does not use oslo logs in built log rotation it does log ratation via cron.
kolla ansible also does not use the configs from the rpms it generates its own nova.conf that is used instead using https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... as the template which is renderd an copied on container start https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... and kolla provides a config override mechaniume to future customise this if needed. https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.opens...
if you look at the nova and nova-cell role config tasks
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com...
you wil see the list of filets that will be merged to generate the final nova.conf
sources: - "{{ role_path }}/templates/nova.conf.j2" - "{{ node_custom_config }}/global.conf" - "{{ node_custom_config }}/nova.conf" - "{{ node_custom_config }}/nova/{{ item.key }}.conf" - "{{ node_custom_config }}/nova/{{ inventory_hostname }}/nova.conf"
kolla renders its template then merges any seconts form the other files if they exist
{{ node_custom_config }} defaults to /etc/kolla/config
so if you want to make global changes to novas log rotation using the oslo-log log rotation capablitys you would add those options to
/etc/kolla/config/nova.conf
you can do it speratly per serive by creating /etc/kolla/config/nova/nova-compute.conf /etc/kolla/config/nova/nova-api.conf ...
and even per host if you want too.
packaging project should ideally avoid deviating from upstream configuration where possible but in a kolla enve the package cofnigs should be unused.
its possible you are using a very old version of kolla before the log rotate configs where intoduced but correct way to apply config changes form a "deployment-philosophy" point of view is to use the config override mechanium not to do it via packaging changes.
There is no easy way to change that in kolla-ansible based deployments.
Is there any specific reason why default logging has been hardcoded like that? Can we agree to change that situation to be able to override logging as expected?
In general, packages (both for rpm-packaging and RDO) try to set some sensible minimum defaults for a few parameters, which are set according to the base OS requirements. This is why we get those defaults in the rpm-packaging Nova spec.
I think the easiest way to ensure the override in /etc/kolla/config/global.conf is enforced would be to parse it last in the nova-api command-line (with a --config-file directive). Would that be possible in your environment?
Regards, Javier
--
Pozdrawiam serdecznie / Best regards,
Piotr Mossakowski
Lead System Engineer, Product Development Services, Cloud Infra & Applications
Email piotr.mossakowski@tietoevry.com, +48795515407 Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com
Thanks for your replies! @Sean: everything you said is true but the situation I described is happening inside the container so: 1) log rotation is applied on /var/log/kolla on the host but not on /var/log/nova inside the container - and this is the biggest pain as it will finally fill out docker partition on the host. 2) kolla-ansible generates /etc/nova/nova.conf inside the container but openstack-nova RPM injects /etc/nova/nova.conf.d/010-nova.conf and this overrides my customizations from /etc/kolla/config/global.conf @Javier: your idea sounds a lot better than updating a ton of 'extend_start.sh' scripts, thanks! -- Pozdrawiam serdecznie / Best regards, *Piotr Mossakowski* Lead System Engineer, Product Development Services, Cloud Infra & Applications Email piotr.mossakowski@tietoevry.com <mailto:piotr.mossakowski@tietoevry.com>, +48795515407 <tel:%2B48795515407> Aleja Piastów 30, 71-064 Szczecin, Poland, www.tietoevry.com <http://www.tieto.com/>
participants (3)
-
Javier Pena
-
Piotr Mossakowski
-
Sean Mooney