[security-sig] Log4j vulnerabilities and OpenStack
Unless you were living under a rock during most of December, you've almost certainly seen the press surrounding the various security vulnerabilities discovered in the Apache Log4j Java library. As everyone reading this list hopefully knows, OpenStack is primarily written in Python, so has little use for Java libraries in the first place, but that hasn't stopped users from asking our VMT members if OpenStack is affected. While OpenStack doesn't require any Java components, I'm aware of one Neutron driver (networking-odl) which relies on an affected third-party service: https://access.redhat.com/solutions/6586821 Additionally, "storm" component of SUSE OpenStack seems to be impacted: https://www.suse.com/c/suse-statement-on-log4j-log4shell-cve-2021-44228-vuln... As does an Elasticsearch component in Sovereign Cloud Stack: https://scs.community/security/2021/12/13/advisory-log4j/ Users should, obviously, rely on their distribution vendors/suppliers to notify them of available updates for these. Is anyone aware of other, similar situations where OpenStack is commonly installed alongside Java software using Log4j in vulnerable ways? -- Jeremy Stanley
On 1/3/22 10:02, Jeremy Stanley wrote:
Unless you were living under a rock during most of December, you've almost certainly seen the press surrounding the various security vulnerabilities discovered in the Apache Log4j Java library. As everyone reading this list hopefully knows, OpenStack is primarily written in Python, so has little use for Java libraries in the first place, but that hasn't stopped users from asking our VMT members if OpenStack is affected.
While OpenStack doesn't require any Java components, I'm aware of one Neutron driver (networking-odl) which relies on an affected third-party service: https://access.redhat.com/solutions/6586821
Additionally, "storm" component of SUSE OpenStack seems to be impacted: https://www.suse.com/c/suse-statement-on-log4j-log4shell-cve-2021-44228-vuln...
As does an Elasticsearch component in Sovereign Cloud Stack: https://scs.community/security/2021/12/13/advisory-log4j/
Users should, obviously, rely on their distribution vendors/suppliers to notify them of available updates for these. Is anyone aware of other, similar situations where OpenStack is commonly installed alongside Java software using Log4j in vulnerable ways?
I don't know if this is common, but if you use Zookeeper for DLM I assume you'd be affected. It's a supported driver in Tooz so it's possible someone would be using it.
On 2022-01-06 10:31:34 -0600 (-0600), Ben Nemec wrote: [...]
I don't know if this is common, but if you use Zookeeper for DLM I assume you'd be affected. It's a supported driver in Tooz so it's possible someone would be using it.
Thanks, that's a good point! I recall when we were investigating it with regard to Zuul (which relies on ZK for state coordination and persistence), the conclusion was that it isn't impacted by the recent vulnerabilities. I found this brief explanation, but maybe that's outdated information? https://issues.apache.org/jira/browse/ZOOKEEPER-4423 -- Jeremy Stanley
On 1/6/22 10:40, Jeremy Stanley wrote:
On 2022-01-06 10:31:34 -0600 (-0600), Ben Nemec wrote: [...]
I don't know if this is common, but if you use Zookeeper for DLM I assume you'd be affected. It's a supported driver in Tooz so it's possible someone would be using it.
Thanks, that's a good point! I recall when we were investigating it with regard to Zuul (which relies on ZK for state coordination and persistence), the conclusion was that it isn't impacted by the recent vulnerabilities. I found this brief explanation, but maybe that's outdated information? https://issues.apache.org/jira/browse/ZOOKEEPER-4423
Ah, so zookeeper was one of the projects using a version of log4j so ancient it wasn't affected. :-) I was just thinking of Java stuff that might be running alongside OpenStack, I don't know anything that contradicts the issue you linked.
On 2022-01-03 16:02:14 +0000 (+0000), Jeremy Stanley wrote: [...]
Is anyone aware of other, similar situations where OpenStack is commonly installed alongside Java software using Log4j in vulnerable ways?
It came to my attention a few moments ago that Kolla installs Elasticsearch[*]. Is there any particular guidance we should be giving Kolla users about mitigating the recent Log4j vulnerabilities in light of this? [*] https://docs.openstack.org/kolla-ansible/latest/reference/logging-and-monito... -- Jeremy Stanley
On Mon, 10 Jan 2022 at 14:42, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-03 16:02:14 +0000 (+0000), Jeremy Stanley wrote: [...]
Is anyone aware of other, similar situations where OpenStack is commonly installed alongside Java software using Log4j in vulnerable ways?
It came to my attention a few moments ago that Kolla installs Elasticsearch[*]. Is there any particular guidance we should be giving Kolla users about mitigating the recent Log4j vulnerabilities in light of this?
Yes, we have already patched the command line [1] so the guidance is to make sure to run the latest and greatest. It would make sense to broadcast this so that users know that log4j is in Elasticsearch. In Kolla, ES is used either standalone or with Monasca (and soon Venus). [1] https://review.opendev.org/c/openstack/kolla-ansible/+/821860 -yoctozepto
[*] https://docs.openstack.org/kolla-ansible/latest/reference/logging-and-monito...
-- Jeremy Stanley
On 2022-01-10 14:47:53 +0100 (+0100), Radosław Piliszek wrote: [...]
Yes, we have already patched the command line [1] so the guidance is to make sure to run the latest and greatest. It would make sense to broadcast this so that users know that log4j is in Elasticsearch. In Kolla, ES is used either standalone or with Monasca (and soon Venus).
[1] https://review.opendev.org/c/openstack/kolla-ansible/+/821860 [...]
Is the presence/absence of Elasticsearch determined by configuration options, or is it always installed and run when Kolla is used? -- Jeremy Stanley
On Mon, 10 Jan 2022 at 14:58, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 14:47:53 +0100 (+0100), Radosław Piliszek wrote: [...]
Yes, we have already patched the command line [1] so the guidance is to make sure to run the latest and greatest. It would make sense to broadcast this so that users know that log4j is in Elasticsearch. In Kolla, ES is used either standalone or with Monasca (and soon Venus).
[1] https://review.opendev.org/c/openstack/kolla-ansible/+/821860 [...]
Is the presence/absence of Elasticsearch determined by configuration options, or is it always installed and run when Kolla is used?
Determined by configuration. It is not present by default - only if installed on demand, by enabling central logging, Monasca or some other dependent component. -yoctozepto
-- Jeremy Stanley
On 2022-01-10 15:44:41 +0100 (+0100), Radosław Piliszek wrote:
On Mon, 10 Jan 2022 at 14:58, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 14:47:53 +0100 (+0100), Radosław Piliszek wrote: [...]
Yes, we have already patched the command line [1] so the guidance is to make sure to run the latest and greatest. It would make sense to broadcast this so that users know that log4j is in Elasticsearch. In Kolla, ES is used either standalone or with Monasca (and soon Venus).
[1] https://review.opendev.org/c/openstack/kolla-ansible/+/821860 [...]
Is the presence/absence of Elasticsearch determined by configuration options, or is it always installed and run when Kolla is used?
Determined by configuration. It is not present by default - only if installed on demand, by enabling central logging, Monasca or some other dependent component.
Thanks for the details, and apologies for all the sudden questions. Is there a list of which components (aside from the aforementioned central logging and Monasca) which pull Elasticsearch into the deployment? Also, does Kolla build/distribute its own Elasticsearch images or reuse some maintained by an outside party? And what version(s) of Elasticsearch and Log4j end up installed? -- Jeremy Stanley
On Mon, 10 Jan 2022 at 17:28, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 15:44:41 +0100 (+0100), Radosław Piliszek wrote:
On Mon, 10 Jan 2022 at 14:58, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 14:47:53 +0100 (+0100), Radosław Piliszek wrote: [...]
Yes, we have already patched the command line [1] so the guidance is to make sure to run the latest and greatest. It would make sense to broadcast this so that users know that log4j is in Elasticsearch. In Kolla, ES is used either standalone or with Monasca (and soon Venus).
[1] https://review.opendev.org/c/openstack/kolla-ansible/+/821860 [...]
Is the presence/absence of Elasticsearch determined by configuration options, or is it always installed and run when Kolla is used?
Determined by configuration. It is not present by default - only if installed on demand, by enabling central logging, Monasca or some other dependent component.
Thanks for the details, and apologies for all the sudden questions. Is there a list of which components (aside from the aforementioned central logging and Monasca) which pull Elasticsearch into the deployment?
Yeah, I just omitted them as they are less frequently used: - osprofiler - skydive - cloudkitty configured to use elasticsearch backend
Also, does Kolla build/distribute its own Elasticsearch images or reuse some maintained by an outside party? And what version(s) of Elasticsearch and Log4j end up installed?
Kolla builds its own images using upstream (ES) rpm/deb packages. We end up with the latest 7.13.x as that's the last truly OSS release of ES. I know it has vulnerable log4j but can't tell the version atm. Let me know if it's crucial. -yoctozepto
-- Jeremy Stanley
On Mon, 10 Jan 2022 at 18:05, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
On Mon, 10 Jan 2022 at 17:28, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 15:44:41 +0100 (+0100), Radosław Piliszek wrote:
On Mon, 10 Jan 2022 at 14:58, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 14:47:53 +0100 (+0100), Radosław Piliszek wrote: [...]
Yes, we have already patched the command line [1] so the guidance is to make sure to run the latest and greatest. It would make sense to broadcast this so that users know that log4j is in Elasticsearch. In Kolla, ES is used either standalone or with Monasca (and soon Venus).
[1] https://review.opendev.org/c/openstack/kolla-ansible/+/821860 [...]
Is the presence/absence of Elasticsearch determined by configuration options, or is it always installed and run when Kolla is used?
Determined by configuration. It is not present by default - only if installed on demand, by enabling central logging, Monasca or some other dependent component.
Thanks for the details, and apologies for all the sudden questions. Is there a list of which components (aside from the aforementioned central logging and Monasca) which pull Elasticsearch into the deployment?
Yeah, I just omitted them as they are less frequently used: - osprofiler - skydive - cloudkitty configured to use elasticsearch backend
Also, does Kolla build/distribute its own Elasticsearch images or reuse some maintained by an outside party? And what version(s) of Elasticsearch and Log4j end up installed?
Kolla builds its own images using upstream (ES) rpm/deb packages. We end up with the latest 7.13.x as that's the last truly OSS release of ES. I know it has vulnerable log4j but can't tell the version atm. Let me know if it's crucial.
For CentOS images, this is bundled into elasticsearch-oss-7.10.2-1.x86_64: /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar Note that according to Elastic, this version is not vulnerable thanks to the use of the Java Security Manager.
On 2022-01-10 18:10:19 +0100 (+0100), Pierre Riteau wrote: [...]
For CentOS images, this is bundled into elasticsearch-oss-7.10.2-1.x86_64:
/usr/share/elasticsearch/lib/log4j-api-2.11.1.jar /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
Note that according to Elastic, this version is not vulnerable thanks to the use of the Java Security Manager.
Thanks! Was there a public statement from Elastic to that effect, so that we can point users at it if they have questions? At this point a lot of enterprises are ripping out or shutting down anything which can't be upgraded to Log4j 2.17.1, due in part to the mixed messages about which older versions are actually impacted and which workarounds can mitigate it. -- Jeremy Stanley
On Mon, 10 Jan 2022 at 18:15, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 18:10:19 +0100 (+0100), Pierre Riteau wrote: [...]
For CentOS images, this is bundled into elasticsearch-oss-7.10.2-1.x86_64:
/usr/share/elasticsearch/lib/log4j-api-2.11.1.jar /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
Note that according to Elastic, this version is not vulnerable thanks to the use of the Java Security Manager.
Thanks! Was there a public statement from Elastic to that effect, so that we can point users at it if they have questions?
https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnera... -yoctozepto
At this point a lot of enterprises are ripping out or shutting down anything which can't be upgraded to Log4j 2.17.1, due in part to the mixed messages about which older versions are actually impacted and which workarounds can mitigate it. -- Jeremy Stanley
On Mon, 10 Jan 2022 at 18:18, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-10 18:10:19 +0100 (+0100), Pierre Riteau wrote: [...]
For CentOS images, this is bundled into elasticsearch-oss-7.10.2-1.x86_64:
/usr/share/elasticsearch/lib/log4j-api-2.11.1.jar /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
Note that according to Elastic, this version is not vulnerable thanks to the use of the Java Security Manager.
Thanks! Was there a public statement from Elastic to that effect, so that we can point users at it if they have questions?
At this point a lot of enterprises are ripping out or shutting down anything which can't be upgraded to Log4j 2.17.1, due in part to the mixed messages about which older versions are actually impacted and which workarounds can mitigate it. -- Jeremy Stanley
https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnera...
Thanks to the excellent feedback from Radosław Piliszek and Pierre Riteau, the list of things for operators to look out for has grown a bit. Is anyone else aware of other, similar situations where OpenStack is commonly installed alongside Java software using Log4j in vulnerable ways? I think this list is becoming extensive enough we could consider publishing it in a security note (OSSN)... Kolla-Ansible Central Logging ----------------------------- If you're deploying with Kolla-Ansible and have enabled central logging, then it's installing a copy of Elasticsearch (7.13.4 currently, which includes Log4j 2.11.1). According to a statement from Elastic's developers, the relevant risks can be mitigated by passing "-Dlog4j2.formatMsgNoLookups=true" on the JVM's command line. All images built after December 21, 2021 have this workaround applied, with the exception of images for Train which did not get that patch merged until January 7, 2021. The statement from Elastic about the workaround can be found here: https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasticsearch/ CloudKitty, Monasca, and OSProfiler ----------------------------------- If you're deploying CloudKitty, Monasca, or OSProfiler, you may be using Elasticsearch as a storage back-end for these services. Make sure you update it or put a suitable mitigation in place. Anyone deploying one or more of these services with Kolla-Ansible is running Elasticsearch, but should be covered so long as they update to the latest available images for their release series, as noted above. Networking-ODL -------------- Neutron's Networking-ODL driver relies on the Java-based OpenDaylight service, which should be updated if used: https://access.redhat.com/solutions/6586821 SUSE OpenStack -------------- The "storm" component of SUSE OpenStack seems to be impacted: https://www.suse.com/c/suse-statement-on-log4j-log4shell-cve-2021-44228-vuln... Sovereign Cloud Stack --------------------- An Elasticsearch component in Sovereign Cloud Stack is affected: https://scs.community/security/2021/12/13/advisory-log4j/ -- Jeremy Stanley
Thank you Jeremy for putting all this information together. Some important comments inline. On Thu, 13 Jan 2022 at 19:49, Jeremy Stanley <fungi@yuggoth.org> wrote:
Thanks to the excellent feedback from Radosław Piliszek and Pierre Riteau, the list of things for operators to look out for has grown a bit. Is anyone else aware of other, similar situations where OpenStack is commonly installed alongside Java software using Log4j in vulnerable ways? I think this list is becoming extensive enough we could consider publishing it in a security note (OSSN)...
Kolla-Ansible Central Logging -----------------------------
If you're deploying with Kolla-Ansible and have enabled central logging, then it's installing a copy of Elasticsearch (7.13.4 currently, which includes Log4j 2.11.1). According to a statement from Elastic's developers, the relevant risks can be mitigated by passing "-Dlog4j2.formatMsgNoLookups=true" on the JVM's command line. All images built after December 21, 2021 have this workaround applied, with the exception of images for Train which did not get that patch merged until January 7, 2021. The statement from Elastic about the workaround can be found here: https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasticsearch/
This part has several issues: - Xena and Wallaby use Elasticsearch OSS 7.x, the latest available package being version 7.10.2 from January 2021, which includes Log4j 2.11.1. Unfortunately it doesn't look like it will ever be updated, so we only have the formatMsgNoLookups to help. The Kolla community is planning to move to OpenSearch as a replacement for Elasticsearch. - Victoria and earlier releases use Elasticsearch OSS 6.x which still gets updated packages. Latest Kolla images for these releases use version 6.8.22, which includes Log4j 2.17.0. I see that Elasticsearch 6.8.23 was released today, which includes Log4j 2.17.1. This should get picked up by Kolla image builds in the coming days. - The formatMsgNoLookups mitigation is applied through Kolla Ansible and doesn't depend on when Kolla images were built. The dates you mentioned above are when the commits merged in Kolla Ansible.
CloudKitty, Monasca, and OSProfiler -----------------------------------
If you're deploying CloudKitty, Monasca, or OSProfiler, you may be using Elasticsearch as a storage back-end for these services. Make sure you update it or put a suitable mitigation in place. Anyone deploying one or more of these services with Kolla-Ansible is running Elasticsearch, but should be covered so long as they update to the latest available images for their release series, as noted above.
As noted in the previous comment, for Xena and Wallaby we don't have a fix using new images, only the mitigation applied through Kolla Ansible is available. Additional Monasca components that include Log4j: - Logstash: similar vulnerability pattern to Elasticsearch, i.e. no RCE out of the box - Kafka: uses an older Log4j 1.x which is only vulnerable to CVE-2021-4104 if configured to use JMSAppender (not the default), according to https://kafka.apache.org/cve-list - Zookeeper: uses Log4j 1.x, not affected by CVE-2021-44228 according to https://blogs.apache.org/security/entry/cve-2021-44228, not sure about the others - Storm: possibly vulnerable? Pull requests in github.com/apache/storm have bumped Log4j versions, but no new release has been issued yet. Kolla uses version 1.2.2. I am looking at adding a mitigation for CVE-2021-45046 based on removing the JndiLookup class from the classpath.
Networking-ODL --------------
Neutron's Networking-ODL driver relies on the Java-based OpenDaylight service, which should be updated if used: https://access.redhat.com/solutions/6586821
SUSE OpenStack --------------
The "storm" component of SUSE OpenStack seems to be impacted: https://www.suse.com/c/suse-statement-on-log4j-log4shell-cve-2021-44228-vuln...
Sovereign Cloud Stack ---------------------
An Elasticsearch component in Sovereign Cloud Stack is affected: https://scs.community/security/2021/12/13/advisory-log4j/
-- Jeremy Stanley
On 2022-01-13 22:17:43 +0100 (+0100), Pierre Riteau wrote: [...]
This part has several issues: [...]
Thanks for the detailed breakdown! I'll try to come up with a summary which retains accuracy while focusing on actionable recommendations, though I'll need to go over it a few more times and think on it for a bit before I can put together a new draft.
- Storm: possibly vulnerable? Pull requests in github.com/apache/storm have bumped Log4j versions, but no new release has been issued yet. Kolla uses version 1.2.2. I am looking at adding a mitigation for CVE-2021-45046 based on removing the JndiLookup class from the classpath. [...]
Could that be the same as this?
SUSE OpenStack --------------
The "storm" component of SUSE OpenStack seems to be impacted: https://www.suse.com/c/suse-statement-on-log4j-log4shell-cve-2021-44228-vuln... [...]
-- Jeremy Stanley
On Thu, 13 Jan 2022 at 22:30, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-13 22:17:43 +0100 (+0100), Pierre Riteau wrote: [...]
This part has several issues: [...]
Thanks for the detailed breakdown! I'll try to come up with a summary which retains accuracy while focusing on actionable recommendations, though I'll need to go over it a few more times and think on it for a bit before I can put together a new draft.
- Storm: possibly vulnerable? Pull requests in github.com/apache/storm have bumped Log4j versions, but no new release has been issued yet. Kolla uses version 1.2.2. I am looking at adding a mitigation for CVE-2021-45046 based on removing the JndiLookup class from the classpath. [...]
Could that be the same as this?
I believe so. This lead me to [1] and [2] which have more details. SUSE opted to remove the JndiLookup class from log4j 2.x jars during build. I've actually already submitted a Kolla patch to apply the same mitigation: https://review.opendev.org/c/openstack/kolla/+/824651 [1] https://lists.suse.com/pipermail/sle-security-updates/2021-December/009911.h... [2] https://bugzilla.suse.com/show_bug.cgi?id=1193641
SUSE OpenStack --------------
The "storm" component of SUSE OpenStack seems to be impacted: https://www.suse.com/c/suse-statement-on-log4j-log4shell-cve-2021-44228-vuln... [...]
-- Jeremy Stanley
participants (4)
-
Ben Nemec
-
Jeremy Stanley
-
Pierre Riteau
-
Radosław Piliszek