[Openstack-operators] Ubuntu Kernel with Meltdown mitigation SSL issues
sorrison at gmail.com
Thu Jan 18 23:59:28 UTC 2018
We have an F5 doing all the SSL in front of our API servers.
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
The majority of the requests that were failing was a glance request
/v2/images?limit=20 (around 25% of requests which is around 1-2 a second)
Glance is on Ocata.
We also saw the same error on the heat and designate running pike and other
We thought it was to do with low entropy on the control VMs as they were
actually low, however we tweaked this and increased entropy to over 3000
and still had issues.
The underlying hypervisor is also running Xenial and the 4.4.0-109 kernel
but it hasn't got the intel-microcode package installed.
Let me know if anyone wants more details of our setup and I'll happily
On Fri, Jan 19, 2018 at 7:18 AM, Logan V. <logan at protiumit.com> wrote:
> We upgraded our control plane to 4.4.0-109 + intel-microcode
> 3.20180108.0~ubuntu16.04.2 several days ago, and are about 1/2 of the
> way thru upgrading our compute hosts with these changes. We use Ocata
> for all services, and no issue like this has been observed yet on our
> env. Control hosts are E5-2600 V2's and the computes are a mix of
> E5-2600 v2/v3/v4's along with some Xeon D1541's.
> On Thu, Jan 18, 2018 at 2:42 AM, Adam Heczko <aheczko at mirantis.com> wrote:
> > Hello Sam, thank you for sharing this information.
> > Could you please provide more information related to your specific setup.
> > How is Keystone API endpoint TLS terminated in your setup?
> > AFAIK in our OpenStack labs we haven't observed anything like this
> > we terminate TLS on Nginx or HAProxy.
> > On Thu, Jan 18, 2018 at 4:36 AM, Sam Morrison <sorrison at gmail.com>
> >> Hi All,
> >> We updated our control infrastructure to the latest Ubuntu Xenial Kernel
> >> (4.4.0-109) which includes the meltdown fixes.
> >> We have found this kernel to have issues with SSL connections with
> >> and have since downgraded. We get errors like:
> >> SSLError: SSL exception connecting to
> >> https://keystone.example.com:35357/v3/auth/tokens: ("bad handshake:
> >> Error([('', 'osrandom_rand_bytes', 'getrandom() initialization
> >> failed.')],)”,)
> >> Full trace: http://paste.openstack.org/show/646803/
> >> This was affecting glance mainly but all API services were having
> >> Our controllers are running inside KVM VMs and the guests see the CPU as
> >> "Intel Xeon E3-12xx v2 (Ivy Bridge)”
> >> This isn’t an openstack issue specifically but hopefully it helps others
> >> who may be seeing similar issues.
> >> Cheers,
> >> Sam
> >> _______________________________________________
> >> OpenStack-operators mailing list
> >> OpenStack-operators at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> > --
> > Adam Heczko
> > Security Engineer @ Mirantis Inc.
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-operators