Open Stack

We have an F5 doing all the SSL in front of our API servers.

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

The majority of the requests that were failing was a glance request
/v2/images?limit=20 (around 25% of requests which is around 1-2 a second)
Glance is on Ocata.

We also saw the same error on the heat and designate running pike and other
services.

We thought it was to do with low entropy on the control VMs as they were
actually low, however we tweaked this and increased entropy to over 3000
and still had issues.

The underlying hypervisor is also running Xenial and the 4.4.0-109 kernel
but it hasn't got the intel-microcode package installed.

Let me know if anyone wants more details of our setup and I'll happily
provide.


Cheers,
Sam




On Fri, Jan 19, 2018 at 7:18 AM, Logan V. <logan at protiumit.com> wrote:

> We upgraded our control plane to 4.4.0-109 + intel-microcode
> 3.20180108.0~ubuntu16.04.2 several days ago, and are about 1/2 of the
> way thru upgrading our compute hosts with these changes. We use Ocata
> for all services, and no issue like this has been observed yet on our
> env. Control hosts are E5-2600 V2's and the computes are a mix of
> E5-2600 v2/v3/v4's along with some Xeon D1541's.
>
> On Thu, Jan 18, 2018 at 2:42 AM, Adam Heczko <aheczko at mirantis.com> wrote:
> > Hello Sam, thank you for sharing this information.
> > Could you please provide more information related to your specific setup.
> > How is Keystone API endpoint TLS terminated in your setup?
> > AFAIK in our OpenStack labs we haven't observed anything like this
> although
> > we terminate TLS on Nginx or HAProxy.
> >
> >
> > On Thu, Jan 18, 2018 at 4:36 AM, Sam Morrison <sorrison at gmail.com>
> wrote:
> >>
> >> Hi All,
> >>
> >> We updated our control infrastructure to the latest Ubuntu Xenial Kernel
> >> (4.4.0-109) which includes the meltdown fixes.
> >>
> >> We have found this kernel to have issues with SSL connections with
> python
> >> and have since downgraded. We get errors like:
> >>
> >> SSLError: SSL exception connecting to
> >> https://keystone.example.com:35357/v3/auth/tokens: ("bad handshake:
> >> Error([('', 'osrandom_rand_bytes', 'getrandom() initialization
> >> failed.')],)”,)
> >>
> >> Full trace:  http://paste.openstack.org/show/646803/
> >>
> >> This was affecting glance mainly but all API services were having
> issues.
> >>
> >> Our controllers are running inside KVM VMs and the guests see the CPU as
> >> "Intel Xeon E3-12xx v2 (Ivy Bridge)”
> >>
> >> This isn’t an openstack issue specifically but hopefully it helps others
> >> who may be seeing similar issues.
> >>
> >>
> >> Cheers,
> >> Sam
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> OpenStack-operators mailing list
> >> OpenStack-operators at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> >
> >
> >
> > --
> > Adam Heczko
> > Security Engineer @ Mirantis Inc.
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180119/0c3dd62e/attachment.html>