<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">Hi,</p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">On wtorek, 19 kwietnia 2022 22:59:03 CEST Ade Lee wrote:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> Hi all,</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> As many have already seen, a number of changes have been merged in</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> OpenStack as part of the effort to allow OpenStack to run on FIPS enabled</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> systems. This effort has been captured in a proposed community goal. [1].</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> One of the requirements for this effort is that md5sum() not be used in a</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> security related context. In fact, python 3.9 has been modified to raise an</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> exception of hashlib.md5sum() is called on a FIPS enabled system, unless it</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> is explicitly annotated with a usedforsecurity=False attribute [2]. We</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> added a wrapper for md5sum in oslo.config to take advantage of this</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> attribute. [3,4,5]</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> Where we have less control is in libraries used by Openstack - and in</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> particular, paramiko. Paramiko fails on FIPS enabled systems because of a</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> call to md5sum() in get_fingerprint(). A patch has been submitted to fix</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> this problem. [6]. Unfortunately, it takes a very long time for paramiko</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> to fix issues.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> In order for us to make progress on FIPS testing, a small monkey-patch for</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> paramiko was checked into tempest. [7]. Because this change was made to a</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> test tool, this patch was relatively uncontroversial.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> A similar change has been found to be needed for manila [8]. I would</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> expect that a similar change will be needed in other components that use</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> paramiko to SSH to other systems (eg. cinder, neutron?) I suspect that the</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> only reason this has not been detected in FIPS testing more widely yet is</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> because the components that use paramiko for SSH are being tested in third</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> party tests that do not, as yet, test FIPS.</p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">I just checked in Neutron and it seems that we are not using paramiko almost anywhere.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">It is used in neutron-tempest-plugin and in rally tests in neutron-vpnaas.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">Except that it's used in os-ken: <a href="https://opendev.org/openstack/os-ken/src/branch/master/os_ken/services/protocols/bgp/operator/ssh.py">https://opendev.org/openstack/os-ken/src/branch/master/os_ken/services/protocols/bgp/operator/ssh.py</a> but that's something what we in neutron are not using at all so that's why we didn't found it during the FIPS testing.</p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> At the request of the manila team, I am bringing this monkey-patch to the</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> attention of the wider OpenStack community to get feedback on the pros and</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> cons of applying this monkey-patch.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> A couple things to note:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> 1. This monkey patch is quite small in scope and only needed until paramiko</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> fixes the issue.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> 2. paramiko is not FIPS compliant, and so we will ultimately need to fix</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> paramiko or replace it with a different library on FIPS enabled systems.</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> When we do this, we would remove the monkey patch.</p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">IMHO it's ok to temporary use it in testing to be sure that everything else is working fine with FIPS (or fix any other issues which will be there). But in longer term I don't think we can say that e.g. Manila is FIPS compliant if it is using paramiko which isn't FIPS compliant. So it will need to be fixed on the paramiko side or Manila will need to move to some other lib to be FIPS compliant.</p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> Thanks,</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> Ade Lee</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [1]</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://opendev.org/openstack/governance/src/branch/master/goals/proposed/fips.rst</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [2] https://bugs.python.org/issue9216</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [3] https://review.opendev.org/c/openstack/oslo.utils/+/750031</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [4] Patches to various projects to use oslo.utils adapter for hashlib.md5</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> (as examples): glance:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/glance/+/756158 nova:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/nova/+/756434 nova:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/nova/+/777686 os-brick:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/os-brick/+/756151 oslo:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/oslo.versionedobjects/+/756153 tooz:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/tooz/+/756432 opensdk:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/openstacksdk/+/767411 octavia:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/octavia/+/798146 designate:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/designate/+/798157 glance_store:</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/glance_store/+/756157</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [5] Swift patch to handle hashlib.md5</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> https://review.opendev.org/c/openstack/swift/+/751966</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [6] https://github.com/paramiko/paramiko/pull/1928</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [7] https://review.opendev.org/c/openstack/tempest/+/822560</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> [8] https://review.opendev.org/c/openstack/manila/+/819375</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">> </p>
<br /><br /><br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">-- </p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">Slawek Kaplonski</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">Principal Software Engineer</p>
<p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">Red Hat</p>
</body>
</html>