<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<tt><font size="-1">Both fernet tokens and JSON Web Tokens (still
under review [0]) require keys to either encrypt or sign the
token payload. These keys are kept on disk by default and are
treated like configuration. During the validation process,
they're loaded from disk, stored as a list, and iterated over
until the correct key decrypts the ciphertext/validates the
signature or the iterable is exhausted [1][2].<br>
<br>
Last week XiYuan raised some concerns about loading all public
keys from disk every time we validate a token [3]. To clarify,
this would be applicable to both fernet keys and asymmetric key
pairs used for JWT. I spent a couple days late last week trying
to recreate the performance bottleneck.<br>
<br>
There were two obvious approaches.<br>
<br>
1. Watch for writable changes to key repositories on disk<br>
<br>
I used a third-party library for this called watchdog [4], but
the inherent downside is that it is specific to file-based key
repositories. For example, this might not work with a proper key
manager, which has been proposed in the past [5]. <br>
<br>
2. Encode a hash of the key used to create the token inside the
token payload<br>
<br>
This would give any keystone server validating the token the
ability preemptively load the correct key for validation instead
of iterating over a list of all possible keys. There was some
hesitation around this approach because the hash would be
visible to anyone with access to the token (it would be outside
of ciphertext in the case of fernet). While hashes are one-way,
it would allow someone to "collect" tokens they know were issued
using the same symmetric private key or signed with the same
asymmetric private key.<br>
<br>
Security concerns aside, I did attempt to write up the second
approach [6]. I was able to get preemptive loading to work, but
I didn't notice a significant performance impact. For
clarification, I was using a single keystone server with one
private key for signing and 100 public keys to simulate a
deployment of 100 API servers that all need to validate tokens
issued by each other. At a certain point, I wondered if the
loading of keys was really the issue, or if it was because we
were iterating over an entire set every time we validate a
token. It's also important to note that I had to turn off
caching to fully test this. Keystone enables caching by default,
which short-circuits the entire key loading/token provider code
path after the token is issued.<br>
<br>
My question is if anyone has additional feedback on how to
recreate performance issues specifically for loading files from
disk, since I wasn't particularly successful in noticing a
difference between repetitive key loading or iterating all keys
on token validation against what we already do.<br>
<br>
<br>
[0] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/614549/">https://review.openstack.org/#/c/614549/</a><br>
[1]
<a class="moz-txt-link-freetext" href="https://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/token_formatters.py?id=053f908853481c00ab28f562a7623f121a7703af#n69">https://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/token_formatters.py?id=053f908853481c00ab28f562a7623f121a7703af#n69</a><br>
[2]
<a class="moz-txt-link-freetext" href="https://github.com/pyca/cryptography/blob/master/src/cryptography/fernet.py#L165-L171">https://github.com/pyca/cryptography/blob/master/src/cryptography/fernet.py#L165-L171</a><br>
[3]
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/614549/13/keystone/token/providers/jws/core.py,unified@103">https://review.openstack.org/#/c/614549/13/keystone/token/providers/jws/core.py,unified@103</a><br>
[4] <a class="moz-txt-link-freetext" href="https://pythonhosted.org/watchdog/">https://pythonhosted.org/watchdog/</a><br>
[5] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/363065/">https://review.openstack.org/#/c/363065/</a><br>
[6] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/636151/">https://review.openstack.org/#/c/636151/</a><br>
</font></tt>
</body>
</html>