[openstack-dev] [nova] Do we still want to lowercase metadata keys?

Jay Pipes jaypipes at gmail.com
Mon Aug 13 13:04:43 UTC 2018


On 08/13/2018 06:06 AM, Matthew Booth wrote:
> Thanks mriedem for answering my previous question, and also pointing
> out the related previous spec around just forcing all metadata to be
> lowercase:
> 
> (Spec: approved in Newton) https://review.openstack.org/#/c/311529/
> (Online migration: not merged, abandoned)
> https://review.openstack.org/#/c/329737/
> 
> There are other code patches, but the above is representative. What I
> had read was the original bug:
> 
> https://bugs.launchpad.net/nova/+bug/1538011
> 
> The tl;dr is that the default collation used by MySQL results in a bug
> when creating 2 metadata keys which differ only in case. The proposal
> was obviously to simply make all metadata keys lower case. However, as
> melwitt pointed out in the bug at the time that's a potentially user
> hostile change. After some lost IRC discussion it seems that folks
> believed at the time that to fix this properly would seriously
> compromise the performance of these queries. The agreed way forward
> was to allow existing keys to keep their case, but force new keys to
> be lower case (so I wonder how the above online migration came
> about?).
> 
> Anyway, as Rajesh's patch shows, it's actually very easy just to fix
> the MySQL misconfiguration:
> 
> https://review.openstack.org/#/c/504885/
> 
> So my question is, given that the previous series remains potentially
> user hostile, the fix isn't as complex as previously believed, and it
> doesn't involve a performance penalty, are there any other reasons why
> we might want to resurrect it rather than just go with Rajesh's patch?
> Or should we ask Rajesh to expand his patch into a series covering
> other metadata?

Keep in mind this patch is only related to *aggregate* metadata, AFAICT.

Any patch series that tries to "fix" this issue needs to include all of 
the following:

* input automatically lower-cased [1]
* inline (note: not online, inline) data migration inside the 
InstanceMeta object's _from_db_object() method for existing 
non-lowercased keys
* change the collation of the aggregate_metadata.key column (note: this 
will require an entire rebuild of the table, since this column is part 
of a unique constraint [3]
* online data migration for migrating non-lowercased keys to their 
lowercased counterpars (essentially doing `UPDATE key = LOWER(key) WHERE 
LOWER(key) != key` once the collation has been changed)

None of the above touches the API layer. I suppose some might argue that 
the REST API should be microversion-bumped since the expected behaviour 
of the API will change (data will be transparently changed in one 
version of the API and not another). I don't personally think that's 
something I would require a microversion for, but who knows what others 
may say.

Best,
-jay

[1] 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L295 
and 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L331 
and 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L356 


[2] 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L248

[3] 
https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/db/sqlalchemy/api_models.py#L64



More information about the OpenStack-dev mailing list