[openstack-dev] [nova] Do we still want to lowercase metadata keys?

Matthew Booth mbooth at redhat.com
Mon Aug 13 14:10:44 UTC 2018


On Mon, 13 Aug 2018 at 14:05, Jay Pipes <jaypipes at gmail.com> wrote:
>
> On 08/13/2018 06:06 AM, Matthew Booth wrote:
> > Thanks mriedem for answering my previous question, and also pointing
> > out the related previous spec around just forcing all metadata to be
> > lowercase:
> >
> > (Spec: approved in Newton) https://review.openstack.org/#/c/311529/
> > (Online migration: not merged, abandoned)
> > https://review.openstack.org/#/c/329737/
> >
> > There are other code patches, but the above is representative. What I
> > had read was the original bug:
> >
> > https://bugs.launchpad.net/nova/+bug/1538011
> >
> > The tl;dr is that the default collation used by MySQL results in a bug
> > when creating 2 metadata keys which differ only in case. The proposal
> > was obviously to simply make all metadata keys lower case. However, as
> > melwitt pointed out in the bug at the time that's a potentially user
> > hostile change. After some lost IRC discussion it seems that folks
> > believed at the time that to fix this properly would seriously
> > compromise the performance of these queries. The agreed way forward
> > was to allow existing keys to keep their case, but force new keys to
> > be lower case (so I wonder how the above online migration came
> > about?).
> >
> > Anyway, as Rajesh's patch shows, it's actually very easy just to fix
> > the MySQL misconfiguration:
> >
> > https://review.openstack.org/#/c/504885/
> >
> > So my question is, given that the previous series remains potentially
> > user hostile, the fix isn't as complex as previously believed, and it
> > doesn't involve a performance penalty, are there any other reasons why
> > we might want to resurrect it rather than just go with Rajesh's patch?
> > Or should we ask Rajesh to expand his patch into a series covering
> > other metadata?
>
> Keep in mind this patch is only related to *aggregate* metadata, AFAICT.

Right, but the original bug pointed out that the same problem applies
equally to a bunch of different metadata stores. I haven't verified,
but the provenance was good ;) There would have to be other patches
for the other metadata stores.

>
> Any patch series that tries to "fix" this issue needs to include all of
> the following:
>
> * input automatically lower-cased [1]
> * inline (note: not online, inline) data migration inside the
> InstanceMeta object's _from_db_object() method for existing
> non-lowercased keys

I suspect I've misunderstood, but I was arguing this is an anti-goal.
There's no reason to do this if the db is working correctly, and it
would violate the principal of least surprise in dbs with legacy
datasets (being all current dbs). These values have always been mixed
case, lets just leave them be and fix the db.

> * change the collation of the aggregate_metadata.key column (note: this
> will require an entire rebuild of the table, since this column is part
> of a unique constraint [3]

Rajesh's patch changes the collation of the table, which I would
assume applies to its columns? I assume this is going to be a
moderately expensive, but one-off, operation similar in cost to adding
a new unique constraint.

> * online data migration for migrating non-lowercased keys to their
> lowercased counterpars (essentially doing `UPDATE key = LOWER(key) WHERE
> LOWER(key) != key` once the collation has been changed)

> None of the above touches the API layer. I suppose some might argue that
> the REST API should be microversion-bumped since the expected behaviour
> of the API will change (data will be transparently changed in one
> version of the API and not another). I don't personally think that's
> something I would require a microversion for, but who knows what others
> may say.

Again, I was considering this is an anti-goal. As I understand,
Rajesh's patch removes the requirement to make this api change. What
did I miss?

Thanks,

Matt

>
> Best,
> -jay
>
> [1]
> https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L295
> and
> https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L331
> and
> https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L356
>
>
> [2]
> https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/objects/aggregate.py#L248
>
> [3]
> https://github.com/openstack/nova/blob/16f89fd093217d22530570e8277b561ea79f46ff/nova/db/sqlalchemy/api_models.py#L64
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Matthew Booth
Red Hat OpenStack Engineer, Compute DFG

Phone: +442070094448 (UK)



More information about the OpenStack-dev mailing list