Open Stack

On 05/10/2018 08:52 AM, Matt Riedemann wrote:
> On 5/9/2018 8:11 PM, Jean-Philippe Méthot wrote:
>> I currently operate a multi-region cloud split between 2 geographic
>> locations. I have updated it to Pike not too long ago, but I've been
>> running into a peculiar issue. Ever since the Pike release, Nova now
>> asks Keystone if a new project exists in Keystone before configuring
>> the project’s quotas. However, there doesn’t seem to be any region
>> restriction regarding which endpoint Nova will query Keystone on. So,
>> right now, if I create a new project in region one, Nova will query
>> Keystone in region two. Because my keystone databases are not synched
>> in real time between each region, the region two Keystone will tell
>> it that the new project doesn't exist, while it exists in region one
>> Keystone.
Are both keystone nodes completely separate? Do they share any information?
>>
>> Thinking that this could be a configuration error, I tried setting
>> the region_name in keystone_authtoken, but that didn’t change much of
>> anything. Right now I am thinking this may be a bug. Could someone
>> confirm that this is indeed a bug and not a configuration error?
>>
>> To circumvent this issue, I am considering either modifying the
>> database by hand or trying to implement realtime replication between
>> both Keystone databases. Would there be another solution? (beside
>> modifying the code for the Nova check)
A variant of this just came up as a proposal for the Forum in a couple
weeks [0]. A separate proposal was also discussed during this week's
keystone meeting [1], which brought up an interesting solution. We
should be seeing a specification soon that covers the proposal in
greater detail and includes use cases. Either way, both sound like they
may be relevant to you.

[0] https://etherpad.openstack.org/p/YVR-edge-keystone-brainstorming
[1]
http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-05-08-16.00.log.html#l-156
>
> This is the specific code you're talking about:
>
> https://github.com/openstack/nova/blob/stable/pike/nova/api/openstack/identity.py#L35
>
>
> I don't see region_name as a config option for talking to keystone in
> Pike:
>
> https://docs.openstack.org/nova/pike/configuration/config.html#keystone
>
> But it is in Queens:
>
> https://docs.openstack.org/nova/queens/configuration/config.html#keystone
>
> That was added in this change:
>
> https://review.openstack.org/#/c/507693/
>
> But I think what you're saying is, since you have multiple regions,
> the project could be in any of them at any given time until they
> synchronize so configuring nova for a specific region isn't probably
> going to help in this case, right?
>
> Isn't this somehow resolved with keystone federation? Granted, I'm not
> at all a keystone person, but I'd think this isn't a unique problem.
Without knowing a whole lot about the current setup, I'm inclined to say
it is. Keystone-to-keystone federation was developed for cases like
this, and it's been something we've been trying to encourage in favor of
building replication tooling outside of the database or over an API. The
main concerns with taking a manual replication approach is that it could
negatively impact overall performance and that keystone already assumes
it will be in control of ID generation for most cases (replicating a
project in RegionOne into RegionTwo will yield a different project ID,
even though it is possible for both to have the same name).
Additionally, there are some things that keystone doesn't expose over
the API that would need to be replicated, like revocation events (I
mentioned this in the etherpad linked above).


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20180510/9d6f08ff/attachment.sig>