Open Stack

Hey all,
   Recently we found an issue again, related to our nova
metadata/system_metadata update.
   This bug already track in this link: https://bugs.launchpad.
net/nova/+bug/1650188 .

   1. For metadata update.
   As we know, the current mechism of metdata as below sequence diagram.
[image: 内嵌图片 1]
   Step [1]: We get instance object(include metadata) in API, we call this
object as *metadata(A)*
   link:
https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/api/openstack/compute/server_metadata.py#L105

   Step [2]: Update metadata object using metadata(A), and we plus the
increacement change.
   link:
https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/compute/api.py#L4014-L4022

   Step [3]: When instance.save, we get instance *metadata(B)* in DB
   link:
https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/db/sqlalchemy/api.py#L2819

   Step [4]: flush change in db: *metadata(A) + increacement - metadata(B)*
   link:
https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/db/sqlalchemy/api.py#L2750-L2778

   As note on diagram:
  In normal case: if metadata(A) == metadata(B), there is no impact on
result.

  However, in concurrent case: we udpate metadata concurrently, metadata(A)
!= metadata(B) in some case,
  because we perhaps already change the metadata between API  step [1] and
step [3] DB, we will lose these changes.

  some discussions in bug link, the ETAG, generation ID and CAS are
mentioned in our discussion.
  *1. ETAG*: the way to avoid API race condition, if check etag failed will
give back a 409 confict. see:
  https://review.openstack.org/#/c/328399/

  *2. Generation ID:* like resource provider done. see:

https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/objects/resource_provider.py#L281
  but it seems we should add this filed the instance object?

 * 3. CAS: *do some pre condition check when we write the record in db. see
instance done:
  https://review.openstack.org/#/c/202593/

2. For system_metadata update.
  it has same problem with metadata update,
  but the difference is the "first get" stuff doesn't happen in API, maybe
in compute or virt driver.
  That is, the ETAG, the way to avoid API race condition, is not suitable
for system_metadata.

Any idea on this issue, pls feel free to append your comments, we need your
advice and help.

Regards,
Yikun
----------------------------------------
Jiang Yikun（Kero）
Mail: yikunkero at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180122/bc9f026e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Nova Metadata update.jpg
Type: image/jpeg
Size: 157958 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180122/bc9f026e/attachment-0001.jpg>