<div dir="ltr"><div>Hey all,<br></div><div><div> Recently we found an issue again, related to our nova metadata/system_metadata update.</div><div> This bug already track in this link: <a href="https://bugs.launchpad.net/nova/+bug/1650188" target="_blank">https://bugs.launchpad.<wbr>net/nova/+bug/1650188</a> .</div><div> </div><div> 1. For metadata update.</div><div> As we know, the current mechism of metdata as below sequence diagram.</div></div><div><img src="cid:ii_1611bb7c43ab6dbd" alt="内嵌图片 1" width="514" height="562"><br></div><div><div> Step [1]: We get instance object(include metadata) in API, we call this object as <b>metadata(A)</b></div><div> link: <a href="https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/api/openstack/compute/server_metadata.py#L105">https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/api/openstack/compute/server_metadata.py#L105</a><b><br></b></div><div><br></div><div> Step [2]: Update metadata object using metadata(A), and we plus the increacement change.</div><div> link: <a href="https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/compute/api.py#L4014-L4022">https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/compute/api.py#L4014-L4022</a></div><div><br></div><div> Step [3]: When instance.save, we get instance <b>metadata(B)</b> in DB</div><div> link: <a href="https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/db/sqlalchemy/api.py#L2819">https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/db/sqlalchemy/api.py#L2819</a><br></div><div><br></div><div> Step [4]: flush change in db: <b>metadata(A) + increacement - metadata(B)</b></div><div> link: <a href="https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/db/sqlalchemy/api.py#L2750-L2778">https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/db/sqlalchemy/api.py#L2750-L2778</a></div><div><br></div><div> As note on diagram:</div></div><div><div> In normal case: if metadata(A) == metadata(B), there is no impact on result.</div><div><br></div><div> However, in concurrent case: we udpate metadata concurrently, metadata(A) != metadata(B) in some case,</div><div> because we perhaps already change the metadata between API step [1] and step [3] DB, we will lose these changes.</div></div><div><br></div><div> some discussions in bug link, the ETAG, generation ID and CAS are mentioned in our discussion.</div><div> <b>1. ETAG</b>: the way to avoid API race condition, if check etag failed will give back a 409 confict. see: </div><div> <a href="https://review.openstack.org/#/c/328399/">https://review.openstack.org/#/c/328399/</a></div><div><br></div><div> <b>2. Generation ID:</b> like resource provider done. see:</div><div> <a href="https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/objects/resource_provider.py#L281">https://github.com/openstack/nova/blob/6d227722d4287726e144e4cf928c8e6ae52a6a4c/nova/objects/resource_provider.py#L281</a></div><div> but it seems we should add this filed the instance object?</div><div><br></div><div> <b> 3. CAS: </b>do some pre condition check<b> </b>when we write the record in db. see instance done:</div><div> <a href="https://review.openstack.org/#/c/202593/">https://review.openstack.org/#/c/202593/</a><br></div><div><br></div><div>2. For system_metadata update.</div><div> it has same problem with metadata update,</div><div> but the difference is the "first get" stuff doesn't happen in API, maybe in compute or virt driver.</div><div> That is, the ETAG, the way to avoid API race condition, is not suitable for system_metadata. </div><div><br></div><div>Any idea on this issue, pls feel free to append your comments, we need your advice and help.</div><div><br></div><div><div class="gmail-m_4537732223946077204gmail-m_-8883037862633497901gmail_signature"><div>Regards,</div><div>Yikun</div><div><font color="#666666">------------------------------<wbr>----------</font></div><div><font color="#666666">Jiang Yikun（</font><span style="color:rgb(102,102,102)">Kero）</span></div><div><font color="#666666">Mail: <a href="mailto:yikunkero@gmail.com" target="_blank">yikunkero@gmail.com</a></font></div></div></div>
</div>