I wanted to get some wider attention on what I think will be a limitation in the rebuild from cell0 spec [1]. The main idea in that spec is the user tries to create a server which fails during scheduling to find a host and puts the instance into ERROR state and "buries" it in cell0. With a new microversion, the user can attempt to rebuild the server from cell0 and go through scheduling again. This is part of a bigger effort to support pre-emptible instances where scheduling fails during server create, an external reaper service tries to make room by destroying some pre-emptible instances, and then the server gets rebuilt.

The issue I've found is that if the server was created with personality files using microversion 2.1, fails, and then the user goes to rebuild it, their personality files will be lost because (1) we don't persist personality files anywhere, not even the RequestSpec and (2) microversion 2.57 deprecated the personality files parameter from the rebuild API [2]. Since you'll have to use a newer microversion > 2.57 to rebuild from cell0, you can't specify the personality files again.

Are we OK with this? We deprecated personality files for a reason [3] so this is a natural extension of that deprecation saying "going forward you should not be using these things and if you do there could be limitations".

This is the nature of per-request versions I think, so I'm okay with it. It's not ideal, but there's not a whole lot that can be done about it.

The user has an obvious recourse: delete the server and re-create it rather than rebuild from cell0.

I don't want to consider un-deprecating personality files in the rebuild API just for this new microversion to rebuild from cell0.

Definitely not.

Another alternative is persisting personality files in the RequestSpec but I'm not sure how much people would support that - although it would fix some other known limitations with personality files, like the fact that if you create a server with files and then shelve offload, the files are gone when you unshelve. I have a similar issue with cross-cell resize [4].

I'd definitely prefer we not do that. Another alternative would be to chuck something into system_metadata for the instance that says "this was created with personality files". We could use that for the rebuild-from-cell0 operation, or any other future one that can't honor those files and just fail the request. "Sorry bro, this instance was created in a way that can't support this request." --Dan