Greetings, I've been working on a tool that requires creating CoreOS nodes on OpenStack. Sadly, I've hit the user data limit, which has blocked the work I'm doing. One big difference between CoreOS images and other cloud images out there is that CoreOS images don't use cloud-init but a different tool called Ignition[0], which uses JSON as its serialization format. The size of the configs that I need to pass to the instance is bigger than the limit imposed by Nova. I've worked on reducing the size as much as possible and even generating a compressed version of it but the data is still bigger than the limit (144 kb vs 65kb). I'd like to understand better what the nature of the limit is (is it because of the field size in the database? Is it purely an API limit? Is it because it causes problems depending on the vendor? As far as I can tell the limit is just being enforced by the API schema[1] and not the DB as it uses a MEDIUMTEXT field. I realize this has been asked before but I wanted to get a feeling of the current opinion about this. Would the Nova team consider increasing the limit of the API considering that more use cases like this may be more common these days? [0] https://coreos.com/ignition/docs/latest/ [1] https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/sch... Thanks, Flavio -- @flaper87 Flavio Percoco
On Tue, 2018-12-04 at 12:08 +0100, Flavio Percoco wrote:
Greetings,
I've been working on a tool that requires creating CoreOS nodes on OpenStack. Sadly, I've hit the user data limit, which has blocked the work I'm doing.
One big difference between CoreOS images and other cloud images out there is that CoreOS images don't use cloud-init but a different tool called Ignition[0], which uses JSON as its serialization format.
The size of the configs that I need to pass to the instance is bigger than the limit imposed by Nova. I've worked on reducing the size as much as possible and even generating a compressed version of it but the data is still bigger than the limit (144 kb vs 65kb).
I'd like to understand better what the nature of the limit is (is it because of the field size in the database? Is it purely an API limit? Is it because it causes problems depending on the vendor? As far as I can tell the limit is just being enforced by the API schema[1] and not the DB as it uses a MEDIUMTEXT field.
I realize this has been asked before but I wanted to get a feeling of the current opinion about this. Would the Nova team consider increasing the limit of the API considering that more use cases like this may be more common these days?
I think EC2 only gives you 1/4 of what Nova does (16KB or so). So it would seem Nova is already being somewhat generous here. I don't see any harm in increasing it so long as the DB supports it (no DB schema change would be required). I wonder if pairing userdata with a token that allowed you to download the information from another (much larger) data source would be a better pattern here though. Then you could make it as large as you needed. Dan
[0] https://coreos.com/ignition/docs/latest/ [1] https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/sch...
Thanks, Flavio
On Tue, Dec 4, 2018 at 2:16 PM Dan Prince <dprince@redhat.com> wrote:
On Tue, 2018-12-04 at 12:08 +0100, Flavio Percoco wrote:
Greetings,
I've been working on a tool that requires creating CoreOS nodes on OpenStack. Sadly, I've hit the user data limit, which has blocked the work I'm doing.
One big difference between CoreOS images and other cloud images out there is that CoreOS images don't use cloud-init but a different tool called Ignition[0], which uses JSON as its serialization format.
The size of the configs that I need to pass to the instance is bigger than the limit imposed by Nova. I've worked on reducing the size as much as possible and even generating a compressed version of it but the data is still bigger than the limit (144 kb vs 65kb).
I'd like to understand better what the nature of the limit is (is it because of the field size in the database? Is it purely an API limit? Is it because it causes problems depending on the vendor? As far as I can tell the limit is just being enforced by the API schema[1] and not the DB as it uses a MEDIUMTEXT field.
I realize this has been asked before but I wanted to get a feeling of the current opinion about this. Would the Nova team consider increasing the limit of the API considering that more use cases like this may be more common these days?
I think EC2 only gives you 1/4 of what Nova does (16KB or so). So it would seem Nova is already being somewhat generous here.
Yeah, I checked before sending this and I thought that regardless of what EC2 is doing, I think it'd be nice for us to consider the use case.
I don't see any harm in increasing it so long as the DB supports it (no DB schema change would be required).
I wonder if pairing userdata with a token that allowed you to download the information from another (much larger) data source would be a better pattern here though. Then you could make it as large as you needed.
This is the current solution, which has allowed me to move forward with the work I'm doing. Regardless, I would like us to discuss this. I'd rather have the limit in Nova increased than adding a dependency on another service that would, very likely, only be used for this specific use case. Flavio
On 12/4/2018 8:14 AM, Flavio Percoco wrote:
This is the current solution, which has allowed me to move forward with the work I'm doing. Regardless, I would like us to discuss this. I'd rather have the limit in Nova increased than adding a dependency on another service that would, very likely, only be used for this specific use case.
As far as the DB limit, it's not just the actual instances.user_data table that matters [1] it's also the build_requests.instance column [2] and the latter is the bigger issue since it's an entire Instance object, including the user_data plus whatever else (like the flavor, metadata and system_metadata) serialized into that single MEDIUMTEXT field. That's what worries me about blowing up that field if we increase the API limit on user_data. As for passing a handle to a thing in another service, we have talked a few times about integrating with a service like Glare to allow users to pass a baremetal node RAID config handle when creating a server and then nova would pull the spec down from Glare and pass it on to the virt driver. We could do the same thing here I would think. Glare is just a generic artifact repository right? I think that would be a better long-term solution for these problems rather than trying to make nova a blob store. [1] https://github.com/openstack/nova/blob/5f648dda49a6d5fe5ecfd7dddcb5f7dc3d6b5... [2] https://github.com/openstack/nova/blob/5f648dda49a6d5fe5ecfd7dddcb5f7dc3d6b5... -- Thanks, Matt
On 12/04/2018 10:09 AM, Matt Riedemann wrote:
On 12/4/2018 8:14 AM, Flavio Percoco wrote:
This is the current solution, which has allowed me to move forward with the work I'm doing. Regardless, I would like us to discuss this. I'd rather have the limit in Nova increased than adding a dependency on another service that would, very likely, only be used for this specific use case.
As far as the DB limit, it's not just the actual instances.user_data table that matters [1] it's also the build_requests.instance column [2] and the latter is the bigger issue since it's an entire Instance object, including the user_data plus whatever else (like the flavor, metadata and system_metadata) serialized into that single MEDIUMTEXT field. That's what worries me about blowing up that field if we increase the API limit on user_data.
How prescient. :) http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000523.... Best, -jay
The limited size of user_data (not even a floppy disk worth of space) is why Octavia has not used it either. We have used a, now deprecated, feature of config drive and nova. So, I am in full support of solving this problem. I would also like to request that we implement this in a way that we can secure the content. We use the config drive mechanism to load per-instance certificates and keys into the instance at boot time and would like to continue to use this mechanism to "seed" the instance. Our current mechanism stores the data along with the instance image, which means it is as protected as the OS image we boot. As we design the future, it would be awesome if we can provide the same or better level of protection for that content. Michael On Tue, Dec 4, 2018 at 7:34 AM Jay Pipes <jaypipes@gmail.com> wrote:
On 12/04/2018 10:09 AM, Matt Riedemann wrote:
On 12/4/2018 8:14 AM, Flavio Percoco wrote:
This is the current solution, which has allowed me to move forward with the work I'm doing. Regardless, I would like us to discuss this. I'd rather have the limit in Nova increased than adding a dependency on another service that would, very likely, only be used for this specific use case.
As far as the DB limit, it's not just the actual instances.user_data table that matters [1] it's also the build_requests.instance column [2] and the latter is the bigger issue since it's an entire Instance object, including the user_data plus whatever else (like the flavor, metadata and system_metadata) serialized into that single MEDIUMTEXT field. That's what worries me about blowing up that field if we increase the API limit on user_data.
How prescient. :)
http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000523....
Best, -jay
Have you looked at vendor data? It gives you a trusted way to inject arbitrary data into a config drive (or metadata server call), where that additional data isn't stored by nova and can be of an arbitrary size. Michael On Wed, Dec 5, 2018 at 9:07 AM Michael Johnson <johnsomor@gmail.com> wrote:
The limited size of user_data (not even a floppy disk worth of space) is why Octavia has not used it either. We have used a, now deprecated, feature of config drive and nova.
So, I am in full support of solving this problem.
I would also like to request that we implement this in a way that we can secure the content. We use the config drive mechanism to load per-instance certificates and keys into the instance at boot time and would like to continue to use this mechanism to "seed" the instance. Our current mechanism stores the data along with the instance image, which means it is as protected as the OS image we boot.
As we design the future, it would be awesome if we can provide the same or better level of protection for that content.
Michael On Tue, Dec 4, 2018 at 7:34 AM Jay Pipes <jaypipes@gmail.com> wrote:
On 12/04/2018 10:09 AM, Matt Riedemann wrote:
On 12/4/2018 8:14 AM, Flavio Percoco wrote:
This is the current solution, which has allowed me to move forward with the work I'm doing. Regardless, I would like us to discuss this. I'd rather have the limit in Nova increased than adding a dependency on another service that would, very likely, only be used for this specific use case.
As far as the DB limit, it's not just the actual instances.user_data table that matters [1] it's also the build_requests.instance column [2] and the latter is the bigger issue since it's an entire Instance
object,
including the user_data plus whatever else (like the flavor, metadata and system_metadata) serialized into that single MEDIUMTEXT field. That's what worries me about blowing up that field if we increase the API limit on user_data.
How prescient. :)
http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000523....
Best, -jay
participants (6)
-
Dan Prince
-
Flavio Percoco
-
Jay Pipes
-
Matt Riedemann
-
Michael Johnson
-
Michael Still