[sdk][ansible][inventory] Server-side filtering capability for nova API
Hi. I saw that there some work regarding filtering in the inventory module. Also, I looked at how it will be used in future versions of ansible (os_server_facts module). I am worried about client-side nature of this filtering. The large deployments can have thousands of vms in one openstack project and such an environment can be very dynamic (vms are created/deleted frequently). The inventory collection can be done from multiple ansible management nodes and all these facts can lead to significant performance impact not only on the client side but on the server side also - a large amount of data will be requested from nova api servers. I want to propose the usage of server-side filtering capability for nova API: https://developer.openstack.org/api-ref/compute/?expanded=list-servers-detai... These filters are used at database level and will add a significant performance boost during inventory collection. I want to propose the usage of filters in ansible openstack inventory plugin but first, such filters should be implemented in openstacksdk level. I suggest splitting filters into two parts - the api_filters and metadata_filters. The api_filters will be used for server-side filtering and metadata_filters for client-side filtering. I have created the snippet with prototype code. There a _split_filters method and related changes for list_hosts and search_hosts methods: https://gist.github.com/ITD27M01/01bc73120bb97b237f53fa418dc83629 What do you think?
On 4/8/19 9:04 AM, Игорь Тиунов wrote:
Hi. I saw that there some work regarding filtering in the inventory module. Also, I looked at how it will be used in future versions of ansible (os_server_facts module).
I am worried about client-side nature of this filtering. The large deployments can have thousands of vms in one openstack project and such an environment can be very dynamic (vms are created/deleted frequently). The inventory collection can be done from multiple ansible management nodes and all these facts can lead to significant performance impact not only on the client side but on the server side also - a large amount of data will be requested from nova api servers. I want to propose the usage of server-side filtering capability for nova API:
https://developer.openstack.org/api-ref/compute/?expanded=list-servers-detai...
These filters are used at database level and will add a significant performance boost during inventory collection. I want to propose the usage of filters in ansible openstack inventory plugin but first, such filters should be implemented in openstacksdk level. I suggest splitting filters into two parts - the api_filters and metadata_filters. The api_filters will be used for server-side filtering and metadata_filters for client-side filtering. I have created the snippet with prototype code. There a _split_filters method and related changes for list_hosts and search_hosts methods:
https://gist.github.com/ITD27M01/01bc73120bb97b237f53fa418dc83629
What do you think?
Generally speaking, I think it's a good idea. However, there's a few things to be aware of that make this a little more complex. Namely - in addition to fire-and-forget programs like the os_ modules or the ansible inventory code, we also support users with long lives that do high volume interactions, such as nodepool. Unfortunately, in terms of efficiency, these two use cases want opposite things. For the short-term programs, as you mention, we want to maximize use of server-side filters. We also want to do things like use GET /servers/{id}. For the long-lived high-volume programs, we want to support a mode that combines caching and batching - rather than making 1000 queries to the server a second each with their own server-side filter conditions, it's more efficient to make one call every 5 seconds, cache the result, and do client-side filtering. We're currently good at the second use case and less good at the first. We took a step towards being better at the first use case with the use_direct_get parameter. Use direct get tells openstacksk to use direct GET /servers/{id} type calls rather than filtering a full list locally when a single resource is wanted by id. We need to fix this for get_server for reasons https://review.openstack.org/#/c/540389/. We should also update the logic for the default value to set use_direct_get=True if caching is enabled, and set it to false if it isn't - that way normal people don't have to care. We should almost CERTAINLY start setting this parameter to True in all of the ansible modules and inventory (I think the cases in which we want use_direct_get=False for ansible are VERY explicit - it involves having caching enabled to use external caching like redis - and I think in those cases we can expect a user to set the flag explicitly in config) So I think what we should potentially do here is expand on that config flag and have it pushdown server-side filters. In fact, if you read the value of that in your _split_filters method, then split filters could either return things in api_filters or not. Then we can take the idea and push it down into the underlying code so that inventory is just calling search_severs ... or something. I'm sure we'll need to figure out some details once we get in to it. One additional thing that will make this slightly more complex - but also I believe might allow us to do a few hard patches and get a LOT of benefit across the SDK. Right now we have the 'cloud' layer which used to be the shade library, and we have the 'resource' layer which came from the openstacksdk library. They share almost nothing, other than the connection object and some Adapters, but we're working to change that. One of the next things on the list is to get the 'cloud' layer to be built on top of the 'resource' layer for server interactions. We're already using the sdk layer for servers: https://review.openstack.org/#/c/530770 (although as mentioned use_direct_get needs fixed) ... What I think would be BEST would be if we taught openstack.resource.Resource about use_direct_get - and then put your _split_filters logic in there (although I believe the equiv logic is already there, so it might just be a matter of harnessing it. I think we can certainly start with inventory - as well as changing the default for use_direct_get in the inventory module and in lib/ansible/module_utils/openstak.py. Monty
Hello mordred I do not understand the need of "use_direct_get" parameter. Currently I can do the conn.list_servers(filters={"status": "ACTIVE", "name": "srv8-"}) And I see the corresponding request on the server side: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ REQ: curl -g -i -X GET " https://openstack:8774/v2.1/40e5d3f91fb44ccaa9056252efc243a7/servers/detail?name=srv8-&status=ACTIVE" -H "Accept: application/json" -H "User-Agent: openstacksdk/0.27.0 keystoneauth1/3.13.1 python-requests/2.21.0 CPython/3.6.5" -H "X-Auth-Token: {SHA256}900ab80fe15448ceec851db2c14e2993ecd9634cefb897e71573c6d9496f701d" RESP: [200] Content-Length: 3645 Content-Type: application/json Date: Thu, 11 Apr 2019 08:15:57 GMT Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-e8622002-fb1d-4deb-8bf2-609d6a4b50c9 X-Openstack-Nova-Api-Version: 2.1 RESP BODY: {"servers": <...> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2019-04-11 11:15:57.997 9810 INFO nova.osapi_compute.wsgi.server [req-e8622002-fb1d-4deb-8bf2-609d6a4b50c9 bbe117ded750704ba460feaabff6190c6b07c9acabc91066878cfca0c8362f48 40e5d3f91fb44ccaa9056252efc243a7 - 95ab4645f0764ac782765cbdcc79bbff 95ab4645f0764ac782765cbdcc79bbff] 172.30.202.194 "GET /v2.1/40e5d3f91fb44ccaa9056252efc243a7/servers/detail?name=srv8-&status=ACTIVE HTTP/1.1" status: 200 len: 3974 time: 0.2514541 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Igor Tiunov пн, 8 апр. 2019 г. в 13:24, Monty Taylor <mordred@inaugust.com>:
On 4/8/19 9:04 AM, Игорь Тиунов wrote:
Hi. I saw that there some work regarding filtering in the inventory module. Also, I looked at how it will be used in future versions of ansible (os_server_facts module).
I am worried about client-side nature of this filtering. The large deployments can have thousands of vms in one openstack project and such an environment can be very dynamic (vms are created/deleted frequently). The inventory collection can be done from multiple ansible management nodes and all these facts can lead to significant performance impact not only on the client side but on the server side also - a large amount of data will be requested from nova api servers. I want to propose the usage of server-side filtering capability for nova API:
https://developer.openstack.org/api-ref/compute/?expanded=list-servers-detai...
These filters are used at database level and will add a significant
I want to propose the usage of filters in ansible openstack inventory
I suggest splitting filters into two parts - the api_filters and
performance boost during inventory collection. plugin but first, such filters should be implemented in openstacksdk level. metadata_filters. The api_filters will be used for server-side filtering and metadata_filters for client-side filtering.
I have created the snippet with prototype code. There a _split_filters method and related changes for list_hosts and search_hosts methods:
https://gist.github.com/ITD27M01/01bc73120bb97b237f53fa418dc83629
What do you think?
Generally speaking, I think it's a good idea. However, there's a few things to be aware of that make this a little more complex. Namely - in addition to fire-and-forget programs like the os_ modules or the ansible inventory code, we also support users with long lives that do high volume interactions, such as nodepool. Unfortunately, in terms of efficiency, these two use cases want opposite things.
For the short-term programs, as you mention, we want to maximize use of server-side filters. We also want to do things like use GET /servers/{id}. For the long-lived high-volume programs, we want to support a mode that combines caching and batching - rather than making 1000 queries to the server a second each with their own server-side filter conditions, it's more efficient to make one call every 5 seconds, cache the result, and do client-side filtering.
We're currently good at the second use case and less good at the first.
We took a step towards being better at the first use case with the use_direct_get parameter. Use direct get tells openstacksk to use direct GET /servers/{id} type calls rather than filtering a full list locally when a single resource is wanted by id.
We need to fix this for get_server for reasons https://review.openstack.org/#/c/540389/.
We should also update the logic for the default value to set use_direct_get=True if caching is enabled, and set it to false if it isn't - that way normal people don't have to care.
We should almost CERTAINLY start setting this parameter to True in all of the ansible modules and inventory (I think the cases in which we want use_direct_get=False for ansible are VERY explicit - it involves having caching enabled to use external caching like redis - and I think in those cases we can expect a user to set the flag explicitly in config)
So I think what we should potentially do here is expand on that config flag and have it pushdown server-side filters. In fact, if you read the value of that in your _split_filters method, then split filters could either return things in api_filters or not. Then we can take the idea and push it down into the underlying code so that inventory is just calling search_severs ... or something. I'm sure we'll need to figure out some details once we get in to it.
One additional thing that will make this slightly more complex - but also I believe might allow us to do a few hard patches and get a LOT of benefit across the SDK.
Right now we have the 'cloud' layer which used to be the shade library, and we have the 'resource' layer which came from the openstacksdk library. They share almost nothing, other than the connection object and some Adapters, but we're working to change that. One of the next things on the list is to get the 'cloud' layer to be built on top of the 'resource' layer for server interactions. We're already using the sdk layer for servers: https://review.openstack.org/#/c/530770 (although as mentioned use_direct_get needs fixed) ...
What I think would be BEST would be if we taught openstack.resource.Resource about use_direct_get - and then put your _split_filters logic in there (although I believe the equiv logic is already there, so it might just be a matter of harnessing it.
I think we can certainly start with inventory - as well as changing the default for use_direct_get in the inventory module and in lib/ansible/module_utils/openstak.py.
Monty
participants (2)
-
Monty Taylor
-
Игорь Тиунов