Open Stack

Hello mordred

I do not understand the need of "use_direct_get" parameter. Currently I can
do the conn.list_servers(filters={"status": "ACTIVE", "name": "srv8-"})

And I see the corresponding request on the server side:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REQ: curl -g -i -X GET "
https://openstack:8774/v2.1/40e5d3f91fb44ccaa9056252efc243a7/servers/detail?name=srv8-&status=ACTIVE"
-H "Accept: application/json" -H "User-Agent: openstacksdk/0.27.0
keystoneauth1/3.13.1 python-requests/2.21.0 CPython/3.6.5" -H
"X-Auth-Token:
{SHA256}900ab80fe15448ceec851db2c14e2993ecd9634cefb897e71573c6d9496f701d"
RESP: [200] Content-Length: 3645 Content-Type: application/json Date: Thu,
11 Apr 2019 08:15:57 GMT Openstack-Api-Version: compute 2.1 Vary:
OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id:
req-e8622002-fb1d-4deb-8bf2-609d6a4b50c9 X-Openstack-Nova-Api-Version: 2.1
RESP BODY: {"servers": <...>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2019-04-11 11:15:57.997 9810 INFO nova.osapi_compute.wsgi.server
[req-e8622002-fb1d-4deb-8bf2-609d6a4b50c9
bbe117ded750704ba460feaabff6190c6b07c9acabc91066878cfca0c8362f48
40e5d3f91fb44ccaa9056252efc243a7 - 95ab4645f0764ac782765cbdcc79bbff
95ab4645f0764ac782765cbdcc79bbff] 172.30.202.194 "GET
/v2.1/40e5d3f91fb44ccaa9056252efc243a7/servers/detail?name=srv8-&status=ACTIVE
HTTP/1.1" status: 200 len: 3974 time: 0.2514541

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Igor Tiunov


пн, 8 апр. 2019 г. в 13:24, Monty Taylor <mordred at inaugust.com>:

>
>
> On 4/8/19 9:04 AM, Игорь Тиунов wrote:
> > Hi.
> > I saw that there some work regarding filtering in the inventory module.
> Also, I looked at how it will be used in future versions of ansible
> (os_server_facts module).
> >
> > I am worried about client-side nature of this filtering. The large
> deployments can have thousands of vms in one openstack project and such an
> environment can be very dynamic (vms are created/deleted frequently). The
> inventory collection can be done from multiple ansible management nodes and
> all these facts can lead to significant performance impact not only on the
> client side but on the server side also - a large amount of data will be
> requested from nova api servers.
> > I want to propose the usage of server-side filtering capability for nova
> API:
> >
> >
> https://developer.openstack.org/api-ref/compute/?expanded=list-servers-detail#list-servers
> >
> > These filters are used at database level and will add a significant
> performance boost during inventory collection.
> > I want to propose the usage of filters in ansible openstack inventory
> plugin but first, such filters should be implemented in openstacksdk level.
> > I suggest splitting filters into two parts - the api_filters and
> metadata_filters. The api_filters will be used for server-side filtering
> and metadata_filters for client-side filtering.
> > I have created the snippet with prototype code. There a _split_filters
> method and related changes for list_hosts and search_hosts methods:
> >
> > https://gist.github.com/ITD27M01/01bc73120bb97b237f53fa418dc83629
> >
> > What do you think?
>
> Generally speaking, I think it's a good idea. However, there's a few
> things to be aware of that make this a little more complex. Namely - in
> addition to fire-and-forget programs like the os_ modules or the ansible
> inventory code, we also support users with long lives that do high
> volume interactions, such as nodepool. Unfortunately, in terms of
> efficiency, these two use cases want opposite things.
>
> For the short-term programs, as you mention, we want to maximize use of
> server-side filters. We also want to do things like use GET
> /servers/{id}. For the long-lived high-volume programs, we want to
> support a mode that combines caching and batching - rather than making
> 1000 queries to the server a second each with their own server-side
> filter conditions, it's more efficient to make one call every 5 seconds,
> cache the result, and do client-side filtering.
>
> We're currently good at the second use case and less good at the first.
>
> We took a step towards being better at the first use case with the
> use_direct_get parameter. Use direct get tells openstacksk to use direct
> GET /servers/{id} type calls rather than filtering a full list locally
> when a single resource is wanted by id.
>
> We need to fix this for get_server for reasons
> https://review.openstack.org/#/c/540389/.
>
> We should also update the logic for the default value to set
> use_direct_get=True if caching is enabled, and set it to false if it
> isn't - that way normal people don't have to care.
>
> We should almost CERTAINLY start setting this parameter to True in all
> of the ansible modules and inventory (I think the cases in which we want
> use_direct_get=False for ansible are VERY explicit - it involves having
> caching enabled to use external caching like redis - and I think in
> those cases we can expect a user to set the flag explicitly in config)
>
> So I think what we should potentially do here is expand on that config
> flag and have it pushdown server-side filters. In fact, if you read the
> value of that in your _split_filters method, then split filters could
> either return things in api_filters or not. Then we can take the idea
> and push it down into the underlying code so that inventory is just
> calling search_severs ... or something. I'm sure we'll need to figure
> out some details once we get in to it.
>
> One additional thing that will make this slightly more complex - but
> also I believe might allow us to do a few hard patches and get a LOT of
> benefit across the SDK.
>
> Right now we have the 'cloud' layer which used to be the shade library,
> and we have the 'resource' layer which came from the openstacksdk
> library. They share almost nothing, other than the connection object and
> some Adapters, but we're working to change that. One of the next things
> on the list is to get the 'cloud' layer to be built on top of the
> 'resource' layer for server interactions. We're already using the sdk
> layer for servers: https://review.openstack.org/#/c/530770 (although as
> mentioned use_direct_get needs fixed) ...
>
> What I think would be BEST would be if we taught
> openstack.resource.Resource about use_direct_get - and then put your
> _split_filters logic in there (although I believe the equiv logic is
> already there, so it might just be a matter of harnessing it.
>
> I think we can certainly start with inventory - as well as changing the
> default for use_direct_get in the inventory module and in
> lib/ansible/module_utils/openstak.py.
>
> Monty
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190411/57c42ae8/attachment-0001.html>