---- On Thu, 01 Aug 2019 17:58:18 +0900 Ian Wienand <iwienand@redhat.com> wrote ----
On Fri, Jul 26, 2019 at 04:53:28PM -0700, Clark Boylan wrote:
Given my change shows this can be so much quicker is there any interest in modifying devstack to be faster here? And if so what do we think an appropriate approach would be?
My first concern was if anyone considered openstack-client setting these things up as actually part of the testing. I'd say not, comments in [1] suggest similar views.
My second concern is that we do keep sufficient track of complexity v speed; obviously doing things in a sequential manner via a script is pretty simple to follow and as we start putting things into scripts we make it harder to debug when a monoscript dies and you have to start pulling apart where it was. With just a little json fiddling we can currently pull good stats from logstash ([2]) so I think as we go it would be good to make sure we account for the time using appropriate wrappers, etc.
I agree on this concern about maintainability and debugging with scripts. Now a days, very less people have good knowledge on devstack code and debugging the failure on job side is much harder for most of the developers. IMO the maintainability and easy to debug is much needed as first priority. If we wanted to convert the OSC with something faster, Tempest service client comes into my mind. They are the very straight call to API directly but the token is requested for each API call. But that is something need PoC about speed improvement especially.
Then the third concern is not to break anything for plugins -- devstack has a very very loose API which basically relies on plugin authors using a combination of good taste and copying other code to decide what's internal or not.
Which made me start thinking I wonder if we look at this closely, even without replacing things we might make inroads?
For example [3]; it seems like SERVICE_DOMAIN_NAME is never not default, so the get_or_create_domain call is always just overhead (the result is never used).
Then it seems that in the gate, basically all of the "get_or_create" calls will really just be "create" calls? Because we're always starting fresh. So we could cut out about half of the calls there pre-checking if we know we're under zuul (proof-of-concept [4]).
Then we have blocks like:
get_or_add_user_project_role $member_role $demo_user $demo_project get_or_add_user_project_role $admin_role $admin_user $demo_project get_or_add_user_project_role $another_role $demo_user $demo_project get_or_add_user_project_role $member_role $demo_user $invis_project
If we wrapped that in something like
start_osc_session ... end_osc_session
which sets a variable that means instead of calling directly, those functions write their arguments to a tmp file. Then at the end call, end_osc_session does
$ osc "$(< tmpfile)"
and uses the inbuilt batching? If that had half the calls by skipping the "get_or" bit, and used common authentication from batching, would that help?
And then I don't know if all the projects and groups are required for every devstack run? Maybe someone skilled in the art could do a bit of an audit and we could cut more of that out too?
Yeah, improving such usused o not required call with the audit is a good call. For example, In most place, devstack need just resource id or name or few fields for created resource so get call which gives complete resource fileds might not be needed and for async call we can have an exception to get resource('addressess' in server). -gmann
So I guess my point is that maybe we could tweak what we have a bit to make some immediate wins, before anyone has to rewrite too much?
-i
[1] https://review.opendev.org/673018 [2] https://ethercalc.openstack.org/rzuhevxz7793 [3] https://review.opendev.org/673941 [4] https://review.opendev.org/673936