[openstack-dev] [Ironic] Random thoughts on asynchronous API spec
Dmitry Tantsur
dtantsur at redhat.com
Wed May 28 18:22:05 UTC 2014
On Wed, 2014-05-28 at 11:08 -0700, Devananda van der Veen wrote:
> While I appreciate the many ideas being discussed here (some of which
> we've explored previously and agreed to continue exploring), there is
> a fundamental difference vs. what I propose in that spec. I believe
> that what I'm proposing will be achievable without any significant
> visible changes in the API -- no new API end points or resources, and
> the client interaction will be nearly the same.
I am just afraid we'll start to get complains like "I ran a request and
it disappeared to nowhere, what should I do?" Are we ready to handle
this kinds of things (happened because of queue overloaded, user too
impatient, some validation failed etc)? I may be misunderstanding, but
it looks to me like we'll simply drop requests we can't fulfill. That
may be frustrating for a user.
> A few status codes may be different in certain circumstances -- but
> it will not require a new major version of the REST API. And it solves
> a scalability and stability problem that folks are encountering today.
> (It seems my spec didn't describe those problems well enough -- I'm
> updating it now.)
Would be great!
>
>
> Cheers,
> Devananda
>
>
>
>
> On Wed, May 28, 2014 at 10:14 AM, Maksym Lobur <mlobur at mirantis.com>
> wrote:
> BTW a very similar discussion is going in Neutron community
> right now, please find a thread under the [openstack-dev]
> [Neutron] Introducing task oriented workflows label.
>
> Best regards,
> Max Lobur,
> Python Developer, Mirantis, Inc.
>
>
> Mobile: +38 (093) 665 14 28
> Skype: max_lobur
>
>
> 38, Lenina ave. Kharkov, Ukraine
> www.mirantis.com
> www.mirantis.ru
>
>
>
> On Wed, May 28, 2014 at 6:56 PM, Maksym Lobur
> <mlobur at mirantis.com> wrote:
> Hi All,
>
>
> You've raised a good discussion, something similar
> already was started back in february. Could someone
> please find the long etherpad with discussion between
> Deva and Lifeless, as I recall most of the points
> mentioned above have a good comments there.
>
>
> Up to this point I have the only idea how to elegantly
> address these problems. This is a tasks concept and
> probably a scheduler service, which not necessarily
> should be separate from the API at the moment (Finally
> we already have a hash ring on the api side which is a
> kind of scheduler right?) It was already proposed
> earlier, but I would like to try to fit all these
> issues into this concept.
>
> 1. "Executability"
> We need to make sure that request can be
> theoretically executed,
> which includes:
> a) Validating request body
>
> We cannot validate everything on the API side, relying
> on the fact that DB state is actual is not a good
> idea, especially under heavy load.
>
>
> In tasks concept we could assume that all the requests
> are executable, and do not perform any validation in
> the API thread at all. Instead of this the API will
> just create a task and return it's ID to the user.
> Task scheduler may perform some minor validations
> before the task is queued or started for convenience,
> but they should be duplicated inside task body because
> there is an arbitrary time between queuing up and
> start ((c)lifeless). I assume the scheduler will have
> it's own thread or even process. The user will need to
> poke received ID to know the current state of his
> submission.
>
> b) For each of entities (e.g. nodes) touched,
> check that they are
> available
> at the moment (at least exist).
> This is arguable, as checking for entity
> existence requires going to
> DB.
>
> Same here, DB round trip is a potential block,
> therefore this will be done inside task (after it's
> queued and started) and will not affect the API. The
> user will just observe the task state by poking the
> API (or using callback as an option).
>
>
> 2. Appropriate state
> For each entity in question, ensure that it's
> either in a proper state
> or
> moving to a proper state.
> It would help avoid users e.g. setting deploy
> twice on the same node
> It will still require some kind of
> NodeInAWrongStateError, but we won't
> necessary need a client retry on this one.
> Allowing the entity to be _moving_ to
> appropriate state gives us a
> problem:
> Imagine OP1 was running and OP2 got scheduled,
> hoping that OP1 will come
> to desired state. What if OP1 fails? What if
> conductor, doing OP1
> crashes?
>
> Let's say OP1 and OP2 are two separate tasks. Each one
> have the initial state validation inside it's body.
> Once OP2 gets its turn it will perform validation and
> fail, which looks reasonable to me.
>
> Similar problem with checking node state.
> Imagine we schedule OP2 while we had OP1 -
> regular checking node state.
> OP1 discovers that node is actually absent and
> puts it to maintenance
> state.
> What to do with OP2?
>
> The task will fail once it get it's turn.
>
> b) Can we make client wait for the results of
> periodic check?
> That is, wait for OP1 _before scheduling_
> OP2?
>
>
> We will just schedule the task and the user will
> observe its progress, once OP1 is finished and OP2
> started - he will see a fail.
>
> 3. Status feedback
> People would like to know, how things are
> going with their task.
> What they know is that their request was
> scheduled. Options:
> a) Poll: return some REQUEST_ID and expect
> users to poll some endpoint.
> Pros:
> - Should be easy to implement
> Cons:
> - Requires persistent storage for tasks.
> Does AMQP allow to do this
> kinds
> of queries? If not, we'll need to
> duplicate tasks in DB.
> - Increased load on API instances and DB
>
> Exactly described the tasks concept :)
>
> b) Callback: take endpoint, call it once task
> is done/fails.
> Pros:
> - Less load on both client and server
> - Answer exactly when it's ready
> Cons:
> - Will not work for cli and similar
> - If conductor crashes, there will be no
> callback.
>
>
> Add to Cons:
> - Callback is not reliable since it may get lost.
> We should have an ability to poke anyway, though I see
> a great benefit from implementing a callbacks - to
> decrease API load.
>
> 4. Debugging consideration
> a) This is an open question: how to debug, if
> we have a lot of requests
> and something went wrong?
>
>
> We will be able to see the queue state (btw what about
> security here, should the user be able to see all the
> tasks, or just his ones, or all but others with hidden
> details).
>
> b) One more thing to consider: how to make
> command like `node-show`
> aware of
> scheduled transitioning, so that people
> don't try operations that are
> doomed to failure.
>
>
> node-show will always show current state of the node,
> though we may check if there are any tasks queued or
> going, which will change the state. If any - add a
> notification to the response.
>
> 5. Performance considerations
> a) With async approach, users will be able to
> schedule nearly unlimited
> number of tasks, thus essentially blocking
> work of Ironic, without
> any
> signs of the problem (at least for some
> time).
> I think there are 2 common answers to this
> problem:
> - Request throttling: disallow user to make
> too many requests in some
> amount of time. Send them 503 with
> Retry-After header set.
>
>
> Can this be achieved by some web-server settings?
> Looks like a typical problem.
>
> - Queue management: watch queue length,
> deny new requests if it's too
> large.
>
>
> Yes, I really like the limited queue size idea. Please
> see my comments in the spec.
> Also, if we have a tasks and the queue, we could merge
> similar tasks
>
>
> b) State framework from (2), if invented, can
> become a bottleneck as
> well.
> Especially with polling approach.
>
>
> True.
> If we have tasks, all the node actions will be done
> through them. We can synchronise node state with DB
> only during the task, and remove periodic syncs.
> Off-course someone may go and turn off the node, in
> this case the Ironic will lie about the node state
> until some task is executed on this node, which may be
> suitable behaviour. Otherwise rare periodic syncs may
> work as well.
>
> 6. Usability considerations
> a) People will be unaware, when and whether
> their request is going to be
> finished. As there will be tempted to
> retry, we may get flooded by
> duplicates. I would suggest at least make
> it possible to request
> canceling
> any task (which will be possible only if it
> is not started yet,
> obviously).
>
>
> Since we will have a limited number of kinds of tasks,
> we could calculate some estimates basing on previous
> similar tasks. Looks like an improvement for a distant
> future. In the end I wouldn't want Ironic to perform
> estimates like windows's copy-paste dialog :)
>
>
> Tasks may be easily interrupted while they are in a
> queue. But if it's already started - there's a
> separate
> dicsussion https://blueprints.launchpad.net/ironic/+spec/make-tasks-interruptible (I'm going to port this bp to the specs repo in some time)
>
> b) We should try to avoid scheduling
> contradictive requests.
>
>
> A task scheduler responsibility: this is basically a
> state check before task is scheduled, and it should be
> done one more time once the task is started, as
> mentioned above.
>
> c) Can we somehow detect duplicated requests
> and ignore them?
> E.g. we won't want user to make 2-3-4
> reboots in a row just because
> the user
> was not patient enough.
>
>
> Queue similar tasks. All the users will be pointed to
> the similar task resource, or maybe to a different
> resources which tied to the same conductor action.
>
> Best regards,
> Max Lobur,
> Python Developer, Mirantis, Inc.
> Mobile: +38 (093) 665 14 28
> Skype: max_lobur
> 38, Lenina ave. Kharkov, Ukraine
> www.mirantis.com
> www.mirantis.ru
>
>
> On Wed, May 28, 2014 at 5:10 PM, Lucas Alvares
> Gomes <lucasagomes at gmail.com> wrote:
> On Wed, May 28, 2014 at 2:02 PM, Dmitry
> Tantsur <dtantsur at redhat.com> wrote:
> > Hi Ironic folks, hi Devananda!
> >
> > I'd like to share with you my thoughts on
> asynchronous API, which is
> > spec https://review.openstack.org/#/c/94923
> > First I was planned this as comments to the
> review, but it proved to be
> > much larger, so I post it for discussion on
> ML.
> >
> > Here is list of different consideration, I'd
> like to take into account
> > when prototyping async support, some are
> reflected in spec already, some
> > are from my and other's comments:
> >
> > 1. "Executability"
> > We need to make sure that request can be
> theoretically executed,
> > which includes:
> > a) Validating request body
> > b) For each of entities (e.g. nodes)
> touched, check that they are
> > available
> > at the moment (at least exist).
> > This is arguable, as checking for entity
> existence requires going to
> > DB.
>
> >
>
> > 2. Appropriate state
> > For each entity in question, ensure that
> it's either in a proper state
> > or
> > moving to a proper state.
> > It would help avoid users e.g. setting
> deploy twice on the same node
> > It will still require some kind of
> NodeInAWrongStateError, but we won't
> > necessary need a client retry on this one.
> >
> > Allowing the entity to be _moving_ to
> appropriate state gives us a
> > problem:
> > Imagine OP1 was running and OP2 got
> scheduled, hoping that OP1 will come
> > to desired state. What if OP1 fails? What if
> conductor, doing OP1
> > crashes?
> > That's why we may want to approve only
> operations on entities that do
> > not
> > undergo state changes. What do you think?
> >
> > Similar problem with checking node state.
> > Imagine we schedule OP2 while we had OP1 -
> regular checking node state.
> > OP1 discovers that node is actually absent
> and puts it to maintenance
> > state.
> > What to do with OP2?
> > a) Obvious answer is to fail it
> > b) Can we make client wait for the results
> of periodic check?
> > That is, wait for OP1 _before scheduling_
> OP2?
> >
> > Anyway, this point requires some state
> framework, that knows about
> > states,
> > transitions, actions and their compatibility
> with each other.
> For {power, provision} state changes should we
> queue the requests? We
> may want to only accept 1 request to change
> the state per time, if a
> second request comes when there's another
> state change mid-operation
> we may just return 409 (Conflict) to indicate
> that a state change is
> already in progress. This is similar of what
> we have today but instead
> of checking the node lock and states on the
> conductor side the API
> service could do it, since it's on the DB.
> >
> > 3. Status feedback
> > People would like to know, how things are
> going with their task.
> > What they know is that their request was
> scheduled. Options:
> > a) Poll: return some REQUEST_ID and expect
> users to poll some endpoint.
> > Pros:
> > - Should be easy to implement
> > Cons:
> > - Requires persistent storage for tasks.
> Does AMQP allow to do this
> > kinds
> > of queries? If not, we'll need to
> duplicate tasks in DB.
> > - Increased load on API instances and DB
> > b) Callback: take endpoint, call it once
> task is done/fails.
> > Pros:
> > - Less load on both client and server
> > - Answer exactly when it's ready
> > Cons:
> > - Will not work for cli and similar
> > - If conductor crashes, there will be no
> callback.
> >
> > Seems like we'd want both (a) and (b) to
> comply with current needs.
> +1, we could allow pooling by default (like
> checking
> nodes/<uuid>/states to know the current and
> target state of the node)
> but we may also want to include a callback
> parameter that users could
> use to input a URL that the conductor will
> call out as soon as the
> operation is finished. So if the callback URl
> exists, the conductor
> will submit a POST request to that URL with
> some data structure
> identifying the operation and the current
> state.
> >
> > If we have a state framework from (2), we
> can also add notifications to
> > it.
> >
> > 4. Debugging consideration
> > a) This is an open question: how to debug,
> if we have a lot of requests
> > and something went wrong?
> > b) One more thing to consider: how to make
> command like `node-show`
> > aware of
> > scheduled transitioning, so that people
> don't try operations that are
> > doomed to failure.
> >
> > 5. Performance considerations
> > a) With async approach, users will be able
> to schedule nearly unlimited
> > number of tasks, thus essentially
> blocking work of Ironic, without
> > any
> > signs of the problem (at least for some
> time).
> > I think there are 2 common answers to
> this problem:
> > - Request throttling: disallow user to
> make too many requests in some
> > amount of time. Send them 503 with
> Retry-After header set.
> > - Queue management: watch queue length,
> deny new requests if it's too
> > large.
> > This means actually getting back error
> 503 and will require retrying
> > again!
> > At least it will be exceptional case, and
> won't affect Tempest run...
> > b) State framework from (2), if invented,
> can become a bottleneck as
> > well.
> > Especially with polling approach.
> >
> > 6. Usability considerations
> > a) People will be unaware, when and whether
> their request is going to be
> > finished. As there will be tempted to
> retry, we may get flooded by
> > duplicates. I would suggest at least make
> it possible to request
> > canceling
> > any task (which will be possible only if
> it is not started yet,
> > obviously).
> > b) We should try to avoid scheduling
> contradictive requests.
> > c) Can we somehow detect duplicated requests
> and ignore them?
> > E.g. we won't want user to make 2-3-4
> reboots in a row just because
> > the user
> > was not patient enough.
> >
> > ------
> >
> > Possible takeaways from this letter:
> > - We'll need at least throttling to avoid
> DoS
> > - We'll still need handling of 503 error,
> though it should not happen
> > under
> > normal conditions
> > - Think about state framework that unifies
> all this complex logic with
> > features:
> > * Track entities, their states and actions
> on entities
> > * Check whether new action is compatible
> with states of entities it
> > touches
> > and with other ongoing and scheduled
> actions on these entities.
> > * Handle notifications for finished and
> failed actions by providing
> > both
> > pull and push approaches.
> > * Track whether started action is still
> executed, perform error
> > notification,
> > if not.
> > * HA and high performance
> > - Think about policies for corner cases
> > - Think, how we can make a user aware of
> what is going on with both
> > request
> > and entity that some requests may touch.
> Also consider canceling
> > requests.
> >
> > Please let me know, what you think.
> >
> > Dmitry.
> >
> >
> >
> _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> + 1
>
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list