Open Stack

Excerpts from Zane Bitter's message of 2014-08-27 08:41:29 -0700:
> On 27/08/14 11:04, Steven Hardy wrote:
> > On Wed, Aug 27, 2014 at 07:54:41PM +0530, Jyoti Ranjan wrote:
> >>     I am little bit skeptical about using Swift for this use case because of
> >>     its eventual consistency issue. I am not sure Swift cluster is good to be
> >>     used for this kind of problem. Please note that Swift cluster may give you
> >>     old data at some point of time.
> >
> > This is probably not a major problem, but it's certainly worth considering.
> >
> > My assumption is that the latency of making the replicas consistent will be
> > small relative to the timeout for things like SoftwareDeployments, so all
> > we need is to ensure that instances  eventually get the new data, act on
> 
> That part is fine, but if they get the new data and then later get the 
> old data back again... that would not be so good.
> 

Agreed, and I had not considered that this can happen.

There is a not-so-simple answer though:

* Heat inserts this as initial metadata:

    {"metadata": {}, "update-url": "xxxxxx", "version": 0}

* Polling goes to update-url and ignores metadata <= 0

* Polling finds new metadata in same format, and continues the loop
without talking to Heat

However, this makes me rethink why we are having performance problems.
MOST of the performance problems have two root causes:

* We parse the entire stack to show metadata, because we have to see if
  there are custom access controls defined in any of the resources used.
  I actually worked on a patch set to deprecate this part of the resource
  plugin API because it is impossible to scale this way.
* We rely on the engine to respond because of the parsing issue.

If however we could just push metadata into the db fully resolved
whenever things in the stack change, and cache the response in the API
using Last-Modified/Etag headers, I think we'd be less inclined to care
so much about swift for polling. However we are still left with the many
thousands of keystone users being created vs. thousands of swift tempurls.

That would also set us up nicely for very easy integration with Zaqar,
as metadata changes would flow naturally into the message queue for the
server through the same mechanism as they flow into the database.