Open Stack

On 02/07/13 19:50, Boris Pavlovic wrote:
> 
>   *) DB Archiving
>      a) create shadow tables
>      b) add tests that checks that shadow and main table are synced.
>      c) add code that work with shadow tables.
> 

Hi Boris & all,

I have a few points regarding db archiving work that I am growing more
concerned about, so I though I might mention them on this thread. I
pointed them out ad-hoc on a recent review
https://review.openstack.org/#/c/34643/ and there is some discussion
there already, although was not very fruitful.

I feel that there were a few design oversights and as a result it has a
couple of rough edges I noticed.

First issue is about the fact that shadow tables do not present a "view
of the world" themselves but are just unconstrained rows copied from
live tables.

This is understandably done for performance reasons while archiving
(with current design ideas in place), but also causes issues when
migrations affect more than one table. Especially if data migrations
need to look at more tables at once, the actual number of table joins
needed in order to consider everything grows exponentially. It could be
argued that these are not that common, but is something that will make
development more difficult and migrations painful once it comes up.

To put it shortly - this property generally makes it harder to reason
about data.

Second point (and it ties in with the first one since it makes it
difficult to fix) - Maybe shadow table migrations should be kept
separate, and made optional? Currently there is a check that will fail
the tests unless the migration is done on both tables, which I think
should be removed in favour of separate migrations. Developers should
still migrate both of course - but deployers should be able to choose
not to do it according to their needs/scale. I am sure there are people
on this list that can chip in more on this subject (I've had a brief
discussion with lifeless on this topic on IRC).

I'm afraid that if you agree that these are in fact problems - you might
also agree that we might want to go back on some of the design decisions
made around db archiving (like having unconstrained tables in the same
db for example).

I'd be happy to hear some of the angles that I may have missed,

Cheers,

Nikola