Open Stack

On 11/19/2014 11:38 AM, Heald, Mike wrote:
> Hello, fellow Storyboarders!
>
> After some discussions with the folks working on oslo.db and going
> over our requirements for paging with krotscheck, we came to the
> conclusion that paging just based on a marker did not give us the
> reliability or functionality that we need in Storyboard. I said I'd
> outline the paging model I'd used before for companies whose
> requirements were quite demanding (e.g. if we miss a record while
> paging, we could potentially lose thousands in profits)

Pretty sure Storyboard paged results are not going to affect anyone's 
profit margin.

> Goals:
>
> 1) Paged list where records do not jump from page to page, even if
> the underlying data changes, so that the user never misses records
> that match search criteria, and never sees the records she is dealing
> with change unexpectedly

Is this use case really something to be worried about? Honestly, I don't 
think so.

 > 2) Able to order the list by column

Nothing about marker/limit pagination prevents ordering by any column.

 > 3) Able to jump to an arbitrary page

What about this last use case is useful? Why would I want to "jump to 
the 7th page of results"? Just being able to do something doesn't 
necessarily make it useful.

> To fully satisfy (1) while having (2) and (3), we need to snapshot
> the result set, and have a mechanism for notifying the user of any
> new records that match the list criteria since they started
> browsing.
>
> This can be achieved by having a three column table, with a record
> ID, the data used to order the results, and an ID for the result set.
> This allows us to left join on the main data table (so that record
> deletions do not affect paging, and the UI can show that a record was
> deleted and not mess up paging). Ordering by the snapshotted data
> means that if records change that data in the main data table, they
> do not jump pages, and the UI can show that the data has changed.
> This does require some resultset metadata too (e.g. column sorted
> by)
>
> Notifications of new records can be done with a web worker that
> checks if there are records that do not appear in the search but do
> match the search criteria. The user can be notified that new records
> have come in, and opt to either view a list of these records, or
> rerun the search to include them.
>
> Downsides of this approach are: - It's relatively heavy. The search
> table for each datatype will have lots of inserts and deletes, so
> careful consideration of partitions, or use of temporary tables, or
> other webscale stuff needs to happen - Housekeeping needs to be
> thought about. Just viewing a list of stuff will result in a snapshot
> so it can be manipulated correctly, so we need to think about if
> there are situations when we *don't* want this to happen (for example
> viewing a list of all stories might not warrant it, but viewing a
> list of stories with a particular status, might), how to clean up
> result set snapshots that are no longer in use, etc.
>
> Upsides are: - It works. The results you are working with remain
> consistent, you get notified of any data changes, and you can page
> through the results easily. You could even use marker based paging
> for the snapshotted results if you wanted to, and it would be
> reliable (although some functionality like jumping to an arbitrary
> page would be sacrified).
>
> Hope that makes sense, what are everyone's thoughts?

I appreciate that you and others have put a lot of thought and time into 
this, but frankly, I just don't see that much benefit to the extra code 
complexity and time to deliver it.

Best,
-jay