Open Stack

On 2017-08-29 06:28 AM, Mikhail Kebich wrote:
> Hello Telemetry Team,
> 
> I see that the Panko API does not provide a way to go through all events 
> in the order they were stored in database. Panko sorts events by 
> generated and message_id fields and uses message_id field as a marker 
> for pagination. So, the following case is possible:
> 
> Before the first request we had the following events in database:
> +-------------------+------------------------------+
> || message_id1 | 2017-08-29T00:05.00 ||
> +-------------------+------------------------------+
> || message_id2 | 2017-08-29T00:06.00 ||
> +-------------------+------------------------------+
> || message_id3 | 2017-08-29T00:08.00 ||
> +-------------------+------------------------------+
> 
> During the first request we got all three events:
> GET /v2/events?limit=3
> 
> After the request some new events were added:
> +-------------------+------------------------------+
> || message_id1 | 2017-08-29T00:05.00 ||
> +-------------------+------------------------------+
> || message_id2 | 2017-08-29T00:06.00 ||
> +-------------------+------------------------------+
> || message_id4 | 2017-08-29T00:07.00 ||
> +-------------------+------------------------------+
> || message_id3 | 2017-08-29T00:08.00 ||
> +-------------------+------------------------------+
> || message_id5 | 2017-08-29T00:09.00 ||
> +-------------------+------------------------------+
> || message_id6 | 2017-08-29T00:09.00 ||
> +-------------------+------------------------------+
> 
> To proceed next we set marker to message_id3:
> GET /v2/events?marker=message_id3&limit=3
> 
> In this case we will get only two new events (message_id5, message_id6). 
> To get message_id4 we need to make the first request again.

i'm having trouble understanding the issue. the marker (message_id in 
this case) denotes the starting point of query. not sure why you would 
expect message_id4 to show up if message_id3 is newer.

> 
> That all makes difficult such scenarios when API clients need to process 
> events as they appear in database.
> 
> I'd like to discuss how we can extend the current API to provide a way 
> to go through all events in the order they were stored in database.
> 
> It looks like cursor API might be useful in this case. We can introduce 
> new "cursor" query parameter. It will work as a marker for pagination, 
> but will determine "special" sorting of events. In this case the request 
> might look like:
> GET /v2/events?cursor=-1&limit=100
> 
> The response will return a list of events and next cursor:
> {
>      events: []
>      next_cursor: 12345
> }
> 
> So, a request to get the next portion of events will look like:
> GET /v2/events?cursor=12345&limit=100
> 

is cursor code part of oslo.db? the one issue i see is we'll need to 
change api results which means we need to do some versioning changes as 
well (not a blocker, just another discussion and more work)

cheers,

-- 
gord