Open Stack

On Dec 6, 2013 9:57 AM, "Maru Newby" <marun at redhat.com> wrote:
>
>
> On Dec 6, 2013, at 1:09 AM, John Dickinson <me at not.mn> wrote:
>
> >
> > On Dec 5, 2013, at 1:36 AM, Maru Newby <marun at redhat.com> wrote:
> >
> >>
> >> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
> >>
> >>>
> >>>
> >>>
> >>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <me at not.mn> wrote:
> >>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in
the outer-most "catch_errors" middleware.
> >>>
> >>> Swift's catch errors middleware is responsible for ensuring that the
transaction id exists on each request, and that all errors previously
uncaught, anywhere in the pipeline, are caught and logged. If there is not
a common way to do this, yet, I submit it as a great template for solving
this problem. It's simple, scalable, and well-tested (ie tests and running
in prod for years).
> >>>
> >>>
https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
> >>>
> >>> Leaving aside error handling and only focusing on the transaction id
(or request id) generation, since OpenStack services are exposed to
untrusted clients, how would you propose communicating the appropriate
transaction id to a different service? I can see great benefit to having a
glance transaction ID carry through to Swift requests (and so on), but how
should the transaction id be communicated? It's not sensitive info, but I
can imagine a pretty big problem when trying to track down errors if a
client application decides to set eg the X-Set-Transaction-Id header on
every request to the same thing.
> >>>
> >>> -1 to cross service request IDs, for the reasons John mentions above.
> >>>
> >>>
> >>> Thanks for bringing this up, and I'd welcome a patch in Swift that
would use a common library to generate the transaction id, if it were
installed. I can see that there would be huge advantage to operators to
trace requests through multiple systems.
> >>>
> >>> Another option would be for each system that calls an another
OpenStack system to expect and log the transaction ID for the request that
was given. This would be looser coupling and be more forgiving for a
heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud
log the transaction id that Swift used (from the Swift response). Likewise,
when Swift makes a call to Keystone, Swift could log the Keystone
transaction id. This wouldn't result in a single transaction id across all
systems, but it would provide markers so an admin could trace the request.
> >>>
> >>> There was a session on this at the summit, and although the notes are
a little scarce this was the conclusion we came up with.  Every time a
cross service call is made, we will log and send a notification for
ceilometer to consume, with the request-ids of both request ids.  One of
the benefits of this approach is that we can easily generate a tree of all
the API calls that are made (and clearly show when multiple calls are made
to the same service), something that just a cross service request id would
have trouble with.
> >>
> >> Is wise to trust anything a client provides to ensure traceability?
 If a user receives a request id back from Nova, then submits that request
id in an unrelated request to Neutron, the traceability would be
effectively corrupted.  If the consensus is that we don't want to securely
deliver request ids for inter-service calls, how about requiring a service
to log its request id along with the request id returned from a call to
another service to achieve the a similar result?
> >
> > Yes, this is what I was proposing. I think this is the best path
forward.

Nova does this for glance client today.  Glance client logs the out glances
request Id and that message is wrapped with novas request id.

Ceilometer wanted notifications of these events as well so it could track
things better.

>
> Ok, great.  And as per your suggestion, a middleware-based error handler
will soon be proposed for oslo that will secondarily ensure that a request
id is added to the request.
>
> >
> >
> >> The catch is that every call point (or client instantiation?) would
have to be modified to pass the request id instead of just logging at one
place in each service.  Is that a cost worth paying?
> >
> > Perhaps this is my ignorance of how other projects work today, but does
this not already happen? Is it possible to get a response from an API call
to an OpenStack project that doesn't include a request id?
>
> We'll have it in Neutron real-soon-now, and then I think the answer will
be 'yes'.
>
> On reflection, it should be easy enough for a given service to ensure
that calls to other services are automatically instrumented to log request
id pairs.  Again, probably something for oslo.
>
>
> m.
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131206/fd4cc744/attachment.html>