Open Stack

On Dec 6, 2013 4:26 PM, "Akihiro Motoki" <motoki at da.jp.nec.com> wrote:
>
>
> (2013/12/06 17:57), Joe Gordon wrote:
> >
> > On Dec 6, 2013 9:57 AM, "Maru Newby" <marun at redhat.com <mailto:
marun at redhat.com>> wrote:
> >  >
> >  >
> >  > On Dec 6, 2013, at 1:09 AM, John Dickinson <me at not.mn <mailto:
me at not.mn>> wrote:
> >  >
> >  > >
> >  > > On Dec 5, 2013, at 1:36 AM, Maru Newby <marun at redhat.com <mailto:
marun at redhat.com>> wrote:
> >  > >
> >  > >>
> >  > >> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gordon0 at gmail.com<mailto:
joe.gordon0 at gmail.com>> wrote:
> >  > >>
> >  > >>>
> >  > >>>
> >  > >>>
> >  > >>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <me at not.mn<mailto:
me at not.mn>> wrote:
> >  > >>> Just to add to the story, Swift uses "X-Trans-Id" and generates
it in the outer-most "catch_errors" middleware.
> >  > >>>
> >  > >>> Swift's catch errors middleware is responsible for ensuring that
the transaction id exists on each request, and that all errors previously
uncaught, anywhere in the pipeline, are caught and logged. If there is not
a common way to do this, yet, I submit it as a great template for solving
this
> > problem. It's simple, scalable, and well-tested (ie tests and running
in prod for years).
> >  > >>>
> >  > >>>
https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
> >  > >>>
> >  > >>> Leaving aside error handling and only focusing on the
transaction id (or request id) generation, since OpenStack services are
exposed to untrusted clients, how would you propose communicating the
appropriate transaction id to a different service? I can see great benefit
to having a glance
> > transaction ID carry through to Swift requests (and so on), but how
should the transaction id be communicated? It's not sensitive info, but I
can imagine a pretty big problem when trying to track down errors if a
client application decides to set eg the X-Set-Transaction-Id header on
every request
> > to the same thing.
> >  > >>>
> >  > >>> -1 to cross service request IDs, for the reasons John mentions
above.
> >  > >>>
> >  > >>>
> >  > >>> Thanks for bringing this up, and I'd welcome a patch in Swift
that would use a common library to generate the transaction id, if it were
installed. I can see that there would be huge advantage to operators to
trace requests through multiple systems.
> >  > >>>
> >  > >>> Another option would be for each system that calls an another
OpenStack system to expect and log the transaction ID for the request that
was given. This would be looser coupling and be more forgiving for a
heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud
log the
> > transaction id that Swift used (from the Swift response). Likewise,
when Swift makes a call to Keystone, Swift could log the Keystone
transaction id. This wouldn't result in a single transaction id across all
systems, but it would provide markers so an admin could trace the request.
> >  > >>>
> >  > >>> There was a session on this at the summit, and although the
notes are a little scarce this was the conclusion we came up with.  Every
time a cross service call is made, we will log and send a notification for
ceilometer to consume, with the request-ids of both request ids.  One of the
> > benefits of this approach is that we can easily generate a tree of all
the API calls that are made (and clearly show when multiple calls are made
to the same service), something that just a cross service request id would
have trouble with.
> >  > >>
> >  > >> Is wise to trust anything a client provides to ensure
traceability?  If a user receives a request id back from Nova, then submits
that request id in an unrelated request to Neutron, the traceability would
be effectively corrupted.  If the consensus is that we don't want to
securely deliver
> > request ids for inter-service calls, how about requiring a service to
log its request id along with the request id returned from a call to
another service to achieve the a similar result?
> >  > >
> >  > > Yes, this is what I was proposing. I think this is the best path
forward.
>
> I think Logging returned request-id at client side is the best way.
> We can track each API call even when multiple API calls are invoked in
one API request.
>  >
>  > Nova does this for glance client today.  Glance client logs the out
glances request Id and that message is wrapped with novas request id.
>
> When I ran devstack, glance client log (with glance request-id) is not
wrapped
> with nova request-id: http://paste.openstack.org/show/54590/
> It is because glanceclient uses standard logging instead of
openstack.common.log.
> It is common to all client libraries and I am not sure why client
libraries
> do not use openstack.common.log.

Woops you are correct, we log both request ids just not on the same line

> >
> > Ceilometer wanted notifications of these events as well so it could
track things better.
>
> Should this feature a part of client libraries or server side which calls
client libraries?

Server side

> At now client libraries do not return information about response headers
including request-id.
> If we support it in server side, a standard way to return request-id in a
response to a caller.
> I don't have good idea on it at the moment and it is just a question.

Agreed, this point is where most of the work will be.

>
> Thanks,
> Akihiro
>
> >
> >  >
> >  > Ok, great.  And as per your suggestion, a middleware-based error
handler will soon be proposed for oslo that will secondarily ensure that a
request id is added to the request.
> >  >
> >  > >
> >  > >
> >  > >> The catch is that every call point (or client instantiation?)
would have to be modified to pass the request id instead of just logging at
one place in each service.  Is that a cost worth paying?
> >  > >
> >  > > Perhaps this is my ignorance of how other projects work today, but
does this not already happen? Is it possible to get a response from an API
call to an OpenStack project that doesn't include a request id?
> >  >
> >  > We'll have it in Neutron real-soon-now, and then I think the answer
will be 'yes'.
> >  >
> >  > On reflection, it should be easy enough for a given service to
ensure that calls to other services are automatically instrumented to log
request id pairs.  Again, probably something for oslo.
> >  >
> >  >
> >  > m.
> >  > _______________________________________________
> >  > OpenStack-dev mailing list
> >  > OpenStack-dev at lists.openstack.org <mailto:
OpenStack-dev at lists.openstack.org>
> >  > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131206/af386af3/attachment.html>