Open Stack

On Wed, Jan 27, 2016, at 05:47 AM, Kuvaja, Erno wrote:
> > -----Original Message-----
> > From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
> > Sent: Wednesday, January 27, 2016 9:56 AM
> > To: openstack-dev at lists.openstack.org
> > Subject: Re: [openstack-dev] [nova][glance][cinder][neutron]How to make
> > use of x-openstack-request-id
> > 
> > 
> > 
> > On 1/27/2016 9:40 AM, Tan, Lin wrote:
> > > Thank you so much. Eron. This really helps me a lot!!
> > >
> > > Tan
> > >
> > > *From:*Kuvaja, Erno [mailto:kuvaja at hpe.com]
> > > *Sent:* Tuesday, January 26, 2016 8:34 PM
> > > *To:* OpenStack Development Mailing List (not for usage questions)
> > > *Subject:* Re: [openstack-dev] [nova][glance][cinder][neutron]How to
> > > make use of x-openstack-request-id
> > >
> > > Hi Tan,
> > >
> > > While the cross project spec was discussed Glance already had
> > > implementation of request ids in place. At the time of the Glance
> > > implementation we assumed that one request id is desired through the
> > > chain of services and we implemented the req id to be accepted as part
> > > of the request. This was mainly driven to have same request id through
> > > the chain between glance-api and glance-registry but as the same code
> > > was used in both api and registry services we got this functionality
> > > across glance.
> > >
> > > The cross project discussion turned this approach down and decided
> > > that only new req id will be returned. We did not want to utilize 2
> > > different code bases to handle req ids in glance-api and
> > > glance-registry, nor we wanted to remove the functionality to allow
> > > the req ids being passed to the service as that was already merged to
> > > our API. Thus is requests are passed without req id defined to the
> > > services they behave (apart from nova having different header name)
> > > same way, but with glance the request maker has the liberty to specify
> > > request id they want to use (within configured length limits).
> > >
> > > Hopefully that clarifies it for you.
> > >
> > > -Erno
> > >
> > > *From:*Tan, Lin [mailto:lin.tan at intel.com]
> > > *Sent:* 26 January 2016 01:26
> > > *To:* OpenStack Development Mailing List (not for usage questions)
> > > *Subject:* Re: [openstack-dev] [nova][glance][cinder][neutron]How to
> > > make use of x-openstack-request-id
> > >
> > > Thanks Kebane, I test glance/neutron/keystone with
> > > ``x-openstack-request-id`` and find something interesting.
> > >
> > > I am able to pass ``x-openstack-request-id``  to glance and it will
> > > use the UUID as its request-id. But it failed with neutron and keystone.
> > >
> > > Here is my test:
> > >
> > > http://paste.openstack.org/show/484644/
> > >
> > > It looks like because keystone and neutron are using
> > > oslo_middleware:RequestId.factory and in this part:
> > >
> > >
> > https://github.com/openstack/oslo.middleware/blob/master/oslo_middlew
> > a
> > > re/request_id.py#L35
> > >
> > > It will always generate an UUID and append to response as
> > > ``x-openstack-request-id`` header.
> > >
> > > My question is should we accept an external passed request-id as the
> > > project's own request-id or having its unique request-id?
> > >
> > > In other words, which one is correct way, glance or neutron/keystone?
> > > There must be something wrong with one of them.
> > >
> > > Thanks
> > >
> > > B.R
> > >
> > > Tan
> > >
> > > *From:*Kekane, Abhishek [mailto:Abhishek.Kekane at nttdata.com]
> > > *Sent:* Wednesday, December 2, 2015 2:24 PM
> > > *To:* OpenStack Development Mailing List
> > > (openstack-dev at lists.openstack.org
> > > <mailto:openstack-dev at lists.openstack.org>)
> > > *Subject:* Re: [openstack-dev] [nova][glance][cinder][neutron]How to
> > > make use of x-openstack-request-id
> > >
> > > Hi Tan,
> > >
> > > Most of the OpenStack RESTful API returns `X-Openstack-Request-Id` in
> > > the API response header but thisrequest id isnotavailable to the
> > > callerfromthe python client.
> > >
> > > When you use --debug option from command from the command prompt
> > using
> > > client, you can see `X-Openstack-Request-Id`on the console but it is
> > > not logged anywhere.
> > >
> > > Currently a cross-project specs [1] is submitted and approved for
> > > returning X-Openstack-Request-Id to the caller and the implementation
> > > for the same is in progress.
> > >
> > > Please go through the specs for detail information which will help you
> > > to understand more about request-ids and current work about the same.
> > >
> > > Please feel free to revert back anytime for your doubts.
> > >
> > > [1]
> > > https://github.com/openstack/openstack-
> > specs/blob/master/specs/return-
> > > request-id.rst
> > >
> > > Thanks,
> > >
> > > Abhishek Kekane
> > >
> > > Hi guys
> > >
> > >          I recently play around with 'x-openstack-request-id' header
> > > but have a dump question about how it works. At beginning, I thought
> > > an action across different services should use a same request-id but
> > > it looks like this is not the true.
> > >
> > > First I read the spec:
> > > https://blueprints.launchpad.net/nova/+spec/cross-service-request-id
> > > which said "This ID and the request ID of the other service will be
> > > logged at service boundaries". and I see cinder/neutron/glance will
> > > attach its context's request-id as the value of "x-openstack-request-id"
> > > header to its response while nova use X-Compute-Request-Id. This is
> > > easy to understand. So It looks like each service should generate its
> > > own request-id and attach to its response, that's all.
> > >
> > > But then I see glance read 'X-Openstack-Request-ID' to generate the
> > > request-id while cinder/neutron/nova read 'openstack.request_id' when
> > > using with keystone. It is try to reuse the request-id from keystone.
> > >
> > > This totally confused me. It would be great if you can correct me or
> > > point me some reference. Thanks a lot
> > >
> > > Best Regards,
> > >
> > > Tan
> > >
> > >
> > >
> > __________________________________________________________
> > ____________
> > > Disclaimer: This email and any attachments are sent in strictest
> > > confidence for the sole use of the addressee and may contain legally
> > > privileged, confidential, and proprietary data. If you are not the
> > > intended recipient, please advise the sender by replying promptly to
> > > this email and then delete and destroy this email and any attachments
> > > without any further use, copying or forwarding.
> > >
> > >
> > >
> > >
> > __________________________________________________________
> > ____________
> > > ____ OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > 
> > I haven't read into the cross-project spec, but I can say that from a debugging
> > perspective, it would be nice if the same request ID was used throughout a
> > call chain, i.e. tempest is making a request to the nova API which is making
> > requests to cinder, glance, keystone and neutron.
> > Something fails in let's say neutron, it'd be nice to be following that request
> > ID throughout the service logs for the same operation that originated in
> > Tempest. Instead of being able to use the request ID, you often times have
> > to use the instance uuid to track the device_id in the neutron logs, or a
> > volume_id in the n-cpu logs back to the c-api/c-vol logs.
> > 
> > Or am I missing something?
> 
> Nope, that's one of the reasons we wanted to have that same id possibly
> initiated by client to follow through the whole life of the request. This
> wasn't good enough for some folks for various reasons; security card was
> obviously played, another big part of discussion was request separation
> when we need to send request to same service multiple times during action
> (for example in Glance creating an image and uploading the data to it are
> two different API calls to glance even it's one action from user).
> Debugging/support/monitoring wise it would be great if one could specify
> desired request ID for action and follow it through, but unfortunately
> that wasn't the agreed approach in the community.

I was involved in some of these discussions years ago but have not kept
up with any of the recent talk/decisions so what I'm saying might be out
of date.

The concern for passing in a request-id isn't really about security but
more about ensuring that the request-ids are  unique.  If clients are
allowed to pass them in it becomes possible for a poor, or malicious,
client to just send the same request-id for every request which makes
log correlation much harder to do.  It is better to leave request-id
generation to the services so that there is control over that.

As mentioned above multiple calls are also a concern.  For example if
Nova makes two calls into Glance but those requests share the same
passed request-id then you need to find another way to separate those
events out during log parsing.  This is the same problem I just
described but we would be intentionally doing it.  The path for moving
forward on this is to have the services generate their own request-id
but pass it back in a consistent manner for every request and then have
the calling service log this request-id with the local request-id in
order to correlate them.  So in Nova logs you would get a log line like
"<Nova-request-id> <Glance-request-id> ..."  This doesn't allow for
jumping straight from a request-id returned by Nova to Tempest to Cinder
logs but does bring consistency to how you get from
Tempest->Nova->Cinder/Neutron/Glance so in theory a log parsing engine
could follow this path.


> 
> - Erno
> 
> > 
> > --
> > 
> > Thanks,
> > 
> > Matt Riedemann
> > 
> > 
> > __________________________________________________________
> > ________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> > request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev