Open Stack

** Description changed:

- This issue is being treated as a potential security risk under embargo.
- Please do not make any public mention of embargoed (private) security
- vulnerabilities before their coordinated publication by the OpenStack
- Vulnerability Management Team in the form of an official OpenStack
- Security Advisory. This includes discussion of the bug or associated
- fixes in public forums such as mailing lists, code review systems and
- bug trackers. Please also avoid private disclosure to other individuals
- not already approved for access to this information, and provide this
- same reminder to those who are made aware of the issue prior to
- publication. All discussion should remain confined to this private bug
- report, and any proposed fixes should be added to the bug as
- attachments.
- 
  Noticed here:
  
  http://logs.openstack.org/08/445308/3/check/gate-tempest-dsvm-py35
  -ubuntu-xenial/7bf0d72/logs/screen-n-api.txt.gz#_2017-03-16_05_31_09_399
  
  I noticed this while investigating public nova bug 1673375, but it looks
  like that bug is caused by a ValueError coming from the oslo.messaging
  notification code, related to a circular reference in the json blob:
  
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/notify/messaging.py", line 70, in notify
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     retry=retry)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/transport.py", line 104, in _send_notification
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     retry=retry)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 509, in send_notification
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     envelope=(version == 2.0), notify=True, retry=retry)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 457, in _send
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     msg = rpc_common.serialize_msg(msg)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/common.py", line 293, in serialize_msg
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_serialization/jsonutils.py", line 190, in dumps
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     return json.dumps(obj, default=default, **kwargs)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     **kw).encode(obj)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     chunks = self.iterencode(o, _one_shot=True)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     return _iterencode(o, 0)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging ValueError: Circular reference detected
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging
  
  The security issue here is that the notification payload that's logged
  has all kinds of auth secrets in it, like tokens and passwords.
  
  From logstash it looks like this is only hitting master (pike) right
  now.

-- 
You received this bug notification because you are a member of OpenStack
Security, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1673569

Title:
  Failed notification payload is dumped in logs with auth secrets

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) mitaka series:
  Fix Released
Status in OpenStack Compute (nova) newton series:
  Fix Released
Status in OpenStack Compute (nova) ocata series:
  Fix Released
Status in OpenStack Security Advisory:
  Confirmed

Bug description:
  Noticed here:

  http://logs.openstack.org/08/445308/3/check/gate-tempest-dsvm-py35
  -ubuntu-
  xenial/7bf0d72/logs/screen-n-api.txt.gz#_2017-03-16_05_31_09_399

  I noticed this while investigating public nova bug 1673375, but it
  looks like that bug is caused by a ValueError coming from the
  oslo.messaging notification code, related to a circular reference in
  the json blob:

  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/notify/messaging.py", line 70, in notify
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     retry=retry)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/transport.py", line 104, in _send_notification
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     retry=retry)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 509, in send_notification
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     envelope=(version == 2.0), notify=True, retry=retry)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 457, in _send
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     msg = rpc_common.serialize_msg(msg)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/common.py", line 293, in serialize_msg
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/local/lib/python3.5/dist-packages/oslo_serialization/jsonutils.py", line 190, in dumps
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     return json.dumps(obj, default=default, **kwargs)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     **kw).encode(obj)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     chunks = self.iterencode(o, _one_shot=True)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging     return _iterencode(o, 0)
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging ValueError: Circular reference detected
  2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging

  The security issue here is that the notification payload that's logged
  has all kinds of auth secrets in it, like tokens and passwords.

  From logstash it looks like this is only hitting master (pike) right
  now.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1673569/+subscriptions