Open Stack

Hi Stackers,

I'm working on the patch 53270 for Swift profiling middleware and was
blocked by a very annoying problem of the python eventlet profile which
inherits from python standard profiler. It sometimes raised AssertionError
of 'bad call' or 'bad return' in trace_dispatch_call and
trace_dispatch_return hooked function. The eventlet profiler extend the
return calls of standard profiler at line 116. This problem will be gone if
I change it back to python standard profile. So I guess it is specific to
eventlet profile. It may not correctly handle some special cases.

The 2 places in the code of standard profiler that complains:
1) https://github.com/python-git/python/blob/master/Lib/profile.py#L299
    assert rframe.f_back is frame.f_back, ("Bad call", rfn, rframe,
rframe.f_back, frame, frame.f_back)

2) https://github.com/python-git/python/blob/master/Lib/profile.py#L330
    assert frame is self.cur[-2].f_back, ("Bad return", self.cur[-3])

I don't understand why to use assert there. What does it mean if it happens
unexpected?
    The profiler crashed and profiling results can't be used because of you
    have an unexpected call?  (based on the practice of using assert
    statement to check your call contract)
    The results still can be used but not precise any more. you may need to
    catch the AssertionError.
    It is actually a bug need to be fixed here?

When I look into the eventlet profile,  I'm also very curious about two
line of code in eventlet profile module. It doesn't make sense to me since
they are not reachable in any case. anybody can explain why?
   https://github.com/eventlet/eventlet/blob/master/eventlet/green/profile.py#L103
   https://github.com/eventlet/eventlet/blob/master/eventlet/green/profile.py#L110


Here's an example of stack trace:
-----------------------------------------------------------
Traceback (most recent call last):

  File "/opt/stack/swift/bin/swift-proxy-server", line 23, in <module>
    sys.exit(run_wsgi(conf_file, 'proxy-server', default_port=8080,
**options))

  File "/opt/stack/swift/swift/common/wsgi.py", line 407, in run_wsgi
    run_server(conf, logger, sock)

  File "/opt/stack/swift/swift/common/wsgi.py", line 335, in run_server
    wsgi.server(sock, app, NullLogger(), custom_pool=pool)

  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 693,
in server
    client_socket = sock.accept()

  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line
183, in accept
    timeout_exc=socket.timeout("timed out"))

  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/__init__.py",
line 155, in trampoline
    return hub.switch()

  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line
187, in switch
    return self.greenlet.switch()

  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line
236, in run
    self.wait(sleep_time)

  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line
113, in wait
    self.block_detect_post()

  File "/usr/lib/python2.7/profile.py", line 211, in trace_dispatch

    if self.dispatch[event](self, frame,t):

  File "/opt/stack/swift/swift/common/middleware/profile.py", line 239, in
trace_dispatch_return_extend_back
    return self.trace_dispatch_return(frame, t);

  File "/usr/lib/python2.7/profile.py", line 312, in trace_dispatch_return
    assert frame is self.cur[-2].f_back, ("Bad return", frame.f_code,
self.cur[-2].f_back.f_code)

AssertionError: ('Bad return', <code object wait at 0x1a15030, file
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 75>,
<code object wait at 0x1191030, file
"/usr/local/lib/python2.7/dist-packages/eventlet/queue.py", line 123>)
-----------------------------------------------------------

Another paste example of stack trace is here.

-Edward Zhang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140219/4b5e327c/attachment.html>