hi folks i recently had to debug a problematic interaction between
eventlet and a native thread in watcher.
while i probably could have printed my way though that eventually i
decided to try and get debugging working
in vscode instead.


in this case i happened to be looking at watcher but this should apply
to any eventlet service installed with devstack.


in my .vscode directoy i created a launch.json that looks like this.

{
     // Use IntelliSense to learn about possible attributes.
     // Hover to view descriptions of existing attributes.
     // For more information, visit:
https://go.microsoft.com/fwlink/?linkid=830387
     "version": "0.2.0",
     "configurations": [
         {
             "name": "Python Debugger: watcher-applier",
             "type": "debugpy",
             "request": "launch",
             "program": "/opt/stack/data/venv/bin/watcher-applier",
             "console": "integratedTerminal",
             "args": [
                 "--config-file",
                 "/etc/watcher/watcher.conf"
             ],
             "gevent": true,
             "env" : {
                 "PYTEST_CURRENT_TEST": "true"
             }
         },
         {
             "name": "Python Debugger: watcher-decision-engine",
             "type": "debugpy",
             "request": "launch",
             "program": "/opt/stack/data/venv/bin/watcher-decision-engine",
             "console": "integratedTerminal",
             "args": [
                 "--config-file",
                 "/etc/watcher/watcher.conf"
             ],
             "gevent": true,
             "env" : {
                 "PYTEST_CURRENT_TEST": "true"
             }
         }
     ]
}


there are a few odd things about this config.

first the program i am launching is the console script run in the
devstack@* systemd service file.
you can find what that is using systemctl show devstack@<name>
to make this work without conflict do `sudo systemctl stop
devstack@<name>` first.
after your done debugging you can just start the systemd service again.

second when using vscode to view the source of a project select the
devstack global virtual env as the project virtual env.

third im telling vscode to enable gevent support.
now this will raise a warning/error as gevent is not installed but you
can ignore that.
for those of you not familiar with gevent its a fork of eventlet that
actually has much better integration
with debuggers due to its wider usage outside of openstack.
eventlet and gevent both use greenlet and the eventlet implementation is
close enough that when you enable gevent debugging
it mostly works.

forth to fix the "mostly" part, specifically the fact that dajngo will
explode on import, we set "PYTEST_CURRENT_TEST": "true" in the terminal env.

PYTEST_CURRENT_TEST is an undocumented environment variable intended
only for unit testing that is check by eventlet
https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772de23fcdb/eventlet/patcher.py#L403-L407
when we set that to true we skip this `isinstance(o, rlock_type)` check
https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772de23fcdb/eventlet/patcher.py#L413
which crashes the debugpy debugger because it tries to load setting form
the danjgo module which get loaded into
the python interpreter as a side effect of debugpy.

i have not tried this on other project yet but it enables me to single
step debug, run to breakpoints and many of the normal
debug action like using the debug console to inspect the current
function scope.
all of the above worked without any code change to watcher or running in
a specal debug mode in the service code.

we occasionally get questions on this list about how to debug openstack
services and i usually respond with

"its possible to do, is often fragile and need a lot of context to do right"

in this case it seam to work reasonable ok in the limited usage i have
made of it so far.
given the current incantation is pretty simple and non invasive i
decided to share this
in case others find it useful.

keep in mind however most service are distributed systems so if you
leave it stopped
on a break point for 5 mins while you inspect things, what ever you were
originally doing will probably time out and
it might fail in odd ways but for local development that may still be
fine. please don't try this in production.

i have not tried this with a wsgi app. that probably will work fine with
the eventlet wsgi web-server via the console scripts

i.e. nova-api binary but probably wont work for uwsig/Apache.


regards

sean