hi folks i recently had to debug a problematic interaction between eventlet and a native thread in watcher. while i probably could have printed my way though that eventually i decided to try and get debugging working in vscode instead. in this case i happened to be looking at watcher but this should apply to any eventlet service installed with devstack. in my .vscode directoy i created a launch.json that looks like this. { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python Debugger: watcher-applier", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-applier", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } }, { "name": "Python Debugger: watcher-decision-engine", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-decision-engine", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } } ] } there are a few odd things about this config. first the program i am launching is the console script run in the devstack@* systemd service file. you can find what that is using systemctl show devstack@<name> to make this work without conflict do `sudo systemctl stop devstack@<name>` first. after your done debugging you can just start the systemd service again. second when using vscode to view the source of a project select the devstack global virtual env as the project virtual env. third im telling vscode to enable gevent support. now this will raise a warning/error as gevent is not installed but you can ignore that. for those of you not familiar with gevent its a fork of eventlet that actually has much better integration with debuggers due to its wider usage outside of openstack. eventlet and gevent both use greenlet and the eventlet implementation is close enough that when you enable gevent debugging it mostly works. forth to fix the "mostly" part, specifically the fact that dajngo will explode on import, we set "PYTEST_CURRENT_TEST": "true" in the terminal env. PYTEST_CURRENT_TEST is an undocumented environment variable intended only for unit testing that is check by eventlet https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772de... when we set that to true we skip this `isinstance(o, rlock_type)` check https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772de... which crashes the debugpy debugger because it tries to load setting form the danjgo module which get loaded into the python interpreter as a side effect of debugpy. i have not tried this on other project yet but it enables me to single step debug, run to breakpoints and many of the normal debug action like using the debug console to inspect the current function scope. all of the above worked without any code change to watcher or running in a specal debug mode in the service code. we occasionally get questions on this list about how to debug openstack services and i usually respond with "its possible to do, is often fragile and need a lot of context to do right" in this case it seam to work reasonable ok in the limited usage i have made of it so far. given the current incantation is pretty simple and non invasive i decided to share this in case others find it useful. keep in mind however most service are distributed systems so if you leave it stopped on a break point for 5 mins while you inspect things, what ever you were originally doing will probably time out and it might fail in odd ways but for local development that may still be fine. please don't try this in production. i have not tried this with a wsgi app. that probably will work fine with the eventlet wsgi web-server via the console scripts i.e. nova-api binary but probably wont work for uwsig/Apache. regards sean