<div dir="ltr">Hi, <div><br></div><div>I found the issue and it was the way I was starting and stopping the listener. I got some more logs below that helped me understand this deadlock behavior. I have fixed issue in my app. </div><div><br></div><div>OPS_WORKER_0--DEBUG-2020-09-02 02:39:30,309-amqpdriver.py-322 - AMQPListener connection consume<br>OPS_WORKER_0--DEBUG-2020-09-02 02:39:30,310-connection.py-712 - heartbeat_tick : for connection b616c1205a61466a94e4ae2e79e6ba84<br>OPS_WORKER_0--DEBUG-2020-09-02 02:39:30,310-connection.py-734 - heartbeat_tick : Prev sent/recv: 8/8, now - 8/8, monotonic - 32.014996083, last_heartbeat_sent - 0.97555832, heartbeat int. - 60 for connection b616c1205a61466a94e4ae2e79e6ba84<br><span style="background-color:rgb(255,255,0)">MainProcess--WARNING-2020-09-02 02:39:34,108-server.py-127 - Possible hang: stop is waiting for start to complete<br></span>MainProcess--DEBUG-2020-09-02 02:39:34,117-server.py-128 - </div><div> File "/testprogs/python/oslo_notif/main.py", line 33, in <module> main()<br> File "/testprogs/python/oslo_notif/main.py", line 29, in main worker.stop()<br> File "/testprogs/python/oslo_notif/oslo_worker.py", line 85, in stop self.__amqp_handler.stop_amqp_event_listener()<br> File "/testprogs/python/oslo_notif/oslo_notif_handler.py", line 184, in stop_amqp_event_listener self.__amqp_listener.stop()<br> File "/pyvenv37/lib/python3.7/site-packages/oslo_messaging/server.py", line 264, in wrapper log_after, timeout_timer)<br> File "/pyvenv37/lib/python3.7/site-packages/oslo_messaging/server.py", line 163, in wait_for_completion msg, log_after, timeout_timer)<br> File "/pyvenv37/lib/python3.7/site-packages/oslo_messaging/server.py", line 128, in _wait<br></div><div><br></div><div>/anil.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 2, 2020 at 12:12 AM Anil Jangam <<a href="mailto:anilj.mailing@gmail.com">anilj.mailing@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi, <div><br></div><div>I have coded OpenStack AMQP listener following the example and it is working fine. </div><div></div><div><a href="https://github.com/gibizer/nova-notification-demo/blob/master/ws_forwarder.py" target="_blank">https://github.com/gibizer/nova-notification-demo/blob/master/ws_forwarder.py</a></div><div><br></div><div>The related code snippets of the NotificationHandler class are shown as follows. <br></div><div><pre style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"JetBrains Mono",monospace;font-size:10.5pt"><span style="color:rgb(128,128,128)"># Initialize the AMQP listener<br></span><span style="color:rgb(204,120,50)">def </span><span style="color:rgb(255,198,109)">init</span>(<span style="color:rgb(148,85,141)">self</span><span style="color:rgb(204,120,50)">, </span>cluster_ip<span style="color:rgb(204,120,50)">, </span>user<span style="color:rgb(204,120,50)">, </span>password<span style="color:rgb(204,120,50)">, </span>port):<br> cfg.CONF()<br> cluster_url = <span style="color:rgb(106,135,89)">"rabbit://" </span>+ user + <span style="color:rgb(106,135,89)">":" </span>+ password + <span style="color:rgb(106,135,89)">"@" </span>+ cluster_ip + <span style="color:rgb(106,135,89)">":" </span>+ port + <span style="color:rgb(106,135,89)">"/"<br></span><span style="color:rgb(106,135,89)"> </span>transport = oslo_messaging.get_notification_transport(cfg.CONF<span style="color:rgb(204,120,50)">, </span><span style="color:rgb(170,73,38)">url</span>=cluster_url)<br> targets = [<br> oslo_messaging.Target(<span style="color:rgb(170,73,38)">topic</span>=<span style="color:rgb(106,135,89)">'versioned_notifications'</span>)<span style="color:rgb(204,120,50)">,<br></span><span style="color:rgb(204,120,50)"> </span>]<br> endpoints = [<span style="color:rgb(148,85,141)">self</span>.__endpoint]<br><br> <span style="color:rgb(128,128,128)"># Initialize the notification listener<br></span><span style="color:rgb(128,128,128)"> </span><span style="color:rgb(204,120,50)">try</span>:<br> <span style="color:rgb(148,85,141)">self</span>.__amqp_listener = oslo_messaging.get_notification_listener(transport<span style="color:rgb(204,120,50)">,<br></span><span style="color:rgb(204,120,50)"> </span>targets<span style="color:rgb(204,120,50)">,<br></span><span style="color:rgb(204,120,50)"> </span>endpoints<span style="color:rgb(204,120,50)">,<br></span><span style="color:rgb(204,120,50)"> </span><span style="color:rgb(170,73,38)">executor</span>=<span style="color:rgb(106,135,89)">'threading'</span>)<br> <span style="color:rgb(204,120,50)">except </span><span style="color:rgb(136,136,198)">NotImplementedError </span><span style="color:rgb(204,120,50)">as </span>err:<br> LOGGER.error(<span style="color:rgb(106,135,89)">"Failed to initialize the notification listener {}"</span>.format(err))<br> <span style="color:rgb(204,120,50)">return False<br></span><span style="color:rgb(204,120,50)"><br></span><span style="color:rgb(204,120,50)"> </span>LOGGER.debug(<span style="color:rgb(106,135,89)">"Initialized the notification listener {}"</span>.format(cluster_url))<br> <span style="color:rgb(204,120,50)">return True<br></span><span style="color:rgb(204,120,50)"><br></span><span style="color:rgb(128,128,128)"># Arm the compute event listeners<br></span><span style="color:rgb(204,120,50)">def </span><span style="color:rgb(255,198,109)">start_amqp_event_listener</span>(<span style="color:rgb(148,85,141)">self</span>):<br> <span style="color:rgb(128,128,128)"># Start the notification handler<br></span><span style="color:rgb(128,128,128)"> </span>LOGGER.debug(<span style="color:rgb(106,135,89)">"Started the OpenStack notification handler"</span>)<br> <span style="color:rgb(148,85,141)">self</span>.__amqp_listener.start()<br><br><span style="color:rgb(128,128,128)"># Disarm the compute event listeners<br></span><span style="color:rgb(204,120,50)">def </span><span style="color:rgb(255,198,109)">stop_amqp_event_listener</span>(<span style="color:rgb(148,85,141)">self</span>):<br> LOGGER.debug(<span style="color:rgb(106,135,89)">"Stopping the OpenStack notification handler"</span>)<br> <span style="color:rgb(204,120,50)">if </span><span style="color:rgb(148,85,141)">self</span>.__amqp_listener <span style="color:rgb(204,120,50)">is not None</span>:<br> <span style="color:rgb(148,85,141)">self</span>.__amqp_listener.stop()</pre>I am using this interface from a new process handler function, however, when I invoke the stop_amqp_eent_listener() method, my process hangs. It does not terminate naturally. </div><div><div>I verified that the <span style="color:rgb(148,85,141)">self</span>.__amqp_listener.stop() function is not returning. Is there anything missing in this code? Is there any specific consideration when calling the listener from a new process? </div><div><br></div><div>Can someone provide a clue? </div><div></div></div><div><pre style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"JetBrains Mono",monospace;font-size:10.5pt"><span style="color:rgb(128,128,128)"># Stop the worker<br></span><span style="color:rgb(204,120,50)">def </span><span style="color:rgb(255,198,109)">stop</span>(<span style="color:rgb(148,85,141)">self</span>):<br> <span style="color:rgb(128,128,128)"># Stop the AMQP notification handler<br></span><span style="color:rgb(128,128,128)"> </span><span style="color:rgb(148,85,141)">self</span>.__amqp_handler.stop_amqp_event_listener()<br> LOGGER.debug(<span style="color:rgb(106,135,89)">"Stopped the worker for {}"</span>.format(<span style="color:rgb(148,85,141)">self</span>.__ops_conn_info.cluster_ip))</pre></div><div><br></div><div>/anil.</div><div><br></div><div><br></div></div>
</blockquote></div>