Thanks for the answer, Michael! That was a planned restart of Octavia that was done via Kolla, so nothing unpredictable happened :)
Ideally we wanted to ACK the message once the TaskFlow is launched
That said, with a graceful shutdown, the controller worker should complete
Absolutely voting for this way ... the processing and load the job into Taskflow before shutting the thread down. For this reason I am not 100% convinced this is the issue, but it is something to investigate more. Well, right now we don't have a proper logging level configured to be able to confirm or deny that this is exactly the case. We just did not see anything about this particular event/action after receiving this event by a worker, so that is why I'm thinking about that ... Currently, I see that we have 1 minute for graceful shutdown and as per logs we did not exceed this time for shutting down, but maybe (again) our log level just not showing enough. But another thing – we've had a little huge pool (and a little huge lb in general), ~50x listeners, 1x pool for each, 40x members for each pool. So one way or another we could spend more than 10-20 seconds preparing the task :) For sure we will increase the log level and continue debugging on our side, I'll let the community know if we will identify something here ...