Hi Florian, That is very unusual. It typically takes less than 30 seconds for a load balancer to be provisioned. It definitely sounds like the mysql instance is having trouble. This can also cause longer term issues if the query response time drops to 10 seconds or more(0.001 is normal), which could trigger unnecessary failovers. In Octavia there are layers of "retries" to attempt to handle clouds that are having trouble. It sounds like database issues are triggering one or more of these retries. There are a few retries that will be in play for database transactions: MySQL internal retries/timeouts such as lock timeouts (logged on the mysql side) oslo.db includes some automatic retries (typically not logged without configuration file settings) Octavia tenacity and flow retries (Typically logged if the configuration file has Debug = True enabled) This may also be a general network connection issue. The default REST timeouts (used when we connect to the amphora agents) is 600, I'm wondering if the lb-mgmt-network is also having an issue. Please check your health manager log files. If there are database query time issues logged, it would point specifically to a mysql issue. In the past we have seen mysql clustering setups that were bad and caused performance issues (flipping primary instance, lock contention between the instances, etc.). You should not be seeing any log messages that the mysql database went away, that is not normal. Michael On Sun, Nov 8, 2020 at 7:06 AM Florian Rommel <florian@datalounges.com> wrote:
Hi, so we have a fully functioning setup of octavia on ussuri and it works nicely, when it competes. So here is what happens: From octavia api to octavia worker takes 20 seconds for the job to be initiated. The loadbalancer gets built quickly and then we get a mysql went away error, the listener gets built and then a member , that works too, then the mysql error comes up with query took too long to execute. Now this is where it gets weird. This is all within the first 2 - 3 minutes. At this point it hangs and takes 10 minutes (600 seconds) for the next step to complete and then another 10 minutes and another 10 until it’s completed. It seems there is a timeout somewhere but even with debug on we do not see what is going on. Does anyone have a mysql 8 running and octavia executing fine? And could send me their redacted octavia or mysql conf files? We didn’t touch them but it seems that there is something off.. especially since it then completes and works extremely nicely. I would highly appreciate it , even off list. Best regards, //f