<div dir="ltr">I understand Nova's requirements, but OVN heartbeat has a significant impact on the Southbound Database.<br><br>We have a related topic on etherpad about this (Vancouver PTG): <br><ul><li>"(labedz) OVN heartbeat mechanism - big mechanism with significant infrastructure impact for ? Why we need to be on OVN southbound with Neutron?"</li></ul><br>Sean mentioned some reasons to use the metadata heartbeat mechanism:<br><ul><li>"i would suggest startign a patch if you belive the current behavior will be probelmatic but keep in mind that addign too much jitter/delay can cause vm boots/migtrations to randomly fail leavign the instance in an error state."</li></ul><br>Maybe we shouldn't try to get the network agent status without considering the OVN backend impact. OVN should take a long time processing messages from ovs-vswitchd daemon on chassis (OVSDB transactions). In this case, the ovn-controller still blocked by the unix socket between ovn-controller <-> ovs-vswitchd, and during this sync the ovn-controller cannot process any "heartbeat" because ovn-controller is busy with the last cfg. In other words, the time to bump the heartbeat cfg is very dependent on the number of resources used (scaling).<br><br>This specific patch is related to "OVN Metadata agent" heartbeat and use the neutron:ovn-metadata-sb-cfg to bump nb_cfg config number:<br><br> table = ('Chassis_Private' if self.agent.has_chassis_private<br> else 'Chassis')<br> self.agent.sb_idl.db_set(<br> table, self.agent.chassis, ('external_ids', {<br> ovn_const.OVN_AGENT_METADATA_SB_CFG_KEY:<br> str(row.nb_cfg)})).execute()<br><br>As I understand, this is very similar to "OVN Controller agent" heartbeat but in ovn-controller case we are talking about the "neutron:liveness_check_at" to bump cfg on NB_Global table.<br> last_ping = self.nb_ovn.nb_global.external_ids.get(<br> ovn_const.OVN_LIVENESS_CHECK_EXT_ID_KEY)<br> <br>In both cases, to transition cfg numbers we need ovn-controller availability... I suppose that being able to customize this value is better for large-scale cases.<br><br>It seems to me that's what we talked about in Vancouver Rodolfo (scalability vs reliability). OVN needs to evolve with I-P (incremental processing) to respond faster to configuration changes, but while that doesn't happen, we'll have to live with bigger timeouts...<br><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em qua., 21 de jun. de 2023 às 18:00, Ihar Hrachyshka <<a href="mailto:ihrachys@redhat.com">ihrachys@redhat.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Jun 19, 2023 at 11:04 AM Roberto Bartzen Acosta <<a href="mailto:roberto.acosta@luizalabs.com" target="_blank">roberto.acosta@luizalabs.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello Neutron folks,<br><br>We discussed in the Operators feedback session about OVN heartbeat and the use of "infinity" values for large-scale deployments because we have a significant infrastructure impact when a short 'agent_down_time' is configured.</div></blockquote><div><br></div><div>This is tangentially related, but note that using "infinity" values for agent_down_time is unsafe: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=2215407" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=2215407</a> (depending on whether your "infinity" value is larger than ~15 days, assuming 32 bit ints used on your platform).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"> <br><br>The merged patch [1] limited the maximum delay to 10 seconds. I understand the requirement to use random values to avoid load spikes, but why does this fix limit the heartbeat to 10 seconds? What is the goal of the agent_down_time parameter in this case? How will it work for someone who has hundreds of compute nodes / metadata agents? <br><br>Regards,<br>Roberto<br><br>[1] - <a href="https://review.opendev.org/c/openstack/neutron/+/883687" target="_blank">https://review.opendev.org/c/openstack/neutron/+/883687</a><br></div>
<br>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:"Open Sans";font-size:14px;line-height:21px;background-color:rgb(255,255,255)"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:"Open Sans";font-size:14px;line-height:21px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><br></div></div></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><i style="font-family:arial,sans-serif;font-size:x-small">‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.</i></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><p style="font-family:arial,sans-serif;text-align:justify"><i><font size="1"> </font></i><i><font size="1">‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.</font></i></p></div></div></div></div></blockquote></div></div>
</blockquote></div>
<br>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:'Open Sans';font-size:14px;line-height:21px;background-color:rgb(255,255,255)"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8000001907349px;line-height:normal"><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:'Open Sans';font-size:14px;line-height:21px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8000001907349px;line-height:normal"><br></div></div></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><i style="font-family:arial,sans-serif;font-size:x-small">‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.</i></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><p style="font-family:arial,sans-serif;text-align:justify"><i><font size="1"> </font></i><i><font size="1">‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.</font></i></p></div></div></div></div>