<div dir="ltr"><div dir="ltr"><br></div>Thanks for your feedback Sean. <br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em seg., 19 de jun. de 2023 às 14:01, <<a href="mailto:smooney@redhat.com">smooney@redhat.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, 2023-06-19 at 12:03 -0300, Roberto Bartzen Acosta wrote:<br>
> Hello Neutron folks,<br>
> <br>
> We discussed in the Operators feedback session about OVN heartbeat and the<br>
> use of "infinity" values for large-scale deployments because we have a<br>
> significant infrastructure impact when a short 'agent_down_time' is<br>
> configured.<br>
agent_down_time is intended to specify how long the heartbeat can be missed before<br>
the agent is considered down. it was not intented to contol the interval at which the heatbeat<br>
was sent.<br>
<br>
<a href="https://opendev.org/openstack/neutron/commit/628442aed7400251f12809a45605bd717f494c4e" rel="noreferrer" target="_blank">https://opendev.org/openstack/neutron/commit/628442aed7400251f12809a45605bd717f494c4e</a><br>
intoduced a colation between the two but it resulted in the agent incorrectly being considered down<br>
and causing port binding failures if the agent_down_time was set too large.<br>
> <br>
> The merged patch [1] limited the maximum delay to 10 seconds. I understand<br>
> the requirement to use random values to avoid load spikes, but why does<br>
> this fix limit the heartbeat to 10 seconds? What is the goal of the<br>
> agent_down_time parameter in this case? How will it work for someone who<br>
> has hundreds of compute nodes / metadata agents?<br>
the change in [1] shoudl just change the delay before _update_chassis is invoked<br>
that at least was the intent. im expecting the interval between heatbeats to be ratlimaited<br>
via the mechim that was used before <br>
<a href="https://opendev.org/openstack/neutron/commit/628442aed7400251f12809a45605bd717f494c4e?style=split&whitespace=show-all" rel="noreferrer" target="_blank">https://opendev.org/openstack/neutron/commit/628442aed7400251f12809a45605bd717f494c4e?style=split&whitespace=show-all</a><br>
was implemented.<br>
<br>
i.e. whwen a SbGlobalUpdateEvent is generated now we are clamping the max wait to 10 seconds instead of<br>
cfg.CONF.agent_down_time // 2 which was causing port binding failures.<br>
<br>
the timer object will run the passed in fucntion after the timer interval has expired.<br>
<br>
<a href="https://docs.python.org/3/library/threading.html#timer-objects" rel="noreferrer" target="_blank">https://docs.python.org/3/library/threading.html#timer-objects</a><br>
<br>
but it will not re run multiple times and the function we are invoking does not loop internally<br>
so only one update will happen per invocation of run.<br>
<br>
i believe the actual heatbeat/reporting interval is controlled by cfg.CONF.AGENT.report_interval<br>
<br>
<a href="https://github.com/openstack/neutron/blob/cbb89fdb1414a1b3a8e8b3a9a4154ef627bb9d1a/neutron/agent/metadata/agent.py#L313-L317" rel="noreferrer" target="_blank">https://github.com/openstack/neutron/blob/cbb89fdb1414a1b3a8e8b3a9a4154ef627bb9d1a/neutron/agent/metadata/agent.py#L313-L317</a><br>
<br>
so i think if you want to reduce the interval in a large envionment to can do that by setting<br>
<br>
[AGENT]<br>
report_interval=<your value><br></blockquote><div><br>I agree that the mechanism for sending heartbeats is controlled by report_interval, however, from what I understand, their original idea would be to configure complementary values: report_interval and agent_down_time would be associated with the status of network agents.<br><br><a href="https://docs.openstack.org/neutron/2023.1/configuration/neutron.html">https://docs.openstack.org/neutron/2023.1/configuration/neutron.html</a><br>report_interval: <span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px">"Seconds between nodes reporting state to server; should be less than agent_down_time, best if it is half or less than agent_down_time."<br></span>agent_down_time: "<span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px">Seconds to regard the agent is down; should be at least twice report_interval, to be sure the agent is down for good."</span><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px"><br></span></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
im not that familiar with this code but that was my original understanding.<br>
the sllep before its rerun is calucated in oslo.service<br>
<a href="https://github.com/openstack/oslo.service/blob/1.38.0/oslo_service/loopingcall.py#L184-L194" rel="noreferrer" target="_blank">https://github.com/openstack/oslo.service/blob/1.38.0/oslo_service/loopingcall.py#L184-L194</a><br>
<a href="https://github.com/openstack/oslo.service/blob/1.38.0/oslo_service/loopingcall.py#L154-L159" rel="noreferrer" target="_blank">https://github.com/openstack/oslo.service/blob/1.38.0/oslo_service/loopingcall.py#L154-L159</a><br>
<br>
the neutron core team can correct me if that is incorrect but i would not expct this to negitivly impact large clouds.<br></blockquote><div><br>Note 1: My point is that the function SbGlobalUpdateEvent seems to be using the agent_down_time disassociated from the original function ( double / half relation).<br><br>Note 2: I'm curious to know the behavior of this modification with more than 200 chassis and with thousands of OVN routers. In this case, with many configurations being applied at the same time (a lot of events in SB_Global) and that require the agent running on Chassis to respond the report_interval at the same time as it is transitioning configs (probably millions of openflow flows entries). Is 10 seconds enough?<br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> <br>
> Regards,<br>
> Roberto<br>
> <br>
> [1] - <a href="https://review.opendev.org/c/openstack/neutron/+/883687" rel="noreferrer" target="_blank">https://review.opendev.org/c/openstack/neutron/+/883687</a><br>
> <br>
<br>
</blockquote></div></div>
<br>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:'Open Sans';font-size:14px;line-height:21px;background-color:rgb(255,255,255)"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8000001907349px;line-height:normal"><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><div style="color:rgb(97,97,97);font-family:'Open Sans';font-size:14px;line-height:21px"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8000001907349px;line-height:normal"><br></div></div></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><i style="font-family:arial,sans-serif;font-size:x-small">‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’.</i></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:1.3em"><p style="font-family:arial,sans-serif;text-align:justify"><i><font size="1"> </font></i><i><font size="1">‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.</font></i></p></div></div></div></div>