I understand Nova's requirements, but OVN heartbeat has a significant impact on the Southbound Database.

We have a related topic on etherpad about this (Vancouver PTG): 

• "(labedz) OVN heartbeat mechanism - big mechanism with significant infrastructure impact for ? Why we need to be on OVN southbound with Neutron?"

Sean mentioned some reasons to use the metadata heartbeat mechanism:

• "i would suggest startign a patch if you belive the current behavior will be probelmatic but keep in mind that addign too much jitter/delay can cause vm boots/migtrations to randomly fail leavign the instance in an error state."

Maybe we shouldn't try to get the network agent status without considering the OVN backend impact. OVN should take a long time processing messages from ovs-vswitchd daemon on chassis (OVSDB transactions). In this case, the ovn-controller still blocked by the unix socket between ovn-controller <-> ovs-vswitchd, and during this sync the ovn-controller cannot process any "heartbeat" because ovn-controller is busy with the last cfg. In other words, the time to bump the heartbeat cfg is very dependent on the number of resources used (scaling).

This specific patch is related to "OVN Metadata agent" heartbeat and use the neutron:ovn-metadata-sb-cfg to bump nb_cfg config number:

            table = ('Chassis_Private' if self.agent.has_chassis_private
                     else 'Chassis')
            self.agent.sb_idl.db_set(
                table, self.agent.chassis, ('external_ids', {
                    ovn_const.OVN_AGENT_METADATA_SB_CFG_KEY:
                        str(row.nb_cfg)})).execute()

As I understand, this is very similar to "OVN Controller agent" heartbeat but in ovn-controller case we are talking about the "neutron:liveness_check_at" to bump cfg on NB_Global table.
        last_ping = self.nb_ovn.nb_global.external_ids.get(
            ovn_const.OVN_LIVENESS_CHECK_EXT_ID_KEY)
            
In both cases, to transition cfg numbers we need ovn-controller availability... I suppose that being able to customize this value is better for large-scale cases.

It seems to me that's what we talked about in Vancouver Rodolfo (scalability vs reliability). OVN needs to evolve with I-P (incremental processing) to respond faster to configuration changes, but while that doesn't happen, we'll have to live with bigger timeouts...