<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Could we replace the refresh from the period task with a timestamp in the network cache of when it was last updated so that we refresh it only when it’s accessed
if older that X ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Aaron Rosen [mailto:aaronorosen@gmail.com]
<br>
<b>Sent:</b> 29 May 2014 01:47<br>
<b>To:</b> Assaf Muller<br>
<b>Cc:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [Nova] [Neutron] heal_instance_info_cache_interval - Can we kill it?</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, May 28, 2014 at 7:39 AM, Assaf Muller <<a href="mailto:amuller@redhat.com" target="_blank">amuller@redhat.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
----- Original Message -----<br>
> Hi,<br>
><br>
> Sorry somehow I missed this email. I don't think you want to disable it,<br>
> though we can definitely have it run less often. The issue with disabling it<br>
> is if one of the notifications from neutron->nova never gets sent<br>
> successfully to nova (neutron-server is restarted before the event is sent<br>
> or some other internal failure). Nova will never update it's cache if the<br>
> heal_instance_info_cache_interval is set to 0.<o:p></o:p></p>
</div>
<p class="MsoNormal">The thing is, this periodic healing doesn't imply correctness either.<br>
In the case where you lose a notification and the compute node hosting<br>
the VM is hosting a non-trivial amount of VMs it can take (With the default<br>
of 60 seconds) dozens of minutes to update the cache, since you only<br>
update a VM a minute. I could understand the use of a sanity check if it<br>
was performed much more often, but as it is now it seems useless to me<br>
since you can't really rely on it.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I agree with you. That's why we implemented the event callback so that the cache would be more up to date. In honesty you can probably safely disable the heal_instance_info_cache_interval and things will probably be fine as we haven't
seen many failures where events from neutron fail to send. If we find out this is the case we can definitely make the event sending notification logic in neutron much more robust by persisting events to the db and implementing retry logic on failure there
to help ensure nova gets the notification. <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><br>
What I'm trying to say is that with the inefficiency of the implementation,<br>
coupled with Neutron's default plugin inability to cope with a "large"<br>
amount of API calls, I feel like the disadvantages outweigh the<br>
advantages when it comes to the cache healing.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Right the current heal_instance implementation has scaling issues as every compute node runs this task querying neutron. The more compute nodes you have the more querying. Hopefully the nova v3 api should solve this issue though as the
networking information will no longer have to live in nova as well. So someone interested in this data network data can query neutron directly and we can avoid these type of caching issues all together :) <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><br>
How would you feel about disabling it, optimizing the implementation<br>
(For example by introducing a new networking_for_instance API verb to Neutron?)<br>
then enabling it again?<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think this is a good idea we should definitely implement something like this so nova can interface with less api calls. <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
> The neutron->nova events help<br>
> to ensure that the nova info_cache is up to date sooner by having neutron<br>
> inform nova whenever a port's data has changed (@Joe Gordon - this happens<br>
> regardless of virt driver).<br>
><br>
> If you're using the libvirt virt driver the neutron->nova events will also be<br>
> used to ensure that the networking is 'ready' before the instance is powered<br>
> on.<br>
><br>
> Best,<br>
><br>
> Aaron<br>
><br>
> P.S: we're working on making the heal_network call to neutron a lot less<br>
> expensive as well in the future.<br>
><br>
><br>
><br>
><br>
> On Tue, May 27, 2014 at 7:25 PM, Joe Gordon < <a href="mailto:joe.gordon0@gmail.com">
joe.gordon0@gmail.com</a> > wrote:<br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Wed, May 21, 2014 at 6:21 AM, Assaf Muller < <a href="mailto:amuller@redhat.com">
amuller@redhat.com</a> > wrote:<br>
><br>
><br>
> Dear Nova aficionados,<br>
><br>
> Please make sure I understand this correctly:<br>
> Each nova compute instance selects a single VM out of all of the VMs<br>
> that it hosts, and every <heal_instance_info_cache_interval> seconds<br>
> queries Neutron for all of its networking information, then updates<br>
> Nova's DB.<br>
><br>
> If the information above is correct, then I fail to see how that<br>
> is in anyway useful. For example, for a compute node hosting 20 VMs,<br>
> it would take 20 minutes to update the last one. Seems unacceptable<br>
> to me.<br>
><br>
> Considering Icehouse's Neutron to Nova notifications, my question<br>
> is if we can change the default to 0 (Disable the feature), deprecate<br>
> it, then delete it in the K cycle. Is there a good reason not to do this?<br>
><br>
> Based on the patch that introduced this function [0] you may be on to<br>
> something, but AFAIK unfortunately the neutron to nova notifications only<br>
> work in libvirt right now [1], so I don' think we can fully deprecate this<br>
> periodic task. That being said turning it off by default may be an option.<br>
> Have you tried disabling this feature and seeing what happens (in the gate<br>
> and/or in production)?<br>
><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">We've disabled it in a scale lab and didn't observe any black holes forming<br>
or other catastrophes.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
><br>
> [0] <a href="https://review.openstack.org/#/c/4269/" target="_blank">https://review.openstack.org/#/c/4269/</a><br>
> [1] <a href="https://wiki.openstack.org/wiki/ReleaseNotes/Icehouse" target="_blank">
https://wiki.openstack.org/wiki/ReleaseNotes/Icehouse</a><br>
><br>
><br>
><br>
><br>
> Assaf Muller, Cloud Networking Engineer<br>
> Red Hat<br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>