<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px; ">
<div style="font-size: 14px; ">Hi Irena,</div>
<div style="font-size: 14px; "><br>
</div>
<div style="font-size: 14px; ">I agree on your first comment. </div>
<div style="font-size: 14px; "><br>
</div>
<div style="font-size: 14px; ">see inline as well.</div>
<div style="font-size: 14px; "><br>
</div>
<div style="font-size: 14px; ">thanks,</div>
<div style="font-size: 14px; ">Robert</div>
<div style="font-size: 14px; "><br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="font-size: 14px; ">
<div>
<div>On 1/27/14 10:54 AM, "Irena Berezovsky" <<a href="mailto:irenab@mellanox.com">irenab@mellanox.com</a>> wrote:</div>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hi Robert, all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">My comments inline<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Irena<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "> Robert Li (baoli) [<a href="mailto:baoli@cisco.com">mailto:baoli@cisco.com</a>]
<br>
<b>Sent:</b> Monday, January 27, 2014 5:05 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [nova][neutron] PCI pass-through SRIOV<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Hi Folks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">In today's meeting, we discussed a scheduler issue for SRIOV. The basic requirement is for coexistence of the following compute nodes in a cloud:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- SRIOV only compute nodes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- non-SRIOV only compute nodes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- Compute nodes that can support both SRIOV and non-SRIOV ports. Lack of a proper name, let's call them compute nodes with hybrid NICs support, or
simply hybrid compute nodes.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">I'm not sure if it's practical in having hybrid compute nodes in a real cloud. But it may be useful in the lab to bench mark the performance differences between
SRIOV, non-SRIOV, and coexistence of both.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">[IrenaB]
<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I would like to clarify a bit on the requirements you stated below.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">As I see it, the hybrid compute nodes actually can be preferred in the real cloud, since one can define VM with one vNIC attached via SR-IOV virtual
function while the other via some vSwitch.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">But it definitely make sense to land VM with ‘virtio’ vNICs only on the non-SRIOV compute node.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Maybe there should be some sort of preference order of suitable nodes in scheduler choice, based on vnic types required for the VM.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">In a cloud that supports SRIOV in some of the compute nodes, a request such as:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> nova boot —flavor m1.large —image <image-uuid> --nic net-id=<net-uuid> vm<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">doesn't require a SRIOV port. However, it's possible for the nova scheduler to place it on a compute node that supports sriov port only. Since neutron plugin
runs on the controller, port-create would succeed unless neutron knows the host doesn't support non-sriov port. But connectivity on the node would not be established since no agent is running on that host to establish such connectivity. <o:p></o:p></span></p>
<p class="MsoNormal"><b><i><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">[IrenaB] I<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Having ML2 plugin as neutron backend, will fail to bind the port, in no agent is running on the Host</span></i></b><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</span>
<div>[ROBERT] If a host supports SRIOV only, and there is an agent running on the host to support SRIOV, would binding succeed in ML2 plugin for the above 'nova boot' request? </div>
<div><br>
</div>
<div>On a hybrid compute node, can we run multiple neutron L2 agents on a single host? It seems possible. </div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="font-size: 14px; ">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<div lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Irena brought up the idea of using host aggregate. This requires creation of a non-SRIOV host aggregate, and use that in the above 'nova boot' command. It
should work.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">The patch I had introduced a new constraint in the existing PCI passthrough filter. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">The consensus seems to be having a better solution in a later release. And for now, people can either use host aggregate or resort to their own means.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Let's keep the discussion going on this. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Robert<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">On 1/24/14 4:50 PM, "Robert Li (baoli)" <<a href="mailto:baoli@cisco.com">baoli@cisco.com</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Hi Folks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Based on Thursday's discussion and a chat with Irena, I took the liberty to add a summary and discussion points for SRIOV on Monday and onwards. Check it
out <a href="https://wiki.openstack.org/wiki/Meetings/Passthrough">https://wiki.openstack.org/wiki/Meetings/Passthrough</a>. Please feel free to update it. Let's try to finalize it next week. The goal is to determine the BPs that need to get approved, and
to start coding. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Robert<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">On 1/22/14 8:03 AM, "Robert Li (baoli)" <<a href="mailto:baoli@cisco.com">baoli@cisco.com</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Sounds great! Let's do it on Thursday.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">--Robert<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">On 1/22/14 12:46 AM, "Irena Berezovsky" <<a href="mailto:irenab@mellanox.com">irenab@mellanox.com</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hi Robert, all,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I would suggest not to delay the SR-IOV discussion to the next week.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Let’s try to cover the SRIOV side and especially the nova-neutron interaction points and interfaces this Thursday.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Once we have the interaction points well defined, we can run parallel patches to cover the full story.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks a lot,</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Irena
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; color: black; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; color: black; "> Robert Li (baoli) [<a href="mailto:baoli@cisco.com">mailto:baoli@cisco.com</a>]
<br>
<b>Sent:</b> Wednesday, January 22, 2014 12:02 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> [openstack-dev] [nova][neutron] PCI passthrough SRIOV</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Hi Folks,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">As the debate about PCI flavor versus host aggregate goes on, I'd like to move forward with the SRIOV side of things in the same time. I know that tomorrow's
IRC will be focusing on the BP review, and it may well continue into Thursday. Therefore, let's start discussing SRIOV side of things on Monday. </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Basically, we need to work out the details on:</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- regardless it's PCI flavor or host aggregate or something else, how to use it to specify a SRIOV port. </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- new parameters for —nic</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- new parameters for neutron net-create/neutron port-create</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- interface between nova and neutron</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- nova side of work</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> -- neutron side of work</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">We should start coding ASAP.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Thanks,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; ">Robert</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; color: black; "> </span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</span>
</body>
</html>