<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Ian,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you for hosting the discussion.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I feel we jumped into the details without formalizing the tenant/provider use cases. I think it worth to define End2End user stories and take gradual steps
for solution. It can definitely be outlined by the blueprint.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think that Quantum related questions are still to be discussed:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-- Is it extension or plugin. If its plugin is it generic or vendor specific<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-- Impact on Nova - Quantum interaction<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-- Does it require Quantum Scheduling
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We can discuss it either prior to the summit (webex, emails) or at the summit.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Irena<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> froggienator@gmail.com [mailto:froggienator@gmail.com]
<b>On Behalf Of </b>Ian Wells<br>
<b>Sent:</b> Wednesday, April 03, 2013 9:35 PM<br>
<b>To:</b> Jiang, Yunhong; Itzik Brown; Vladimir Popovski; Irena Berezovsky; niilo.minkkinen.ext@nsn.com; Chuck Short; OpenStack Development Mailing List<br>
<b>Cc:</b> Ian Wells (iawells@cisco.com)<br>
<b>Subject:</b> Re: PCI-passthrough - preliminary meeting this WEDNESDAY 10am PST / 7pm CET<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Thanks for attending, everyone!<br>
<br>
Here's the notes from the meeting today (generally formatted as title - slide content - comments from meeting). Out of attachment-phobia I've taken the text out of the presentation. They're a bit brief, so feel free to ask for clarification if it's needed.<br>
<br>
As I said, if there's enough interest we could organise another meeting next week; and maybe it would be a good idea to get the outline of a blueprint ready for discussion at the Summit session.<o:p></o:p></p>
</div>
<p class="MsoNormal">Cheers,<br>
-- <o:p></o:p></p>
</div>
<p class="MsoNormal">Ian.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
Problems to solve<br>
- Configuring available devices<br>
- Permitting users to use passthrough<br>
- Marking passthrough as supported<br>
- Booting with passthrough<br>
- Finding a free device<br>
- Mapping the device<br>
- Networking-specific issues<br>
Additional:<br>
- Migration – possible but hard; not for the first release<br>
- Run-time assignment – possible but not even supported for virtual devices at the moment; not for the first release<br>
<br>
<br>
Configuring available devices<br>
- On the compute server:<br>
pci_dev={JSON describing the device}<br>
Includes PCI path<br>
Includes PCI device type (or nova-compute could discover this via the hypervisor driver)<br>
May include undiscoverable attributes like where a network card is connected<br>
<br>
.. What attributes needs specifying?<br>
Other hypervisors than KVM? (Xen should be possible directly via libvirt; xenapi, VMWare, …)<br>
Given meeting attendance, likely our first target would be libvirt/KVM, though<br>
Direct PCI versus SR-IOV (but SR-IOV function count is typically static so it they could be listed in the Nova config)<br>
Another option: don’t map the (network) device directly but use a vNIC linked directly to it (see Mellanox code, below)<br>
<br>
<br>
Permitting passthrough<br>
- Nova must have extra markers in the flavor specifying that the VM is entitled to PCI devices<br>
- Customers can be charged differently<br>
<br>
In the future, maybe ‘start with PCI if PCI is available?’ rather than ‘start with PCI and don’t start if none is available’<br>
<br>
<br>
Marking support on the Glance image<br>
- (Optional)<br>
- Show which images support a device type; refuse to start unless the device is supported<br>
- Mark an image with properties<br>
<br>
<br>
Booting with passthrough<br>
- Extra arguments to the ‘nova boot’ command<br>
- May be general purpose arguments…<br>
-passthrough-dev=gpu<br>
... or special specific arguments<br>
-nic net_id=XXX,passthrough_dev=ixgbe<br>
<br>
Likely that we have to specify a list of constraints (driver type, device class, tagged in a certain way) rather than a simple type<br>
Could put characteristics on Quantum port and not on nic parameter<br>
Some things will already have to be loaded off the port, e.g. network discriminator, firewalling<br>
Some things will be available there in the future, likely QoS information<br>
Connection between PCI port and CPU (i.e. that the PCI device is local to the CPU to which VM is bound to avoid putting the QPI interprocessor interconnect in the PCI datapath) - of future interest<br>
<br>
<br>
Finding a free device<br>
- New scheduler behaviour<br>
- Compute devices report passthrough resources<br>
- Scheduler tries to find a free resource<br>
- Resources updated when VMs use and free them<br>
- New information in DB<br>
- Need to keep track of the above resources<br>
<br>
Needs checking – does this information get stored to the DB or is it in the scheduler only?<br>
Capabilities, properties - ‘a device connected to this network’, ‘a device of this type’<br>
<br>
<br>
Mapping the device<br>
- Changes to the hypervisor driver to configure the VM<br>
We’ve changed libvirt/driver.py<br>
Changing the XML is easy<br>
- Bare-metal guys may also want to use the scheduler<br>
If their baremetal machines are described they’ll get free PCI device selection from the same scheduler<br>
The device will be present whether you use it or not; some VMs will get ‘freebies’<br>
<br>
<br>
Networking-specific cases<br>
- We want to integrate a NIC into a Quantum network<br>
This would involve a PCI-enabled NIC driver<br>
pci_dev in config may say how the NIC’s attached to the network<br>
Hypervisor driver tells Quantum this information<br>
Quantum configures the switch that the network device is attached to, instead of the OVS that the vNIC is attached to<br>
Or Quantum may indicate that the PF should be set to encapsulate all that VF’s traffic (with VLAN tags, for instance)<br>
<br>
Firewalling will be a problem – may need programming on the NIC or the switch, may simply not be available<br>
Should we be programming the VF encap from the Quantum agent or from information passed to Nova from the Quantum plugging call?<br>
<br>
<br>
Zadara Storage/Cisco Code<br>
- Code that Cisco received from Zadara and updated (Folsom-based)<br>
- Currently experimental, we’re only using it for networking<br>
- Has enabling, scheduling, mapping<br>
- We use it for networking; code can’t attach to specific networks<br>
- Not the end result, merely something that works for now<br>
<br>
<br>
Mellanox code<br>
- Available now, attempting to merge into Quantum<br>
- Nova side just maps NICs into libvirt VMs<br>
- Quantum side does most of the work – NICs are programmed to use the correct encap for the network they’re used on via the Quantum agent<br>
- 2 modes – one is SRIOV and passthrough, one uses an SRIOV device with macvtap and a virtual NIC in the VM<br>
- No scheduling – assumption is that there are more SRIOV devices available than there are VMs using them<br>
<br>
<br>
Tieto code<br>
- collaboration with Intel<br>
- ...? (short on details, sorry<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 2 April 2013 01:13, Ian Wells <<a href="mailto:ijw.ubuntu@cack.org.uk" target="_blank">ijw.ubuntu@cack.org.uk</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Here's the webex details for anyone that would care to discuss PCI device passthrough in Openstack. This will be a preliminary meeting to see who's interested, what their use cases are, and for anyone who has code to give a quick briefing
on what they have.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Ian.<br>
<br>
Ian Wells invites you to attend this online meeting. <br>
<br>
Topic: Openstack and PCI passthrough <br>
Date: Wednesday, April 3, 2013 <br>
Time: 7:00 pm, Europe Summer Time (Amsterdam, GMT+02:00) <br>
Meeting Number: 206 594 344 <br>
Meeting Password: cisco<br>
<br>
------------------------------------------------------- <br>
To join the online meeting (Now from mobile devices!) <br>
------------------------------------------------------- <br>
1. Go to <a href="https://cisco.webex.com/ciscosales/j.php?ED=222105092&UID=0&RT=MiMyMg%3D%3D" target="_blank">
https://cisco.webex.com/ciscosales/j.php?ED=222105092&UID=0&RT=MiMyMg%3D%3D</a> <br>
2. Enter your name and email address. <br>
3. Enter the meeting password: Please obtain your meeting password from your host.
<br>
4. Click "Join Now". <br>
<br>
To view in other time zones or languages, please click the link: <br>
<a href="https://cisco.webex.com/ciscosales/j.php?ED=222105092&UID=0&ORT=MiMyMg%3D%3D" target="_blank">https://cisco.webex.com/ciscosales/j.php?ED=222105092&UID=0&ORT=MiMyMg%3D%3D</a>
<br>
<br>
---------------------------------------------------------------- <br>
ALERT:Toll-Free Dial Restrictions for (408) and (919) Area Codes <br>
---------------------------------------------------------------- <br>
<br>
The affected toll free numbers are: <a href="tel:%28866%29%20432-9903" target="_blank">
(866) 432-9903</a> for the San Jose/Milpitas area and <a href="tel:%28866%29%20349-3520" target="_blank">
(866) 349-3520</a> for the RTP area. <br>
<br>
Please dial the local access number for your area from the list below: <br>
- San Jose/Milpitas (408) area: 525-6800 <br>
- RTP (919) area: 392-3330 <br>
<br>
------------------------------------------------------- <br>
To join the teleconference only <br>
------------------------------------------------------- <br>
1. Dial into Cisco WebEx (view all Global Access Numbers at <br>
<a href="http://cisco.com/en/US/about/doing_business/conferencing/index.html" target="_blank">http://cisco.com/en/US/about/doing_business/conferencing/index.html</a>
<br>
2. Follow the prompts to enter the Meeting Number (listed above) or Access Code followed by the # sign.
<br>
<br>
San Jose, CA: <a href="tel:%2B1.408.525.6800" target="_blank">+1.408.525.6800</a> RTP:
<a href="tel:%2B1.919.392.3330" target="_blank">+1.919.392.3330</a> <br>
US/Canada: <a href="tel:%2B1.866.432.9903" target="_blank">+1.866.432.9903</a> United Kingdom:
<a href="tel:%2B44.20.8824.0117" target="_blank">+44.20.8824.0117</a> <br>
India: <a href="tel:%2B91.80.4350.1111" target="_blank">+91.80.4350.1111</a> Germany:
<a href="tel:%2B49.619.6773.9002" target="_blank">+49.619.6773.9002</a> <br>
Japan: <a href="tel:%2B81.3.5763.9394" target="_blank">+81.3.5763.9394</a> China:
<a href="tel:%2B86.10.8515.5666" target="_blank">+86.10.8515.5666</a> <br>
<br>
------------------------------------------------------- <br>
For assistance <br>
------------------------------------------------------- <br>
1. Go to <a href="https://cisco.webex.com/ciscosales/mc" target="_blank">https://cisco.webex.com/ciscosales/mc</a>
<br>
2. On the left navigation bar, click "Support". <br>
<br>
You can contact me at: <br>
<a href="mailto:iawells@cisco.com" target="_blank">iawells@cisco.com</a> <br>
<br>
To add this meeting to your calendar program (for example Microsoft Outlook), click this link:
<br>
<a href="https://cisco.webex.com/ciscosales/j.php?ED=222105092&UID=0&ICS=MI&LD=1&RD=2&ST=1&SHA2=AAAAAiiZzgrg8m975SaxKJ04c8ZMUb5MfxDZM1l66EploCxX&RT=MiMyMg%3D%3D" target="_blank">https://cisco.webex.com/ciscosales/j.php?ED=222105092&UID=0&ICS=MI&LD=1&RD=2&ST=1&SHA2=AAAAAiiZzgrg8m975SaxKJ04c8ZMUb5MfxDZM1l66EploCxX&RT=MiMyMg%3D%3D</a>
<br>
<br>
The playback of UCF (Universal Communications Format) rich media files requires appropriate players. To view this type of rich media files in the meeting, please check whether you have the players installed on your computer by going to
<a href="https://cisco.webex.com/ciscosales/systemdiagnosis.php" target="_blank">
https://cisco.webex.com/ciscosales/systemdiagnosis.php</a>. <br>
<br>
<a href="http://www.webex.com" target="_blank">http://www.webex.com</a> <br>
<br>
CCP:+14085256800x206594344# <br>
<br>
IMPORTANT NOTICE: This WebEx service includes a feature that allows audio and any documents and other materials exchanged or viewed during the session to be recorded. By joining this session, you automatically consent to such recordings. If you do not consent
to the recording, discuss your concerns with the meeting host prior to the start of the recording or do not join the session. Please note that any such recordings may be subject to discovery in the event of litigation.
<o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>