<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">在 2015年03月11日 22:15, jacob jacob 写道:<br>
      Hi, jacob<br>
      <br>
      I'm trying  to find someone to check it, if there any feed back, i
      update you.<br>
      <br>
      Yongli He<br>
    </div>
    <blockquote
cite="mid:CA+AkT2hSyU=cX=Ek=YhuNEyeTAnsLQH_61267LMsQM_pK4j-Lg@mail.gmail.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <div dir="ltr"><br>
        <div class="gmail_quote">---------- Forwarded message ----------<br>
          From: <b class="gmail_sendername">jacob jacob</b> <span
            dir="ltr"><<a moz-do-not-send="true"
              href="mailto:opstkusr@gmail.com">opstkusr@gmail.com</a>></span><br>
          Date: Tue, Mar 10, 2015 at 6:00 PM<br>
          Subject: PCI passthrough of 40G ethernet interface<br>
          To: <a moz-do-not-send="true"
            href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
          <br>
          <br>
          <div dir="ltr"><br>
            <div class="gmail_quote">
              <div dir="ltr"><span class=""><span style="font-size:13px">Hi,</span>
                  <div style="font-size:13px">I'm interested in finding
                    out if anyone has successfully tested PCI
                    passthrough functionality for 40G interfaces in
                    Openstack(KVM hypervisor).</div>
                  <div style="font-size:13px"><br>
                  </div>
                  <div style="font-size:13px">I am trying this out on a
                    Fedora 21 host  with Fedora 21 VM
                    image.(3.18.7-200.fc21.x86_64)</div>
                  <div style="font-size:13px"><br>
                  </div>
                  <div style="font-size:13px">Was able to successfully
                    test PCI passthrough of 10 G interfaces:</div>
                  <div style="font-size:13px">  Ethernet controller:
                    Intel Corporation 82599ES 10-Gigabit SFI/SFP+
                    Network Connection (rev 01)</div>
                  <div style="font-size:13px"><br>
                  </div>
                  <div style="font-size:13px">With 40G interface
                    testing, the PCI device is passed through to the VM
                    but data transfer is failing.</div>
                  <div style="font-size:13px">    0a:00.1 Ethernet
                    controller: Intel Corporation Ethernet Controller
                    XL710 for 40GbE QSFP+ (rev 01)</div>
                  <div style="font-size:13px"><br>
                  </div>
                  <div style="font-size:13px">Tried this with both the
                    i40e driver and latest dpdk driver but no luck so
                    far.</div>
                  <div style="font-size:13px"><br>
                  </div>
                </span>
                <div style="font-size:13px">With the i40e driver, the
                  data transfer fails.</div>
                <div>
                  <div class="h5">
                    <div style="font-size:13px"> Relevant dmesg output:</div>
                    <div style="font-size:13px"> [   11.544088] i40e
                      0000:00:05.0 eth1: NIC Link is Up 40 Gbps Full
                      Duplex, Flow Control: None</div>
                    <div style="font-size:13px">[   11.689178] i40e
                      0000:00:06.0 eth2: NIC Link is Up 40 Gbps Full
                      Duplex, Flow Control: None</div>
                    <div style="font-size:13px">[   16.704071]
                      ------------[ cut here ]------------</div>
                    <div style="font-size:13px">[   16.705053] WARNING:
                      CPU: 1 PID: 0 at net/sched/sch_generic.c:303
                      dev_watchdog+0x23e/0x250()</div>
                    <div style="font-size:13px">[   16.705053] NETDEV
                      WATCHDOG: eth1 (i40e): transmit queue 1 timed out</div>
                    <div style="font-size:13px">[   16.705053] Modules
                      linked in: cirrus ttm drm_kms_helper i40e drm
                      ppdev serio_raw i2c_piix4 virtio_net parport_pc
                      ptp virtio_balloon crct10dif_pclmul pps_core
                      parport pvpanic crc32_pclmul ghash_clmulni_intel
                      virtio_blk crc32c_intel virtio_pci virtio_ring
                      virtio ata_generic pata_acpi</div>
                    <div style="font-size:13px">[   16.705053] CPU: 1
                      PID: 0 Comm: swapper/1 Not tainted
                      3.18.7-200.fc21.x86_64 #1</div>
                    <div style="font-size:13px">[   16.705053] Hardware
                      name: Fedora Project OpenStack Nova, BIOS
                      1.7.5-20140709_153950- 04/01/2014</div>
                    <div style="font-size:13px">[   16.705053]
                       0000000000000000 2e5932b294d0c473
                      ffff88043fc83d48 ffffffff8175e686</div>
                    <div style="font-size:13px">[   16.705053]
                       0000000000000000 ffff88043fc83da0
                      ffff88043fc83d88 ffffffff810991d1</div>
                    <div style="font-size:13px">[   16.705053]
                       ffff88042958f5c0 0000000000000001
                      ffff88042865f000 0000000000000001</div>
                    <div style="font-size:13px">[   16.705053] Call
                      Trace:</div>
                    <div style="font-size:13px">[   16.705053]
                       <IRQ>  [<ffffffff8175e686>]
                      dump_stack+0x46/0x58</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff810991d1>]
                      warn_slowpath_common+0x81/0xa0</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff81099245>]
                      warn_slowpath_fmt+0x55/0x70</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8166e62e>]
                      dev_watchdog+0x23e/0x250</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8166e3f0>] ?
                      dev_graft_qdisc+0x80/0x80</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff810fd52a>]
                      call_timer_fn+0x3a/0x120</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8166e3f0>] ?
                      dev_graft_qdisc+0x80/0x80</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff810ff692>]
                      run_timer_softirq+0x212/0x2f0</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8109d7a4>]
                      __do_softirq+0x124/0x2d0</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8109db75>] irq_exit+0x125/0x130</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff817681d8>]
                      smp_apic_timer_interrupt+0x48/0x60</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff817662bd>]
                      apic_timer_interrupt+0x6d/0x80</div>
                    <div style="font-size:13px">[   16.705053]
                       <EOI>  [<ffffffff811005c8>] ?
                      hrtimer_start+0x18/0x20</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8105ca96>] ?
                      native_safe_halt+0x6/0x10</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff810f81d3>] ?
                      rcu_eqs_enter+0xa3/0xb0</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8101ec7f>] default_idle+0x1f/0xc0</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8101f64f>] arch_cpu_idle+0xf/0x20</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff810dad35>]
                      cpu_startup_entry+0x3c5/0x410</div>
                    <div style="font-size:13px">[   16.705053]
                       [<ffffffff8104a2af>]
                      start_secondary+0x1af/0x1f0</div>
                    <div style="font-size:13px">[   16.705053] ---[ end
                      trace 7bda53aeda558267 ]---</div>
                    <div style="font-size:13px">[   16.705053] i40e
                      0000:00:05.0 eth1: tx_timeout recovery level 1</div>
                    <div style="font-size:13px">[   16.705053] i40e
                      0000:00:05.0: i40e_vsi_control_tx: VSI seid 519 Tx
                      ring 0 disable timeout</div>
                    <div style="font-size:13px">[   16.744198] i40e
                      0000:00:05.0: i40e_vsi_control_tx: VSI seid 520 Tx
                      ring 64 disable timeout</div>
                    <div style="font-size:13px">[   16.779322] i40e
                      0000:00:05.0: i40e_ptp_init: added PHC on eth1</div>
                    <div style="font-size:13px">[   16.791819] i40e
                      0000:00:05.0: PF 40 attempted to control timestamp
                      mode on port 1, which is owned by PF 1</div>
                    <div style="font-size:13px">[   16.933869] i40e
                      0000:00:05.0 eth1: NIC Link is Up 40 Gbps Full
                      Duplex, Flow Control: None</div>
                    <div style="font-size:13px">[   18.853624] SELinux:
                      initialized (dev tmpfs, type tmpfs), uses
                      transition SIDs</div>
                    <div style="font-size:13px">[   22.720083] i40e
                      0000:00:05.0 eth1: tx_timeout recovery level 2</div>
                    <div style="font-size:13px">[   22.826993] i40e
                      0000:00:05.0: i40e_vsi_control_tx: VSI seid 519 Tx
                      ring 0 disable timeout</div>
                    <div style="font-size:13px">[   22.935288] i40e
                      0000:00:05.0: i40e_vsi_control_tx: VSI seid 520 Tx
                      ring 64 disable timeout</div>
                    <div style="font-size:13px">[   23.669555] i40e
                      0000:00:05.0: i40e_ptp_init: added PHC on eth1</div>
                    <div style="font-size:13px">[   23.682067] i40e
                      0000:00:05.0: PF 40 attempted to control timestamp
                      mode on port 1, which is owned by PF 1</div>
                    <div style="font-size:13px">[   23.722423] i40e
                      0000:00:05.0 eth1: NIC Link is Up 40 Gbps Full
                      Duplex, Flow Control: None</div>
                    <div style="font-size:13px">[   23.800206] i40e
                      0000:00:06.0: i40e_ptp_init: added PHC on eth2</div>
                    <div style="font-size:13px">[   23.813804] i40e
                      0000:00:06.0: PF 48 attempted to control timestamp
                      mode on port 0, which is owned by PF 0</div>
                    <div style="font-size:13px">[   23.855275] i40e
                      0000:00:06.0 eth2: NIC Link is Up 40 Gbps Full
                      Duplex, Flow Control: None</div>
                    <div style="font-size:13px">[   38.720091] i40e
                      0000:00:05.0 eth1: tx_timeout recovery level 3</div>
                    <div style="font-size:13px">[   38.725844] random:
                      nonblocking pool is initialized</div>
                    <div style="font-size:13px">[   38.729874] i40e
                      0000:00:06.0: HMC error interrupt</div>
                    <div style="font-size:13px">[   38.733425] i40e
                      0000:00:06.0: i40e_vsi_control_tx: VSI seid 518 Tx
                      ring 0 disable timeout</div>
                    <div style="font-size:13px">[   38.738886] i40e
                      0000:00:06.0: i40e_vsi_control_tx: VSI seid 521 Tx
                      ring 64 disable timeout</div>
                    <div style="font-size:13px">[   39.689569] i40e
                      0000:00:06.0: i40e_ptp_init: added PHC on eth2</div>
                    <div style="font-size:13px">[   39.704197] i40e
                      0000:00:06.0: PF 48 attempted to control timestamp
                      mode on port 0, which is owned by PF 0</div>
                    <div style="font-size:13px">[   39.746879] i40e
                      0000:00:06.0 eth2: NIC Link is Down</div>
                    <div style="font-size:13px">[   39.838356] i40e
                      0000:00:05.0: i40e_ptp_init: added PHC on eth1</div>
                    <div style="font-size:13px">[   39.851788] i40e
                      0000:00:05.0: PF 40 attempted to control timestamp
                      mode on port 1, which is owned by PF 1</div>
                    <div style="font-size:13px">[   39.892822] i40e
                      0000:00:05.0 eth1: NIC Link is Down</div>
                    <div style="font-size:13px">[   43.011610] i40e
                      0000:00:06.0 eth2: NIC Link is Up 40 Gbps Full
                      Duplex, Flow Control: None</div>
                    <div style="font-size:13px">[   43.059976] i40e
                      0000:00:05.0 eth1: NIC Link is Up 40 Gbps Full
                      Duplex, Flow Control: None</div>
                    <div><br>
                    </div>
                    <div style="font-size:13px"><br>
                    </div>
                    <div style="font-size:13px">Similarly, with dpdk
                      driver, no packet transfer is happening.</div>
                    <div style="font-size:13px"><br>
                    </div>
                    <div style="font-size:13px"><br>
                    </div>
                    <div style="font-size:13px">The host was booted
                      with iommu=pt intel_iommu=on.</div>
                    <div style="font-size:13px"><br>
                    </div>
                    <div style="font-size:13px">It is to be noted that
                      everything works fine on the host itself. The
                      issue is seen only when the devices are passed
                      through to the VM.</div>
                    <div style="font-size:13px"><br>
                    </div>
                    <div style="font-size:13px">Would really appreciate
                      any additional information on how to address or
                      further debug this issue.</div>
                    <div style="font-size:13px"><br>
                    </div>
                    <div style="font-size:13px">Thanks</div>
                  </div>
                </div>
              </div>
              <span class="HOEnZb"><font color="#888888">
                </font></span></div>
            <span class="HOEnZb"><font color="#888888">Jacob</font></span></div>
        </div>
        <br>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>