<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
There is a weekly meeting for PCI Passthrough:<br>
<a class="moz-txt-link-freetext" href="https://wiki.openstack.org/wiki/Meetings/Passthrough">https://wiki.openstack.org/wiki/Meetings/Passthrough</a><br>
<br>
I think it's a good idea to attend the meetings and share ideas
there.<br>
<br>
BR,<br>
Itzik<br>
<br>
<div class="moz-cite-prefix">On 02/06/2015 07:14 AM, Akilesh K
wrote:<br>
</div>
<blockquote
cite="mid:CAFafR-=cC_96avAgncY8R1iXT48NL8LVAM97=5Ce0Y4T9s4aXw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>I do understand that and that is why I believe it
should not be that way. profile should rather be populated
by the sriov-nic-switch-agent that is running on the
compute node. That way It is possible to do
interface-attach because the profile is already populated
and nova doesn't have to do it and also the agent can keep
track of the devices that are available instead of nova
tracking it. <br>
<br>
</div>
There is already a discussion on cleaning up the interaction
between nova and neutron and probably our case can also be a
part of it. <br>
<br>
</div>
Thank you,<br>
</div>
Ageeleshwar K<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Feb 5, 2015 at 6:45 PM, Irena
Berezovsky <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:irenab.dev@gmail.com" target="_blank">irenab.dev@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span class="">On Thu, Feb 5,
2015 at 9:38 AM, Akilesh K <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:akilesh1597@gmail.com"
target="_blank">akilesh1597@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>I know that vif_type is binding_failed on
a multinode setup and I also know why it
happens. <br>
<br>
</div>
As for interface-attach I got it work for
sriov ports and even verified it works inside
the instance. The trick was to specify profile
with pci_slot and pci_vendor_info during port
create. In case any one else wants to do this.</div>
</div>
</blockquote>
<div> </div>
</span>
<div>binding:profile is not supposed to be populated
by the user, it can be set only under admin
credentials and actually for SR-IOV case should be
populated by nova. Manual population of the profile
with pci_slot details can be very dangerous, since
you skip the phase when this pci slot is reserved by
nova. The system may become inconsistent.</div>
<div>
<div class="h5">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div> <br>
<br>
Thank you,<br>
</div>
Ageeleshwar K<br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Feb 5,
2015 at 12:19 PM, Irena Berezovsky <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:irenab.dev@gmail.com"
target="_blank">irenab.dev@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Hi Akilesh,
<div>Please see my responses inline.</div>
<div>Hope this help,</div>
<div><br>
</div>
<div>BR,</div>
<div>Irena</div>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On
Thu, Feb 5, 2015 at 6:14 AM,
Akilesh K <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:akilesh1597@gmail.com"
target="_blank">akilesh1597@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Irena,<br>
<br>
</div>
<div>Issue 1 - I
agree. You are
correct.<br>
</div>
<div><br>
</div>
<div>Issue 2<br>
</div>
The behavior you
outlined<br>
1. <span
style="color:rgb(116,27,71)">When
port is created
with
vnic_type=direct,
the vif_type is
'unbound'. The
pci_vendor_info
will be available
during port update
when 'nova boot'
command is invoked
and PCI device is
allocated. </span><br>
</div>
This happens when the
controller and compute
are on the same host.
Not when they are on
the different host. On
a multiserver setup
vif_type is set to
binging_failed during
port create.<br>
<br>
</div>
</div>
</div>
</blockquote>
</span>
<div>This is strange, since
port-create operation is pure
neutron API call and it should
not differ whether you are in
the multiserver or all-in-one
setup.</div>
<span>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>Second is i am not
doing nova boot. Instead
I am doing nova
interface-attach. In
this case the
pci_vendor_info is not
updated by anyone but
me. and pci_slot is also
not populated.<br>
<br>
</div>
</div>
</blockquote>
</span>
<div>interface-attach is
currently not supported for
SR-IOV ports. There is a
proposed blueprint to support
this: <a
moz-do-not-send="true"
href="https://review.openstack.org/#/c/139910/"
target="_blank">https://review.openstack.org/#/c/139910/</a>.</div>
<div>So for now, the only option
to provide PCI passthrough
vNIC is according to what is
described in the previously
referenced wiki page: create
neutron port with vnic_type=
direct and then 'nova boot'
with pre-created port.</div>
<div>
<div>
<div><br>
</div>
<div>Do you still think this
is correct ?</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div><br>
<br>
</div>
</div>
</div>
</div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Feb 4, 2015
at 8:08 PM, Irena
Berezovsky <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:irenab.dev@gmail.com"
target="_blank">irenab.dev@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Hi
Akilesh,
<div>please
see inline<br>
<div
class="gmail_extra"><br>
<div
class="gmail_quote"><span>On
Wed, Feb 4,
2015 at 11:32
AM, Akilesh K
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:akilesh1597@gmail.com"
target="_blank">akilesh1597@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hi,<br>
Issue 1:<br>
</div>
I do not
understand
what you mean.
I did specify
the
physical_network.
What I am
trying to say
is some
physical
networks
exists only on
the compute
node and not
on the network
node. We are
unable to
create a
network on
those
physnets. The
work around
was to fake
their
existance on
the network
node too.
Which I
believe is the
wrong way to
do.<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
<div>Every
physical
network
should be
defined at the
Controller
node,
including
range of
segmentation
ids (i.e. vlan
ids) available
for
allocation.</div>
<div>When
virtual
network is
created, you
should verify
that it has
associated
network type
and
segmentation
id (assuming
you are using
provider
network
extension).</div>
<span>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div><br>
</div>
Issue2:<br>
</div>
I looked
directly into
the code after
looking at the
logs. <br>
<br>
1. What
neutron (sriov
mech driver )
is doing is
loading the
default list
of
'supported_pci_vendor_devs'
, then it
picks up the
profile->pci_vendor_info
from the port
defenition we
sent in the
port create
request and
checks if it
is supported.
If not it says
'binding_failed'<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
<div>When port
is created
with
vnic_type=direct,
the vif_type
is 'unbound'.
The
pci_vendor_info
will be
available
during port
update when
'nova boot'
command is
invoked and
PCI device is
allocated. </div>
<span>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div><br>
</div>
I am fine with
this<br>
<br>
</div>
2. Then when I
attach the
created port
to a host
nova's vif
driver
(hv_veb) is
looking for
profile->pci_slot
in the context
of the port
that was
supplied and
fails to
attach to the
instance if it
is not
present. <br>
<br>
</div>
</div>
</div>
</blockquote>
</span>
<div>nova vif
driver
receives
profile->pci_slot
from neutron,
but it was
actually filed
earlier by
nova during
port-update. </div>
<span>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div>this is
what I think
should be done
by neutron
itself.
neutron's
sriov mech
driver should
have updated
the port with
the pci_slot
details when
the port got
created. and
this does
happen on a
single machine
install. We
need to find
why it does
not happen on
a multi node
install,
possibly
because the
mech driver is
not running on
the host with
sriov devices
and fix it. <br>
<br>
</div>
</div>
</blockquote>
</span>
<div>I suggest
to follow <a
moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking"
style="font-size:12.8000001907349px" target="_blank">https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking</a> instructions,
this should
work for you.</div>
<div>
<div>
<div><br>
</div>
<div>I hope
you guys can
understand
what I mean.</div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
Thank you,<br>
Ageeleshwar K<br>
<div>
<div>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Feb 4,
2015 at 2:49
PM, Itzik
Brown <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:itzikb@redhat.com" target="_blank">itzikb@redhat.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Hi,<br>
<br>
Issue 1;<br>
You must
specify the
physical
networks.<br>
Please look
at:<br>
<a
moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking"
target="_blank">https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking</a><b><br>
<br>
</b>Issue 2:<br>
AFAIK the
agent is
supported by
only one
vendor.<br>
Can you please
look for
errors in
Neutron's log?<br>
<b><br>
</b>Thanks,<br>
Itzik
<div>
<div><br>
<div>On
02/04/2015
09:12 AM,
Akilesh K
wrote:<br>
</div>
</div>
</div>
<blockquote
type="cite">
<div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>Hi,<br>
</div>
I found two
issues with
the way
neutron
behaves on a
multi server
install. I got
it to work but
I do not this
this is the
right way to
do it. It
might be a bug
we might want
to fix and for
which I could
volunteer. <br>
<br>
</div>
<div>Setup -
Multiserver
juno on
ubuntu.<br>
<br>
</div>
<div>Machine 1
- Controller<br>
</div>
<div>All api
servers , l3,
dhcp and ovs
agent<br>
<br>
</div>
<div>Machine 2
- Compute <br>
</div>
<div>nova
compute,
neutron-ovs-agent,
neutron sriov
agent.<br>
</div>
<div><br>
</div>
<div><br>
</div>
Issue 1:<br>
<br>
</div>
Controller
node has
physnets
'External',
'Internal'
configured in
ml2<br>
<br>
</div>
Compute node
has physnets
'Internal',
'Physnet1',
'Physnet2'
configured in
ml2<br>
<br>
</div>
When I do
neutron
net-create
--provider:physicalnetwork
Physnet1, It
complains that
'Physnet1' is
not available.
<br>
<br>
</div>
Offcourse its
not available
on the
controller but
is available
on the compute
node and there
is no way to
tell neutron
to host that
network on
compute node
alone<br>
<br>
</div>
Work around<br>
I had to
include
'Physnet1' in
the controller
node also to
get it to
work, except
that there is
not bridge
mapings for
this physnet.<br>
<br>
<br>
</div>
Issue 2:<br>
<div>
<div>
<div><br>
</div>
<div>This is
related to
sriov agent.
This agent is
configured
only on the
compute node
as that node
alone has
supported
devices. <br>
<br>
</div>
<div>When I do
a port create
--binding:vnic_type
direct
--binding:host_id
<compute
node> The
port is
created but
with
binding:vif_type:
<b>'binding-failed'</b>.
and naturally
I could not
attach it to
any instance.<br>
<br>
</div>
<div>I looked
at the code
and figured
out that
neutron api is
expecting
binding:profile
also in the
format <br>
{"pci_slot":
"0000:03:10.1",
"pci_vendor_info":
"8086:10ed"}<br>
<br>
</div>
<div>Is this
how it should
be. Because on
a single
machine
install I did
not have to do
this. However
on a
multiserver I
had to even
give the pci
address is the
exact format
to get it to
work. <br>
<br>
</div>
<div>I have a
serious
feeling that
this could be
lot simpler if
neutron could
take care of
finding the
details in a
smart way
rather than
relying on the
administrator
to find which
device is
available and
configure it.
<br>
<br>
<br>
</div>
<div>Note:<br>
</div>
<div>1. If I
can get some
expert advice
I can fix both
these. <br>
</div>
<div>2. I am
not sure if
this question
should rather
be sent to
openstack-dev
group. Let me
know.<br>
</div>
<div><br>
<br>
</div>
<div>Thank
you,<br>
</div>
<div>Ageeleshwar
K<br>
<br>
<br>
</div>
<div><br>
<br>
<br>
</div>
<div><br>
<br>
<br>
<br>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
Mailing list: <a moz-do-not-send="true" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
Post to : <a moz-do-not-send="true" href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>
Unsubscribe : <a moz-do-not-send="true" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Mailing list:
<a
moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to :
<a
moz-do-not-send="true"
href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe :
<a
moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>