<div dir="ltr">Thanks Gary.<div><br></div><div style>I have looked independently at this issue and came to the same conclusion.</div><div style>How do you reckon we should go to fix this?</div><div style>I think we agree on the diagnosis, but I reckon the severity is bigger than just a usability issue.</div>
<div style><br></div><div style>This might affect anyone developing diagnostic/reporting/billing tools on quantum API.</div><div style>For instance, if I was a subscriber I would refuse to pay my service provider for usage of floating IP, as I would tell them: "look, they've been down all the time!".</div>
<div style>Luckily, not all people are cheeky and dishonest like me.</div><div style><br></div><div style>Salvatore</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 20 June 2013 13:03, Gary Kotton <span dir="ltr"><<a href="mailto:gkotton@redhat.com" target="_blank">gkotton@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div class="im">
On 06/20/2013 11:59 AM, Gary Kotton wrote:
<blockquote type="cite">
On 06/19/2013 07:00 PM, Samuel Winchenbach wrote:
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default">Here are ALL the warnings and errors I see
when restarting nova and quantum: <a href="http://pastie.org/pastes/8059837/text" target="_blank">http://pastie.org/pastes/8059837/text</a></div>
<div class="gmail_default">Here are ALL the warnings and errors I see
when creating both networks (public and internal), the
router that bridges them, assigning the public network as
the gateway, adding the internal interface to it, launching
a VM, allocating a floating IP, and assigning it to the VM: <a href="http://pastie.org/pastes/8059850/text" target="_blank">http://pastie.org/pastes/8059850/text</a></div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">I am not sure about the nbd stuff, but I
think think that is just because the qemu disk image is not
resized to the disk size yet. The ONLY warning that I
think might effect me is: </div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default"><font face="courier new, monospace">Jun
19 11:53:32 test1 dnsmasq[28913]: warning: no upstream
servers configured</font><br>
</div>
<div class="gmail_default"><font face="courier new, monospace"><br>
</font></div>
<div class="gmail_default"><font face="courier new,
monospace">Any idea if that could be causing my DOWN ports
issue? <br>
</font></div>
</div>
</blockquote>
<br>
Sorry to chime in pretty late. This is a bug. In certain plugins
when a port is created the status of the port is set to DOWN. This
port is set to UP when the plugin receives an indication that the
port is actually UP (for example when a agent detects that a tap
device has been created). This is not treated well when dealing
with gateway ports and flotaing IP's.<br>
</blockquote>
<br></div>
Please note that this is just a usability issue. The port is
actually up and forwarding traffic. <br><div><div class="h5">
<br>
<blockquote type="cite"> <br>
I'll take care of this.<br>
Thanks<br>
Gary<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default"><font face="courier new,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="courier new,
monospace">Thanks,</font></div>
<div class="gmail_default"><font face="courier new,
monospace">Sam</font></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Jun 19, 2013 at 11:03 AM,
Samuel Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I
<div class="gmail_default">
did that, and I went one step further... I deleted and
recreated ALL the databases to ensure that I would be
starting from scratch. I also deleted all bridges,
stopped openvswitch, deleted conf.db, started
openvswitch and recreated all the bridges.</div>
<div>
<div class="gmail_default"><br>
</div>
</div>
<div>
<div class="gmail_default">This
solved the reference to "test1-int" but the ports
are still stuck in the down state :(</div>
</div>
<div>
<div class="gmail_default"><br>
</div>
</div>
<div>
<div class="gmail_default">
Sam</div>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Jun 18, 2013 at
4:39 PM, Filipe Manco <span dir="ltr"><<a href="mailto:filipe.manco@gmail.com" target="_blank">filipe.manco@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Stop quantum agents (not
quantum-api) and openvswitch service. Delete <span>/etc/openvswitch/conf.db</span><span style="font-size:13px"><font face="arial,
helvetica, sans-serif"> and delte all
agents using </font><font face="courier
new, monospace">quantum agent-delete
<id></font><font face="arial,
helvetica, sans-serif">. Start openvswitch
service and then quantum agents. If ports
still down check quantum logs mainly
quantum l3-agent. If you don't find
anything interesting delete the networks
and recreate them.</font></span>
<div> <span style="font-size:13px"><font face="arial, helvetica, sans-serif"><br>
</font></span></div>
<div><span style="font-size:13px"><font face="arial, helvetica, sans-serif">If
you still have references to "test1-int"
check the quantum database </font></span><span style="font-family:arial,helvetica,sans-serif">ovs_tunnel_endpoints
table and manually remove any reference.</span></div>
</div>
<div>
<div>
<div class="gmail_extra"><br clear="all">
<div>
<div dir="ltr">Filipe Manco<br>
<div><a href="http://about.me/fmanco" target="_blank">http://about.me/fmanco</a></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2013/6/18 Samuel
Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default">Hmmm I used both
of those commands, but no matter
what I do I can not remove
references to "test1-int" in
/etc/openvswitch/conf.db</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Should I just
manually replace those with the
IP? Delete the file? </div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue,
Jun 18, 2013 at 1:14 PM,
Filipe Manco <span dir="ltr"><<a href="mailto:filipe.manco@gmail.com" target="_blank">filipe.manco@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Honestly I'm
not sure because I've
always used IPs. But
according to the logs it
looks so. After changing
configurations you should
probably run
quantum-netns-cleanup and
quantum-ovs-cleanup before
starting the services.<br>
</div>
<div>
<div>
<div class="gmail_extra"><br clear="all">
<div>
<div dir="ltr">Filipe
Manco<br>
<div><a href="http://about.me/fmanco" target="_blank">http://about.me/fmanco</a></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2013/6/18
Samuel Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default">I think I may be onto something: <a href="http://pastie.org/pastes/8056137/text" style="font-family:arial" target="_blank">http://pastie.org/pastes/8056137/text</a></div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">from syslog</div>
<div class="gmail_default"><font face="courier
new,
monospace">Jun
18 12:57:26
test1
ovs-vsctl:
00001|vsctl|INFO|Called
as
/usr/bin/ovs-vsctl
-- --may-exist
add-port
br-int
qvo3eb6d144-07
-- set
Interface
qvo3eb6d144-07
external-ids:iface-id=3eb6d144-077e-42cf-ad2e-57c50aa00399
external-ids:iface-status=active
external-ids:attached-mac=fa:16:3e:92:31:1e
external-ids:vm-uuid=add44e48-6f42-4ede-a646-f29e74ccc02d</font></div>
<div class="gmail_default">
<div class="gmail_default"><font face="courier
new,
monospace">Jun
18 12:57:26
test1
ovs-vswitchd:
03753|socket_util|ERR|"test1-int"
is not a valid
IP address</font></div>
<div class="gmail_default">
<font face="courier
new,
monospace">Jun
18 12:57:26
test1
ovs-vswitchd:
03755|netdev_vport|ERR|gre-1:
gre type
requires valid
'remote_ip'
argument<br>
Jun 18
12:57:30 test1
ovs-vsctl:
00001|vsctl|INFO|Called
as
/usr/bin/ovs-vsctl
--timeout=2
set Port
qvo3eb6d144-07
tag=1<br>
</font></div>
<div><font face="courier
new,
monospace">Jun
18 12:57:30
test1
ovs-vswitchd:
03768|netdev_vport|ERR|gre-1:
gre type
requires valid
'remote_ip'
argument</font><br>
</div>
<div><font face="courier
new,
monospace"><br>
</font></div>
<div><font face="courier
new,
monospace"><br>
</font></div>
<div><font face="courier
new,
monospace">Looks
like you might
not be able to
use entries
from
/etc/hosts in
the config
files?</font></div>
<div> <font face="courier
new,
monospace"><br>
</font></div>
<div><font face="courier
new,
monospace"><br>
</font></div>
<div><font face="courier
new,
monospace"><br>
</font></div>
<div><font face="courier
new,
monospace"><br>
</font></div>
</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default"><br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Tue, Jun 18,
2013 at 10:42
AM, Samuel
Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default"><font face="courier
new,
monospace">I
have three
agents running
(Open vSwitch
agent, DHCP
agent, and L3
agent): <a href="http://pastie.org/pastes/8055658/text" target="_blank">http://pastie.org/pastes/8055658/text</a></font><br>
</div>
<div class="gmail_default"><font face="courier
new,
monospace">The
agents listed
on test3 are
there because
ubuntu starts
them
automatically.
L3 agent will
never run on
test3 because
it doesn't
even have an
external
interface.
Right now I am
just trying to
limit it to
one node.</font></div>
<div class="gmail_default"><font face="courier
new,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="courier
new,
monospace">Here
is my
l3_agent.ini:
<a href="http://pastie.org/pastes/8055674/text" target="_blank">http://pastie.org/pastes/8055674/text</a></font></div>
<div class="gmail_default"><font face="courier
new,
monospace">Here
are a list of
bridges (eth1
is my external
interface) <a href="http://pastie.org/pastes/8055678/text" target="_blank">http://pastie.org/pastes/8055678/text</a></font></div>
<div class="gmail_default"><font face="courier
new,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="courier
new,
monospace">Thanks
again for all
your help!</font></div>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On
Tue, Jun 18,
2013 at 10:32
AM, Filipe
Manco <span dir="ltr"><<a href="mailto:filipe.manco@gmail.com" target="_blank">filipe.manco@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><font face="arial,
helvetica,
sans-serif">What
is the status
of <font color="#000000">quantum
agent-list? I
see on your
node test3 the
agents are
down and you
don't have
openvswitch
agent.</font></font>
<div> <font face="arial,
helvetica,
sans-serif"><font color="#000000">I would check for the logs of the l3 agent? Have you
configured the
external
network id on
the l3 agent
config file?</font></font><br>
</div>
</div>
<div>
<div>
<div class="gmail_extra">
<br clear="all">
<div>
<div dir="ltr">Filipe
Manco<br>
<div><a href="http://about.me/fmanco" target="_blank">http://about.me/fmanco</a></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2013/6/18
Samuel
Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_default">Hi Filipe,</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default"> Thanks for the response. I already had the
/etc/sudoers.d/quantum_sudoers
file. On a
whim I added
"root_helper =
sudo
quantum-rootwrap
/etc/quantum/rootwrap.conf"
to
/etc/quantum/dhcp_agent.ini
and that took
care of that
problem.</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">I managed to remove the libvirt errors by disabling
apparmor. </div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">All the ports on my public network are still listed as
"DOWN" I
have managed
to remove all
of the errors
and warnings
from quantum
but those
ports will
still not come
up. I really
am lost. </div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Thanks again for the post, I am not sure what to try
next :/</div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default">Sam</div>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">
On Tue, Jun
18, 2013 at
10:13 AM,
Filipe Manco <span dir="ltr"><<a href="mailto:filipe.manco@gmail.com" target="_blank">filipe.manco@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">From
what I can see
in the logs
you must
create the
file <font face="courier
new,
monospace">/etc/sudoers.d/quantum_sudoers</font>
with the
following
contents:
<div><br>
</div>
<div>
<div><font face="courier
new,
monospace">Defaults:quantum
!requiretty</font></div>
<div><font face="courier
new,
monospace">quantum
ALL = (root)
NOPASSWD:
/usr/bin/quantum-rootwrap</font></div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>About the
libvirt error
edit the file
<font face="courier
new,
monospace">/etc/libvirt/qemu.conf</font>
and add the
following:</div>
<div><br>
</div>
<div>
<div><font face="courier
new,
monospace">cgroup_device_acl
= [</font></div>
<div><font face="courier
new,
monospace">
"/dev/null",
"/dev/full",
"/dev/zero",</font></div>
<div><font face="courier
new,
monospace">
"/dev/random",
"/dev/urandom",</font></div>
<div><font face="courier
new,
monospace">
"/dev/ptmx",
"/dev/kvm",
"/dev/kqemu",</font></div>
<div><font face="courier
new,
monospace">
"/dev/rtc","/dev/hpet"
,
"/dev/net/tun"</font></div>
<div><font face="courier
new,
monospace">]</font></div>
<div><br>
</div>
</div>
<div>Probably
this won't fix
all of your
issues. The
logs ofI don't
the l3 agent
will be
helpful.</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div dir="ltr">Filipe
Manco<br>
<div><a href="http://about.me/fmanco" target="_blank">http://about.me/fmanco</a></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2013/6/18
Samuel
Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div dir="ltr">
<div class="gmail_default"><font face="courier
new,
monospace">I
may have found
the cause of
my problem,
but I am
unsure of the
solution. In
my libvirt log
file I found
many error
messages similar to
this: </font></div>
<div class="gmail_default"><br>
</div>
<div class="gmail_default"><font face="courier
new,
monospace">2013-06-18
13:12:19.812+0000:
8353: warning
:
virAuditSend:135
: Failed to
send audit
message
virt=kvm
resrc=net
reason=open
vm="instance-00000033"
uuid=bca8a09e-46aa-408b-81cd-2432068361c1
net=FA:16:3E:71:7F:68
path="/dev/net/tun"
rdev=0A:C8:
Operation not
permitted</font><br>
</div>
<div class="gmail_default"><font face="courier
new,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="courier
new,
monospace">Sam</font></div>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">
On Mon, Jun
17, 2013 at
8:52 PM,
Samuel
Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div class="gmail_default">
<div class="gmail_default"><font face="courier
new,
monospace">Here
is a bunch
more
information
from quantum:
<a href="http://pastie.org/pastes/8053820/text" target="_blank">http://pastie.org/pastes/8053820/text</a></font></div>
<div class="gmail_default"><font face="courier
new,
monospace"><br>
</font></div>
<div class="gmail_default"><font face="courier
new,
monospace">If
anyone has any
ideas I would
really
appreciate it.
Thanks!</font></div>
<div> <br>
</div>
</div>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Mon, Jun 17,
2013 at 5:07
PM, Samuel
Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi
All,<br>
<br>
I have been
stuck on a
problem for a
couple of days
now. I am
using Grizzly
on Ubuntu
12.04 LTS. I
can launch
vms, create
networks,
subnets,
routers, etc.
The problem
is quantum
reports that
all fors on
the public
network are
"DOWN" for
example: <a href="http://pastie.org/pastes/8053283/text" target="_blank">http://pastie.org/pastes/8053283/text</a><br>
<br>
Does anyone
have any hints
or tips on
what might be
causing this,
or the errors
listed below
in the quantum
logs? Thanks!<br>
<br>
quantum
configuration:
<a href="http://pastie.org/pastes/8053100/text" target="_blank">http://pastie.org/pastes/8053100/text</a><br>
<br>
nova
configuration:
<a href="http://pastie.org/pastes/8043800/text" target="_blank">http://pastie.org/pastes/8043800/text</a><br>
<br>
quantum logs:
<a href="http://pastie.org/pastes/8053269/text" target="_blank">http://pastie.org/pastes/8053269/text</a>
(this is
everything
logged during
the creation
the networks,
launching of
vm instances,
allocating the
floating ip
and assigning
it to a VM)<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
</div>
</div>
<div>_______________________________________________<br>
Mailing list:
<a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to :
<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe :
<a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help :
<a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a>
</pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br></div>