<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Ok, lemme try to summarize. <br>
You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks
to Pacemaker. <br>
This setup is relying on two hosts, test1 (10.21.0.1) and test2
(10.21.0.2).<br>
Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP.<br>
<br>
Are you sure your my.cnf is actually the same in between both DRBD
nodes ? (I would recommend to symlink it to a physical file hosted
on the DRBD device).<br>
<br>
One thing is hurting me : you told me that nova is also
pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ?
It should be pointing to nova-ha (assuming 10.21.2.4 as per
/etc/hosts).<br>
<br>
Also, as per my understanding of Pacemaker, DRBD partition is
setup by default on test2, correct ?<br>
<br>
<br>
Sorry, as per my first reading, I can't see anything obvious. That
said, I'm not sure this is a Nova bug, as the tcpdump trace is
seeing a correct MySQL connection attempt. But maybe I'm wrong ?<br>
<br>
Anyway, are you sure you only have *one* MySQL engine running
(either on test1 or test2) and nova-manage trying to access this
right one ?<br>
<br>
Perms look good to me. As it a test setup, you could try to
unleash the grants by deleting them and allowing nova@'%' to see
if it's a basic dns mapping issue.<br>
<br>
-Sylvain<br>
<br>
<br>
<br>
Le 11/03/2013 16:09, Samuel Winchenbach a écrit :<br>
</div>
<blockquote
cite="mid:CAK9s_NxvMcJDcUcHfOSmZwzwZgYGxpzHEwdm79f44Jux1Q197g@mail.gmail.com"
type="cite">
<div dir="ltr">I
<div style="font-family:'courier new',monospace;display:inline"
class="gmail_default"> enabled general_log in
/etc/mysql/my.cnf Here are the results of connecting from
"test1", "test2" and using the client:</div>
<div>
<div style="display:inline" class="gmail_default"><font
face="courier new, monospace"><a moz-do-not-send="true"
href="http://paste2.org/p/3115525">http://paste2.org/p/3115525</a></font></div>
</div>
<div><span style="font-family:'courier new',monospace">I
purposefully used the real password in case there is a
problem with it.
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default"> I
changed before submitting post.</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default"><br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
here is a raw packet TCP dump (tcpdump -w rawdump port
3306) of an attempted "nova-manage service list" from
test1: </div>
</span></div>
<div>
<div style="display:inline" class="gmail_default"><font
face="courier new, monospace"><a moz-do-not-send="true"
href="https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump">https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump</a></font></div>
<span style="font-family:'courier new',monospace"> </span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
I looked at it with wireshark and couldn't see anything
that jumped out at me as incorrect. I have not yet tried
to recreate the salted password.</div>
</span><br>
</div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
<br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">Here
is my pacemaker configuration for mysql. I stripped out
openstack services, rabbitmq and others for clarity. All
resources are currently disabled (other than MySQL): </div>
</span><span style="font-family:'courier new',monospace"><a
moz-do-not-send="true" href="http://paste2.org/p/3115685">http://paste2.org/p/3115685</a>
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default"><br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
Please don't yell at me for having STONITH disabled :P
This is a testing cluster and I am working on getting
routed to the IPMI interface.</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
<br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">/etc/hosts: </div>
</span><font face="courier new, monospace"><a
moz-do-not-send="true" href="http://paste2.org/p/3115713">http://paste2.org/p/3115713</a>
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
</div>
</font></div>
<div><font face="courier new, monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">/etc/nova/nova.conf: </div>
<a moz-do-not-send="true" href="http://paste2.org/p/3115739">http://paste2.org/p/3115739</a>
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
</div>
</font></div>
<div><span style="font-family:'courier new',monospace"><br>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
If there is anything else I can provide you, please let me
know! I have pulled out most of my hair at this point!</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">
<br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div style="font-family:'courier
new',monospace;display:inline" class="gmail_default">Sam</div>
</span><br>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Mar 11, 2013 at 10:11 AM,
Sylvain Bauza <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:sylvain.bauza@digimind.com" target="_blank">sylvain.bauza@digimind.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>So as to reproduce the nova-manage SQL command, I
would recommand to tcpdump -A port 3306 on the host and
get the SQL trace on what's failing.<br>
<br>
Could you please explain further what is your HA config
? Are you using pacemaker/heartbeat or any VIP ?<br>
<br>
-Sylvain<br>
<br>
Le 11/03/2013 14:23, Samuel Winchenbach a écrit :<br>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default">Does anyone think this
could be an openstack bug? I just want to check
before submitting a bug report.</div>
<div class="gmail_default"> <br>
</div>
<div class="gmail_default">Sam</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Mar 8, 2013 at
4:02 PM, Jay Pipes <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">Sorry, I really can't
think of anything :(<br>
<div><br>
On 03/08/2013 03:52 PM, Samuel Winchenbach
wrote:<br>
> I dropped those users and no change.<br>
><br>
> I also set up general logging in mysql
but it really doesn't provide any<br>
> additional information. Any idea for a
next step I could take?<br>
><br>
> I am almost at the point of taking a
tcpdump and trying to recreate the<br>
> salted password. :/<br>
><br>
> Thanks for the help<br>
><br>
> Sam<br>
><br>
><br>
><br>
><br>
> On Fri, Mar 8, 2013 at 3:38 PM, Jay
Pipes <<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a><br>
</div>
<div>> <mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>>
wrote:<br>
><br>
> I'm stumped :( Looks like
everything is set up correctly to me. What
is<br>
> interested is that your nova user
access works from test2, but there is<br>
> no nova@test2 user in the
mysql.user table. What about doing a DROP
USER<br>
> nova@test1; FLUSH PRIVILEGES; and
then see if that fixes things... since<br>
</div>
> the <a moz-do-not-send="true"
href="http://nova@10.21.0.0/255.255.0.0"
target="_blank">nova@10.21.0.0/255.255.0.0</a>
<<a moz-do-not-send="true"
href="http://nova@10.21.0.0/255.255.0.0"
target="_blank">http://nova@10.21.0.0/255.255.0.0</a>><br>
<div>> user is clearly working for the
access<br>
> from test2.<br>
><br>
> Also, I'd recommend highly removing
the nova@% user.<br>
><br>
> Best,<br>
> -jay<br>
><br>
> On 03/08/2013 03:09 PM, Samuel
Winchenbach wrote:<br>
> ><br>
> > <a moz-do-not-send="true"
href="http://paste2.org/p/3085807"
target="_blank">http://paste2.org/p/3085807</a><br>
> ><br>
> ><br>
> > On Fri, Mar 8, 2013 at 2:46
PM, Jay Pipes <<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>><br>
</div>
<div>> > <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>>>
wrote:<br>
> ><br>
</div>
<div>
<div>> > Please paste the
results of SELECT User, Host, Password
FROM<br>
> mysql.user<br>
> > when running as root...<br>
> ><br>
> > Thanks!<br>
> > -jay<br>
> ><br>
> > On 03/08/2013 02:25 PM,
Samuel Winchenbach wrote:<br>
> > > Here are my grants.
I don't know if this helps, but I did<br>
> verify that<br>
> > > the password was
identical for each grant:<br>
> > <a
moz-do-not-send="true"
href="http://paste2.org/p/3085361"
target="_blank">http://paste2.org/p/3085361</a><br>
> > ><br>
> > ><br>
> > > On Fri, Mar 8, 2013
at 2:17 PM, Samuel Winchenbach<br>
> > <<a
moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>>><br>
> > > <mailto:<a
moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:swinchen@gmail.com"
target="_blank">swinchen@gmail.com</a>>>>>
wrote:<br>
> > ><br>
> > >
root@test1:/var/log# mysql -hmysql-ha
-unova<br>
> > >
-p********************************
-e"SELECT User, Host,<br>
> Password<br>
> > > FROM
mysql.user;"<br>
> > > ERROR 1142
(42000) at line 1: SELECT command denied
to user<br>
> > > 'nova'@'test1'
for table 'user'<br>
> > ><br>
> > ><br>
> > > On Fri, Mar 8,
2013 at 2:06 PM, Jay Pipes<br>
> <<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>><br>
> > <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>><br>
</div>
</div>
<div>
<div>> > > <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>>>>
wrote:<br>
> > ><br>
> > > What does
this show?<br>
> > ><br>
> > > mysql
-hmysql-ha -unova -p<PASS> -e"SELECT
User, Host,<br>
> > Password FROM<br>
> > > mysql.user"<br>
> > ><br>
> > > -jay<br>
> > ><br>
> > > On
03/08/2013 01:46 PM, Samuel Winchenbach
wrote:<br>
> > > > Sorry,
that must have been a copy and paste
error.<br>
> Here<br>
> > is what I<br>
> > > >
actually ran:<br>
> > > ><br>
> > > > <a
moz-do-not-send="true"
href="http://paste2.org/p/3084996"
target="_blank">http://paste2.org/p/3084996</a><br>
> > > ><br>
> > > ><br>
> > > > On
Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes<br>
> > <<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>><br>
> > > <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>>><br>
> > > >
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>><br>
> > <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>
<mailto:<a moz-do-not-send="true"
href="mailto:jaypipes@gmail.com"
target="_blank">jaypipes@gmail.com</a>>>>>>
wrote:<br>
> > > ><br>
> > > > On
03/08/2013 12:19 PM, Samuel Winchenbach
wrote:<br>
> > > >
> Hi All,<br>
> > > >
><br>
> > > >
> I have two nodes (test1 and test2)
that I am<br>
> trying to<br>
> > > set up in a<br>
> > > >
> highly available configuration.<br>
> > > >
><br>
> > > >
> During the setup process I tried
running<br>
> "nova-manage<br>
> > > service
list" on<br>
> > > >
> both nodes. It worked fine on
test2, but<br>
> fails on<br>
> > > test1 even<br>
> > > >
though I<br>
> > > >
> can connect to the database with the
mysql<br>
> client from<br>
> > > test1.<br>
> > > >
><br>
> > > >
> Here is a screen capture that shows
the setup on<br>
> > the two<br>
> > > nodes are<br>
> > > >
> basically identical:<br>
> <a moz-do-not-send="true"
href="http://paste2.org/p/3084223"
target="_blank">http://paste2.org/p/3084223</a><br>
> > > ><br>
> > > > In
the above paste you are doing:<br>
> > > ><br>
> > > >
mysql -unova - hmysql-ha -u root
nova<br>
> > > >
-p********************************<br>
> > > ><br>
> > > >
Note you are supplying 2 -u arguments, and
mysql<br>
> > will take<br>
> > > the second<br>
> > > >
(root).<br>
> > > ><br>
> > > >
-jay<br>
> > > ><br>
> > > >
_______________________________________________<br>
> > > >
Mailing list: <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
> > > >
Post to : <a moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>>><br>
> > > <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>>>><br>
> > > >
<mailto:<a moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>>><br>
> > > <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a
moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net"
target="_blank">openstack@lists.launchpad.net</a>>>>><br>
> > > >
Unsubscribe : <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack"
target="_blank">https://launchpad.net/~openstack</a><br>
> > > >
More help : <a moz-do-not-send="true"
href="https://help.launchpad.net/ListHelp"
target="_blank">https://help.launchpad.net/ListHelp</a><br>
> > > ><br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Mailing list: <a moz-do-not-send="true" href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a>
Post to : <a moz-do-not-send="true" href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>
Unsubscribe : <a moz-do-not-send="true" href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a>
More help : <a moz-do-not-send="true" href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Mailing list: <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a moz-do-not-send="true"
href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>