<div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace">#1 - No change</div><div class="gmail_default" style="font-family:'courier new',monospace">#2 - All of grants are in the ip/mask form such as: 'nova'@'<a href="http://10.21.0.0/255.255.0.0">10.21.0.0/255.255.0.0</a>' I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No change.</div>
<div class="gmail_default" style="font-family:'courier new',monospace">#3 - I changed the SQL connection string over to IP instead of hostname. No change. I didn't restart nova-api because it isn't running. If I understand correctly nova-manage communicated directly with the db, bypassing nova-api. This would appear true seeing "nova-manage service list" works correctly on test2.</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">
:(</div><div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">Thanks for the help!</div><div class="gmail_default" style="font-family:'courier new',monospace">
Sam</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza <span dir="ltr"><<a href="mailto:sylvain.bauza@digimind.com" target="_blank">sylvain.bauza@digimind.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>When looking at MySQL 5.1 refman
(<a href="http://dev.mysql.com/doc/refman/5.1/en/access-denied.html" target="_blank">http://dev.mysql.com/doc/refman/5.1/en/access-denied.html</a> ), I
would suggest to follow the procedure : <br>
1. 'mysqladmin flush-hosts'<br>
2. replace DNS entries in mysql.user table by IP addresses
instead<br>
3. modify /etc/nova/nova.conf with IP address of HA Mysql instead
(and restart nova-api !)<br>
<br>
I wouldn't bet on it, but I would say this is due to some name
resolution which is incorrect. <br>
<br>
-Sylvain<br>
<br>
<br>
Le 11/03/2013 17:00, Sylvain Bauza a écrit :<br>
</div><div><div class="h5">
<blockquote type="cite">
<div>Ok, lemme try to summarize. <br>
You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1
thanks to Pacemaker. <br>
This setup is relying on two hosts, test1 (10.21.0.1) and test2
(10.21.0.2).<br>
Your nova.conf is pointing to mysql://<a href="http://10.21.1.1" target="_blank">10.21.1.1</a> which is the
VIP.<br>
<br>
Are you sure your my.cnf is actually the same in between both
DRBD nodes ? (I would recommend to symlink it to a physical file
hosted on the DRBD device).<br>
<br>
One thing is hurting me : you told me that nova is also
pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ?
It should be pointing to nova-ha (assuming 10.21.2.4 as per
/etc/hosts).<br>
<br>
Also, as per my understanding of Pacemaker, DRBD partition is
setup by default on test2, correct ?<br>
<br>
<br>
Sorry, as per my first reading, I can't see anything obvious.
That said, I'm not sure this is a Nova bug, as the tcpdump trace
is seeing a correct MySQL connection attempt. But maybe I'm
wrong ?<br>
<br>
Anyway, are you sure you only have *one* MySQL engine running
(either on test1 or test2) and nova-manage trying to access this
right one ?<br>
<br>
Perms look good to me. As it a test setup, you could try to
unleash the grants by deleting them and allowing nova@'%' to see
if it's a basic dns mapping issue.<br>
<br>
-Sylvain<br>
<br>
<br>
<br>
Le 11/03/2013 16:09, Samuel Winchenbach a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">I
<div class="gmail_default">
enabled general_log in /etc/mysql/my.cnf Here are the
results of connecting from "test1", "test2" and using the
client:</div>
<div>
<div style="display:inline" class="gmail_default"><font face="courier new, monospace"><a href="http://paste2.org/p/3115525" target="_blank">http://paste2.org/p/3115525</a></font></div>
</div>
<div><span style="font-family:'courier new',monospace">I
purposefully used the real password in case there is a
problem with it.
<div class="gmail_default"> I
changed before submitting post.</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default"><br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default">
here is a raw packet TCP dump (tcpdump -w rawdump port
3306) of an attempted "nova-manage service list" from
test1: </div>
</span></div>
<div>
<div style="display:inline" class="gmail_default"><font face="courier new, monospace"><a href="https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump" target="_blank">https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump</a></font></div>
<span style="font-family:'courier new',monospace"> </span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default"> I
looked at it with wireshark and couldn't see anything
that jumped out at me as incorrect. I have not yet
tried to recreate the salted password.</div>
</span><br>
</div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default"> <br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default">Here
is my pacemaker configuration for mysql. I stripped out
openstack services, rabbitmq and others for clarity.
All resources are currently disabled (other than
MySQL): </div>
</span><span style="font-family:'courier new',monospace"><a href="http://paste2.org/p/3115685" target="_blank">http://paste2.org/p/3115685</a>
<div class="gmail_default"> </div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default"><br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default">
Please don't yell at me for having STONITH disabled :P
This is a testing cluster and I am working on getting
routed to the IPMI interface.</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default"> <br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default">/etc/hosts: </div>
</span><font face="courier new, monospace"><a href="http://paste2.org/p/3115713" target="_blank">http://paste2.org/p/3115713</a>
<div class="gmail_default"> </div>
</font></div>
<div><font face="courier new, monospace">
<div class="gmail_default">/etc/nova/nova.conf: </div>
<a href="http://paste2.org/p/3115739" target="_blank">http://paste2.org/p/3115739</a>
<div class="gmail_default"> </div>
</font></div>
<div><span style="font-family:'courier new',monospace"><br>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default"> If
there is anything else I can provide you, please let me
know! I have pulled out most of my hair at this point!</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default"> <br>
</div>
</span></div>
<div><span style="font-family:'courier new',monospace">
<div class="gmail_default">Sam</div>
</span><br>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Mar 11, 2013 at 10:11 AM,
Sylvain Bauza <span dir="ltr"><<a href="mailto:sylvain.bauza@digimind.com" target="_blank">sylvain.bauza@digimind.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>So as to reproduce the nova-manage SQL command, I
would recommand to tcpdump -A port 3306 on the host
and get the SQL trace on what's failing.<br>
<br>
Could you please explain further what is your HA
config ? Are you using pacemaker/heartbeat or any VIP
?<br>
<br>
-Sylvain<br>
<br>
Le 11/03/2013 14:23, Samuel Winchenbach a écrit :<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default">Does anyone think
this could be an openstack bug? I just want
to check before submitting a bug report.</div>
<div class="gmail_default"> <br>
</div>
<div class="gmail_default">Sam</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Mar 8, 2013 at
4:02 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sorry, I really
can't think of anything :(<br>
<div><br>
On 03/08/2013 03:52 PM, Samuel Winchenbach
wrote:<br>
> I dropped those users and no change.<br>
><br>
> I also set up general logging in
mysql but it really doesn't provide any<br>
> additional information. Any idea for
a next step I could take?<br>
><br>
> I am almost at the point of taking a
tcpdump and trying to recreate the<br>
> salted password. :/<br>
><br>
> Thanks for the help<br>
><br>
> Sam<br>
><br>
><br>
><br>
><br>
> On Fri, Mar 8, 2013 at 3:38 PM, Jay
Pipes <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
</div>
<div>> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>>
wrote:<br>
><br>
> I'm stumped :( Looks like
everything is set up correctly to me. What
is<br>
> interested is that your nova user
access works from test2, but there is<br>
> no nova@test2 user in the
mysql.user table. What about doing a DROP
USER<br>
> nova@test1; FLUSH PRIVILEGES; and
then see if that fixes things... since<br>
</div>
> the <a href="http://nova@10.21.0.0/255.255.0.0" target="_blank">nova@10.21.0.0/255.255.0.0</a>
<<a href="http://nova@10.21.0.0/255.255.0.0" target="_blank">http://nova@10.21.0.0/255.255.0.0</a>><br>
<div>> user is clearly working for
the access<br>
> from test2.<br>
><br>
> Also, I'd recommend highly
removing the nova@% user.<br>
><br>
> Best,<br>
> -jay<br>
><br>
> On 03/08/2013 03:09 PM, Samuel
Winchenbach wrote:<br>
> ><br>
> > <a href="http://paste2.org/p/3085807" target="_blank">http://paste2.org/p/3085807</a><br>
> ><br>
> ><br>
> > On Fri, Mar 8, 2013 at 2:46
PM, Jay Pipes <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>><br>
</div>
<div>> > <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>>>
wrote:<br>
> ><br>
</div>
<div>
<div>> > Please paste the
results of SELECT User, Host, Password
FROM<br>
> mysql.user<br>
> > when running as
root...<br>
> ><br>
> > Thanks!<br>
> > -jay<br>
> ><br>
> > On 03/08/2013 02:25
PM, Samuel Winchenbach wrote:<br>
> > > Here are my
grants. I don't know if this helps, but
I did<br>
> verify that<br>
> > > the password was
identical for each grant:<br>
> > <a href="http://paste2.org/p/3085361" target="_blank">http://paste2.org/p/3085361</a><br>
> > ><br>
> > ><br>
> > > On Fri, Mar 8,
2013 at 2:17 PM, Samuel Winchenbach<br>
> > <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>
<mailto:<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>><br>
> <mailto:<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>
<mailto:<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>>><br>
> > > <mailto:<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>
<mailto:<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>><br>
> <mailto:<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>
<mailto:<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>>>>>
wrote:<br>
> > ><br>
> > >
root@test1:/var/log# mysql -hmysql-ha
-unova<br>
> > >
-p********************************
-e"SELECT User, Host,<br>
> Password<br>
> > > FROM
mysql.user;"<br>
> > > ERROR 1142
(42000) at line 1: SELECT command denied
to user<br>
> > >
'nova'@'test1' for table 'user'<br>
> > ><br>
> > ><br>
> > > On Fri, Mar
8, 2013 at 2:06 PM, Jay Pipes<br>
> <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>><br>
> > <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>><br>
</div>
</div>
<div>
<div>> > >
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>>>>
wrote:<br>
> > ><br>
> > > What does
this show?<br>
> > ><br>
> > > mysql
-hmysql-ha -unova -p<PASS>
-e"SELECT User, Host,<br>
> > Password FROM<br>
> > >
mysql.user"<br>
> > ><br>
> > > -jay<br>
> > ><br>
> > > On
03/08/2013 01:46 PM, Samuel Winchenbach
wrote:<br>
> > > >
Sorry, that must have been a copy and
paste error.<br>
> Here<br>
> > is what I<br>
> > > >
actually ran:<br>
> > > ><br>
> > > > <a href="http://paste2.org/p/3084996" target="_blank">http://paste2.org/p/3084996</a><br>
> > > ><br>
> > > ><br>
> > > > On
Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes<br>
> > <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>><br>
> > >
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>>><br>
> > > >
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>><br>
> > <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>><br>
> <mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>>>>>
wrote:<br>
> > > ><br>
> > > >
On 03/08/2013 12:19 PM, Samuel
Winchenbach wrote:<br>
> > > >
> Hi All,<br>
> > > >
><br>
> > > >
> I have two nodes (test1 and test2)
that I am<br>
> trying to<br>
> > > set up in
a<br>
> > > >
> highly available configuration.<br>
> > > >
><br>
> > > >
> During the setup process I tried
running<br>
> "nova-manage<br>
> > > service
list" on<br>
> > > >
> both nodes. It worked fine on
test2, but<br>
> fails on<br>
> > > test1
even<br>
> > > >
though I<br>
> > > >
> can connect to the database with
the mysql<br>
> client from<br>
> > > test1.<br>
> > > >
><br>
> > > >
> Here is a screen capture that shows
the setup on<br>
> > the two<br>
> > > nodes are<br>
> > > >
> basically identical:<br>
> <a href="http://paste2.org/p/3084223" target="_blank">http://paste2.org/p/3084223</a><br>
> > > ><br>
> > > >
In the above paste you are doing:<br>
> > > ><br>
> > > >
mysql -unova - hmysql-ha -u root
nova<br>
> > > >
-p********************************<br>
> > > ><br>
> > > >
Note you are supplying 2 -u arguments,
and mysql<br>
> > will take<br>
> > > the
second<br>
> > > >
(root).<br>
> > > ><br>
> > > >
-jay<br>
> > > ><br>
> > > >
_______________________________________________<br>
> > > >
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
> > > >
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>>><br>
> > >
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>>>><br>
> > > >
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>>><br>
> > >
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
> > <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>>>>><br>
> > > >
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
> > > >
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
> > > ><br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>