[Openstack] Incredibly odd mysql permission error
Sylvain Bauza
sylvain.bauza at digimind.com
Mon Mar 11 16:00:06 UTC 2013
Ok, lemme try to summarize.
You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to
Pacemaker.
This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2).
Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP.
Are you sure your my.cnf is actually the same in between both DRBD nodes
? (I would recommend to symlink it to a physical file hosted on the DRBD
device).
One thing is hurting me : you told me that nova is also pacemake'd. If
so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing
to nova-ha (assuming 10.21.2.4 as per /etc/hosts).
Also, as per my understanding of Pacemaker, DRBD partition is setup by
default on test2, correct ?
Sorry, as per my first reading, I can't see anything obvious. That said,
I'm not sure this is a Nova bug, as the tcpdump trace is seeing a
correct MySQL connection attempt. But maybe I'm wrong ?
Anyway, are you sure you only have *one* MySQL engine running (either on
test1 or test2) and nova-manage trying to access this right one ?
Perms look good to me. As it a test setup, you could try to unleash the
grants by deleting them and allowing nova@'%' to see if it's a basic dns
mapping issue.
-Sylvain
Le 11/03/2013 16:09, Samuel Winchenbach a écrit :
> I
> enabled general_log in /etc/mysql/my.cnf Here are the results of
> connecting from "test1", "test2" and using the client:
> http://paste2.org/p/3115525
> I purposefully used the real password in case there is a problem with it.
> I changed before submitting post.
>
> here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an
> attempted "nova-manage service list" from test1:
> https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump
> I looked at it with wireshark and couldn't see anything that jumped
> out at me as incorrect. I have not yet tried to recreate the salted
> password.
>
>
> Here is my pacemaker configuration for mysql. I stripped out
> openstack services, rabbitmq and others for clarity. All resources
> are currently disabled (other than MySQL):
> http://paste2.org/p/3115685
>
> Please don't yell at me for having STONITH disabled :P This is a
> testing cluster and I am working on getting routed to the IPMI interface.
>
> /etc/hosts:
> http://paste2.org/p/3115713
> /etc/nova/nova.conf:
> http://paste2.org/p/3115739
>
> If there is anything else I can provide you, please let me know! I
> have pulled out most of my hair at this point!
>
> Sam
>
>
>
>
> On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza
> <sylvain.bauza at digimind.com <mailto:sylvain.bauza at digimind.com>> wrote:
>
> So as to reproduce the nova-manage SQL command, I would recommand
> to tcpdump -A port 3306 on the host and get the SQL trace on
> what's failing.
>
> Could you please explain further what is your HA config ? Are you
> using pacemaker/heartbeat or any VIP ?
>
> -Sylvain
>
> Le 11/03/2013 14:23, Samuel Winchenbach a écrit :
>> Does anyone think this could be an openstack bug? I just want to
>> check before submitting a bug report.
>>
>> Sam
>>
>>
>> On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>> wrote:
>>
>> Sorry, I really can't think of anything :(
>>
>> On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
>> > I dropped those users and no change.
>> >
>> > I also set up general logging in mysql but it really
>> doesn't provide any
>> > additional information. Any idea for a next step I could take?
>> >
>> > I am almost at the point of taking a tcpdump and trying to
>> recreate the
>> > salted password. :/
>> >
>> > Thanks for the help
>> >
>> > Sam
>> >
>> >
>> >
>> >
>> > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes
>> <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>> wrote:
>> >
>> > I'm stumped :( Looks like everything is set up
>> correctly to me. What is
>> > interested is that your nova user access works from
>> test2, but there is
>> > no nova at test2 user in the mysql.user table. What about
>> doing a DROP USER
>> > nova at test1; FLUSH PRIVILEGES; and then see if that
>> fixes things... since
>> > the nova at 10.21.0.0/255.255.0.0
>> <http://nova@10.21.0.0/255.255.0.0>
>> <http://nova@10.21.0.0/255.255.0.0>
>> > user is clearly working for the access
>> > from test2.
>> >
>> > Also, I'd recommend highly removing the nova@% user.
>> >
>> > Best,
>> > -jay
>> >
>> > On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
>> > >
>> > > http://paste2.org/p/3085807
>> > >
>> > >
>> > > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes
>> <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>> > > <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>>>> wrote:
>> > >
>> > > Please paste the results of SELECT User, Host,
>> Password FROM
>> > mysql.user
>> > > when running as root...
>> > >
>> > > Thanks!
>> > > -jay
>> > >
>> > > On 03/08/2013 02:25 PM, Samuel Winchenbach wrote:
>> > > > Here are my grants. I don't know if this
>> helps, but I did
>> > verify that
>> > > > the password was identical for each grant:
>> > > http://paste2.org/p/3085361
>> > > >
>> > > >
>> > > > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach
>> > > <swinchen at gmail.com <mailto:swinchen at gmail.com>
>> <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>>
>> > <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>
>> <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>>>
>> > > > <mailto:swinchen at gmail.com
>> <mailto:swinchen at gmail.com> <mailto:swinchen at gmail.com
>> <mailto:swinchen at gmail.com>>
>> > <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>
>> <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>>>>> wrote:
>> > > >
>> > > > root at test1:/var/log# mysql -hmysql-ha -unova
>> > > > -p******************************** -e"SELECT
>> User, Host,
>> > Password
>> > > > FROM mysql.user;"
>> > > > ERROR 1142 (42000) at line 1: SELECT
>> command denied to user
>> > > > 'nova'@'test1' for table 'user'
>> > > >
>> > > >
>> > > > On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
>> > <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>> > > <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>>>
>> > > > <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>>> wrote:
>> > > >
>> > > > What does this show?
>> > > >
>> > > > mysql -hmysql-ha -unova -p<PASS>
>> -e"SELECT User, Host,
>> > > Password FROM
>> > > > mysql.user"
>> > > >
>> > > > -jay
>> > > >
>> > > > On 03/08/2013 01:46 PM, Samuel
>> Winchenbach wrote:
>> > > > > Sorry, that must have been a copy and
>> paste error.
>> > Here
>> > > is what I
>> > > > > actually ran:
>> > > > >
>> > > > > http://paste2.org/p/3084996
>> > > > >
>> > > > >
>> > > > > On Fri, Mar 8, 2013 at 12:40 PM, Jay
>> Pipes
>> > > <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>
>> > > > <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>>
>> > > > > <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>
>> > > <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>>
>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>>>>
>> wrote:
>> > > > >
>> > > > > On 03/08/2013 12:19 PM, Samuel
>> Winchenbach wrote:
>> > > > > > Hi All,
>> > > > > >
>> > > > > > I have two nodes (test1 and test2)
>> that I am
>> > trying to
>> > > > set up in a
>> > > > > > highly available configuration.
>> > > > > >
>> > > > > > During the setup process I tried
>> running
>> > "nova-manage
>> > > > service list" on
>> > > > > > both nodes. It worked fine on
>> test2, but
>> > fails on
>> > > > test1 even
>> > > > > though I
>> > > > > > can connect to the database with
>> the mysql
>> > client from
>> > > > test1.
>> > > > > >
>> > > > > > Here is a screen capture that shows
>> the setup on
>> > > the two
>> > > > nodes are
>> > > > > > basically identical:
>> > http://paste2.org/p/3084223
>> > > > >
>> > > > > In the above paste you are doing:
>> > > > >
>> > > > > mysql -unova - hmysql-ha -u
>> root nova
>> > > > > -p********************************
>> > > > >
>> > > > > Note you are supplying 2 -u
>> arguments, and mysql
>> > > will take
>> > > > the second
>> > > > > (root).
>> > > > >
>> > > > > -jay
>> > > > >
>> > > > >
>> _______________________________________________
>> > > > > Mailing list:
>> https://launchpad.net/~openstack
>> <https://launchpad.net/%7Eopenstack>
>> > > > > Post to :
>> openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>>
>> > > > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>>>
>> > > > > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>>
>> > > > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>>>>
>> > > > > Unsubscribe :
>> https://launchpad.net/~openstack
>> <https://launchpad.net/%7Eopenstack>
>> > > > > More help :
>> https://help.launchpad.net/ListHelp
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>>
>>
>> _______________________________________________
>> Mailing list:https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
>> Post to :openstack at lists.launchpad.net <mailto:openstack at lists.launchpad.net>
>> Unsubscribe :https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
>> More help :https://help.launchpad.net/ListHelp
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> Post to : openstack at lists.launchpad.net
> <mailto:openstack at lists.launchpad.net>
> Unsubscribe : https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> More help : https://help.launchpad.net/ListHelp
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130311/4d1766b7/attachment.html>
More information about the Openstack
mailing list