[Openstack] Incredibly odd mysql permission error
Sylvain Bauza
sylvain.bauza at digimind.com
Mon Mar 11 16:24:40 UTC 2013
When looking at MySQL 5.1 refman
(http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would
suggest to follow the procedure :
1. 'mysqladmin flush-hosts'
2. replace DNS entries in mysql.user table by IP addresses instead
3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and
restart nova-api !)
I wouldn't bet on it, but I would say this is due to some name
resolution which is incorrect.
-Sylvain
Le 11/03/2013 17:00, Sylvain Bauza a écrit :
> Ok, lemme try to summarize.
> You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to
> Pacemaker.
> This setup is relying on two hosts, test1 (10.21.0.1) and test2
> (10.21.0.2).
> Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP.
>
> Are you sure your my.cnf is actually the same in between both DRBD
> nodes ? (I would recommend to symlink it to a physical file hosted on
> the DRBD device).
>
> One thing is hurting me : you told me that nova is also pacemake'd. If
> so, why can I still see my_ip=10.21.0.2 (test2) ? It should be
> pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts).
>
> Also, as per my understanding of Pacemaker, DRBD partition is setup by
> default on test2, correct ?
>
>
> Sorry, as per my first reading, I can't see anything obvious. That
> said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing
> a correct MySQL connection attempt. But maybe I'm wrong ?
>
> Anyway, are you sure you only have *one* MySQL engine running (either
> on test1 or test2) and nova-manage trying to access this right one ?
>
> Perms look good to me. As it a test setup, you could try to unleash
> the grants by deleting them and allowing nova@'%' to see if it's a
> basic dns mapping issue.
>
> -Sylvain
>
>
>
> Le 11/03/2013 16:09, Samuel Winchenbach a écrit :
>> I
>> enabled general_log in /etc/mysql/my.cnf Here are the results of
>> connecting from "test1", "test2" and using the client:
>> http://paste2.org/p/3115525
>> I purposefully used the real password in case there is a problem with
>> it.
>> I changed before submitting post.
>>
>> here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an
>> attempted "nova-manage service list" from test1:
>> https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump
>> I looked at it with wireshark and couldn't see anything that jumped
>> out at me as incorrect. I have not yet tried to recreate the salted
>> password.
>>
>>
>> Here is my pacemaker configuration for mysql. I stripped out
>> openstack services, rabbitmq and others for clarity. All resources
>> are currently disabled (other than MySQL):
>> http://paste2.org/p/3115685
>>
>> Please don't yell at me for having STONITH disabled :P This is a
>> testing cluster and I am working on getting routed to the IPMI interface.
>>
>> /etc/hosts:
>> http://paste2.org/p/3115713
>> /etc/nova/nova.conf:
>> http://paste2.org/p/3115739
>>
>> If there is anything else I can provide you, please let me know! I
>> have pulled out most of my hair at this point!
>>
>> Sam
>>
>>
>>
>>
>> On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza
>> <sylvain.bauza at digimind.com <mailto:sylvain.bauza at digimind.com>> wrote:
>>
>> So as to reproduce the nova-manage SQL command, I would recommand
>> to tcpdump -A port 3306 on the host and get the SQL trace on
>> what's failing.
>>
>> Could you please explain further what is your HA config ? Are you
>> using pacemaker/heartbeat or any VIP ?
>>
>> -Sylvain
>>
>> Le 11/03/2013 14:23, Samuel Winchenbach a écrit :
>>> Does anyone think this could be an openstack bug? I just want
>>> to check before submitting a bug report.
>>>
>>> Sam
>>>
>>>
>>> On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>> wrote:
>>>
>>> Sorry, I really can't think of anything :(
>>>
>>> On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
>>> > I dropped those users and no change.
>>> >
>>> > I also set up general logging in mysql but it really
>>> doesn't provide any
>>> > additional information. Any idea for a next step I could
>>> take?
>>> >
>>> > I am almost at the point of taking a tcpdump and trying to
>>> recreate the
>>> > salted password. :/
>>> >
>>> > Thanks for the help
>>> >
>>> > Sam
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes
>>> <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>
>>> wrote:
>>> >
>>> > I'm stumped :( Looks like everything is set up
>>> correctly to me. What is
>>> > interested is that your nova user access works from
>>> test2, but there is
>>> > no nova at test2 user in the mysql.user table. What about
>>> doing a DROP USER
>>> > nova at test1; FLUSH PRIVILEGES; and then see if that
>>> fixes things... since
>>> > the nova at 10.21.0.0/255.255.0.0
>>> <http://nova@10.21.0.0/255.255.0.0>
>>> <http://nova@10.21.0.0/255.255.0.0>
>>> > user is clearly working for the access
>>> > from test2.
>>> >
>>> > Also, I'd recommend highly removing the nova@% user.
>>> >
>>> > Best,
>>> > -jay
>>> >
>>> > On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
>>> > >
>>> > > http://paste2.org/p/3085807
>>> > >
>>> > >
>>> > > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes
>>> <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>>> > > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>>>> wrote:
>>> > >
>>> > > Please paste the results of SELECT User, Host,
>>> Password FROM
>>> > mysql.user
>>> > > when running as root...
>>> > >
>>> > > Thanks!
>>> > > -jay
>>> > >
>>> > > On 03/08/2013 02:25 PM, Samuel Winchenbach wrote:
>>> > > > Here are my grants. I don't know if this
>>> helps, but I did
>>> > verify that
>>> > > > the password was identical for each grant:
>>> > > http://paste2.org/p/3085361
>>> > > >
>>> > > >
>>> > > > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach
>>> > > <swinchen at gmail.com <mailto:swinchen at gmail.com>
>>> <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>>
>>> > <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>
>>> <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>>>
>>> > > > <mailto:swinchen at gmail.com
>>> <mailto:swinchen at gmail.com> <mailto:swinchen at gmail.com
>>> <mailto:swinchen at gmail.com>>
>>> > <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>
>>> <mailto:swinchen at gmail.com <mailto:swinchen at gmail.com>>>>>
>>> wrote:
>>> > > >
>>> > > > root at test1:/var/log# mysql -hmysql-ha -unova
>>> > > > -p******************************** -e"SELECT
>>> User, Host,
>>> > Password
>>> > > > FROM mysql.user;"
>>> > > > ERROR 1142 (42000) at line 1: SELECT
>>> command denied to user
>>> > > > 'nova'@'test1' for table 'user'
>>> > > >
>>> > > >
>>> > > > On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
>>> > <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>>> > > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>>>
>>> > > > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>>
>>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>>>
>>> wrote:
>>> > > >
>>> > > > What does this show?
>>> > > >
>>> > > > mysql -hmysql-ha -unova -p<PASS>
>>> -e"SELECT User, Host,
>>> > > Password FROM
>>> > > > mysql.user"
>>> > > >
>>> > > > -jay
>>> > > >
>>> > > > On 03/08/2013 01:46 PM, Samuel
>>> Winchenbach wrote:
>>> > > > > Sorry, that must have been a copy
>>> and paste error.
>>> > Here
>>> > > is what I
>>> > > > > actually ran:
>>> > > > >
>>> > > > > http://paste2.org/p/3084996
>>> > > > >
>>> > > > >
>>> > > > > On Fri, Mar 8, 2013 at 12:40 PM, Jay
>>> Pipes
>>> > > <jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>
>>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>
>>> > > > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>
>>> > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>
>>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>>
>>> > > > > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>
>>> > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>
>>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>
>>> > > <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com> <mailto:jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>>
>>> > <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>
>>> <mailto:jaypipes at gmail.com <mailto:jaypipes at gmail.com>>>>>>
>>> wrote:
>>> > > > >
>>> > > > > On 03/08/2013 12:19 PM, Samuel
>>> Winchenbach wrote:
>>> > > > > > Hi All,
>>> > > > > >
>>> > > > > > I have two nodes (test1 and test2)
>>> that I am
>>> > trying to
>>> > > > set up in a
>>> > > > > > highly available configuration.
>>> > > > > >
>>> > > > > > During the setup process I tried
>>> running
>>> > "nova-manage
>>> > > > service list" on
>>> > > > > > both nodes. It worked fine on
>>> test2, but
>>> > fails on
>>> > > > test1 even
>>> > > > > though I
>>> > > > > > can connect to the database with
>>> the mysql
>>> > client from
>>> > > > test1.
>>> > > > > >
>>> > > > > > Here is a screen capture that
>>> shows the setup on
>>> > > the two
>>> > > > nodes are
>>> > > > > > basically identical:
>>> > http://paste2.org/p/3084223
>>> > > > >
>>> > > > > In the above paste you are doing:
>>> > > > >
>>> > > > > mysql -unova - hmysql-ha -u
>>> root nova
>>> > > > > -p********************************
>>> > > > >
>>> > > > > Note you are supplying 2 -u
>>> arguments, and mysql
>>> > > will take
>>> > > > the second
>>> > > > > (root).
>>> > > > >
>>> > > > > -jay
>>> > > > >
>>> > > > >
>>> _______________________________________________
>>> > > > > Mailing list:
>>> https://launchpad.net/~openstack
>>> <https://launchpad.net/%7Eopenstack>
>>> > > > > Post to :
>>> openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>
>>> > > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>>
>>> > > > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>
>>> > > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>>>
>>> > > > >
>>> <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>
>>> > > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>>
>>> > > > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>
>>> > > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>
>>> > <mailto:openstack at lists.launchpad.net
>>> <mailto:openstack at lists.launchpad.net>>>>>
>>> > > > > Unsubscribe :
>>> https://launchpad.net/~openstack
>>> <https://launchpad.net/%7Eopenstack>
>>> > > > > More help :
>>> https://help.launchpad.net/ListHelp
>>> > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> >
>>> >
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list:https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
>>> Post to :openstack at lists.launchpad.net <mailto:openstack at lists.launchpad.net>
>>> Unsubscribe :https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
>>> More help :https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> <https://launchpad.net/%7Eopenstack>
>> Post to : openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> Unsubscribe : https://launchpad.net/~openstack
>> <https://launchpad.net/%7Eopenstack>
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130311/a18a8841/attachment.html>
More information about the Openstack
mailing list