[Openstack] Error on upgrading to latest Keystone - Ubuntu 12.04.4 + UCA
Martinx - ジェームズ
thiagocmartinsc at gmail.com
Sun Mar 9 02:29:45 UTC 2014
Almost everything is working... Only 1 tenant doesn't work anymore, its
instances have IP but it can not ping the gateway (Neutron L3 unreachable
only for 1 Tenant, very weird), others tenants are working as expected...
I'll keep testing it... Cheers!
On 8 March 2014 21:07, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
> Guys,
>
> Please, ignore my last message... I'm thinking that I'm facing a different
> problem, out off this topic.
>
> Now, instances can not ping its gateway (Neutron L3 is unreachable). I'll
> make a different post if necessary.
>
> Sorry about the buzz.
>
> Tks,
> Thiago
>
>
> On 8 March 2014 16:53, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
>
>> Well, I think I just destroyed the entire Cloud Computing environment...
>> Security Groups isn't working, it is impossible to reach the tenant's
>> subnet, they are just closed (i.e. not working), all sites hosted here, are
>> down. Damn... What a mess... :-/
>>
>> Is there any way to try to fix this? I have the dump of keystone db,
>> before dropping the INDEXes...
>>
>>
>> On 8 March 2014 16:12, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
>>
>>> Stackers,
>>>
>>> I'll mark this as fixed, look:
>>>
>>> ---
>>> root at controller-1:~# apt-get -f install
>>> Reading package lists... Done
>>> Building dependency tree
>>> Reading state information... Done
>>> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>>> 1 not fully installed or removed.
>>> After this operation, 0 B of additional disk space will be used.
>>> *Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...*
>>> ---
>>>
>>> Keystone seems to be upgraded now...
>>>
>>> Thanks!
>>> Thiago
>>>
>>>
>>> On 8 March 2014 16:06, Martinx - ジェームズ <thiagocmartinsc at gmail.com>wrote:
>>>
>>>> Well,
>>>>
>>>> I just did a shot in the dark:
>>>>
>>>>
>>>> > ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires_valid`;
>>>>
>>>> And "keystone-manage db_sync" seems to be running again... But, I have
>>>> no idea if this DROP I did will break my OpenStack, or not...
>>>>
>>>> I don't want to waste more time from you guys with this but, maybe it
>>>> is not a big deal... What do you guys think?!
>>>>
>>>> Thanks!
>>>> Thiago
>>>>
>>>>
>>>> On 8 March 2014 15:52, Martinx - ジェームズ <thiagocmartinsc at gmail.com>wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> I'm seeing some progress here, look:
>>>>>
>>>>> ---
>>>>> root at controller-1:~# pt-duplicate-key-checker --host localhost
>>>>> --databases=keystone --ask-pass
>>>>> Enter password:
>>>>> #
>>>>> ########################################################################
>>>>> # keystone.token
>>>>>
>>>>> #
>>>>> ########################################################################
>>>>>
>>>>> # ix_token_expires is a left-prefix of ix_token_expires_valid
>>>>> # Key definitions:
>>>>> # KEY `ix_token_expires` (`expires`),
>>>>> # KEY `ix_token_expires_valid` (`expires`,`valid`)
>>>>> # Column types:
>>>>> # `expires` datetime default null
>>>>> # `valid` tinyint(1) not null
>>>>> # To remove this duplicate index, execute:
>>>>> ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires`;
>>>>>
>>>>> #
>>>>> ########################################################################
>>>>> # Summary of indexes
>>>>>
>>>>> #
>>>>> ########################################################################
>>>>>
>>>>> # Size Duplicate Indexes 1321470
>>>>> # Total Duplicate Indexes 1
>>>>> # Total Indexes 42
>>>>> ----
>>>>>
>>>>> ---
>>>>> mysql> ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires`;
>>>>> Query OK, 0 rows affected (1.03 sec)
>>>>> Records: 0 Duplicates: 0 Warnings: 0
>>>>> ---
>>>>>
>>>>> ---
>>>>> No more duplicate key found by "pt-duplicate-key-checker" but, same
>>>>> problem with "keystone-manage "db_sync", look:
>>>>>
>>>>> root at controller-1:~# keystone-manage db_sync
>>>>> 2014-03-08 15:46:59.186 8043 CRITICAL keystone [-] (OperationalError)
>>>>> (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX
>>>>> ix_token_expires_valid ON token (expires, valid)' ()
>>>>> ---
>>>>>
>>>>> Guys, what can I do!? I really need to keep this OpenStack
>>>>> installation intact, I can not afford losing it...
>>>>>
>>>>> Tks!
>>>>> Thiago
>>>>>
>>>>>
>>>>> On 8 March 2014 04:39, Martinx - ジェームズ <thiagocmartinsc at gmail.com>wrote:
>>>>>
>>>>>> Lance,
>>>>>>
>>>>>> I think I know what had happened... The Keystone upgrade process died
>>>>>> in the middle of the upgrade.
>>>>>>
>>>>>> Now, the Keystone SQL database is stucked at some point "in the
>>>>>> middle" of both versions...
>>>>>>
>>>>>> My "keystone" MySQL db dump is huge, more than >1G...
>>>>>>
>>>>>> Any tips about how to fix this?!
>>>>>>
>>>>>> I really appreciate any help!
>>>>>>
>>>>>> Tks!
>>>>>> Thiago
>>>>>>
>>>>>>
>>>>>> On 8 March 2014 02:55, Martinx - ジェームズ <thiagocmartinsc at gmail.com>wrote:
>>>>>>
>>>>>>> Lance,
>>>>>>>
>>>>>>> I can provide everything, no problem... I'm doing this to reproduce
>>>>>>> it:
>>>>>>>
>>>>>>>
>>>>>>> root at controller-1:~# keystone-manage db_sync
>>>>>>> 2014-03-08 02:44:46.723 5407 CRITICAL keystone [-]
>>>>>>> (OperationalError) (1061, "Duplicate key name 'ix_token_expires_valid'")
>>>>>>> 'CREATE INDEX ix_token_expires_valid ON token (expires, valid)' ()
>>>>>>>
>>>>>>>
>>>>>>> Log shows:
>>>>>>>
>>>>>>> ==> /var/log/keystone/keystone.log <==
>>>>>>> 2014-03-08 02:44:46.723 5407 CRITICAL keystone [-]
>>>>>>> (OperationalError) (1061, "Duplicate key name 'ix_token_expires_valid'")
>>>>>>> 'CREATE INDEX ix_token_expires_valid ON token (expires, valid)' ()
>>>>>>>
>>>>>>>
>>>>>>> root at controller-1:~# dpkg -l | grep keystone
>>>>>>> iF keystone 1:2013.2.2-0ubuntu1~cloud0
>>>>>>> OpenStack identity service - Daemons
>>>>>>> ii python-keystone 1:2013.2.2-0ubuntu1~cloud0
>>>>>>> OpenStack identity service - Python library
>>>>>>> ii python-keystoneclient 1:0.3.2-0ubuntu1~cloud0
>>>>>>> Client library for OpenStack Identity API
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> If I run "apt-get -f install", to try to fix it ("iF" dpkg status
>>>>>>> above), same problem, look:
>>>>>>>
>>>>>>> ---
>>>>>>> root at controller-1:~# apt-get -f install
>>>>>>> Reading package lists... Done
>>>>>>> Building dependency tree
>>>>>>> Reading state information... Done
>>>>>>> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>>>>>>> 1 not fully installed or removed.
>>>>>>> After this operation, 0 B of additional disk space will be used.
>>>>>>> Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...
>>>>>>> 2014-03-08 02:49:11.050 5469 CRITICAL keystone [-]
>>>>>>> (OperationalError) (1061, "Duplicate key name 'ix_token_expires_valid'")
>>>>>>> 'CREATE INDEX ix_token_expires_valid ON token (expires, valid)' ()
>>>>>>> dpkg: error processing keystone (--configure):
>>>>>>> subprocess installed post-installation script returned error exit
>>>>>>> status 1
>>>>>>> Errors were encountered while processing:
>>>>>>> keystone
>>>>>>> E: Sub-process /usr/bin/dpkg returned an error code (1)
>>>>>>> ---
>>>>>>>
>>>>>>> I'm lost... :-/
>>>>>>>
>>>>>>> Maybe the upgrade procedure got stucked, then the second time I
>>>>>>> tried, it triggered this error... I'm not sure...
>>>>>>>
>>>>>>> Tks!
>>>>>>> Thiago
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 8 March 2014 01:55, Lance D Bragstad <ldbragst at us.ibm.com> wrote:
>>>>>>>
>>>>>>>> Thiago,
>>>>>>>>
>>>>>>>> Are you able to provide a little more information? Log files maybe?
>>>>>>>> What are the steps you're taking to recreate the issue?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>>
>>>>>>>> Lance Bragstad
>>>>>>>> ldbragst at us.ibm.com
>>>>>>>>
>>>>>>>> Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote on 03/07/2014
>>>>>>>> 08:34:11 PM:
>>>>>>>>
>>>>>>>>
>>>>>>>> > From: Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>>>>>>>
>>>>>>>> > To: Lance D Bragstad/Rochester/IBM at IBMUS,
>>>>>>>> > Cc: "openstack at lists.openstack.org" <
>>>>>>>> openstack at lists.openstack.org>
>>>>>>>> > Date: 03/07/2014 08:34 PM
>>>>>>>> > Subject: Re: [Openstack] Error on upgrading to latest Keystone -
>>>>>>>> > Ubuntu 12.04.4 + UCA
>>>>>>>> >
>>>>>>>> > Thanks Lance!
>>>>>>>>
>>>>>>>> >
>>>>>>>> > I just did: "keystone-manage db_sync" and the output is the same:
>>>>>>>> >
>>>>>>>> > ---
>>>>>>>> > 2014-03-07 23:23:48.564 2258 CRITICAL keystone [-]
>>>>>>>> > (OperationalError) (1061, "Duplicate key name
>>>>>>>> > 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid
>>>>>>>> ON
>>>>>>>> > token (expires, valid)' ()
>>>>>>>> > ---
>>>>>>>> >
>>>>>>>> > I just read [1] and [2] but, not a clue about how to fix it.
>>>>>>>> >
>>>>>>>> > My Keystone is still running and I have no idea if I can restart
>>>>>>>> it
>>>>>>>> > or not... What if it doesn't come back?! My entire Cloud will
>>>>>>>> become
>>>>>>>> > useless... This is a production environment, it was working
>>>>>>>> smoothly
>>>>>>>> > for months... :'(
>>>>>>>> >
>>>>>>>> > Do you know if there is something that I can do directly on MySQL
>>>>>>>> to
>>>>>>>> > try to fix it?!
>>>>>>>> >
>>>>>>>> > Best,
>>>>>>>> > Thiago
>>>>>>>> >
>>>>>>>>
>>>>>>>> > On 7 March 2014 23:18, Lance D Bragstad <ldbragst at us.ibm.com>
>>>>>>>> wrote:
>>>>>>>> > Thiago,
>>>>>>>> >
>>>>>>>> > Your database might be out of sync with the latest Keystone code
>>>>>>>> if
>>>>>>>> > you just performed an update. Try migrating your database [1].
>>>>>>>> The
>>>>>>>> > link gives some information on how to do that, but you'll
>>>>>>>> > essentially want to do a `keystone-manage db_sync`, also check
>>>>>>>> the
>>>>>>>> > man pages [2]. This should ensure your Keystone database has the
>>>>>>>> > proper schema for the Keystone code you're running.
>>>>>>>> >
>>>>>>>> > [1] http://docs.openstack.org/developer/keystone/
>>>>>>>> > configuration.html#step-2-sync-your-new-empty-database
>>>>>>>> > [2]
>>>>>>>> http://docs.openstack.org/developer/keystone/man/keystone-manage.html
>>>>>>>> >
>>>>>>>> > Best Regards,
>>>>>>>> >
>>>>>>>> > Lance Bragstad
>>>>>>>> > ldbragst at us.ibm.com
>>>>>>>> >
>>>>>>>> > Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote on 03/07/2014
>>>>>>>> > 06:50:01 PM:
>>>>>>>> >
>>>>>>>> > > From: Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>>>>>>> > > To: "openstack at lists.openstack.org" <
>>>>>>>> openstack at lists.openstack.org>,
>>>>>>>> > > Date: 03/07/2014 07:03 PM
>>>>>>>> > > Subject: [Openstack] Error on upgrading to latest Keystone -
>>>>>>>> Ubuntu
>>>>>>>> > > 12.04.4 + UCA
>>>>>>>> >
>>>>>>>> > >
>>>>>>>> > > Guys,
>>>>>>>> > >
>>>>>>>> > > I just did:
>>>>>>>> > >
>>>>>>>> > > apt-get update
>>>>>>>> > > apt-get dist-upgrade
>>>>>>>> > >
>>>>>>>> > > And I'm seeing the following error:
>>>>>>>> > >
>>>>>>>> > > ---
>>>>>>>> > > Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...
>>>>>>>> > > 2014-03-07 21:45:39.473 15540 CRITICAL keystone [-]
>>>>>>>> > > (OperationalError) (1061, "Duplicate key name
>>>>>>>> > > 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid
>>>>>>>> ON
>>>>>>>> > > token (expires, valid)' ()
>>>>>>>> > > dpkg: error processing keystone (--configure):
>>>>>>>> > > subprocess installed post-installation script returned error
>>>>>>>> exit status 1
>>>>>>>> > > Errors were encountered while processing:
>>>>>>>> > > keystone
>>>>>>>> > > E: Sub-process /usr/bin/dpkg returned an error code (1)
>>>>>>>> > > ---
>>>>>>>> > >
>>>>>>>> > > What can I do?!
>>>>>>>> > >
>>>>>>>> > > Tks!
>>>>>>>> > > Thiago_______________________________________________
>>>>>>>> > > Mailing list:
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>> > > Post to : openstack at lists.openstack.org
>>>>>>>> > > Unsubscribe :
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140308/2c0e5416/attachment.html>
More information about the Openstack
mailing list