Open Stack

Well, I think I just destroyed the entire Cloud Computing environment...
Security Groups isn't working, it is impossible to reach the tenant's
subnet, they are just closed (i.e. not working), all sites hosted here, are
down. Damn... What a mess...   :-/

Is there any way to try to fix this? I have the dump of keystone db, before
dropping the INDEXes...

On 8 March 2014 16:12, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:

> Stackers,
>
> I'll mark this as fixed, look:
>
> ---
> root at controller-1:~# apt-get -f install
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> 1 not fully installed or removed.
> After this operation, 0 B of additional disk space will be used.
> *Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...*
> ---
>
> Keystone seems to be upgraded now...
>
> Thanks!
> Thiago
>
>
> On 8 March 2014 16:06, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
>
>> Well,
>>
>> I just did a shot in the dark:
>>
>>
>> > ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires_valid`;
>>
>> And "keystone-manage db_sync" seems to be running again... But, I have no
>> idea if this DROP I did will break my OpenStack, or not...
>>
>> I don't want to waste more time from you guys with this but, maybe it is
>> not a big deal... What do you guys think?!
>>
>> Thanks!
>> Thiago
>>
>>
>> On 8 March 2014 15:52, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:
>>
>>> Hi!
>>>
>>> I'm seeing some progress here, look:
>>>
>>> ---
>>> root at controller-1:~# pt-duplicate-key-checker --host localhost
>>> --databases=keystone --ask-pass
>>> Enter password:
>>> #
>>> ########################################################################
>>> # keystone.token
>>>
>>> #
>>> ########################################################################
>>>
>>> # ix_token_expires is a left-prefix of ix_token_expires_valid
>>> # Key definitions:
>>> #   KEY `ix_token_expires` (`expires`),
>>> #   KEY `ix_token_expires_valid` (`expires`,`valid`)
>>> # Column types:
>>> #         `expires` datetime default null
>>> #         `valid` tinyint(1) not null
>>> # To remove this duplicate index, execute:
>>> ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires`;
>>>
>>> #
>>> ########################################################################
>>> # Summary of indexes
>>>
>>> #
>>> ########################################################################
>>>
>>> # Size Duplicate Indexes   1321470
>>> # Total Duplicate Indexes  1
>>> # Total Indexes            42
>>> ----
>>>
>>> ---
>>> mysql> ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires`;
>>> Query OK, 0 rows affected (1.03 sec)
>>> Records: 0  Duplicates: 0  Warnings: 0
>>> ---
>>>
>>> ---
>>> No more duplicate key found by "pt-duplicate-key-checker" but, same
>>> problem with "keystone-manage "db_sync", look:
>>>
>>> root at controller-1:~# keystone-manage db_sync
>>> 2014-03-08 15:46:59.186 8043 CRITICAL keystone [-] (OperationalError)
>>> (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX
>>> ix_token_expires_valid ON token (expires, valid)' ()
>>> ---
>>>
>>> Guys, what can I do!? I really need to keep this OpenStack installation
>>> intact, I can not afford losing it...
>>>
>>> Tks!
>>> Thiago
>>>
>>>
>>> On 8 March 2014 04:39, Martinx - ジェームズ <thiagocmartinsc at gmail.com>wrote:
>>>
>>>> Lance,
>>>>
>>>> I think I know what had happened... The Keystone upgrade process died
>>>> in the middle of the upgrade.
>>>>
>>>> Now, the Keystone SQL database is stucked at some point "in the middle"
>>>> of both versions...
>>>>
>>>> My "keystone" MySQL db dump is huge, more than >1G...
>>>>
>>>> Any tips about how to fix this?!
>>>>
>>>> I really appreciate any help!
>>>>
>>>> Tks!
>>>> Thiago
>>>>
>>>>
>>>> On 8 March 2014 02:55, Martinx - ジェームズ <thiagocmartinsc at gmail.com>wrote:
>>>>
>>>>> Lance,
>>>>>
>>>>> I can provide everything, no problem... I'm doing this to reproduce it:
>>>>>
>>>>>
>>>>> root at controller-1:~# keystone-manage db_sync
>>>>> 2014-03-08 02:44:46.723 5407 CRITICAL keystone [-] (OperationalError)
>>>>> (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX
>>>>> ix_token_expires_valid ON token (expires, valid)' ()
>>>>>
>>>>>
>>>>> Log shows:
>>>>>
>>>>> ==> /var/log/keystone/keystone.log <==
>>>>> 2014-03-08 02:44:46.723 5407 CRITICAL keystone [-] (OperationalError)
>>>>> (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX
>>>>> ix_token_expires_valid ON token (expires, valid)' ()
>>>>>
>>>>>
>>>>> root at controller-1:~# dpkg -l | grep keystone
>>>>> iF  keystone                         1:2013.2.2-0ubuntu1~cloud0
>>>>>                    OpenStack identity service - Daemons
>>>>> ii  python-keystone                  1:2013.2.2-0ubuntu1~cloud0
>>>>>                    OpenStack identity service - Python library
>>>>> ii  python-keystoneclient            1:0.3.2-0ubuntu1~cloud0
>>>>>                   Client library for OpenStack Identity API
>>>>>
>>>>> ...
>>>>>
>>>>> If I run "apt-get -f install", to try to fix it ("iF" dpkg status
>>>>> above), same problem, look:
>>>>>
>>>>> ---
>>>>> root at controller-1:~# apt-get -f install
>>>>> Reading package lists... Done
>>>>> Building dependency tree
>>>>> Reading state information... Done
>>>>> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>>>>> 1 not fully installed or removed.
>>>>> After this operation, 0 B of additional disk space will be used.
>>>>> Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...
>>>>> 2014-03-08 02:49:11.050 5469 CRITICAL keystone [-] (OperationalError)
>>>>> (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX
>>>>> ix_token_expires_valid ON token (expires, valid)' ()
>>>>>  dpkg: error processing keystone (--configure):
>>>>>  subprocess installed post-installation script returned error exit
>>>>> status 1
>>>>> Errors were encountered while processing:
>>>>>  keystone
>>>>> E: Sub-process /usr/bin/dpkg returned an error code (1)
>>>>> ---
>>>>>
>>>>> I'm lost...    :-/
>>>>>
>>>>> Maybe the upgrade procedure got stucked, then the second time I tried,
>>>>> it triggered this error... I'm not sure...
>>>>>
>>>>> Tks!
>>>>> Thiago
>>>>>
>>>>>
>>>>>
>>>>> On 8 March 2014 01:55, Lance D Bragstad <ldbragst at us.ibm.com> wrote:
>>>>>
>>>>>> Thiago,
>>>>>>
>>>>>> Are you able to provide a little more information? Log files maybe?
>>>>>> What are the steps you're taking to recreate the issue?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best Regards,
>>>>>>
>>>>>> Lance Bragstad
>>>>>> ldbragst at us.ibm.com
>>>>>>
>>>>>> Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote on 03/07/2014
>>>>>> 08:34:11 PM:
>>>>>>
>>>>>>
>>>>>> > From: Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>>>>>
>>>>>> > To: Lance D Bragstad/Rochester/IBM at IBMUS,
>>>>>> > Cc: "openstack at lists.openstack.org" <openstack at lists.openstack.org>
>>>>>> > Date: 03/07/2014 08:34 PM
>>>>>> > Subject: Re: [Openstack] Error on upgrading to latest Keystone -
>>>>>> > Ubuntu 12.04.4 + UCA
>>>>>> >
>>>>>> > Thanks Lance!
>>>>>>
>>>>>> >
>>>>>> > I just did: "keystone-manage db_sync" and the output is the same:
>>>>>> >
>>>>>> > ---
>>>>>> > 2014-03-07 23:23:48.564 2258 CRITICAL keystone [-]
>>>>>> > (OperationalError) (1061, "Duplicate key name
>>>>>> > 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid ON
>>>>>> > token (expires, valid)' ()
>>>>>> > ---
>>>>>> >
>>>>>> > I just read [1] and [2] but, not a clue about how to fix it.
>>>>>> >
>>>>>> > My Keystone is still running and I have no idea if I can restart it
>>>>>> > or not... What if it doesn't come back?! My entire Cloud will become
>>>>>> > useless... This is a production environment, it was working smoothly
>>>>>> > for months...    :'(
>>>>>> >
>>>>>> > Do you know if there is something that I can do directly on MySQL to
>>>>>> > try to fix it?!
>>>>>> >
>>>>>> > Best,
>>>>>> > Thiago
>>>>>> >
>>>>>>
>>>>>> > On 7 March 2014 23:18, Lance D Bragstad <ldbragst at us.ibm.com>
>>>>>> wrote:
>>>>>> > Thiago,
>>>>>> >
>>>>>> > Your database might be out of sync with the latest Keystone code if
>>>>>> > you just performed an update. Try migrating your database [1]. The
>>>>>> > link gives some information on how to do that, but you'll
>>>>>> > essentially want to do a `keystone-manage db_sync`, also check the
>>>>>> > man pages [2]. This should ensure your Keystone database has the
>>>>>> > proper schema for the Keystone code you're running.
>>>>>> >
>>>>>> > [1] http://docs.openstack.org/developer/keystone/
>>>>>> > configuration.html#step-2-sync-your-new-empty-database
>>>>>> > [2]
>>>>>> http://docs.openstack.org/developer/keystone/man/keystone-manage.html
>>>>>> >
>>>>>> > Best Regards,
>>>>>> >
>>>>>> > Lance Bragstad
>>>>>> > ldbragst at us.ibm.com
>>>>>> >
>>>>>> > Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote on 03/07/2014
>>>>>> > 06:50:01 PM:
>>>>>> >
>>>>>> > > From: Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>>>>>> > > To: "openstack at lists.openstack.org" <
>>>>>> openstack at lists.openstack.org>,
>>>>>> > > Date: 03/07/2014 07:03 PM
>>>>>> > > Subject: [Openstack] Error on upgrading to latest Keystone -
>>>>>> Ubuntu
>>>>>> > > 12.04.4 + UCA
>>>>>> >
>>>>>> > >
>>>>>> > > Guys,
>>>>>> > >
>>>>>> > > I just did:
>>>>>> > >
>>>>>> > > apt-get update
>>>>>> > > apt-get dist-upgrade
>>>>>> > >
>>>>>> > > And I'm seeing the following error:
>>>>>> > >
>>>>>> > > ---
>>>>>> > > Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...
>>>>>> > > 2014-03-07 21:45:39.473 15540 CRITICAL keystone [-]
>>>>>> > > (OperationalError) (1061, "Duplicate key name
>>>>>> > > 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid
>>>>>> ON
>>>>>> > > token (expires, valid)' ()
>>>>>> > > dpkg: error processing keystone (--configure):
>>>>>> > >  subprocess installed post-installation script returned error
>>>>>> exit status 1
>>>>>> > > Errors were encountered while processing:
>>>>>> > >  keystone
>>>>>> > > E: Sub-process /usr/bin/dpkg returned an error code (1)
>>>>>> > > ---
>>>>>> > >
>>>>>> > > What can I do?!
>>>>>> > >
>>>>>> > > Tks!
>>>>>> > > Thiago_______________________________________________
>>>>>> > > Mailing list:
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>> > > Post to     : openstack at lists.openstack.org
>>>>>> > > Unsubscribe :
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140308/fb9a2680/attachment.html>