<div dir="ltr">Guys,<div><br></div><div>Please, ignore my last message... I'm thinking that I'm facing a different problem, out off this topic.<div><br></div><div>Now, instances can not ping its gateway (Neutron L3 is unreachable). I'll make a different post if necessary.</div>
<div><br></div><div>Sorry about the buzz.</div><div><br></div><div>Tks,</div><div>Thiago</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 8 March 2014 16:53, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Well, I think I just destroyed the entire Cloud Computing environment... Security Groups isn't working, it is impossible to reach the tenant's subnet, they are just closed (i.e. not working), all sites hosted here, are down. Damn... What a mess... :-/
<div class="gmail_extra"><br></div><div class="gmail_extra">Is there any way to try to fix this? I have the dump of keystone db, before dropping the INDEXes...<div><div class="h5"><br><br><div class="gmail_quote">On 8 March 2014 16:12, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Stackers,<div><br></div><div>I'll mark this as fixed, look:</div><div><div><br></div><div>---</div>
<div><div>root@controller-1:~# apt-get -f install</div><div>Reading package lists... Done</div><div>Building dependency tree </div>
<div>Reading state information... Done</div><div>0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.</div><div>1 not fully installed or removed.</div><div>After this operation, 0 B of additional disk space will be used.</div>
<div><b>Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...</b></div></div></div><div>---</div><div><br></div><div>Keystone seems to be upgraded now...</div><div><br></div><div>Thanks!</div><div>Thiago</div></div><div>
<div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 8 March 2014 16:06, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Well,<div><br></div><div>I just did a shot in the dark:</div><div><br></div><div><br></div><div>> ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires_valid`;<br></div><div><br></div><div>And "keystone-manage db_sync" seems to be running again... But, I have no idea if this DROP I did will break my OpenStack, or not...</div>
<div><br></div><div>I don't want to waste more time from you guys with this but, maybe it is not a big deal... What do you guys think?!</div><div><br></div><div>Thanks!</div><div>Thiago</div></div><div>
<div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 8 March 2014 15:52, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi!<div><br></div><div>I'm seeing some progress here, look:</div><div><br></div><div>---</div><div><div>root@controller-1:~# pt-duplicate-key-checker --host localhost --databases=keystone --ask-pass</div>
<div>Enter password: </div><div># ########################################################################</div><div># keystone.token </div><div># ########################################################################</div>
<div><br></div><div># ix_token_expires is a left-prefix of ix_token_expires_valid</div><div># Key definitions:</div><div># KEY `ix_token_expires` (`expires`),</div><div># KEY `ix_token_expires_valid` (`expires`,`valid`)</div>
<div># Column types:</div><div># `expires` datetime default null</div><div># `valid` tinyint(1) not null</div><div># To remove this duplicate index, execute:</div><div>ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires`;</div>
<div><br></div><div># ########################################################################</div><div># Summary of indexes </div><div># ########################################################################</div>
<div><br></div><div># Size Duplicate Indexes 1321470</div><div># Total Duplicate Indexes 1</div><div># Total Indexes 42</div></div><div>----</div><div><br></div><div>---</div><div><div>mysql> ALTER TABLE `keystone`.`token` DROP INDEX `ix_token_expires`;</div>
<div>Query OK, 0 rows affected (1.03 sec)</div><div>Records: 0 Duplicates: 0 Warnings: 0</div></div><div>---</div><div><br></div><div>---</div><div>No more duplicate key found by "pt-duplicate-key-checker" but, same problem with "keystone-manage "db_sync", look:</div>
<div><br></div><div><div>root@controller-1:~# keystone-manage db_sync</div><div>2014-03-08 15:46:59.186 8043 CRITICAL keystone [-] (OperationalError) (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid ON token (expires, valid)' ()</div>
</div><div>---</div><div><br></div><div>Guys, what can I do!? I really need to keep this OpenStack installation intact, I can not afford losing it...</div><div><br></div><div>Tks!</div><div>Thiago</div></div><div>
<div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 8 March 2014 04:39, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>Lance,</div><div><br></div>I think I know what had happened... The Keystone upgrade process died in the middle of the upgrade.<div><br></div><div>Now, the Keystone SQL database is stucked at some point "in the middle" of both versions...</div>
<div><br></div><div>My "keystone" MySQL db dump is huge, more than >1G...</div><div><br></div><div>Any tips about how to fix this?! <br></div><div><br></div><div>I really appreciate any help!</div><div><br></div>
<div>Tks!</div><div>Thiago</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On 8 March 2014 02:55, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Lance,<div><br></div><div>I can provide everything, no problem... I'm doing this to reproduce it:</div>
<div><br></div><div><div><br></div><div>root@controller-1:~# keystone-manage db_sync</div><div>2014-03-08 02:44:46.723 5407 CRITICAL keystone [-] (OperationalError) (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid ON token (expires, valid)' ()</div>
</div><div><br></div><div><br></div><div>Log shows:<br></div><div><br></div><div><div>==> /var/log/keystone/keystone.log <==</div><div>2014-03-08 02:44:46.723 5407 CRITICAL keystone [-] (OperationalError) (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid ON token (expires, valid)' ()</div>
</div><div><br></div><div><br></div><div><div>root@controller-1:~# dpkg -l | grep keystone</div><div>iF keystone 1:2013.2.2-0ubuntu1~cloud0 OpenStack identity service - Daemons</div>
<div>ii python-keystone 1:2013.2.2-0ubuntu1~cloud0 OpenStack identity service - Python library</div><div>ii python-keystoneclient 1:0.3.2-0ubuntu1~cloud0 Client library for OpenStack Identity API</div>
</div><div><br></div><div>...</div><div><br></div><div>If I run "apt-get -f install", to try to fix it ("iF" dpkg status above), same problem, look:</div><div><br></div><div>---</div><div><div>root@controller-1:~# apt-get -f install</div>
<div>Reading package lists... Done</div><div>Building dependency tree </div><div>Reading state information... Done</div><div>0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.</div><div>1 not fully installed or removed.</div>
<div>After this operation, 0 B of additional disk space will be used.</div><div><div>Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...</div></div><div>2014-03-08 02:49:11.050 5469 CRITICAL keystone [-] (OperationalError) (1061, "Duplicate key name 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid ON token (expires, valid)' ()</div>
<div>
<div>dpkg: error processing keystone (--configure):</div><div> subprocess installed post-installation script returned error exit status 1</div><div>Errors were encountered while processing:</div><div> keystone</div><div>
E: Sub-process /usr/bin/dpkg returned an error code (1)</div>
</div></div><div>---</div><div><br></div><div>I'm lost... :-/</div><div><br></div><div>Maybe the upgrade procedure got stucked, then the second time I tried, it triggered this error... I'm not sure...</div><div>
<br></div>
<div>Tks!</div><div>Thiago</div><div><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On 8 March 2014 01:55, Lance D Bragstad <span dir="ltr"><<a href="mailto:ldbragst@us.ibm.com" target="_blank">ldbragst@us.ibm.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<p><font face="sans-serif">Thiago, </font><br>
<br>
<font face="sans-serif">Are you able to provide a little more information? Log files maybe? What are the steps you're taking to recreate the issue?</font></p><div><br>
<font face="sans-serif"><br>
<br>
Best Regards, <br>
<br>
Lance Bragstad <br>
<a href="mailto:ldbragst@us.ibm.com" target="_blank">ldbragst@us.ibm.com</a></font><br>
<br>
</div><tt><font>Martinx - ジェームズ <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>> wrote on 03/07/2014 08:34:11 PM:<div><br>
<br>
> From: Martinx - ジェームズ <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></div></font></tt><br>
<tt><font>> To: Lance D Bragstad/Rochester/IBM@IBMUS, </font></tt><br>
<tt><font>> Cc: "<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>" <<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>></font></tt><br>
<tt><font>> Date: 03/07/2014 08:34 PM</font></tt><br>
<tt><font>> Subject: Re: [Openstack] Error on upgrading to latest Keystone - <br>
> Ubuntu 12.04.4 + UCA</font></tt><br>
<tt><font>> <br>
> Thanks Lance!</font></tt><div><div><br>
<tt><font>> <br>
> I just did: "keystone-manage db_sync" and the output is the same:</font></tt><br>
<tt><font>> <br>
> ---</font></tt><br>
<tt><font>> 2014-03-07 23:23:48.564 2258 CRITICAL keystone [-] <br>
> (OperationalError) (1061, "Duplicate key name <br>
> 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid ON <br>
> token (expires, valid)' ()</font></tt><br>
<tt><font>> ---</font></tt><br>
<tt><font>> <br>
> I just read [1] and [2] but, not a clue about how to fix it.</font></tt><br>
<tt><font>> <br>
> My Keystone is still running and I have no idea if I can restart it <br>
> or not... What if it doesn't come back?! My entire Cloud will become<br>
> useless... This is a production environment, it was working smoothly<br>
> for months... :'(</font></tt><br>
<tt><font>> <br>
> Do you know if there is something that I can do directly on MySQL to<br>
> try to fix it?!</font></tt><br>
<tt><font>> <br>
> Best,</font></tt><br>
<tt><font>> Thiago</font></tt><br>
<tt><font>> <br>
</font></tt><br>
<tt><font>> On 7 March 2014 23:18, Lance D Bragstad <<a href="mailto:ldbragst@us.ibm.com" target="_blank">ldbragst@us.ibm.com</a>> wrote:</font></tt><br>
<tt><font>> Thiago, <br>
> <br>
> Your database might be out of sync with the latest Keystone code if <br>
> you just performed an update. Try migrating your database [1]. The <br>
> link gives some information on how to do that, but you'll <br>
> essentially want to do a `keystone-manage db_sync`, also check the <br>
> man pages [2]. This should ensure your Keystone database has the <br>
> proper schema for the Keystone code you're running.<br>
> <br>
> [1] <a href="http://docs.openstack.org/developer/keystone/" target="_blank">http://docs.openstack.org/developer/keystone/</a><br>
> configuration.html#step-2-sync-your-new-empty-database<br>
> [2] <a href="http://docs.openstack.org/developer/keystone/man/keystone-manage.html" target="_blank">http://docs.openstack.org/developer/keystone/man/keystone-manage.html</a><br>
> <br>
> Best Regards, <br>
> <br>
> Lance Bragstad <br>
> <a href="mailto:ldbragst@us.ibm.com" target="_blank">ldbragst@us.ibm.com</a><br>
> <br>
> Martinx - ジェームズ <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>> wrote on 03/07/2014 <br>
> 06:50:01 PM:<br>
> <br>
> > From: Martinx - ジェームズ <<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>><br>
> > To: "<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>" <<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>>, <br>
> > Date: 03/07/2014 07:03 PM<br>
> > Subject: [Openstack] Error on upgrading to latest Keystone - Ubuntu <br>
> > 12.04.4 + UCA</font></tt><br>
<tt><font>> <br>
> > <br>
> > Guys,<br>
> > <br>
> > I just did:<br>
> > <br>
> > apt-get update<br>
> > apt-get dist-upgrade<br>
> > <br>
> > And I'm seeing the following error:<br>
> > <br>
> > ---<br>
> > Setting up keystone (1:2013.2.2-0ubuntu1~cloud0) ...<br>
> > 2014-03-07 21:45:39.473 15540 CRITICAL keystone [-] <br>
> > (OperationalError) (1061, "Duplicate key name <br>
> > 'ix_token_expires_valid'") 'CREATE INDEX ix_token_expires_valid ON <br>
> > token (expires, valid)' ()<br>
> > dpkg: error processing keystone (--configure):<br>
> > subprocess installed post-installation script returned error exit status 1<br>
> > Errors were encountered while processing:<br>
> > keystone<br>
> > E: Sub-process /usr/bin/dpkg returned an error code (1)<br>
> > ---<br>
> > <br>
> > What can I do?!<br>
> > <br>
> > Tks!</font></tt><br>
<tt><font>> > Thiago_______________________________________________<br>
> > Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> > Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
> > Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></font></tt></div></div><p></p></div></blockquote>
</div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>