Open Stack

If you can't isolate the exact thing you need to get cleaned up here it can
be difficult to unwind. You'll either need to read the code to see what's
triggering the db setup (which is probably the package installs) or start
on a clean box. I'd recommend the latter.

On Thu, Feb 4, 2016 at 10:35 AM, Ptacek, MichalX <michalx.ptacek at intel.com>
wrote:

> Hi Emilien,
>
>
>
> It seems that keystone database is not populated, because of something,
> which happened on previous runs (e.g. some packages installation),
>
>
>
> Following rows are visible just in log from first attempt
>
> Debug: Executing '/usr/bin/mysql -e CREATE USER 'keystone'@'127.0.0.1'
> IDENTIFIED BY PASSWORD '*936E8F7AB2E21B47F6C9A7E5D9FE14DBA2255E5A''
>
> Debug: Executing '/usr/bin/mysql -e GRANT USAGE ON *.* TO 'keystone'@'127.0.0.1'
> WITH MAX_USER_CONNECTIONS 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_QUERIES_PER_HOUR
> 0 MAX_UPDATES_PER_HOUR 0'
>
> ….
>
> ….
>
> I tried to clean databases & uninstall packages installed during
> deployment, but maybe I miss something as it simply doesn’t  work J
>
>
>
> Is there any procedure, how can I restore system to “vanilla state” before
> puppet modules installation ?
>
> It looks to me that when deployment failed, it’s very difficult to
> “unstack” it
>
>
>
> Thanks in advance,
>
> Michal
>
>
>
> *From:* Ptacek, MichalX
> *Sent:* Thursday, February 04, 2016 11:14 AM
> *To:* OpenStack Development Mailing List (not for usage questions) <
> openstack-dev at lists.openstack.org>
> *Subject:* RE: [openstack-dev] [puppet] compatibility of puppet upstream
> modules
>
>
>
>
>
>
>
> -----Original Message-----
> From: Emilien Macchi [mailto:emilien at redhat.com <emilien at redhat.com>]
> Sent: Thursday, February 04, 2016 10:06 AM
> To: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [puppet] compatibility of puppet upstream
> modules
>
>
>
>
>
>
>
> On 02/03/2016 04:03 PM, Ptacek, MichalX wrote:
>
> > Hi all,
>
> >
>
> >
>
> >
>
> > I have one general question,
>
> >
>
> > currently I am deploying liberty openstack as described in
>
> > https://wiki.openstack.org/wiki/Puppet/Deploy
>
> >
>
> > Unfortunately puppet modules specified in
>
> > puppet-openstack-integration/Puppetfile are not compatible
>
>
>
> Did you take the file from stable/liberty branch?
>
>
> https://github.com/openstack/puppet-openstack-integration/tree/stable/liberty
>
>
>
> *[Michal Ptacek]*  I am deploying scenario003 with stable/liberty
>
> >
>
> > and some are also missing as visible from following output of “puppet
>
> > module list”
>
> >
>
> >
>
> >
>
> > Warning: Setting templatedir is deprecated. See
>
> > http://links.puppetlabs.com/env-settings-deprecations
>
> >
>
> >    (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in
>
> > `issue_deprecation_warning')
>
> >
>
> > Warning: Module 'openstack-openstacklib' (v7.0.0) fails to meet some
>
> > dependencies:
>
> >
>
> >   'openstack-barbican' (v0.0.1) requires 'openstack-openstacklib'
>
> > (>=6.0.0 <7.0.0)
>
> >
>
> >   'openstack-zaqar' (v0.0.1) requires 'openstack-openstacklib'
>
> > (>=6.0.0
>
> > <7.0.0)
>
> >
>
> > Warning: Module 'puppetlabs-postgresql' (v4.4.2) fails to meet some
>
> > dependencies:
>
> >
>
> >   'openstack-openstacklib' (v7.0.0) requires 'puppetlabs-postgresql'
>
> > (>=3.3.0 <4.0.0)
>
> >
>
> > Warning: Missing dependency 'deric-storm':
>
> >
>
> >   'openstack-monasca' (v1.0.0) requires 'deric-storm' (>=0.0.1 <1.0.0)
>
> >
>
> > Warning: Missing dependency 'deric-zookeeper':
>
> >
>
> >   'openstack-monasca' (v1.0.0) requires 'deric-zookeeper' (>=0.0.1
>
> > <1.0.0)
>
> >
>
> > Warning: Missing dependency 'dprince-qpid':
>
> >
>
> >   'openstack-cinder' (v7.0.0) requires 'dprince-qpid' (>=1.0.0 <2.0.0)
>
> >
>
> >   'openstack-manila' (v7.0.0) requires 'dprince-qpid' (>=1.0.0 <2.0.0)
>
> >
>
> >   'openstack-nova' (v7.0.0) requires 'dprince-qpid' (>=1.0.0 <2.0.0)
>
> >
>
> > Warning: Missing dependency 'jdowning-influxdb':
>
> >
>
> >   'openstack-monasca' (v1.0.0) requires 'jdowning-influxdb' (>=0.3.0
>
> > <1.0.0)
>
> >
>
> > Warning: Missing dependency 'opentable-kafka':
>
> >
>
> >   'openstack-monasca' (v1.0.0) requires 'opentable-kafka' (>=1.0.0
>
> > <2.0.0)
>
> >
>
> > Warning: Missing dependency 'puppetlabs-stdlib':
>
> >
>
> >   'antonlindstrom-powerdns' (v0.0.5) requires 'puppetlabs-stdlib' (>=
>
> > 0.0.0)
>
> >
>
> > Warning: Missing dependency 'puppetlabs-corosync':
>
> >
>
> >   'openstack-openstack_extras' (v7.0.0) requires 'puppetlabs-corosync'
>
> > (>=0.1.0 <1.0.0)
>
> >
>
> > /etc/puppet/modules
>
> >
>
> > ├──antonlindstrom-powerdns (v0.0.5)
>
> >
>
> > ├──duritong-sysctl (v0.0.11)
>
> >
>
> > ├──nanliu-staging (v1.0.4)
>
> >
>
> > ├──openstack-barbican (v0.0.1)
>
> >
>
> > ├──openstack-ceilometer (v7.0.0)
>
> >
>
> > ├──openstack-cinder (v7.0.0)
>
> >
>
> > ├──openstack-designate (v7.0.0)
>
> >
>
> > ├──openstack-glance (v7.0.0)
>
> >
>
> > ├──openstack-gnocchi (v7.0.0)
>
> >
>
> > ├──openstack-heat (v7.0.0)
>
> >
>
> > ├──openstack-horizon (v7.0.0)
>
> >
>
> > ├──openstack-ironic (v7.0.0)
>
> >
>
> > ├──openstack-keystone (v7.0.0)
>
> >
>
> > ├──openstack-manila (v7.0.0)
>
> >
>
> > ├──openstack-mistral (v0.0.1)
>
> >
>
> > ├──openstack-monasca (v1.0.0)
>
> >
>
> > ├──openstack-murano (v7.0.0)
>
> >
>
> > ├──openstack-neutron (v7.0.0)
>
> >
>
> > ├──openstack-nova (v7.0.0)
>
> >
>
> > ├──openstack-openstack_extras (v7.0.0)
>
> >
>
> > ├──openstack-openstacklib (v7.0.0)  invalid
>
> >
>
> > ├──openstack-sahara (v7.0.0)
>
> >
>
> > ├──openstack-swift (v7.0.0)
>
> >
>
> > ├──openstack-tempest (v7.0.0)
>
> >
>
> > ├──openstack-trove (v7.0.0)
>
> >
>
> > ├──openstack-tuskar (v7.0.0)
>
> >
>
> > ├──openstack-vswitch (v3.0.0)
>
> >
>
> > ├──openstack-zaqar (v0.0.1)
>
> >
>
> > ├──openstack_integration (???)
>
> >
>
> > ├──puppet-aodh (v7.0.0)
>
> >
>
> > ├──puppet-corosync (v0.8.0)
>
> >
>
> > ├──puppetlabs-apache (v1.4.1)
>
> >
>
> > ├──puppetlabs-apt (v2.1.1)
>
> >
>
> > ├──puppetlabs-concat (v1.2.5)
>
> >
>
> > ├──puppetlabs-firewall (v1.6.0)
>
> >
>
> > ├──puppetlabs-inifile (v1.4.3)
>
> >
>
> > ├──puppetlabs-mongodb (v0.11.0)
>
> >
>
> > ├──puppetlabs-mysql (v3.6.2)
>
> >
>
> > ├──puppetlabs-postgresql (v4.4.2)  invalid
>
> >
>
> > ├──puppetlabs-rabbitmq (v5.2.3)
>
> >
>
> > ├──puppetlabs-rsync (v0.4.0)
>
> >
>
> > ├──puppetlabs-stdlib (v4.6.0)
>
> >
>
> > ├──puppetlabs-vcsrepo (v1.3.2)
>
> >
>
> > ├──puppetlabs-xinetd (v1.5.0)
>
> >
>
> > ├──qpid (???)
>
> >
>
> > ├──saz-memcached (v2.8.1)
>
> >
>
> > ├──stankevich-python (v1.8.0)
>
> >
>
> > └── theforeman-dns (v3.0.0)
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > Most of the warning can be probably ignored, e.g I assume that latest
>
> > barbican & zaqar are compatible with liberty (7.0) version of
>
> > openstack-openstacklib
>
> >
>
> >   'openstack-barbican' (v0.0.1) requires 'openstack-openstacklib'
>
> > (>=6.0.0 <7.0.0)
>
> >
>
> >   'openstack-zaqar' (v0.0.1) requires 'openstack-openstacklib'
>
> > (>=6.0.0
>
> > <7.0.0)
>
> >
>
> >
>
> >
>
> > Am I right or I need to get rid of all of these compatibility warnings
>
> > before proceeding further ?
>
> >
>
>
>
> If you look at our CI jobs, we also have some warnings:
>
>
> http://logs.openstack.org/36/275836/1/gate/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/15a5ead/console.html#_2016-02-03_21_56_40_945
>
>
>
> *[Michal Ptacek]* Great, it seems that it’s not that big problem.  I can
> stick on versions from Puppetfile (if it’s working in CI)
>
>
>
> >
>
> > I tried both,  but during subsequent deployments I reached some
>
> > intermediate issue with number of parallel mysql connections
>
> >
>
> >
>
> >
>
> > 2016-02-03 00:01:03.326 90406 DEBUG oslo_db.api [-] Loading backend
>
> > 'sqlalchemy' from 'nova.db.sqlalchemy.api' _load_backend
>
> > /usr/lib/python2.7/dist-packages/oslo_db/api.py:238
>
> >
>
> > 2016-02-03 00:01:03.333 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 10 attempts left.
>
> >
>
> > 2016-02-03 00:01:13.345 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 9 attempts left.
>
> >
>
> > 2016-02-03 00:01:23.358 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 8 attempts left.
>
> >
>
> > 2016-02-03 00:01:33.361 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 7 attempts left.
>
> >
>
> > 2016-02-03 00:01:43.374 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 6 attempts left.
>
> >
>
> > 2016-02-03 00:01:53.387 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 5 attempts left.
>
> >
>
> > 2016-02-03 00:02:03.400 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 4 attempts left.
>
> >
>
> > 2016-02-03 00:02:13.412 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 3 attempts left.
>
> >
>
> > 2016-02-03 00:02:23.415 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 2 attempts left.
>
> >
>
> > 2016-02-03 00:02:33.429 90406 WARNING oslo_db.sqlalchemy.engines [-]
>
> > SQL connection failed. 1 attempts left.
>
> >
>
> > *2016-02-03 00:02:43.440 90406 CRITICAL nova [-] DBConnectionError:
>
> > (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server
>
> > on '127.0.0.1' ((1040, u'Too many connections'))")*
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova Traceback (most recent call
> last):
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File "/usr/bin/nova-manage",
>
> > line 10, in <module>
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     sys.exit(main())
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1445, in
>
> > main
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     ret = fn(*fn_args,
> **fn_kwargs)
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 949, in
>
> > archive_deleted_rows
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     table_to_rows_archived =
>
> > db.archive_deleted_rows(max_rows)
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 1922, in
>
> > archive_deleted_rows
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     return
>
> > IMPL.archive_deleted_rows(max_rows=max_rows)
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line
>
> > 6075, in archive_deleted_rows
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     meta =
>
> > MetaData(get_engine(use_slave=True))
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line
>
> > 162, in get_engine
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     return
>
> > main_context_manager._factory.get_legacy_facade().get_engine(
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py",
>
> > line 256, in get_legacy_facade
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     self._start()
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py",
>
> > line 338, in _start
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     engine_args, maker_args)
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py",
>
> > line 362, in _setup_for_connection
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova
>
> > sql_connection=sql_connection, **engine_kwargs)
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line
>
> > 152, in create_engine
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     test_conn =
>
> > _test_connection(engine, max_retries, retry_interval)
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File
>
> > "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line
>
> > 334, in _test_connection
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova     six.reraise(type(de_ref),
>
> > de_ref)
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova   File "<string>", line 2, in
>
> > reraise
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova DBConnectionError:
>
> > (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server
>
> > on '127.0.0.1' ((1040, u'Too many connections'))")
>
> >
>
> > 2016-02-03 00:02:43.440 90406 ERROR nova
>
> >
>
> > 2016-02-03 12:32:54.682 16003 CRITICAL nova [-] KeyError:
>
> > <VerNum(312)>
>
> >
>
> >
>
>
>
> Sounds like related to MySQL config, not really related to Puppet
> OpenStack modules.
>
>
>
> *[Michal Ptacek]* Thanks, you’re right. I resolved that issue already.
> Now I am getting following error from keystone
>
> /usr/bin/openstack endpoint list --quiet --format csv
>
> Could not find project: openstack (Disable debug mode to suppress these
> details.) (HTTP 401) (Request-ID: req-981ac293-15e0-45c7-81da-63c44bdc28b8)
>
>
>
> Keystone db looks quite empty …
>
> mysql> use keystone;
>
> Reading table information for completion of table and column names
>
> You can turn off this feature to get a quicker startup with -A
>
>
>
> Database changed
>
> mysql> select * from user;
>
> Empty set (0.00 sec)
>
>
>
> mysql> select * from project;
>
> Empty set (0.00 sec)
>
>
>
> From log  it looks that relevant part passed,
>
> Debug: Automatically imported keystone::roles::admin from
> keystone/roles/admin into production
>
> but not sure from where exactly keystone db should be populated:
>
>
>
> Thanks for any comment/suggestion,
>
> Michal
>
>
>
> >
>
> >
>
> > which didn’t disappear even after changing max_connection limit in
>
> > /etc/mysql/my.cnf ….
>
> >
>
> >
>
> >
>
> > mysql> select @@global.max_connections;
>
> >
>
> > +--------------------------+
>
> >
>
> > | @@global.max_connections |
>
> >
>
> > +--------------------------+
>
> >
>
> > |                     1024 |
>
> >
>
> > +--------------------------+
>
> >
>
> > 1 row in set (0.00 sec)
>
> >
>
> >
>
> >
>
> > mysql> use nova;
>
> >
>
> > Reading table information for completion of table and column names
>
> >
>
> > You can turn off this feature to get a quicker startup with -A
>
> >
>
> >
>
> >
>
> > Database changed
>
> >
>
> > mysql>
>
> >
>
> >
>
> >
>
> > Please help me to understand, how people should use this integration
>
> > project properly and how to avoid avoidable issues J
>
> >
>
>
>
> Well, your error does not seem related to Puppet OpenStack modules, but
> we're willing to help if you provide your manifests so we can try to see
> what is wrong in your deployment.
>
>
>
> FWIW, our current CI is running integration jobs on both liberty & mitaka,
> using https://github.com/openstack/puppet-openstack-integration
>
> scripts.
>
>
>
> HTH
>
>
>
> >
>
> > Thanks in advance,
>
> >
>
> > Michal
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > --------------------------------------------------------------
>
> > Intel Research and Development Ireland Limited Registered in Ireland
>
> > Registered Office: Collinstown Industrial Park, Leixlip, County
>
> > Kildare Registered Number: 308263
>
> >
>
> > This e-mail and any attachments may contain confidential material for
>
> > the sole use of the intended recipient(s). Any review or distribution
>
> > by others is strictly prohibited. If you are not the intended
>
> > recipient, please contact the sender and delete all copies.
>
> >
>
> >
>
> >
>
> > ______________________________________________________________________
>
> > ____ OpenStack Development Mailing List (not for usage questions)
>
> > Unsubscribe:
>
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> >
>
>
>
> --
>
> Emilien Macchi
>
>
>
> --------------------------------------------------------------
> Intel Research and Development Ireland Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
>
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by others
> is strictly prohibited. If you are not the intended recipient, please
> contact the sender and delete all copies.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160204/9532bcaa/attachment.html>