[Openstack] Fuel-9.2 - Broken deployment on one node stops all other nodes too

Raja T Nair rtnair at gmail.com
Tue Oct 31 07:27:05 UTC 2017


Hi,

I built the repos locally, using the instructions in this page:
https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-install-guide/local-repo.html

Now, while trying to deploy to 2 newly added nodes, it fails with this
error:

2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure)
E: Unable to locate package fuel-ha-utils
2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure)
Reading state information...
2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure)
Building dependency tree...
2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure)
change from purged to present failed: Execution of '/usr/bin/apt-get -q -y
-o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1
install fuel-ha-utils' returned 100: Reading package lists...
2017-10-31 06:45:11ERR/usr/bin/puppet:8:in `<main>'2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in
`execute'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in
`run'2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in
`exit_on_fail'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in
run'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in
`plugin_hook'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2
levels) in run'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in
`run_command'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in
`main'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in
`override'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in
`block in main'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in
`apply_catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in
`run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in
`override'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in
`block in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in
`run_internal'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in
`apply_catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in
`benchmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in
`block in benchmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in
`block in apply_catalog'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in
`apply'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in
`as_logging_destination'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in
`with_destination'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in
`block in apply'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in
`traverse'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in
evaluate'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in
`thinmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in
`block in thinmark'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2
levels) in evaluate'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in
`eval_resource'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in
`evaluate'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in
`perform_changes'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in
`sync_if_needed'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in
`sync'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in
`sync'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in
`set'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in
`call_valuemethod'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/type/package.rb:73:in `block (3
levels) in <module:Puppet>'
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:72:in
`install'
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:72:in
`each'
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:74:in
`block in install'
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:66:in
`wait_for_lock'
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:75:in
`block (2 levels) in install'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider/package/apt.rb:73:in
`install'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider.rb:463:in
`block in create_class_and_instance_method'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:in
`block in has_command'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider/command.rb:23:in
`execute'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:in
`execute'
2017-10-31 06:45:11ERRE: Unable to locate package fuel-ha-utils
2017-10-31 06:45:11ERRReading state information...
2017-10-31 06:45:11ERRBuilding dependency tree...

I can see fuel-ha-utils deb package in /var/www/ tree:

[root at fuel ~]# cd /var/www/nailgun/
[root at fuel nailgun]# find . -name fuel-ha-utils*
./mitaka-9.0/mos-centos/x86_64/Packages/fuel-ha-utils-9.0.0-1.mos8460.noarch.rpm
./mitaka-9.0/ubuntu/x86_64/pool/main/f/fuel-library9.0/fuel-ha-utils_9.0.0-1~u14.04+mos8460_all.deb
[root at fuel nailgun]#

On a node which was installed before building local repo, I can see the
package fuel-ha-utils installed, but a different version i think:
root at node-4:~# dpkg -l fuel-ha-utils
Desired=Unknown/Install/Remove/Purge/Hold
|
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version
Architecture         Description
+++-==============================-====================-====================-==================================================================
ii  fuel-ha-utils                  9.0.0-1~u14.04+mos87
all                  Fuel Library HA utils
root at node-4:~#

Is there a way to fix this problem? Or should I just remove all nodes and
start the deployment from scratch - since local repo was built after
deploying 5 nodes?

Please advise.

Best Regards,
Raja.

On 30 October 2017 at 19:27, Evgeniy L <eli at mirantis.com> wrote:

> Hi,
>
> fuel_pkgs/9 means that "fuel_pkgs" task failed on node with id 9, find the
> node with this id using "fuel node" (run it from Fuel node), ssh to the
> nodes and look for errors in /var/log/puppet.log.
>
> If Nova service was running before, it should be running even after
> fuel_pkgs task failed on one of the nodes, see nova logs, to find what
> happened to nova.
>
> Thanks,
>
> On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair <rtnair at gmail.com> wrote:
>
>> Hello All,
>>
>> Trying to add one more node to the group, and it breaks with error:
>> Failed tasks: Task[fuel_pkgs/9] Stopping the deployment process!
>>
>> But now the entire cluster is stopped - nova service on all nodes are
>> down.
>> Is this expected behaviour? How do I get back my cluster active?
>>
>> Please help me to resolve this problem.
>>
>> Regards,
>> Raja.
>>
>>
>> --
>> :^)
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>


-- 
:^)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171031/fb5c9dd3/attachment.html>


More information about the Openstack mailing list