<div dir="ltr"><div><div><div><div><div>Hi,<br><br></div>I built the repos locally, using the instructions in this page:</div><div><a href="https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-install-guide/local-repo.html">https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-install-guide/local-repo.html</a><br></div><div><br></div>Now, while trying to deploy to 2 newly added nodes, it fails with this error:<br><br>2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure) E: Unable to locate package fuel-ha-utils<br>2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure) Reading state information...<br>2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure) Building dependency tree...<br>2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuel_pkgs::Fuel_pkgs/Package[fuel-ha-utils]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install fuel-ha-utils' returned 100: Reading package lists...</div><div>2017-10-31 06:45:11ERR/usr/bin/puppet:8:in `<main>'2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/type/package.rb:73:in `block (3 levels) in <module:Puppet>'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:72:in `install'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:72:in `each'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:74:in `block in install'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:66:in `wait_for_lock'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:75:in `block (2 levels) in install'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider/package/apt.rb:73:in `install'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider.rb:463:in `block in create_class_and_instance_method'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:in `block in has_command'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider/command.rb:23:in `execute'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:in `execute'</div><div>2017-10-31 06:45:11ERRE: Unable to locate package fuel-ha-utils</div><div>2017-10-31 06:45:11ERRReading state information...</div><div>2017-10-31 06:45:11ERRBuilding dependency tree...</div><div></div><div><br></div>I can see fuel-ha-utils deb package in /var/www/ tree:<br><br>[root@fuel ~]# cd /var/www/nailgun/<br>[root@fuel nailgun]# find . -name fuel-ha-utils*<br>./mitaka-9.0/mos-centos/x86_64/Packages/fuel-ha-utils-9.0.0-1.mos8460.noarch.rpm<br>./mitaka-9.0/ubuntu/x86_64/pool/main/f/fuel-library9.0/fuel-ha-utils_9.0.0-1~u14.04+mos8460_all.deb<br>[root@fuel nailgun]# <br><br></div>On a node which was installed before building local repo, I can see the package fuel-ha-utils installed, but a different version i think:<br>root@node-4:~# dpkg -l fuel-ha-utils<br>Desired=Unknown/Install/Remove/Purge/Hold<br>| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend<br>|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)<br>||/ Name                           Version              Architecture         Description<br>+++-==============================-====================-====================-==================================================================<br>ii  fuel-ha-utils                  9.0.0-1~u14.04+mos87 all                  Fuel Library HA utils<br>root@node-4:~#<br></div><div><br></div><div>Is there a way to fix this problem? Or should I just remove all nodes and start the deployment from scratch - since local repo was built after deploying 5 nodes?</div><div><br></div><div>Please advise.</div><div><br></div><div>Best Regards,</div><div>Raja.<br><div><div><div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 30 October 2017 at 19:27, Evgeniy L <span dir="ltr"><<a href="mailto:eli@mirantis.com" target="_blank">eli@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div><span style="font-size:12.8px">fuel_pkgs/9 means that "fuel_pkgs" task failed on node with id 9, find the node with this id using "fuel node" (run it from Fuel node), ssh to the nodes and look for errors in /var/log/puppet.log.</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">If Nova service was running before, it should be running even after fuel_pkgs task failed on one of the nodes, see nova logs, to find what happened to nova.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks,</span></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-h5">On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair <span dir="ltr"><<a href="mailto:rtnair@gmail.com" target="_blank">rtnair@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5"><div dir="ltr"><div><div><div><div><div>Hello All,<br><br></div>Trying to add one more node to the group, and it breaks with error:<br><span>Failed tasks: Task[fuel_pkgs/9] </span><span><span>Stopping the deployment process!</span></span></div><span><br>But now the entire cluster is stopped - nova service on all nodes are down.<br></span></div><span>Is this expected behaviour? How do I get back my cluster active?<br></span></div><div><span><br></span></div><div><span>Please help me to resolve this problem.<br></span></div><div><span><br></span></div><span>Regards,<br></span></div><span>Raja.<span class="gmail-m_-8596960065969626969HOEnZb"><font color="#888888"><br></font></span></span><span class="gmail-m_-8596960065969626969HOEnZb"><font color="#888888"><div><div><div><div><span><br></span><div><div><br>-- <br><div class="gmail-m_-8596960065969626969m_-6108219991736169004gmail_signature">:^)</div>
</div></div></div></div></div></div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to     : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">:^)</div>
</div></div>