<div dir="ltr">Run "apt-cache policy fuel-ha-utils" and see from which repository does it come from, if your mirrors configured correctly, the package should come from your mirror.<div>It also shows what command it failed to run, "<span style="font-size:12.8px">/usr/bin/apt-get -q -y -o DPkg::Options::=--force-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">confold -o APT::Get::</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">AllowUnauthenticated=1 install fuel-ha-utils" you can run it manually on the node and see what the error is.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks,</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 31, 2017 at 12:27 AM, Raja T Nair <span dir="ltr"><<a href="mailto:rtnair@gmail.com" target="_blank">rtnair@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi,<br><br></div>I built the repos locally, using the instructions in this page:</div><div><a href="https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-install-guide/local-repo.html" target="_blank">https://docs.openstack.org/<wbr>fuel-docs/latest/userdocs/<wbr>fuel-install-guide/local-repo.<wbr>html</a><br></div><div><br></div>Now, while trying to deploy to 2 newly added nodes, it fails with this error:<br><br>2017-10-31 06:45:11ERR(/Stage[main]/<wbr>Osnailyfacter::Fuel_pkgs::<wbr>Fuel_pkgs/Package[fuel-ha-<wbr>utils]/ensure) E: Unable to locate package fuel-ha-utils<br>2017-10-31 06:45:11ERR(/Stage[main]/<wbr>Osnailyfacter::Fuel_pkgs::<wbr>Fuel_pkgs/Package[fuel-ha-<wbr>utils]/ensure) Reading state information...<br>2017-10-31 06:45:11ERR(/Stage[main]/<wbr>Osnailyfacter::Fuel_pkgs::<wbr>Fuel_pkgs/Package[fuel-ha-<wbr>utils]/ensure) Building dependency tree...<br>2017-10-31 06:45:11ERR(/Stage[main]/<wbr>Osnailyfacter::Fuel_pkgs::<wbr>Fuel_pkgs/Package[fuel-ha-<wbr>utils]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-<wbr>confold -o APT::Get::<wbr>AllowUnauthenticated=1 install fuel-ha-utils' returned 100: Reading package lists...</div><div>2017-10-31 06:45:11ERR/usr/bin/puppet:8:<wbr>in `<main>'2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util/<wbr>command_line.rb:92:in `execute'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util/<wbr>command_line.rb:146:in `run'2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application.rb:381:in `run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util.rb:<wbr>496:in `exit_on_fail'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application.rb:381:in `block in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application.rb:507:in `plugin_hook'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application.rb:381:in `block (2 levels) in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application/apply.rb:159:in `run_command'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application/apply.rb:198:in `main'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet.rb:246:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/context.rb:<wbr>64:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application/apply.rb:236:in `block in main'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>application/apply.rb:302:in `apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/configurer.<wbr>rb:133:in `run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet.rb:246:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/context.rb:<wbr>64:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/configurer.<wbr>rb:134:in `block in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/configurer.<wbr>rb:227:in `run_internal'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/configurer.<wbr>rb:119:in `apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util.rb:<wbr>160:in `benchmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.<wbr>1/benchmark.rb:295:in `realtime'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util.rb:<wbr>161:in `block in benchmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/configurer.<wbr>rb:120:in `block in apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/resource/<wbr>catalog.rb:168:in `apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction/report.rb:112:in `as_logging_destination'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util/log.<wbr>rb:149:in `with_destination'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/resource/<wbr>catalog.rb:169:in `block in apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction.rb:138:in `evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/graph/<wbr>relationship_graph.rb:118:in `traverse'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction.rb:147:in `block in evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util.rb:<wbr>334:in `thinmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.<wbr>1/benchmark.rb:295:in `realtime'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util.rb:<wbr>335:in `block in thinmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction.rb:147:in `block (2 levels) in evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction.rb:147:in `call'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction.rb:217:in `eval_resource'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction.rb:204:in `apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction/resource_harness.<wbr>rb:20:in `evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction/resource_harness.<wbr>rb:81:in `perform_changes'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction/resource_harness.<wbr>rb:128:in `sync_if_needed'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/<wbr>transaction/resource_harness.<wbr>rb:204:in `sync'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/property.<wbr>rb:581:in `sync'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/property.<wbr>rb:498:in `set'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/property.<wbr>rb:197:in `call_valuemethod'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/type/<wbr>package.rb:73:in `block (3 levels) in <module:Puppet>'</div><div>2017-10-31 06:45:11ERR/etc/puppet/<wbr>modules/osnailyfacter/lib/<wbr>puppet/provider/package/apt_<wbr>fuel.rb:72:in `install'</div><div>2017-10-31 06:45:11ERR/etc/puppet/<wbr>modules/osnailyfacter/lib/<wbr>puppet/provider/package/apt_<wbr>fuel.rb:72:in `each'</div><div>2017-10-31 06:45:11ERR/etc/puppet/<wbr>modules/osnailyfacter/lib/<wbr>puppet/provider/package/apt_<wbr>fuel.rb:74:in `block in install'</div><div>2017-10-31 06:45:11ERR/etc/puppet/<wbr>modules/osnailyfacter/lib/<wbr>puppet/provider/package/apt_<wbr>fuel.rb:66:in `wait_for_lock'</div><div>2017-10-31 06:45:11ERR/etc/puppet/<wbr>modules/osnailyfacter/lib/<wbr>puppet/provider/package/apt_<wbr>fuel.rb:75:in `block (2 levels) in install'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/provider/<wbr>package/apt.rb:73:in `install'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/provider.<wbr>rb:463:in `block in create_class_and_instance_<wbr>method'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/provider.<wbr>rb:237:in `block in has_command'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/provider/<wbr>command.rb:23:in `execute'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/<wbr>vendor_ruby/puppet/util/<wbr>execution.rb:219:in `execute'</div><div>2017-10-31 06:45:11ERRE: Unable to locate package fuel-ha-utils</div><div>2017-10-31 06:45:11ERRReading state information...</div><div>2017-10-31 06:45:11ERRBuilding dependency tree...</div><div></div><div><br></div>I can see fuel-ha-utils deb package in /var/www/ tree:<br><br>[root@fuel ~]# cd /var/www/nailgun/<br>[root@fuel nailgun]# find . -name fuel-ha-utils*<br>./mitaka-9.0/mos-centos/x86_<wbr>64/Packages/fuel-ha-utils-9.0.<wbr>0-1.mos8460.noarch.rpm<br>./mitaka-9.0/ubuntu/x86_64/<wbr>pool/main/f/fuel-library9.0/<wbr>fuel-ha-utils_9.0.0-1~u14.04+<wbr>mos8460_all.deb<br>[root@fuel nailgun]# <br><br></div>On a node which was installed before building local repo, I can see the package fuel-ha-utils installed, but a different version i think:<br>root@node-4:~# dpkg -l fuel-ha-utils<br>Desired=Unknown/Install/<wbr>Remove/Purge/Hold<br>| Status=Not/Inst/Conf-files/<wbr>Unpacked/halF-conf/Half-inst/<wbr>trig-aWait/Trig-pend<br>|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)<br>||/ Name Version Architecture Description<br>+++-==========================<wbr>====-====================-====<wbr>================-=============<wbr>==============================<wbr>=======================<br>ii fuel-ha-utils 9.0.0-1~u14.04+mos87 all Fuel Library HA utils<br>root@node-4:~#<br></div><div><br></div><div>Is there a way to fix this problem? Or should I just remove all nodes and start the deployment from scratch - since local repo was built after deploying 5 nodes?</div><div><br></div><div>Please advise.</div><div><br></div><div>Best Regards,</div><div>Raja.<br><div><div><div></div></div></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On 30 October 2017 at 19:27, Evgeniy L <span dir="ltr"><<a href="mailto:eli@mirantis.com" target="_blank">eli@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div><span style="font-size:12.8px">fuel_pkgs/9 means that "fuel_pkgs" task failed on node with id 9, find the node with this id using "fuel node" (run it from Fuel node), ssh to the nodes and look for errors in /var/log/puppet.log.</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">If Nova service was running before, it should be running even after fuel_pkgs task failed on one of the nodes, see nova logs, to find what happened to nova.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks,</span></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_654365184472788751gmail-h5">On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair <span dir="ltr"><<a href="mailto:rtnair@gmail.com" target="_blank">rtnair@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_654365184472788751gmail-h5"><div dir="ltr"><div><div><div><div><div>Hello All,<br><br></div>Trying to add one more node to the group, and it breaks with error:<br><span>Failed tasks: Task[fuel_pkgs/9] </span><span><span>Stopping the deployment process!</span></span></div><span><br>But now the entire cluster is stopped - nova service on all nodes are down.<br></span></div><span>Is this expected behaviour? How do I get back my cluster active?<br></span></div><div><span><br></span></div><div><span>Please help me to resolve this problem.<br></span></div><div><span><br></span></div><span>Regards,<br></span></div><span>Raja.<span class="m_654365184472788751gmail-m_-8596960065969626969HOEnZb"><font color="#888888"><br></font></span></span><span class="m_654365184472788751gmail-m_-8596960065969626969HOEnZb"><font color="#888888"><div><div><div><div><span><br></span><div><div><br>-- <br><div class="m_654365184472788751gmail-m_-8596960065969626969m_-6108219991736169004gmail_signature">:^)</div>
</div></div></div></div></div></div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br></div></div>-- <br><div class="m_654365184472788751gmail_signature">:^)</div>
</div></div>
</blockquote></div><br></div>