<div dir="ltr"><div><div><div><div><div><div>Hi Evgeniy,<br><br></div>apt-cache policy fuel-ha-utils says that the package is not to be found.<br><br></div>Now I re-installed fuel on another node(as there was a lot of issues faced on the old one), and created local repo using these commands:<br><br>fuel-mirror create -P ubuntu -G ubuntu<br>fuel-mirror apply -G ubuntu -I /usr/share/fuel-mirror/ubuntu.yaml --replace --default<br>fuel-mirror apply -G mos -I /usr/share/fuel-mirror/ubuntu.yaml --default<br><br></div>my /usr/share/fuel-mirror directory looks like this:<br><br>[root@fuel92 nginx]# ll /usr/share/fuel-mirror/<br>total 12<br>-rwxr-xr-x. 1 root root 1344 May 24 07:47 centos.yaml<br>-rwxr-xr-x. 1 root root 4288 May 24 07:47 ubuntu.yaml<br>[root@fuel92 nginx]#<br><br></div>While doing the ``connectivity check'', it failed with following error lines:<br><span>Verification failed.</span><br>Repo
availability verification failed on following nodes Fuel master node,
Untitled (0b:34), Untitled (71:28), Untitled (2b:52), Untitled (ae:40),
Untitled (a8:73).<span></span><br>Following repos are not available -
<a href="http://172.22.66.11:8080/mirrors/mos-repos/ubuntu/9.0">http://172.22.66.11:8080/mirrors/mos-repos/ubuntu/9.0</a>,
<a href="http://172.22.66.11:8080/mirrors/mos-repos/ubuntu/9.2">http://172.22.66.11:8080/mirrors/mos-repos/ubuntu/9.2</a>.<br><br></div>nginx logs say this:<br>2017/11/02 10:22:47 [error] 1188#1188: *12256 open() "/var/www/nailgun/mirrors/mos-repos/ubuntu/9.0" failed (2: No such file or directory), client<br>: 172.22.66.24, server: localhost, request: "GET /mirrors/mos-repos/ubuntu/9.0 HTTP/1.1", host: "<a href="http://172.22.66.11:8080">172.22.66.11:8080</a>"<br><br></div>And the directory mos-repos actually does not exist:<br>[root@fuel92 nginx]# ls /var/www/nailgun/mirrors/<br>ubuntu<br><div><br></div><div>I suppose I need to build the mos-repos repo also locally. If yes, how do I do this? I thought it would have built it with the above <br></div><div>"fuel-mirror create -P ubuntu -G ubuntu" command.</div><div><br></div><div>Attaching screenshot of settings page, which shows currently configured repos.<br></div><div></div><div><br></div><div>Please help me with this one.<br></div><div><br></div><div>Regards,</div><div>Raja.<br><div><div><div><br></div><div><br></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 31 October 2017 at 19:29, Evgeniy L <span dir="ltr"><<a href="mailto:eli@mirantis.com" target="_blank">eli@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Run "apt-cache policy fuel-ha-utils" and see from which repository does it come from, if your mirrors configured correctly, the package should come from your mirror.<div>It also shows what command it failed to run, "<span style="font-size:12.8px">/usr/bin/apt-get -q -y -o DPkg::Options::=--force-</span><span style="font-size:12.8px">confol<wbr>d -o APT::Get::</span><span style="font-size:12.8px">AllowUnauthenticated<wbr>=1 install fuel-ha-utils" you can run it manually on the node and see what the error is.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks,</span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 31, 2017 at 12:27 AM, Raja T Nair <span dir="ltr"><<a href="mailto:rtnair@gmail.com" target="_blank">rtnair@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi,<br><br></div>I built the repos locally, using the instructions in this page:</div><div><a href="https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-install-guide/local-repo.html" target="_blank">https://docs.openstack.org/fue<wbr>l-docs/latest/userdocs/fuel-<wbr>install-guide/local-repo.html</a><br></div><div><br></div>Now, while trying to deploy to 2 newly added nodes, it fails with this error:<br><br>2017-10-31 06:45:11ERR(/Stage[main]/Osnai<wbr>lyfacter::Fuel_pkgs::Fuel_<wbr>pkgs/Package[fuel-ha-utils]/<wbr>ensure) E: Unable to locate package fuel-ha-utils<br>2017-10-31 06:45:11ERR(/Stage[main]/Osnai<wbr>lyfacter::Fuel_pkgs::Fuel_<wbr>pkgs/Package[fuel-ha-utils]/<wbr>ensure) Reading state information...<br>2017-10-31 06:45:11ERR(/Stage[main]/Osnai<wbr>lyfacter::Fuel_pkgs::Fuel_<wbr>pkgs/Package[fuel-ha-utils]/<wbr>ensure) Building dependency tree...<br>2017-10-31 06:45:11ERR(/Stage[main]/Osnai<wbr>lyfacter::Fuel_pkgs::Fuel_<wbr>pkgs/Package[fuel-ha-utils]/<wbr>ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confol<wbr>d -o APT::Get::AllowUnauthenticated<wbr>=1 install fuel-ha-utils' returned 100: Reading package lists...</div><div>2017-10-31 06:45:11ERR/usr/bin/puppet:8:i<wbr>n `<main>'2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util/command_<wbr>line.rb:92:in `execute'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util/command_<wbr>line.rb:146:in `run'2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application.rb:<wbr>381:in `run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util.rb:496:in `exit_on_fail'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application.rb:<wbr>381:in `block in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application.rb:<wbr>507:in `plugin_hook'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application.rb:<wbr>381:in `block (2 levels) in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application/<wbr>apply.rb:159:in `run_command'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application/<wbr>apply.rb:198:in `main'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet.rb:246:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/context.rb:64:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application/<wbr>apply.rb:236:in `block in main'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/application/<wbr>apply.rb:302:in `apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/configurer.rb:<wbr>133:in `run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet.rb:246:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/context.rb:64:in `override'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/configurer.rb:<wbr>134:in `block in run'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/configurer.rb:<wbr>227:in `run_internal'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/configurer.rb:<wbr>119:in `apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util.rb:160:in `benchmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1<wbr>/benchmark.rb:295:in `realtime'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util.rb:161:in `block in benchmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/configurer.rb:<wbr>120:in `block in apply_catalog'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/resource/catalog<wbr>.rb:168:in `apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction/<wbr>report.rb:112:in `as_logging_destination'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util/log.rb:149:<wbr>in `with_destination'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/resource/catalog<wbr>.rb:169:in `block in apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction.rb:<wbr>138:in `evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/graph/relationsh<wbr>ip_graph.rb:118:in `traverse'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction.rb:<wbr>147:in `block in evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util.rb:334:in `thinmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1<wbr>/benchmark.rb:295:in `realtime'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util.rb:335:in `block in thinmark'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction.rb:<wbr>147:in `block (2 levels) in evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction.rb:<wbr>147:in `call'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction.rb:<wbr>217:in `eval_resource'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction.rb:<wbr>204:in `apply'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction/<wbr>resource_harness.rb:20:in `evaluate'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction/<wbr>resource_harness.rb:81:in `perform_changes'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction/<wbr>resource_harness.rb:128:in `sync_if_needed'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/transaction/<wbr>resource_harness.rb:204:in `sync'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/property.rb:581:<wbr>in `sync'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/property.rb:498:<wbr>in `set'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/property.rb:197:<wbr>in `call_valuemethod'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/type/package.rb:<wbr>73:in `block (3 levels) in <module:Puppet>'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules<wbr>/osnailyfacter/lib/puppet/<wbr>provider/package/apt_fuel.rb:<wbr>72:in `install'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules<wbr>/osnailyfacter/lib/puppet/<wbr>provider/package/apt_fuel.rb:<wbr>72:in `each'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules<wbr>/osnailyfacter/lib/puppet/<wbr>provider/package/apt_fuel.rb:<wbr>74:in `block in install'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules<wbr>/osnailyfacter/lib/puppet/<wbr>provider/package/apt_fuel.rb:<wbr>66:in `wait_for_lock'</div><div>2017-10-31 06:45:11ERR/etc/puppet/modules<wbr>/osnailyfacter/lib/puppet/<wbr>provider/package/apt_fuel.rb:<wbr>75:in `block (2 levels) in install'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/provider/package<wbr>/apt.rb:73:in `install'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/provider.rb:463:<wbr>in `block in create_class_and_instance_meth<wbr>od'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/provider.rb:237:<wbr>in `block in has_command'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/provider/command<wbr>.rb:23:in `execute'</div><div>2017-10-31 06:45:11ERR/usr/lib/ruby/vendo<wbr>r_ruby/puppet/util/execution.<wbr>rb:219:in `execute'</div><div>2017-10-31 06:45:11ERRE: Unable to locate package fuel-ha-utils</div><div>2017-10-31 06:45:11ERRReading state information...</div><div>2017-10-31 06:45:11ERRBuilding dependency tree...</div><div></div><div><br></div>I can see fuel-ha-utils deb package in /var/www/ tree:<br><br>[root@fuel ~]# cd /var/www/nailgun/<br>[root@fuel nailgun]# find . -name fuel-ha-utils*<br>./mitaka-9.0/mos-centos/x86_64<wbr>/Packages/fuel-ha-utils-9.0.0-<wbr>1.mos8460.noarch.rpm<br>./mitaka-9.0/ubuntu/x86_64/poo<wbr>l/main/f/fuel-library9.0/fuel-<wbr>ha-utils_9.0.0-1~u14.04+mos846<wbr>0_all.deb<br>[root@fuel nailgun]# <br><br></div>On a node which was installed before building local repo, I can see the package fuel-ha-utils installed, but a different version i think:<br>root@node-4:~# dpkg -l fuel-ha-utils<br>Desired=Unknown/Install/Remove<wbr>/Purge/Hold<br>| Status=Not/Inst/Conf-files/Unp<wbr>acked/halF-conf/Half-inst/trig<wbr>-aWait/Trig-pend<br>|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)<br>||/ Name Version Architecture Description<br>+++-==========================<wbr>====-====================-====<wbr>================-=============<wbr>==============================<wbr>=======================<br>ii fuel-ha-utils 9.0.0-1~u14.04+mos87 all Fuel Library HA utils<br>root@node-4:~#<br></div><div><br></div><div>Is there a way to fix this problem? Or should I just remove all nodes and start the deployment from scratch - since local repo was built after deploying 5 nodes?</div><div><br></div><div>Please advise.</div><div><br></div><div>Best Regards,</div><div>Raja.<br><div><div><div></div></div></div></div><div class="gmail_extra"><div><div class="m_4239743263674334626h5"><br><div class="gmail_quote">On 30 October 2017 at 19:27, Evgeniy L <span dir="ltr"><<a href="mailto:eli@mirantis.com" target="_blank">eli@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div><span style="font-size:12.8px">fuel_pkgs/9 means that "fuel_pkgs" task failed on node with id 9, find the node with this id using "fuel node" (run it from Fuel node), ssh to the nodes and look for errors in /var/log/puppet.log.</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">If Nova service was running before, it should be running even after fuel_pkgs task failed on one of the nodes, see nova logs, to find what happened to nova.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks,</span></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_4239743263674334626m_654365184472788751gmail-h5">On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair <span dir="ltr"><<a href="mailto:rtnair@gmail.com" target="_blank">rtnair@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_4239743263674334626m_654365184472788751gmail-h5"><div dir="ltr"><div><div><div><div><div>Hello All,<br><br></div>Trying to add one more node to the group, and it breaks with error:<br><span>Failed tasks: Task[fuel_pkgs/9] </span><span><span>Stopping the deployment process!</span></span></div><span><br>But now the entire cluster is stopped - nova service on all nodes are down.<br></span></div><span>Is this expected behaviour? How do I get back my cluster active?<br></span></div><div><span><br></span></div><div><span>Please help me to resolve this problem.<br></span></div><div><span><br></span></div><span>Regards,<br></span></div><span>Raja.<span class="m_4239743263674334626m_654365184472788751gmail-m_-8596960065969626969HOEnZb"><font color="#888888"><br></font></span></span><span class="m_4239743263674334626m_654365184472788751gmail-m_-8596960065969626969HOEnZb"><font color="#888888"><div><div><div><div><span><br></span><div><div><br>-- <br><div class="m_4239743263674334626m_654365184472788751gmail-m_-8596960065969626969m_-6108219991736169004gmail_signature">:^)</div>
</div></div></div></div></div></div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br></div></div>-- <br><div class="m_4239743263674334626m_654365184472788751gmail_signature">:^)</div>
</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">:^)</div>
</div>