[Openstack] Mirantis - Fuel 9.2 Failed Deployment
Raja T Nair
rtnair at gmail.com
Wed Oct 25 07:36:59 UTC 2017
Hello All,
I tried deploying again, without making any change.
It went past the 42% mark where it was stuck last time, and after around 2
hours, errored out again, with the following message:
*Error*
Deployment has failed. All nodes are finished. Failed tasks:
Task[primary-keystone/1] Stopping the deployment process!
keystone-all.log in the errored node - controller - shows only these lines
marked as error:
<43>Oct 25 07:24:35 node-1 keystone_wsgi_admin_error [Wed Oct 25
07:24:29.560129 2017] [wsgi:error] [pid 193507:tid 139913962108672] Option
"verbose" from group "DEFAULT" is deprecated for removal. Its value may be
silently ignored in the future.
The above line had repeated 23 times.
I can attach the complete log file if needed.
Doing this for the first time, I do not know where to look for possible
issues. Please help me get past this.
Many Thanks,
Raja.
On 25 October 2017 at 07:14, Raja T Nair <rtnair at gmail.com> wrote:
> Thanks Evgeniy, for the support.
>
> I ran the command on shell, and it immediately came out, leaving a line in
> the logfile which said that all required images are present.
> After this I re-ran the deployment. It went past the point, but errored
> out, with the following info on the dashboard:
>
> Error
> Deployment has failed. All nodes are finished. Failed tasks:
> Task[fuel_pkgs/3], Task[fuel_pkgs/2], Task[fuel_pkgs/4] Stopping the
> deployment process!
>
> 3 of the nodes show `Error Deployment' and 2 of them show `Stopped'
> Log tab gives the following lines for nodes:
>
> <>
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/bin/puppet:8:in `<main>'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels)
> in run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in
> main'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in
> `apply_catalog'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in
> apply_catalog'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in
> `as_logging_destination'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in
> apply'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in
> `traverse'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels)
> in evaluate'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:221:in `eval_resource'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:19:in
> `process_events'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in
> `queued_events'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in `each'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:93:in
> `block in queued_events'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:20:in
> `block in process_events'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:101:in
> `process_callback'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:583:in `refresh'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `sync'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `times'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:130:in `block in sync'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:45:in `run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `chdir'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:64:in `block in run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:68:in `block (2 levels)
> in run'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `execute'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `waitpid2'
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> Command exceeded timeout
> 2017-10-24 17:13:43 ERR (/Stage[main]/Apt::Update/Exec[apt_update])
> Failed to call refresh: Command exceeded timeout
> </>
>
> Inside individual nodes, puppet.log has these lines, which shows of some
> unmet dependency:
>
> 2017-10-24 17:14:00 +0000 /Package[fuel-misc] (info): Starting to evaluate
> the resource
> 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/dpkg-query
> -W --showformat '${Status} ${P
> ackage} ${Version}\n' fuel-misc'
> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug):
> Call: install
> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug):
> Call: wait_for_lock
> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug):
> Call: locked?
> 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/apt-get -q
> -y -o DPkg::Options::=--force-
> confold -o APT::Get::AllowUnauthenticated=1 install fuel-misc'
> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel)
> (warning): Attempt '1' of '3' has faile
> d: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold
> -o APT::Get::AllowUnauthenti
> cated=1 install fuel-misc' returned 100: Reading package lists...
> Building dependency tree...
> Reading state information...
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
>
> The following packages have unmet dependencies:
> fuel-misc : Depends: socat but it is not installable
> E: Unable to correct problems, you have held broken packages.
>
> But dpkg shows that both are installed:
>
> root at node-1:/var/log# dpkg -l fuel-misc socat
> Desired=Unknown/Install/Remove/Purge/Hold
> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/
> trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name Version Architecture Description
> +++-===================-==============-==============-======
> ======================================
> ii fuel-misc 9.0.0-1~u14.04 all Misc Fuel library
> scripts
> ii socat 1.7.2.3-1 amd64 multipurpose relay
> for bidirectional data tr
> root at node-1:/var/log#
>
>
> Please help me to get this deployment completed.
>
> Many Thanks,
>
> Raja.
>
> On 24 October 2017 at 21:13, Evgeniy L <eli at mirantis.com> wrote:
>
>> Hi,
>>
>> Yes, execution expired, means that it could not build an image within
>> required period of time.
>> The message you see can be used to build an image (copy paste into
>> console "command"), also there is a file with logs
>> /var/log/fuel-agent-env-1.log, which you can use to try to identify
>> where the problem is.
>>
>> Thanks,
>>
>> On Tue, Oct 24, 2017 at 7:48 AM, Raja T Nair <rtnair at gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> We are trying to deploy one environment using Fuel 9.2, and it failed at
>>> around 23%.
>>>
>>> The error says:
>>>
>>> <>
>>> Error
>>> Provision has failed. Failed to execute hook 'shell'
>>>
>>> command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp
>>> --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image
>>> --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "
>>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz",
>>> "format": "ext2"}, "/": {"container": "gzip", "uri": "
>>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz",
>>> "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos":
>>> [{"name": "ubuntu", "section": "main universe multiverse", "uri": "
>>> http://archive.ubuntu.com/ubuntu/", "priority": null, "suite":
>>> "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main
>>> universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/",
>>> "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name":
>>> "ubuntu-security", "section": "main universe multiverse", "uri": "
>>> http://archive.ubuntu.com/ubuntu/", "priority": null, "suite":
>>> "trusty-security", "type": "deb"}, {"name": "mos", "section": "main
>>> restricted", "uri": "http://172.22.66.2:8080/mitaka-9.0/ubuntu/x86_64",
>>> "priority": 1050, "suite": "mos9.0", "type": "deb"}, {"name":
>>> "mos-updates", "section": "main restricted", "uri": "
>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/", "priority": 1050,
>>> "suite": "mos9.0-updates", "type": "deb"}, {"name": "mos-security",
>>> "section": "main restricted", "uri": "http://mirror.fuel-infra.org/
>>> mos-repos/ubuntu/9.0/", "priority": 1050, "suite": "mos9.0-security",
>>> "type": "deb"}, {"name": "mos9.2-updates", "section": "main restricted",
>>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2", "priority":
>>> 1050, "suite": "mos9.0-updates", "type": "deb"}, {"name":
>>> "mos9.2-security", "section": "main restricted", "uri": "
>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2", "priority": 1050,
>>> "suite": "mos9.0-security", "type": "deb"}, {"name": "Auxiliary",
>>> "section": "main restricted", "uri": "http://172.22.66.2:8080/mitak
>>> a-9.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary",
>>> "type": "deb"}, {"name": "mos9.2-holdback", "section": "main restricted",
>>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2", "priority":
>>> 1100, "suite": "mos9.0-holdback", "type": "deb"}], "packages": ["acl",
>>> "anacron", "bash-completion", "bridge-utils", "bsdmainutils",
>>> "build-essential", "cloud-init", "curl", "daemonize", "debconf-utils",
>>> "gdisk", "grub-pc", "hwloc", "i40e-dkms", "linux-firmware",
>>> "linux-firmware-nonfree", "linux-headers-generic-lts-xenial",
>>> "linux-image-generic-lts-xenial", "lvm2", "mcollective", "mdadm",
>>> "multipath-tools", "multipath-tools-boot", "nailgun-agent",
>>> "nailgun-mcagents", "network-checker", "ntp", "openssh-client",
>>> "openssh-server", "puppet", "python-amqp", "ruby-augeas", "ruby-ipaddress",
>>> "ruby-json", "ruby-netaddr", "ruby-openstack", "ruby-shadow", "ruby-stomp",
>>> "telnet", "ubuntu-minimal", "ubuntu-standard", "uuid-runtime", "vim",
>>> "virt-what", "vlan"], "codename": "trusty"}'
>>>
>>> Task: a84a3af7-2ed4-4d5e-9911-a12dfc0f05e9: overall timeout error:
>>> execution expired
>>> Task timeout: 3600, Retries: 1
>>>
>>> </>
>>>
>>> I understand it was a timeout, probably because of a slow Internet
>>> connection, while building a required image - please correct me if I'm
>>> wrong here.
>>>
>>> Might be stupid question - but is there a way for me to build such
>>> images before clicking the deploy button, thereby avoiding the timeout
>>> issue?
>>>
>>> Can anyone guide me or point me to a document to understand what is
>>> going wrong here?
>>>
>>> Many thanks in advance.
>>>
>>> Regards,
>>> Raja.
>>>
>>> --
>>> :^)
>>>
>>> _______________________________________________
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack at lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>>
>>>
>>
>
>
> --
> :^)
>
--
:^)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171025/7397be2f/attachment.html>
More information about the Openstack
mailing list