Open Stack

Wed Oct 25 09:45:19 UTC 2017

Tried once again, this time its a different error:
*Error*
Deployment has failed. All nodes are finished. Failed tasks:
Task[top-role-compute/3], Task[top-role-compute/2],
Task[top-role-compute/5], Task[top-role-compute/4] Stopping the deployment
process!

I don't know if I should continue doing this in the hope that it will
complete eventually.

Am I missing something? Are all these known bugs and I had to apply some
patch at some stage? A guiding note will be highly appreciated.

Many Thanks,
Raja.

On 25 October 2017 at 13:06, Raja T Nair <rtnair at gmail.com> wrote:

> Hello All,
>
> I tried deploying again, without making any change.
> It went past the 42% mark where it was stuck last time, and after around 2
> hours, errored out again, with the following message:
>
> *Error*
> Deployment has failed. All nodes are finished. Failed tasks:
> Task[primary-keystone/1] Stopping the deployment process!
>
> keystone-all.log in the errored node - controller - shows only these lines
> marked as error:
>
> <43>Oct 25 07:24:35 node-1 keystone_wsgi_admin_error [Wed Oct 25
> 07:24:29.560129 2017] [wsgi:error] [pid 193507:tid 139913962108672] Option
> "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be
> silently ignored in the future.
>
> The above line had repeated 23 times.
> I can attach the complete log file if needed.
>
> Doing this for the first time, I do not know where to look for possible
> issues. Please help me get past this.
>
> Many Thanks,
> Raja.
>
>
>
> On 25 October 2017 at 07:14, Raja T Nair <rtnair at gmail.com> wrote:
>
>> Thanks Evgeniy, for the support.
>>
>> I ran the command on shell, and it immediately came out, leaving a line
>> in the logfile which said that all required images are present.
>> After this I re-ran the deployment. It went past the point, but errored
>> out, with the following info on the dashboard:
>>
>> Error
>> Deployment has failed. All nodes are finished. Failed tasks:
>> Task[fuel_pkgs/3], Task[fuel_pkgs/2], Task[fuel_pkgs/4] Stopping the
>> deployment process!
>>
>> 3 of the nodes show `Error Deployment' and 2 of them show `Stopped'
>> Log tab gives the following lines for nodes:
>>
>> <>
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/bin/puppet:8:in `<main>'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels)
>> in run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in
>> `run_command'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in
>> main'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in
>> `apply_catalog'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in
>> apply_catalog'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in
>> `as_logging_destination'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in
>> apply'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in
>> `traverse'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in
>> evaluate'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels)
>> in evaluate'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:221:in `eval_resource'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:19:in
>> `process_events'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in
>> `queued_events'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:92:in
>> `each'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:93:in
>> `block in queued_events'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:20:in
>> `block in process_events'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/transaction/event_manager.rb:101:in
>> `process_callback'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:583:in `refresh'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `sync'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:127:in `times'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:130:in `block in sync'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/provider/exec/posix.rb:45:in `run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:29:in `chdir'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:64:in `block in run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/provider/exec.rb:68:in `block (2
>> levels) in run'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `execute'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> /usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:186:in `waitpid2'
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> Command exceeded timeout
>> 2017-10-24 17:13:43    ERR    (/Stage[main]/Apt::Update/Exec[apt_update])
>> Failed to call refresh: Command exceeded timeout
>> </>
>>
>> Inside individual nodes, puppet.log has these lines, which shows of some
>> unmet dependency:
>>
>> 2017-10-24 17:14:00 +0000 /Package[fuel-misc] (info): Starting to
>> evaluate the resource
>> 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/dpkg-query
>> -W --showformat '${Status} ${P
>> ackage} ${Version}\n' fuel-misc'
>> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug):
>> Call: install
>> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug):
>> Call: wait_for_lock
>> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel) (debug):
>> Call: locked?
>> 2017-10-24 17:14:00 +0000 Puppet (debug): Executing '/usr/bin/apt-get -q
>> -y -o DPkg::Options::=--force-
>> confold -o APT::Get::AllowUnauthenticated=1 install fuel-misc'
>> 2017-10-24 17:14:00 +0000 Package[fuel-misc](provider=apt_fuel)
>> (warning): Attempt '1' of '3' has faile
>> d: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold
>> -o APT::Get::AllowUnauthenti
>> cated=1 install fuel-misc' returned 100: Reading package lists...
>> Building dependency tree...
>> Reading state information...
>> Some packages could not be installed. This may mean that you have
>> requested an impossible situation or if you are using the unstable
>> distribution that some required packages have not yet been created
>> or been moved out of Incoming.
>> The following information may help to resolve the situation:
>>
>> The following packages have unmet dependencies:
>>  fuel-misc : Depends: socat but it is not installable
>> E: Unable to correct problems, you have held broken packages.
>>
>> But dpkg shows that both are installed:
>>
>> root at node-1:/var/log# dpkg -l fuel-misc socat
>> Desired=Unknown/Install/Remove/Purge/Hold
>> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig
>> -aWait/Trig-pend
>> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>> ||/ Name                Version        Architecture   Description
>> +++-===================-==============-==============-======
>> ======================================
>> ii  fuel-misc           9.0.0-1~u14.04 all            Misc Fuel library
>> scripts
>> ii  socat               1.7.2.3-1      amd64          multipurpose relay
>> for bidirectional data tr
>> root at node-1:/var/log#
>>
>>
>> Please help me to get this deployment completed.
>>
>> Many Thanks,
>>
>> Raja.
>>
>> On 24 October 2017 at 21:13, Evgeniy L <eli at mirantis.com> wrote:
>>
>>> Hi,
>>>
>>> Yes, execution expired, means that it could not build an image within
>>> required period of time.
>>> The message you see can be used to build an image (copy paste into
>>> console "command"), also there is a file with logs
>>> /var/log/fuel-agent-env-1.log, which you can use to try to identify
>>> where the problem is.
>>>
>>> Thanks,
>>>
>>> On Tue, Oct 24, 2017 at 7:48 AM, Raja T Nair <rtnair at gmail.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> We are trying to deploy one environment using Fuel 9.2, and it failed
>>>> at around 23%.
>>>>
>>>> The error says:
>>>>
>>>> <>
>>>> Error
>>>> Provision has failed. Failed to execute hook 'shell'
>>>>
>>>> command: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp
>>>> --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image
>>>> --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "
>>>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd6
>>>> 4-boot.img.gz", "format": "ext2"}, "/": {"container": "gzip", "uri": "
>>>> http://172.22.66.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz",
>>>> "format": "ext4"}}, "output": "/var/www/nailgun/targetimages",
>>>> "repos": [{"name": "ubuntu", "section": "main universe multiverse", "uri": "
>>>> http://archive.ubuntu.com/ubuntu/", "priority": null, "suite":
>>>> "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main
>>>> universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/",
>>>> "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name":
>>>> "ubuntu-security", "section": "main universe multiverse", "uri": "
>>>> http://archive.ubuntu.com/ubuntu/", "priority": null, "suite":
>>>> "trusty-security", "type": "deb"}, {"name": "mos", "section": "main
>>>> restricted", "uri": "http://172.22.66.2:8080/mitaka-9.0/ubuntu/x86_64",
>>>> "priority": 1050, "suite": "mos9.0", "type": "deb"}, {"name":
>>>> "mos-updates", "section": "main restricted", "uri": "
>>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/", "priority": 1050,
>>>> "suite": "mos9.0-updates", "type": "deb"}, {"name": "mos-security",
>>>> "section": "main restricted", "uri": "http://mirror.fuel-infra.org/
>>>> mos-repos/ubuntu/9.0/", "priority": 1050, "suite": "mos9.0-security",
>>>> "type": "deb"}, {"name": "mos9.2-updates", "section": "main restricted",
>>>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2",
>>>> "priority": 1050, "suite": "mos9.0-updates", "type": "deb"}, {"name":
>>>> "mos9.2-security", "section": "main restricted", "uri": "
>>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2", "priority": 1050,
>>>> "suite": "mos9.0-security", "type": "deb"}, {"name": "Auxiliary",
>>>> "section": "main restricted", "uri": "http://172.22.66.2:8080/mitak
>>>> a-9.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary",
>>>> "type": "deb"}, {"name": "mos9.2-holdback", "section": "main restricted",
>>>> "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2",
>>>> "priority": 1100, "suite": "mos9.0-holdback", "type": "deb"}], "packages":
>>>> ["acl", "anacron", "bash-completion", "bridge-utils", "bsdmainutils",
>>>> "build-essential", "cloud-init", "curl", "daemonize", "debconf-utils",
>>>> "gdisk", "grub-pc", "hwloc", "i40e-dkms", "linux-firmware",
>>>> "linux-firmware-nonfree", "linux-headers-generic-lts-xenial",
>>>> "linux-image-generic-lts-xenial", "lvm2", "mcollective", "mdadm",
>>>> "multipath-tools", "multipath-tools-boot", "nailgun-agent",
>>>> "nailgun-mcagents", "network-checker", "ntp", "openssh-client",
>>>> "openssh-server", "puppet", "python-amqp", "ruby-augeas", "ruby-ipaddress",
>>>> "ruby-json", "ruby-netaddr", "ruby-openstack", "ruby-shadow", "ruby-stomp",
>>>> "telnet", "ubuntu-minimal", "ubuntu-standard", "uuid-runtime", "vim",
>>>> "virt-what", "vlan"], "codename": "trusty"}'
>>>>
>>>> Task: a84a3af7-2ed4-4d5e-9911-a12dfc0f05e9: overall timeout error:
>>>> execution expired
>>>> Task timeout: 3600, Retries: 1
>>>>
>>>> </>
>>>>
>>>> I understand it was a timeout, probably because of a slow Internet
>>>> connection, while building a required image - please correct me if I'm
>>>> wrong here.
>>>>
>>>> Might be stupid question - but is there a way for me to build such
>>>> images before clicking the deploy button, thereby avoiding the timeout
>>>> issue?
>>>>
>>>> Can anyone guide me or point me to a document to understand what is
>>>> going wrong here?
>>>>
>>>> Many thanks in advance.
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>> --
>>>> :^)
>>>>
>>>> _______________________________________________
>>>> Mailing list: http://lists.openstack.org/cgi
>>>> -bin/mailman/listinfo/openstack
>>>> Post to     : openstack at lists.openstack.org
>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>> -bin/mailman/listinfo/openstack
>>>>
>>>>
>>>
>>
>>
>> --
>> :^)
>>
>
>
>
> --
> :^)
>

-- 
:^)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171025/6eb439d4/attachment.html>

Open Stack

[Openstack] Mirantis - Fuel 9.2 Failed Deployment

OpenStack

Community

Documentation

Branding & Legal