[Openstack] Fuel openstack deployment failing.

Makrand makrandsanap at gmail.com
Sat Feb 6 07:57:02 UTC 2016 

Guys,

Just saw the emails. (was busy with real life activities :)).

For my case this seems to be problem with slow internet connection. Plus
may be disk io (have laptop with amd quad core A8 6140 and 16GB ram....but
the disk is 1TB 5400 rpm. this all offcourse running on top of virtual box
latest version).

Although, as mentioned in point A in my second email, when I hit button
deploy button second time, the error never reappeared. I kept the process
running and it installed openstack on all nodes (after OS installation). It
went to 100%, but it threw damn exception in end telling something like

Deployment has failed. Method granular_deploy. Failed to execute hook
> 'shell' command: cd / && ruby
> /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb


Any idea what this might be.

I will give fuel one more run. need to start all VMs again.


One quick question, If I've to retain the successful ostack deployment on
fuel,  in which order I should shutdown/start the VMs? I can't obviously
keep laptop running 24x7.

I am thinking more like

Shutdown= Compute>>Controller>>Fuel master
Start= Fule Mastter>>Controller>>Compute




--
Best,
Makrand


On Thu, Feb 4, 2016 at 10:54 PM, Vladimir Kozhukalov <
vkozhukalov at mirantis.com> wrote:

> Laurent,
>
> Nice to read you have found a solution. According to your question about
> this local repository: it come with the ISO and is to be copied from ISO to
> the local file system during master node provisioning. I will take a look
> at the diagnostic snapshot you attached to the bug and will comment later.
>
> Thanks you very much for not giving up using Fuel.
>
>
>
> Vladimir Kozhukalov
>
> On Thu, Feb 4, 2016 at 6:46 PM, Laurent Tupin <laurent.tupin at apalia.net>
> wrote:
>
>> Hi Makrand,
>>
>> MU1 installation resolve my problem.
>>
>> 1.    Back up your data with dockerctl backup. This will save the data to
>> /var/backup/fuel/.
>> 2.    Run yum update.
>> 3.    Run docker load -i /var/www/nailgun/docker/images/fuel-images.tar.
>> 4.    Run dockerctl destroy all.
>> 5.    Run dockerctl start all.
>> 6.    Run puppet apply -dv
>> /etc/puppet/modules/nailgun/examples/host-only.pp.
>>
>>
>> <https://docs.mirantis.com/openstack/fuel/fuel-7.0/maintenance-updates.html>
>> https://docs.mirantis.com/openstack/fuel/fuel-7.0/maintenance-updates.html
>>
>>
>> Cordialement,
>>
>> Laurent Tupin
>>
>> * <laurent.tupin at apalia.net>*
>> On 04/02/2016 14:51, Laurent Tupin wrote:
>>
>> Hi,
>>
>> I opened a case for same topic in the case of broken local repo:
>> https://bugs.launchpad.net/fuel/+bug/1541843
>>
>> I´m sure that´s the reason of my problem because empty files shouldn't be
>> empty.
>>
>> Here an example for the same file:
>>
>> Local repo
>>
>>
>> MOS repo
>>
>>
>> I count more than 200 empty files in local repo
>> /var/www/nailgun/2015.1.0-7.0/ubuntu :
>> [root at fuelmos7 ubuntu]# find . -type f -size 0 | wc
>>     223     223   19668
>>
>> Notice, that is a fresh fuel installation.
>>
>> @Vladimir
>> Just a question, do you know how this local repository 2015.1.0-7.0 is
>> made ? I found files in /opt/fuel-createmirror-7.0/config/ seems not have
>> any relation with this 2015.1.0-7.0 repository ?
>>
>> @Makrand
>> Did you check you´re deployment log to know if you have same error ?
>>
>> Cordialement,
>>
>> Laurent Tupin
>>
>> * <laurent.tupin at apalia.net>*
>> On 04/02/2016 10:42, Vladimir Kozhukalov wrote:
>>
>> Makrand,
>>
>> The error you are experiencing could occur for several reasons. This
>> command  fa_build_image that exceeds allowed time builds OS image on the
>> master node and as you can see the maximum time is set by default to 3600
>> seconds which should usually be enough.
>>
>> One possible reason why this error occurs is that you use environment
>> with slow disks. Although we disable ext4 journaling for images when
>> building them, there could be still too many IO operations for a particular
>> VM that makes the whole image building process very slow. If you use
>> libvirt, then try to use cache='unsafe' (for details read this [0]) but it
>> can be only used FOR TEST ENVIRONMENTS, because it is UNSAFE.
>>
>> Second possible reason (and most likely) is that you have slow internet
>> connection. The majority of packages are to be downloaded from unstream
>> mirrors by default, so the speed should allow to download about 700M from
>> unstream in less than an hour.
>>
>> Third reason is that something could go wrong while the master node
>> deployment and local repos are broken for some reason. As Laurent wrote the
>> time out error could occur due to errors like 'Size mismatch'. Anyway,
>> please file a bug in launchpad [1] and attach diagnostic snapshot (you can
>> get it clicking the button 'Generate diagnostic snapshot' on the Support
>> tab on UI).
>>
>> As for fuel-createmirror, we have re-worked it totally in Fuel 8.0 (to
>> address bugs and some limitations) and it will be much more reliable and
>> much easier to use.
>>
>> Please, feel free to contanct me directly if you have detailed questions.
>>
>> [0] https://libvirt.org/formatdomain.html
>> [1] https://bugs.launchpad.net/fuel/+filebug
>>
>>
>> Vladimir Kozhukalov
>>
>> On Wed, Feb 3, 2016 at 8:12 PM, Laurent Tupin <laurent.tupin at apalia.net>
>> wrote:
>>
>>> Just finding another indice. The packages in error in deployment logs
>>> are EMPTY.
>>>
>>>
>>>
>>> trying to delete them and synchro again...
>>>
>>> .
>>> Laurent
>>>
>>> On 03/02/2016 18:02, Laurent Tupin wrote:
>>>
>>> Hi,
>>>
>>> I think the time out is not a cause but a consequence of multiples retry
>>> to get those packages....
>>>
>>> Forget to talk about it...
>>> Already tried...
>>> Unfortunately that won´t change anything, problem is still there.
>>>
>>> .
>>> Laurent
>>>
>>> On 03/02/2016 17:58, Makrand wrote:
>>>
>>> Hi
>>>
>>> I did some research and found that the error happening because of time
>>> out.
>>>
>>> A) There is a timer which runs for 3600 sec and this timer expires after
>>> an hour of you starting the process but the image creation task does not
>>> get completed by then ,so when the FUEL master nodes starts looking for the
>>> image to install on the slave nodes it does not find them and hence the
>>> error .
>>>
>>> When you hit the error, don't do anything. Don't delete the environment.
>>> Wait for 40-60 mins and just hit deploy button again. I did same and in
>>> middle of deployment again. Fingers crossed this time :D.
>>>
>>> B) Or it may be bug as well as described below. This is mainly because
>>> of older docker version.
>>>
>>> <https://bugs.launchpad.net/fuel/+bug/1528498/comments/8>
>>> https://bugs.launchpad.net/fuel/+bug/1528498/comments/8
>>>
>>> There is work around for that as well, as mentioned in comment above.
>>>
>>>
>>>
>>> --
>>> Best,
>>> Makrand
>>>
>>>
>>> On Wed, Feb 3, 2016 at 10:14 PM, Laurent Tupin <laurent.tupin at apalia.net
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> I am experimenting exactly the same problem here...
>>>> In deployment logs I find errors regarding *size mismatch* for various
>>>> packages :
>>>>
>>>> */var/log/docker-logs/fuel-agent-env-2.log*
>>>> *Stderr: 'E: Failed to fetch
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/p/python-pbr/python-pbr_0.10.0-0u%7Eu14.04+mos2_all.deb>http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/p/python-pbr/python-pbr_0.10.0-0u~u14.04+mos2_all.deb
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/p/python-pbr/python-pbr_0.10.0-0u~u14.04+mos2_all.deb>
>>>> Size mismatch\n\nE: Failed to fetch
>>>> http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/s/six/python-six_1.9.0-1~u14.04+mos2_all.deb
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/s/six/python-six_1.9.0-1%7Eu14.04+mos2_all.deb>
>>>> Size mismatch\n\nE: Failed to fetch
>>>> http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/s/stevedore/python-stevedore_1.3.0-0u~u14.04+mos2_all.deb
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/s/stevedore/python-stevedore_1.3.0-0u%7Eu14.04+mos2_all.deb>
>>>> Size mismatch\n\nE: Failed to fetch
>>>> http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/p/python-urllib3/python-urllib3_1.8.3-1~u14.04+mos1_all.deb
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/p/python-urllib3/python-urllib3_1.8.3-1%7Eu14.04+mos1_all.deb>
>>>> Size mismatch\n\nE: Failed to fetch
>>>> http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/r/ruby-cstruct/ruby-cstruct_1.0.1-1~u14.04+mos2_all.deb
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/r/ruby-cstruct/ruby-cstruct_1.0.1-1%7Eu14.04+mos2_all.deb>
>>>> Size mismatch\n\n...*
>>>>
>>>> I think there is a repository problem when local repo are creating
>>>> because errors are always regarding it, same as your example.
>>>>
>>>> I am trying to make a fuel-createmirror and if it s still not working :
>>>>
>>>>    -     trying only with external repos and if its still not working
>>>>    - trying to copy with rsync or something mainstream repo locally
>>>>
>>>>
>>>> Please guys for who its working... what are your repos ?
>>>>
>>>> Mine are:
>>>> *ubuntu deb
>>>> <http://archive.ubuntu.com/ubuntu/>http://archive.ubuntu.com/ubuntu/
>>>> <http://archive.ubuntu.com/ubuntu/> trusty main universe multiverse*
>>>> *ubuntu-updates deb
>>>> <http://archive.ubuntu.com/ubuntu/>http://archive.ubuntu.com/ubuntu/
>>>> <http://archive.ubuntu.com/ubuntu/> trusty-updates main universe multiverse*
>>>> *ubuntu-security deb
>>>> <http://archive.ubuntu.com/ubuntu/>http://archive.ubuntu.com/ubuntu/
>>>> <http://archive.ubuntu.com/ubuntu/> trusty-security main universe
>>>> multiverse*
>>>> *mos-updates deb
>>>> <http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/>http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/
>>>> <http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/> mos7.0-updates main
>>>> restricted 1050*
>>>> *mos-security deb
>>>> <http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/>http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/
>>>> <http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/> mos7.0-security main
>>>> restricted 1050*
>>>> *mos deb
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64>http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64
>>>> <http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64> mos7.0 main restricted
>>>> 1050*
>>>>
>>>>
>>>> *mos-holdback deb
>>>> <http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/>http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/
>>>> <http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/> mos7.0-holdback main
>>>> restricted 1100 *Maybe I´m on a wrong way....
>>>>
>>>> .
>>>> Laurent
>>>>
>>>>
>>>>
>>>> On 03/02/2016 16:58, Makrand wrote:
>>>>
>>>> Hi there,
>>>>
>>>> I am trying to setup demo for openstack using the fuel 7.0 iso. My fuel
>>>> master and slave VMs are ready on OVB. Slave VMs can boot from PXE over
>>>> network. I can create the deployment environment frame. But when I hit
>>>> deploy option from fuel master GUI, it just fails with following error.
>>>> Tried multiple times, same thing happening. Anyone have any idea how to get
>>>> around this. Please note, I don't want to create the local ubuntu mirror on
>>>> fuel master as of now.
>>>>
>>>> =================================
>>>> Error
>>>> Failed to execute hook 'shell' command: cd / && fa_build_image
>>>> --image_build_dir /var/lib/fuel/ibp --log-file
>>>> /var/log/fuel-agent-env-2.log --data_driver nailgun_build_image
>>>> --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "
>>>> <http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64-boot.img.gz>
>>>> http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64-boot.img.gz",
>>>> "format": "ext2"}, "/": {"container": "gzip", "uri": "
>>>> <http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64.img.gz>
>>>> http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64.img.gz",
>>>> "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos":
>>>> [{"name": "ubuntu", "section": "main universe multiverse", "uri": "
>>>> <http://archive.ubuntu.com/ubuntu/>http://archive.ubuntu.com/ubuntu/",
>>>> "priority": null, "suite": "trusty", "type": "deb"}, {"name":
>>>> "ubuntu-updates", "section": "main universe multiverse", "uri": "
>>>> <http://archive.ubuntu.com/ubuntu/>http://archive.ubuntu.com/ubuntu/",
>>>> "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name":
>>>> "ubuntu-security", "section": "main universe multiverse", "uri": "
>>>> <http://archive.ubuntu.com/ubuntu/>http://archive.ubuntu.com/ubuntu/",
>>>> "priority": null, "suite": "trusty-security", "type": "deb"}, {"name":
>>>> "mos", "section": "main restricted", "uri": "
>>>> <http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64>
>>>> http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64", "priority": 1050,
>>>> "suite": "mos7.0", "type": "deb"}, {"name": "mos-updates", "section": "main
>>>> restricted", "uri": "
>>>> <http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/>
>>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/", "priority": 1050,
>>>> "suite": "mos7.0-updates", "type": "deb"}, {"name": "mos-security",
>>>> "section": "main restricted", "uri": "
>>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/", "priority": 1050,
>>>> "suite": "mos7.0-security", "type": "deb"}, {"name": "mos-holdback",
>>>> "section": "main restricted", "uri": "
>>>> http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/", "priority": 1100,
>>>> "suite": "mos7.0-holdback", "type": "deb"}, {"name": "Auxiliary",
>>>> "section": "main restricted", "uri": "
>>>> http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/auxiliary", "priority":
>>>> 1150, "suite": "auxiliary", "type": "deb"}], "codename": "trusty"}'
>>>>
>>>> Task: ddb86944-18ec-473a-a671-8edc7eab9e6d: shell timeout error: execution expired
>>>> Task timeout: 3600, Retries: 1
>>>>
>>>> =================================
>>>>
>>>> --
>>>> Best,
>>>> Makrand
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> Post to     : openstack at lists.openstack.org
>>>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160206/bf846989/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 53921 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160206/bf846989/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 31042 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160206/bf846989/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 57617 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160206/bf846989/attachment-0002.png>

More information about the Openstack mailing list