[Openstack] Fuel openstack deployment failing.

Laurent Tupin laurent.tupin at apalia.net
Thu Feb 4 15:46:56 UTC 2016


Hi Makrand,

MU1 installation resolve my problem.

1.    Back up your data with dockerctl backup. This will save the data
to /var/backup/fuel/.
2.    Run yum update.
3.    Run docker load -i /var/www/nailgun/docker/images/fuel-images.tar.
4.    Run dockerctl destroy all.
5.    Run dockerctl start all.
6.    Run puppet apply -dv
/etc/puppet/modules/nailgun/examples/host-only.pp.

https://docs.mirantis.com/openstack/fuel/fuel-7.0/maintenance-updates.html


Cordialement,

Laurent Tupin

_<mailto:laurent.tupin at apalia.net>_
On 04/02/2016 14:51, Laurent Tupin wrote:
> Hi,
>
> I opened a case for same topic in the case of broken local repo:
> https://bugs.launchpad.net/fuel/+bug/1541843
>
> I´m sure that´s the reason of my problem because empty files shouldn't
> be empty.
>
> Here an example for the same file:
>
> Local repo
>
>
> MOS repo
>
>
> I count more than 200 empty files in local repo
> /var/www/nailgun/2015.1.0-7.0/ubuntu :
> [root at fuelmos7 ubuntu]# find . -type f -size 0 | wc
>     223     223   19668
>
> Notice, that is a fresh fuel installation.
>
> @Vladimir
> Just a question, do you know how this local repository 2015.1.0-7.0 is
> made ? I found files in /opt/fuel-createmirror-7.0/config/ seems not
> have any relation with this 2015.1.0-7.0 repository ?
>
> @Makrand
> Did you check you´re deployment log to know if you have same error ?
>
> Cordialement,
>
> Laurent Tupin
>
> _<mailto:laurent.tupin at apalia.net>_
> On 04/02/2016 10:42, Vladimir Kozhukalov wrote:
>> Makrand,
>>
>> The error you are experiencing could occur for several reasons. This
>> command  fa_build_image that exceeds allowed time builds OS image on
>> the master node and as you can see the maximum time is set by default
>> to 3600 seconds which should usually be enough. 
>>
>> One possible reason why this error occurs is that you use environment
>> with slow disks. Although we disable ext4 journaling for images when
>> building them, there could be still too many IO operations for a
>> particular VM that makes the whole image building process very slow.
>> If you use libvirt, then try to use cache='unsafe' (for details read
>> this [0]) but it can be only used FOR TEST ENVIRONMENTS, because it
>> is UNSAFE.
>>
>> Second possible reason (and most likely) is that you have slow
>> internet connection. The majority of packages are to be downloaded
>> from unstream mirrors by default, so the speed should allow to
>> download about 700M from unstream in less than an hour. 
>>
>> Third reason is that something could go wrong while the master node
>> deployment and local repos are broken for some reason. As Laurent
>> wrote the time out error could occur due to errors like 'Size
>> mismatch'. Anyway, please file a bug in launchpad [1] and attach
>> diagnostic snapshot (you can get it clicking the button 'Generate
>> diagnostic snapshot' on the Support tab on UI).
>>
>> As for fuel-createmirror, we have re-worked it totally in Fuel 8.0
>> (to address bugs and some limitations) and it will be much more
>> reliable and much easier to use. 
>>
>> Please, feel free to contanct me directly if you have detailed
>> questions. 
>>
>> [0] https://libvirt.org/formatdomain.html
>> [1] https://bugs.launchpad.net/fuel/+filebug
>>  
>>
>> Vladimir Kozhukalov
>>
>> On Wed, Feb 3, 2016 at 8:12 PM, Laurent Tupin
>> <laurent.tupin at apalia.net <mailto:laurent.tupin at apalia.net>> wrote:
>>
>>     Just finding another indice. The packages in error in deployment
>>     logs are EMPTY.
>>
>>
>>
>>     trying to delete them and synchro again...
>>
>>     .
>>     Laurent
>>
>>     __
>>     On 03/02/2016 18:02, Laurent Tupin wrote:
>>>     Hi,
>>>
>>>     I think the time out is not a cause but a consequence of
>>>     multiples retry to get those packages....
>>>
>>>     Forget to talk about it...
>>>     Already tried...
>>>     Unfortunately that won´t change anything, problem is still there.
>>>
>>>     .
>>>     Laurent
>>>
>>>     __
>>>     On 03/02/2016 17:58, Makrand wrote:
>>>>     Hi
>>>>
>>>>     I did some research and found that the error happening because
>>>>     of time out.
>>>>
>>>>     A) There is a timer which runs for 3600 sec and this timer
>>>>     expires after an hour of you starting the process but the image
>>>>     creation task does not get completed by then ,so when the FUEL
>>>>     master nodes starts looking for the image to install on the
>>>>     slave nodes it does not find them and hence the error .
>>>>
>>>>     When you hit the error, don't do anything. Don't delete
>>>>     the environment. Wait for 40-60 mins and just hit deploy button
>>>>     again. I did same and in middle of deployment again. Fingers
>>>>     crossed this time :D.
>>>>
>>>>     B) Or it may be bug as well as described below. This is mainly
>>>>     because of older docker version.
>>>>
>>>>     https://bugs.launchpad.net/fuel/+bug/1528498/comments/8
>>>>
>>>>     There is work around for that as well, as mentioned in comment
>>>>     above.
>>>>
>>>>
>>>>
>>>>     --
>>>>     Best,
>>>>     Makrand
>>>>
>>>>
>>>>     On Wed, Feb 3, 2016 at 10:14 PM, Laurent Tupin
>>>>     <laurent.tupin at apalia.net> wrote:
>>>>
>>>>         Hi,
>>>>
>>>>         I am experimenting exactly the same problem here...
>>>>         In deployment logs I find errors regarding *size mismatch*
>>>>         for various packages :
>>>>
>>>>         //var/log/docker-logs/fuel-agent-env-2.log//
>>>>         //Stderr: 'E: Failed to fetch
>>>>         http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/p/python-pbr/python-pbr_0.10.0-0u~u14.04+mos2_all.deb 
>>>>         Size mismatch\n\nE: Failed to fetch
>>>>         http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/s/six/python-six_1.9.0-1~u14.04+mos2_all.deb 
>>>>         Size mismatch\n\nE: Failed to fetch
>>>>         http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/s/stevedore/python-stevedore_1.3.0-0u~u14.04+mos2_all.deb 
>>>>         Size mismatch\n\nE: Failed to fetch
>>>>         http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/p/python-urllib3/python-urllib3_1.8.3-1~u14.04+mos1_all.deb 
>>>>         Size mismatch\n\nE: Failed to fetch
>>>>         http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64/pool/main/r/ruby-cstruct/ruby-cstruct_1.0.1-1~u14.04+mos2_all.deb 
>>>>         Size mismatch\n\n.../
>>>>
>>>>         I think there is a repository problem when local repo are
>>>>         creating because errors are always regarding it, same as
>>>>         your example.
>>>>
>>>>         I am trying to make a fuel-createmirror and if it s still
>>>>         not working :
>>>>
>>>>           *     trying only with external repos and if its still
>>>>             not working
>>>>               o trying to copy with rsync or something mainstream
>>>>                 repo locally
>>>>
>>>>
>>>>         Please guys for who its working... what are your repos ?
>>>>
>>>>         Mine are:
>>>>         /ubuntu deb http://archive.ubuntu.com/ubuntu/ trusty main
>>>>         universe multiverse//
>>>>         //ubuntu-updates deb http://archive.ubuntu.com/ubuntu/
>>>>         trusty-updates main universe multiverse//
>>>>         //ubuntu-security deb http://archive.ubuntu.com/ubuntu/
>>>>         trusty-security main universe multiverse//
>>>>         //mos-updates deb
>>>>         http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/
>>>>         mos7.0-updates main restricted 1050//
>>>>         //mos-security deb
>>>>         http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/
>>>>         mos7.0-security main restricted 1050//
>>>>         //mos deb http://10.20.4.2:8080/2015.1.0-7.0/ubuntu/x86_64
>>>>         mos7.0 main restricted 1050//
>>>>         //mos-holdback deb
>>>>         http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/
>>>>         mos7.0-holdback main restricted 1100
>>>>
>>>>         /Maybe I´m on a wrong way....
>>>>
>>>>         .
>>>>         Laurent
>>>>
>>>>
>>>>
>>>>         __
>>>>         On 03/02/2016 16:58, Makrand wrote:
>>>>>         Hi there,
>>>>>
>>>>>         I am trying to setup demo for openstack using the fuel 7.0
>>>>>         iso. My fuel master and slave VMs are ready on OVB. Slave
>>>>>         VMs can boot from PXE over network. I can create the
>>>>>         deployment environment frame. But when I hit deploy option
>>>>>         from fuel master GUI, it just fails with following error.
>>>>>         Tried multiple times, same thing happening. Anyone have
>>>>>         any idea how to get around this. Please note, I don't want
>>>>>         to create the local ubuntu mirror on fuel master as of now.
>>>>>
>>>>>         =================================
>>>>>         Error
>>>>>         Failed to execute hook 'shell' command: cd / &&
>>>>>         fa_build_image --image_build_dir /var/lib/fuel/ibp
>>>>>         --log-file /var/log/fuel-agent-env-2.log --data_driver
>>>>>         nailgun_build_image --input_data '{"image_data": {"/boot":
>>>>>         {"container": "gzip", "uri":
>>>>>         "http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64-boot.img.gz",
>>>>>         "format": "ext2"}, "/": {"container": "gzip", "uri":
>>>>>         "http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64.img.gz",
>>>>>         "format": "ext4"}}, "output":
>>>>>         "/var/www/nailgun/targetimages", "repos": [{"name":
>>>>>         "ubuntu", "section": "main universe multiverse", "uri":
>>>>>         "http://archive.ubuntu.com/ubuntu/", "priority": null,
>>>>>         "suite": "trusty", "type": "deb"}, {"name":
>>>>>         "ubuntu-updates", "section": "main universe multiverse",
>>>>>         "uri": "http://archive.ubuntu.com/ubuntu/", "priority":
>>>>>         null, "suite": "trusty-updates", "type": "deb"}, {"name":
>>>>>         "ubuntu-security", "section": "main universe multiverse",
>>>>>         "uri": "http://archive.ubuntu.com/ubuntu/", "priority":
>>>>>         null, "suite": "trusty-security", "type": "deb"}, {"name":
>>>>>         "mos", "section": "main restricted", "uri":
>>>>>         "http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64",
>>>>>         "priority": 1050, "suite": "mos7.0", "type": "deb"},
>>>>>         {"name": "mos-updates", "section": "main restricted",
>>>>>         "uri":
>>>>>         "http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/",
>>>>>         "priority": 1050, "suite": "mos7.0-updates", "type":
>>>>>         "deb"}, {"name": "mos-security", "section": "main
>>>>>         restricted", "uri":
>>>>>         "http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/",
>>>>>         "priority": 1050, "suite": "mos7.0-security", "type":
>>>>>         "deb"}, {"name": "mos-holdback", "section": "main
>>>>>         restricted", "uri":
>>>>>         "http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/",
>>>>>         "priority": 1100, "suite": "mos7.0-holdback", "type":
>>>>>         "deb"}, {"name": "Auxiliary", "section": "main
>>>>>         restricted", "uri":
>>>>>         "http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/auxiliary",
>>>>>         "priority": 1150, "suite": "auxiliary", "type": "deb"}],
>>>>>         "codename": "trusty"}'
>>>>>         Task: ddb86944-18ec-473a-a671-8edc7eab9e6d: shell timeout error: execution expired
>>>>>         Task timeout: 3600, Retries: 1
>>>>>         =================================
>>>>>
>>>>>         --
>>>>>         Best,
>>>>>         Makrand
>>>>>
>>>>>
>>>>>
>>>>>         _______________________________________________
>>>>>         Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>         Post to     : openstack at lists.openstack.org
>>>>>         <mailto:openstack at lists.openstack.org>
>>>>>         Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>
>>>>
>>>
>>
>>
>>     _______________________________________________
>>     Mailing list:
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>     Post to     : openstack at lists.openstack.org
>>     <mailto:openstack at lists.openstack.org>
>>     Unsubscribe :
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160204/ed93fcd4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 31042 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160204/ed93fcd4/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 53921 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160204/ed93fcd4/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 57617 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160204/ed93fcd4/attachment-0002.png>


More information about the Openstack mailing list