Open Stack

Hello OpenStack community:

I'm trying to learn OpenStack for the 1st time. I've been trying to setup
an environment following the instructions for Juno here:

http://docs.openstack.org/juno/install-guide/install/yum/content/ch_preface.html

On Fedora 20. I have 3 systems, with the following Openstack services:

controller:
- keystone
- glance-{api,registry}
- nova-{api,cert,conductor,consoleauth,novncproxy,scheduler}
- cinder-{api,scheduler}
- neutron-server
- MariaDB
- rabbitmq

compute:
- nova-compute
- cinder-volume
- neutron-openvswitch-agent
- neutron-ovs-cleanup

network:
- neutron-dhcp-agent
- neutron-l3-agent
- neutron-metadata-agent
- neutron-openvswitch-agent
- neutron-ovs-cleanup

The only real hardware server above is the "compute" which is why we're
using it for nova-compute and cinder-volume (it has an attached disk
array). The other two systems are virtual machines.

I've gotten as far as step 8. (Add the Block Storage services) and all the
verification steps have passed. However, when I try to launch an instance
with a block storage volume, it fails with the following from the nova.log
on compute:

2015-03-25 16:59:22.154 10056 AUDIT nova.compute.manager
[req-b653adda-3fb2-47d2-a060-577061a5f65d None] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Starting instance...
2015-03-25 16:59:22.218 10056 AUDIT nova.compute.claims [-] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Attempting claim: memory 4096 MB,
disk 40 GB
2015-03-25 16:59:22.218 10056 AUDIT nova.compute.claims [-] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Total memory: 63419 MB, used: 1024.00
MB
2015-03-25 16:59:22.218 10056 AUDIT nova.compute.claims [-] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] memory limit: 95128.50 MB, free:
94104.50 MB
2015-03-25 16:59:22.218 10056 AUDIT nova.compute.claims [-] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Total disk: 14 GB, used: 1.00 GB
2015-03-25 16:59:22.219 10056 AUDIT nova.compute.claims [-] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] disk limit not specified, defaulting
to unlimited
2015-03-25 16:59:22.228 10056 AUDIT nova.compute.claims [-] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Claim successful
2015-03-25 16:59:22.384 10056 INFO nova.scheduler.client.report [-]
Compute_service record updated for ('aurora', 'aurora.bbky.org')
2015-03-25 16:59:22.583 10056 INFO nova.scheduler.client.report [-]
Compute_service record updated for ('aurora', 'aurora.bbky.org')
2015-03-25 16:59:22.816 10056 AUDIT nova.virt.block_device
[req-118f83bf-bd5d-49bd-aab8-162587db2a88 None] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Booting with volume
6a8d0e5f-f18a-4b1e-96d2-5234ad0abbb1 at /dev/vda
2015-03-25 16:59:22.817 10056 WARNING nova.volume.cinder
[req-118f83bf-bd5d-49bd-aab8-162587db2a88 None] Cinder V1 API is deprecated
as of the Juno release, and Nova is still configured to use it. Enable the
V2 API in Cinder and set cinder_catalog_info in nova.conf to use it.
2015-03-25 16:59:23.057 10056 INFO nova.scheduler.client.report [-]
Compute_service record updated for ('aurora', 'aurora.bbky.org')
2015-03-25 17:00:14.188 10056 AUDIT nova.compute.resource_tracker [-]
Auditing locally available compute resources
2015-03-25 17:00:14.424 10056 AUDIT nova.compute.resource_tracker [-] Total
physical ram (MB): 63419, total allocated virtual ram (MB): 5120
2015-03-25 17:00:14.425 10056 AUDIT nova.compute.resource_tracker [-] Free
disk (GB): -27
2015-03-25 17:00:14.425 10056 AUDIT nova.compute.resource_tracker [-] Total
usable vcpus: 8, total allocated vcpus: 0
2015-03-25 17:00:14.425 10056 AUDIT nova.compute.resource_tracker [-] PCI
stats: []
2015-03-25 17:00:14.524 10056 INFO nova.scheduler.client.report [-]
Compute_service record updated for ('aurora', 'aurora.bbky.org')
2015-03-25 17:00:14.524 10056 INFO nova.compute.resource_tracker [-]
Compute_service record updated for aurora:aurora.bbky.org
2015-03-25 17:01:16.189 10056 AUDIT nova.compute.resource_tracker [-]
Auditing locally available compute resources
2015-03-25 17:01:16.444 10056 AUDIT nova.compute.resource_tracker [-] Total
physical ram (MB): 63419, total allocated virtual ram (MB): 5120
2015-03-25 17:01:16.444 10056 AUDIT nova.compute.resource_tracker [-] Free
disk (GB): -27
2015-03-25 17:01:16.444 10056 AUDIT nova.compute.resource_tracker [-] Total
usable vcpus: 8, total allocated vcpus: 0
2015-03-25 17:01:16.445 10056 AUDIT nova.compute.resource_tracker [-] PCI
stats: []
2015-03-25 17:01:16.445 10056 INFO nova.compute.resource_tracker [-]
Compute_service record updated for aurora:aurora.bbky.org
2015-03-25 17:02:17.188 10056 AUDIT nova.compute.resource_tracker [-]
Auditing locally available compute resources
2015-03-25 17:02:17.429 10056 AUDIT nova.compute.resource_tracker [-] Total
physical ram (MB): 63419, total allocated virtual ram (MB): 5120
2015-03-25 17:02:17.429 10056 AUDIT nova.compute.resource_tracker [-] Free
disk (GB): -27
2015-03-25 17:02:17.430 10056 AUDIT nova.compute.resource_tracker [-] Total
usable vcpus: 8, total allocated vcpus: 0
2015-03-25 17:02:17.430 10056 AUDIT nova.compute.resource_tracker [-] PCI
stats: []
2015-03-25 17:02:17.430 10056 INFO nova.compute.resource_tracker [-]
Compute_service record updated for aurora:aurora.bbky.org
2015-03-25 17:03:19.218 10056 AUDIT nova.compute.resource_tracker [-]
Auditing locally available compute resources
2015-03-25 17:03:19.456 10056 AUDIT nova.compute.resource_tracker [-] Total
physical ram (MB): 63419, total allocated virtual ram (MB): 5120
2015-03-25 17:03:19.456 10056 AUDIT nova.compute.resource_tracker [-] Free
disk (GB): -27
2015-03-25 17:03:19.457 10056 AUDIT nova.compute.resource_tracker [-] Total
usable vcpus: 8, total allocated vcpus: 0
2015-03-25 17:03:19.457 10056 AUDIT nova.compute.resource_tracker [-] PCI
stats: []
2015-03-25 17:03:19.457 10056 INFO nova.compute.resource_tracker [-]
Compute_service record updated for aurora:aurora.bbky.org
2015-03-25 17:03:45.331 10056 ERROR nova.compute.manager
[req-118f83bf-bd5d-49bd-aab8-162587db2a88 None] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Instance failed block device setup
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Traceback (most recent call last):
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1822, in
_prep_block_device
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     do_check_attach=do_check_attach) +
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 407, in
attach_block_devices
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     map(_log_and_attach,
block_device_mapping)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 405, in
_log_and_attach
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     bdm.attach(*attach_args,
**attach_kwargs)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in
wrapped
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     ret_val = method(obj, context,
*args, **kwargs)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 237, in
attach
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     connector)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 185, in
wrapper
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     res = method(self, ctx,
volume_id, *args, **kwargs)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 355, in
initialize_connection
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     connector)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 321, in
initialize_connection
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     {'connector':
connector})[1]['connection_info']
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 250, in
_action
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     return self.api.client.post(url,
body=body)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 305, in post
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     return self._cs_request(url,
'POST', **kwargs)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 269, in
_cs_request
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     **kwargs)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 252, in
request
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     raise
exceptions.from_response(resp, body)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] ClientException: The server has
either erred or is incapable of performing the requested operation. (HTTP
500) (Request-ID: req-6d373795-7566-4421-8c2f-f2d66d014d1c)
2015-03-25 17:03:45.331 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]
2015-03-25 17:03:45.338 10056 ERROR nova.compute.manager
[req-118f83bf-bd5d-49bd-aab8-162587db2a88 None] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Failure prepping block device
2015-03-25 17:03:45.338 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Traceback (most recent call last):
2015-03-25 17:03:45.338 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2221, in
_build_resources
2015-03-25 17:03:45.338 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     block_device_mapping)
2015-03-25 17:03:45.338 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1850, in
_prep_block_device
2015-03-25 17:03:45.338 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     raise exception.InvalidBDM()
2015-03-25 17:03:45.338 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] InvalidBDM: Block Device Mapping is
Invalid.
2015-03-25 17:03:45.338 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]
2015-03-25 17:03:45.441 10056 INFO nova.scheduler.client.report
[req-118f83bf-bd5d-49bd-aab8-162587db2a88 None] Compute_service record
updated for ('aurora', 'aurora.bbky.org')
2015-03-25 17:03:45.443 10056 ERROR nova.compute.manager
[req-118f83bf-bd5d-49bd-aab8-162587db2a88 None] [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Build of instance
3f91262a-74dd-4c36-81b4-f874744de1d8 aborted: Failure prepping block device.
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] Traceback (most recent call last):
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2033, in
_do_build_and_run_instance
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     filter_properties)
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2132, in
_build_and_run_instance
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     'create.error', fault=e)
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
82, in __exit__
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     six.reraise(self.type_,
self.value, self.tb)
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2105, in
_build_and_run_instance
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     block_device_mapping) as
resources:
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     return self.gen.next()
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]   File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2243, in
_build_resources
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]     reason=msg)
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8] BuildAbortException: Build of
instance 3f91262a-74dd-4c36-81b4-f874744de1d8 aborted: Failure prepping
block device.
2015-03-25 17:03:45.443 10056 TRACE nova.compute.manager [instance:
3f91262a-74dd-4c36-81b4-f874744de1d8]


I can create the volumes via the dashboard and 'cinder' command without
issue. So, I think cinder is working. I see this in cinder's logs:

2015-03-25 16:59:23.342 12404 INFO cinder.brick.iscsi.iscsi
[req-efb02477-2628-418a-961b-f12b3102a1aa 1d280b39d00e4cb5bdbc1928d161ae9a
39e0d28062a74f288cc4894fb8b15d74 - - -] Creating iscsi_target for volume:
volume-6a8d0e5f-f18a-4b1e-96d2-5234ad0abbb1

Which shows that cinder is providing the iscsi target. But the volume never
successfully attaches to the instances i'm launching. The "InvalidBDM:
Block Device Mapping is Invalid." is the message that stands out to me in
the logs above, but I don't know what that means or how to troubleshoot
further??

I would very much appreciate it if anyone could provide some guidance on
how to troubleshoot this further so I can resolve the problem.

Thanks in advance...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150325/d1daa475/attachment.html>