Re: [Ussuri] [openstack-ansible] [cinder] Can't attach volumes to instances
I recommend you run the attach request with the --debug flag to get the request id, that way you can easily track the request and see where it failed.
Then you check the logs like Dmitriy mentions and see where things failed.
Running with the --debug flag returned the following ``` Starting new HTTP connection (1): 192.168.110.201:8774 http://192.168.110.201:8774 "GET /v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54 HTTP/1.1" 200 1781 RESP: [200] Connection: close Content-Length: 1781 Content-Type: application/json OpenStack-API-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 x-openstack-request-id: req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 RESP BODY: {"server": {"id": "2fd482ac-7a93-4626-955f-77b78f783d54", "name": "ubuntu_test", "status": "ACTIVE", "tenant_id": "5fbba868d04b47fa87a72b9dd821ee12", "user_id": "f78c008ff22d40d2870f1c34919c93ad", "metadata": {}, "hostId": "7d7e25a2fe22479152026feff6bd71cf32a46ea2ee78ec841c5973f8", "image": {"id": "84e94029-c737-4bb6-84a7-195002e6dbe9", "links": [{"rel": "bookmark", "href": "http://192.168.110.201:8774/images/84e94029-c737-4bb6-84a7-195002e6dbe9"}]}, "flavor": {"id": "94b75111-dfa2-4565-af5b-64e25e220861", "links": [{"rel": "bookmark", "href": "http://192.168.110.201:8774/flavors/94b75111-dfa2-4565-af5b-64e25e220861"}]}, "created": "2020-10-07T09:48:09Z", "updated": "2020-10-07T10:03:23Z", "addresses": {"test-001": [{"version": 4, "addr": "192.168.32.170", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9b:a5:fe"}, {"version": 4, "addr": "192.168.113.29", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9b:a5:fe"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54"}, {"rel": "bookmark", "href": "http://192.168.110.201:8774/servers/2fd482ac-7a93-4626-955f-77b78f783d54"}], "OS-DCF:diskConfig": "AUTO", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "hiwi-PC", "OS-SRV-USG:launched_at": "2020-10-07T10:03:22.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}], "OS-EXT-SRV-ATTR:host": "bc1blade14", "OS-EXT-SRV-ATTR:instance_name": "instance-00000007", "OS-EXT-SRV-ATTR:hypervisor_hostname": "bc1blade14.openstack.local", "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}} GET call to compute for http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d5... used request id req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 REQ: curl -g -i -X GET http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e1... -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1" Starting new HTTP connection (1): 192.168.110.201:8776 http://192.168.110.201:8776 "GET /v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d HTTP/1.1" 200 1033 RESP: [200] Connection: close Content-Length: 1033 Content-Type: application/json OpenStack-API-Version: volume 3.0 Vary: OpenStack-API-Version x-compute-request-id: req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 x-openstack-request-id: req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 RESP BODY: {"volume": {"id": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "status": "available", "size": 30, "availability_zone": "nova", "created_at": "2020-10-07T11:38:01.000000", "updated_at": "2020-10-12T09:57:53.000000", "attachments": [], "name": "test_volume_002", "description": "", "volume_type": "lvm", "snapshot_id": null, "source_volid": null, "metadata": {}, "links": [{"rel": "self", "href": "http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"}, {"rel": "bookmark", "href": "http://192.168.110.201:8776/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"}], "user_id": "f78c008ff22d40d2870f1c34919c93ad", "bootable": "false", "encrypted": false, "replication_status": null, "consistencygroup_id": null, "multiattach": false, "migration_status": null, "os-vol-tenant-attr:tenant_id": "5fbba868d04b47fa87a72b9dd821ee12", "os-vol-host-attr:host": "bc1bl10@lvm#LVM_iSCSI", "os-vol-mig-status-attr:migstat": null, "os-vol-mig-status-attr:name_id": null}} GET call to volumev3 for http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e1... used request id req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 REQ: curl -g -i -X POST http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d5... -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1" -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"volumeAttachment": {"volumeId": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}}' Resetting dropped connection: 192.168.110.201 http://192.168.110.201:8774 "POST /v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments HTTP/1.1" 200 194 RESP: [200] Connection: close Content-Length: 194 Content-Type: application/json OpenStack-API-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a x-openstack-request-id: req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a RESP BODY: {"volumeAttachment": {"id": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "serverId": "2fd482ac-7a93-4626-955f-77b78f783d54", "volumeId": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}} POST call to compute for http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d5... used request id req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a clean_up AddServerVolume: END return value: 0 ``` However, I could not read the logs the way Dmitriy suggested, it simply returns -- Logs begin at Mon 2020-10-05 14:30:22 UTC, end at Mon 2020-10-12 11:17:20 UTC. -- -- No entries -- so I couldn't investigate the request. ( I tried this on both utility and cinder api containers on the management node and on the storage and respective compute node to be sure)
It's important that the target_ip_address can be accessed from the Nova compute using the interface for the IP defined in my_ip in nova.conf
my_ip in nova.conf seems to be the address of the nova_api container on the management node, target_ip_address from cinder.conf seems to be the (storage-net) address of the cinder-api container on the management node (the value is different from the iscsi_ip_address specified in the playbook).
IIRC, for lvm storage, cinder-volumes should be launched on every nova compute node in order to attach volumes to instances, since it's not shared storage.
I did not try this yet since the documentation didn't mention it and for our current production system (mitaka, which someone else set up ages ago) this doesn't seem to be the case but volumes can still be attached. Kind regards, Oliver
On 12/10, Oliver Wenz wrote:
I recommend you run the attach request with the --debug flag to get the request id, that way you can easily track the request and see where it failed.
Then you check the logs like Dmitriy mentions and see where things failed.
Running with the --debug flag returned the following
``` Starting new HTTP connection (1): 192.168.110.201:8774 http://192.168.110.201:8774 "GET /v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54 HTTP/1.1" 200 1781 RESP: [200] Connection: close Content-Length: 1781 Content-Type: application/json OpenStack-API-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 x-openstack-request-id: req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 RESP BODY: {"server": {"id": "2fd482ac-7a93-4626-955f-77b78f783d54", "name": "ubuntu_test", "status": "ACTIVE", "tenant_id": "5fbba868d04b47fa87a72b9dd821ee12", "user_id": "f78c008ff22d40d2870f1c34919c93ad", "metadata": {}, "hostId": "7d7e25a2fe22479152026feff6bd71cf32a46ea2ee78ec841c5973f8", "image": {"id": "84e94029-c737-4bb6-84a7-195002e6dbe9", "links": [{"rel": "bookmark", "href": "http://192.168.110.201:8774/images/84e94029-c737-4bb6-84a7-195002e6dbe9"}]}, "flavor": {"id": "94b75111-dfa2-4565-af5b-64e25e220861", "links": [{"rel": "bookmark", "href": "http://192.168.110.201:8774/flavors/94b75111-dfa2-4565-af5b-64e25e220861"}]}, "created": "2020-10-07T09:48:09Z", "updated": "2020-10-07T10:03:23Z", "addresses": {"test-001": [{"version": 4, "addr": "192.168.32.170", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9b:a5:fe"}, {"version": 4, "addr": "192.168.113.29", "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9b:a5:fe"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54"}, {"rel": "bookmark", "href": "http://192.168.110.201:8774/servers/2fd482ac-7a93-4626-955f-77b78f783d54"}], "OS-DCF:diskConfig": "AUTO", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": "hiwi-PC", "OS-SRV-USG:launched_at": "2020-10-07T10:03:22.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}], "OS-EXT-SRV-ATTR:host": "bc1blade14", "OS-EXT-SRV-ATTR:instance_name": "instance-00000007", "OS-EXT-SRV-ATTR:hypervisor_hostname": "bc1blade14.openstack.local", "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}} GET call to compute for http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d5... used request id req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 REQ: curl -g -i -X GET http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e1... -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1" Starting new HTTP connection (1): 192.168.110.201:8776 http://192.168.110.201:8776 "GET /v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d HTTP/1.1" 200 1033 RESP: [200] Connection: close Content-Length: 1033 Content-Type: application/json OpenStack-API-Version: volume 3.0 Vary: OpenStack-API-Version x-compute-request-id: req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 x-openstack-request-id: req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 RESP BODY: {"volume": {"id": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "status": "available", "size": 30, "availability_zone": "nova", "created_at": "2020-10-07T11:38:01.000000", "updated_at": "2020-10-12T09:57:53.000000", "attachments": [], "name": "test_volume_002", "description": "", "volume_type": "lvm", "snapshot_id": null, "source_volid": null, "metadata": {}, "links": [{"rel": "self", "href": "http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"}, {"rel": "bookmark", "href": "http://192.168.110.201:8776/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"}], "user_id": "f78c008ff22d40d2870f1c34919c93ad", "bootable": "false", "encrypted": false, "replication_status": null, "consistencygroup_id": null, "multiattach": false, "migration_status": null, "os-vol-tenant-attr:tenant_id": "5fbba868d04b47fa87a72b9dd821ee12", "os-vol-host-attr:host": "bc1bl10@lvm#LVM_iSCSI", "os-vol-mig-status-attr:migstat": null, "os-vol-mig-status-attr:name_id": null}} GET call to volumev3 for http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e1... used request id req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 REQ: curl -g -i -X POST http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d5... -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1" -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"volumeAttachment": {"volumeId": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}}' Resetting dropped connection: 192.168.110.201 http://192.168.110.201:8774 "POST /v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments HTTP/1.1" 200 194 RESP: [200] Connection: close Content-Length: 194 Content-Type: application/json OpenStack-API-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id: req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a x-openstack-request-id: req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a RESP BODY: {"volumeAttachment": {"id": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "serverId": "2fd482ac-7a93-4626-955f-77b78f783d54", "volumeId": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}} POST call to compute for http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d5... used request id req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a clean_up AddServerVolume: END return value: 0 ```
However, I could not read the logs the way Dmitriy suggested, it simply returns
-- Logs begin at Mon 2020-10-05 14:30:22 UTC, end at Mon 2020-10-12 11:17:20 UTC. -- -- No entries --
so I couldn't investigate the request. ( I tried this on both utility and cinder api containers on the management node and on the storage and respective compute node to be sure)
Hi, I really don't know anything about those Ansible playbooks, so you'll have to either check them, check how the container is being run, check the cinder configuration to see if there's a specific logging option set, or check the output of the stdout of the container on startup to see the logging options output by Cinder on startup (on DEBUG mode).
It's important that the target_ip_address can be accessed from the Nova compute using the interface for the IP defined in my_ip in nova.conf
my_ip in nova.conf seems to be the address of the nova_api container on the management node, target_ip_address from cinder.conf seems to be the (storage-net) address of the cinder-api container on the management node (the value is different from the iscsi_ip_address specified in the playbook).
Can you ping from the address defined in my_ip on your compute node to the target_ip_address being used by Cinder? If you cannot, then that's the problem. Cheers, Gorka.
IIRC, for lvm storage, cinder-volumes should be launched on every nova compute node in order to attach volumes to instances, since it's not shared storage.
I did not try this yet since the documentation didn't mention it and for our current production system (mitaka, which someone else set up ages ago) this doesn't seem to be the case but volumes can still be attached.
Kind regards, Oliver
participants (2)
-
Gorka Eguileor
-
Oliver Wenz