Open Stack

> I recommend you run the attach request with the --debug flag to get the
> request id, that way you can easily track the request and see where it
> failed.
> 
> Then you check the logs like Dmitriy mentions and see where things
> failed.

Running with the --debug flag returned the following

```
Starting new HTTP connection (1): 192.168.110.201:8774
http://192.168.110.201:8774 "GET
/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54 HTTP/1.1" 200 1781
RESP: [200] Connection: close Content-Length: 1781 Content-Type:
application/json OpenStack-API-Version: compute 2.1 Vary:
OpenStack-API-Version, X-OpenStack-Nova-API-Version
X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id:
req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 x-openstack-request-id:
req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6
RESP BODY: {"server": {"id": "2fd482ac-7a93-4626-955f-77b78f783d54",
"name": "ubuntu_test", "status": "ACTIVE", "tenant_id":
"5fbba868d04b47fa87a72b9dd821ee12", "user_id":
"f78c008ff22d40d2870f1c34919c93ad", "metadata": {}, "hostId":
"7d7e25a2fe22479152026feff6bd71cf32a46ea2ee78ec841c5973f8", "image":
{"id": "84e94029-c737-4bb6-84a7-195002e6dbe9", "links": [{"rel":
"bookmark", "href":
"http://192.168.110.201:8774/images/84e94029-c737-4bb6-84a7-195002e6dbe9"}]},
"flavor": {"id": "94b75111-dfa2-4565-af5b-64e25e220861", "links":
[{"rel": "bookmark", "href":
"http://192.168.110.201:8774/flavors/94b75111-dfa2-4565-af5b-64e25e220861"}]},
"created": "2020-10-07T09:48:09Z", "updated": "2020-10-07T10:03:23Z",
"addresses": {"test-001": [{"version": 4, "addr": "192.168.32.170",
"OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr":
"fa:16:3e:9b:a5:fe"}, {"version": 4, "addr": "192.168.113.29",
"OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr":
"fa:16:3e:9b:a5:fe"}]}, "accessIPv4": "", "accessIPv6": "", "links":
[{"rel": "self", "href":
"http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54"},
{"rel": "bookmark", "href":
"http://192.168.110.201:8774/servers/2fd482ac-7a93-4626-955f-77b78f783d54"}],
"OS-DCF:diskConfig": "AUTO", "progress": 0,
"OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name":
"hiwi-PC", "OS-SRV-USG:launched_at": "2020-10-07T10:03:22.000000",
"OS-SRV-USG:terminated_at": null, "security_groups": [{"name":
"default"}], "OS-EXT-SRV-ATTR:host": "bc1blade14",
"OS-EXT-SRV-ATTR:instance_name": "instance-00000007",
"OS-EXT-SRV-ATTR:hypervisor_hostname": "bc1blade14.openstack.local",
"OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active",
"OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}
GET call to compute for
http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54
used request id req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6
REQ: curl -g -i -X GET
http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d
-H "Accept: application/json" -H "User-Agent: python-cinderclient" -H
"X-Auth-Token:
{SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1"
Starting new HTTP connection (1): 192.168.110.201:8776
http://192.168.110.201:8776 "GET
/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d
HTTP/1.1" 200 1033
RESP: [200] Connection: close Content-Length: 1033 Content-Type:
application/json OpenStack-API-Version: volume 3.0 Vary:
OpenStack-API-Version x-compute-request-id:
req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 x-openstack-request-id:
req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1
RESP BODY: {"volume": {"id": "a9e188a4-7981-4174-a31d-a258de2a7b3d",
"status": "available", "size": 30, "availability_zone": "nova",
"created_at": "2020-10-07T11:38:01.000000", "updated_at":
"2020-10-12T09:57:53.000000", "attachments": [], "name":
"test_volume_002", "description": "", "volume_type": "lvm",
"snapshot_id": null, "source_volid": null, "metadata": {}, "links":
[{"rel": "self", "href":
"http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"},
{"rel": "bookmark", "href":
"http://192.168.110.201:8776/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"}],
"user_id": "f78c008ff22d40d2870f1c34919c93ad", "bootable": "false",
"encrypted": false, "replication_status": null, "consistencygroup_id":
null, "multiattach": false, "migration_status": null,
"os-vol-tenant-attr:tenant_id": "5fbba868d04b47fa87a72b9dd821ee12",
"os-vol-host-attr:host": "bc1bl10 at lvm#LVM_iSCSI",
"os-vol-mig-status-attr:migstat": null,
"os-vol-mig-status-attr:name_id": null}}
GET call to volumev3 for
http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d
used request id req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1
REQ: curl -g -i -X POST
http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments
-H "Accept: application/json" -H "Content-Type: application/json" -H
"User-Agent: python-novaclient" -H "X-Auth-Token:
{SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1"
-H "X-OpenStack-Nova-API-Version: 2.1" -d '{"volumeAttachment":
{"volumeId": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}}'
Resetting dropped connection: 192.168.110.201
http://192.168.110.201:8774 "POST
/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments
HTTP/1.1" 200 194
RESP: [200] Connection: close Content-Length: 194 Content-Type:
application/json OpenStack-API-Version: compute 2.1 Vary:
OpenStack-API-Version, X-OpenStack-Nova-API-Version
X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id:
req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a x-openstack-request-id:
req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a
RESP BODY: {"volumeAttachment": {"id":
"a9e188a4-7981-4174-a31d-a258de2a7b3d", "serverId":
"2fd482ac-7a93-4626-955f-77b78f783d54", "volumeId":
"a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}}
POST call to compute for
http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments
used request id req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a
clean_up AddServerVolume:
END return value: 0
```

However, I could not read the logs the way Dmitriy suggested, it simply
returns

-- Logs begin at Mon 2020-10-05 14:30:22 UTC, end at Mon 2020-10-12
11:17:20 UTC. --
-- No entries --

so I couldn't investigate the request. ( I tried this on both utility
and cinder api containers on the management node and on the storage and
respective compute node to be sure)

> It's important that the target_ip_address can be accessed from the Nova
> compute using the interface for the IP defined in my_ip in nova.conf

my_ip in nova.conf seems to be the address of the nova_api container on
the management node, target_ip_address from cinder.conf seems to be the
(storage-net) address of the cinder-api container on the management node
(the value is different from the iscsi_ip_address specified in the
playbook).

> IIRC, for lvm storage, cinder-volumes should be launched on every nova
> compute node in order to attach volumes
> to instances, since it's not shared storage.

I did not try this yet since the documentation didn't mention it and for
our current production system (mitaka, which someone else set up ages
ago) this doesn't seem to be the case but volumes can still be attached.

Kind regards,
Oliver