Open Stack

On 12/10, Oliver Wenz wrote:
> > I recommend you run the attach request with the --debug flag to get the
> > request id, that way you can easily track the request and see where it
> > failed.
> >
> > Then you check the logs like Dmitriy mentions and see where things
> > failed.
>
> Running with the --debug flag returned the following
>
> ```
> Starting new HTTP connection (1): 192.168.110.201:8774
> http://192.168.110.201:8774 "GET
> /v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54 HTTP/1.1" 200 1781
> RESP: [200] Connection: close Content-Length: 1781 Content-Type:
> application/json OpenStack-API-Version: compute 2.1 Vary:
> OpenStack-API-Version, X-OpenStack-Nova-API-Version
> X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id:
> req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6 x-openstack-request-id:
> req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6
> RESP BODY: {"server": {"id": "2fd482ac-7a93-4626-955f-77b78f783d54",
> "name": "ubuntu_test", "status": "ACTIVE", "tenant_id":
> "5fbba868d04b47fa87a72b9dd821ee12", "user_id":
> "f78c008ff22d40d2870f1c34919c93ad", "metadata": {}, "hostId":
> "7d7e25a2fe22479152026feff6bd71cf32a46ea2ee78ec841c5973f8", "image":
> {"id": "84e94029-c737-4bb6-84a7-195002e6dbe9", "links": [{"rel":
> "bookmark", "href":
> "http://192.168.110.201:8774/images/84e94029-c737-4bb6-84a7-195002e6dbe9"}]},
> "flavor": {"id": "94b75111-dfa2-4565-af5b-64e25e220861", "links":
> [{"rel": "bookmark", "href":
> "http://192.168.110.201:8774/flavors/94b75111-dfa2-4565-af5b-64e25e220861"}]},
> "created": "2020-10-07T09:48:09Z", "updated": "2020-10-07T10:03:23Z",
> "addresses": {"test-001": [{"version": 4, "addr": "192.168.32.170",
> "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr":
> "fa:16:3e:9b:a5:fe"}, {"version": 4, "addr": "192.168.113.29",
> "OS-EXT-IPS:type": "floating", "OS-EXT-IPS-MAC:mac_addr":
> "fa:16:3e:9b:a5:fe"}]}, "accessIPv4": "", "accessIPv6": "", "links":
> [{"rel": "self", "href":
> "http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54"},
> {"rel": "bookmark", "href":
> "http://192.168.110.201:8774/servers/2fd482ac-7a93-4626-955f-77b78f783d54"}],
> "OS-DCF:diskConfig": "AUTO", "progress": 0,
> "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name":
> "hiwi-PC", "OS-SRV-USG:launched_at": "2020-10-07T10:03:22.000000",
> "OS-SRV-USG:terminated_at": null, "security_groups": [{"name":
> "default"}], "OS-EXT-SRV-ATTR:host": "bc1blade14",
> "OS-EXT-SRV-ATTR:instance_name": "instance-00000007",
> "OS-EXT-SRV-ATTR:hypervisor_hostname": "bc1blade14.openstack.local",
> "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active",
> "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": []}}
> GET call to compute for
> http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54
> used request id req-56b2bf8f-f338-4d25-a54e-ce9cfb4e71d6
> REQ: curl -g -i -X GET
> http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d
> -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H
> "X-Auth-Token:
> {SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1"
> Starting new HTTP connection (1): 192.168.110.201:8776
> http://192.168.110.201:8776 "GET
> /v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d
> HTTP/1.1" 200 1033
> RESP: [200] Connection: close Content-Length: 1033 Content-Type:
> application/json OpenStack-API-Version: volume 3.0 Vary:
> OpenStack-API-Version x-compute-request-id:
> req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1 x-openstack-request-id:
> req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1
> RESP BODY: {"volume": {"id": "a9e188a4-7981-4174-a31d-a258de2a7b3d",
> "status": "available", "size": 30, "availability_zone": "nova",
> "created_at": "2020-10-07T11:38:01.000000", "updated_at":
> "2020-10-12T09:57:53.000000", "attachments": [], "name":
> "test_volume_002", "description": "", "volume_type": "lvm",
> "snapshot_id": null, "source_volid": null, "metadata": {}, "links":
> [{"rel": "self", "href":
> "http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"},
> {"rel": "bookmark", "href":
> "http://192.168.110.201:8776/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d"}],
> "user_id": "f78c008ff22d40d2870f1c34919c93ad", "bootable": "false",
> "encrypted": false, "replication_status": null, "consistencygroup_id":
> null, "multiattach": false, "migration_status": null,
> "os-vol-tenant-attr:tenant_id": "5fbba868d04b47fa87a72b9dd821ee12",
> "os-vol-host-attr:host": "bc1bl10 at lvm#LVM_iSCSI",
> "os-vol-mig-status-attr:migstat": null,
> "os-vol-mig-status-attr:name_id": null}}
> GET call to volumev3 for
> http://192.168.110.201:8776/v3/5fbba868d04b47fa87a72b9dd821ee12/volumes/a9e188a4-7981-4174-a31d-a258de2a7b3d
> used request id req-3d35fab2-88b2-46cf-a39a-f8942f3ee1c1
> REQ: curl -g -i -X POST
> http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments
> -H "Accept: application/json" -H "Content-Type: application/json" -H
> "User-Agent: python-novaclient" -H "X-Auth-Token:
> {SHA256}7fa6e3ac57b32176603c393dd97a5ea04813739efb0d1d4bff4f9680ede427c1"
> -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"volumeAttachment":
> {"volumeId": "a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}}'
> Resetting dropped connection: 192.168.110.201
> http://192.168.110.201:8774 "POST
> /v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments
> HTTP/1.1" 200 194
> RESP: [200] Connection: close Content-Length: 194 Content-Type:
> application/json OpenStack-API-Version: compute 2.1 Vary:
> OpenStack-API-Version, X-OpenStack-Nova-API-Version
> X-OpenStack-Nova-API-Version: 2.1 x-compute-request-id:
> req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a x-openstack-request-id:
> req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a
> RESP BODY: {"volumeAttachment": {"id":
> "a9e188a4-7981-4174-a31d-a258de2a7b3d", "serverId":
> "2fd482ac-7a93-4626-955f-77b78f783d54", "volumeId":
> "a9e188a4-7981-4174-a31d-a258de2a7b3d", "device": "/dev/vdb"}}
> POST call to compute for
> http://192.168.110.201:8774/v2.1/servers/2fd482ac-7a93-4626-955f-77b78f783d54/os-volume_attachments
> used request id req-87ba31ae-3511-41be-a2b9-f6a0ccdb092a
> clean_up AddServerVolume:
> END return value: 0
> ```
>
> However, I could not read the logs the way Dmitriy suggested, it simply
> returns
>
> -- Logs begin at Mon 2020-10-05 14:30:22 UTC, end at Mon 2020-10-12
> 11:17:20 UTC. --
> -- No entries --
>
> so I couldn't investigate the request. ( I tried this on both utility
> and cinder api containers on the management node and on the storage and
> respective compute node to be sure)

Hi,

I really don't know anything about those Ansible playbooks, so you'll
have to either check them, check how the container is being run, check
the cinder configuration to see if there's a specific logging option
set, or check the output of the stdout of the container on startup to
see the logging options output by Cinder on startup (on DEBUG mode).


>
> > It's important that the target_ip_address can be accessed from the Nova
> > compute using the interface for the IP defined in my_ip in nova.conf
>
> my_ip in nova.conf seems to be the address of the nova_api container on
> the management node, target_ip_address from cinder.conf seems to be the
> (storage-net) address of the cinder-api container on the management node
> (the value is different from the iscsi_ip_address specified in the
> playbook).

Can you ping from the address defined in my_ip on your compute node to
the target_ip_address being used by Cinder?

If you cannot, then that's the problem.

Cheers,
Gorka.


>
> > IIRC, for lvm storage, cinder-volumes should be launched on every nova
> > compute node in order to attach volumes
> > to instances, since it's not shared storage.
>
> I did not try this yet since the documentation didn't mention it and for
> our current production system (mitaka, which someone else set up ages
> ago) this doesn't seem to be the case but volumes can still be attached.
>
> Kind regards,
> Oliver
>
>