[kolla-ansible] Multipath unsuccesfull in nova
Hello, I have a openstack deployed with kolla-ansible 2024.2. I'm trying to configure cinder and nova to use multipath when attaching volumes from the Hitachi SAN array. Without multipathing everything works good. When I enabled multipath in kolla-ansible (enable_multipathd: "yes"), deployment is without errors, volume can be created, but attachment of volume ends up with error in nova_compute container: ... 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] raise exc_type(*result[2]) 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Command: multipath -C dm-0 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Exit code: 1 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stdout: '' 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stderr: '3601817.750747 | /etc/multipath.conf does not exist, blacklisting all devices.\n3601817.750753 | You can run "/sbin/mpathconf --enable" to create\n3601817.750754 | /etc/multipath.conf. See man mpathconf(8) for more details\n3601817.751393 | sdd: error 2 in pathinfo, discarding path\n3601817.751501 | sde: error 2 in pathinfo, discarding path\n3601817.751504 | 360060e8012b142005040b14200000ee2: removing empty pathgroup 0\n3601817.751505 | dm-0: no usable paths found\n' When I manually added /etc/multipath.conf to nova_compute container, the attachment passed without any issues. I do not want to build special image only because of that single config file. My question is, how to make kolla-ansible to copy/add /etc/multipath.conf file into nova_compute service? Is there any mechanism for that? Thanks. Kamil Madac
Hello Kamil, There’s no testing for the multi path case upstream in Kolla-Ansible, and there’s no functionality at the moment to copy multi path.conf to nova_compute container. Feel free to raise a bug in Launchpad (https://bugs.launchpad.net/kolla-ansible) - but since you’re the best person to fix that in K-A code - we encourage you to contribute that fix upstream. If you would like to discuss that or need any help with that - we’re happy to help on #openstack-kolla on OFTC IRC Best regards, Michal
On 23 Jan 2025, at 09:37, Kamil Madac <kamil.madac@gmail.com> wrote:
Hello,
I have a openstack deployed with kolla-ansible 2024.2. I'm trying to configure cinder and nova to use multipath when attaching volumes from the Hitachi SAN array. Without multipathing everything works good. When I enabled multipath in kolla-ansible (enable_multipathd: "yes"), deployment is without errors, volume can be created, but attachment of volume ends up with error in nova_compute container:
... 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] raise exc_type(*result[2]) 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Command: multipath -C dm-0 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Exit code: 1 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stdout: '' 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stderr: '3601817.750747 | /etc/multipath.conf does not exist, blacklisting all devices.\n3601817.750753 | You can run "/sbin/mpathconf --enable" to create\n3601817.750754 | /etc/multipath.conf. See man mpathconf(8) for more details\n3601817.751393 | sdd: error 2 in pathinfo, discarding path\n3601817.751501 | sde: error 2 in pathinfo, discarding path\n3601817.751504 | 360060e8012b142005040b14200000ee2: removing empty pathgroup 0\n3601817.751505 | dm-0: no usable paths found\n'
When I manually added /etc/multipath.conf to nova_compute container, the attachment passed without any issues. I do not want to build special image only because of that single config file.
My question is, how to make kolla-ansible to copy/add /etc/multipath.conf file into nova_compute service? Is there any mechanism for that?
Thanks.
Kamil Madac
Hi Kamil, In our cluster, we have enabled multipath support and have a separate container which is responsible for handling multipath [1]. Additionally, we need to remember to keep the multipath.conf configuration file inside /etc/kolla/config, and based on that, our configuration is propagated to nova_compute and multipathd containers. Please also check if nova.conf inside /etc/kolla/config contains the parameter volume_use_multipath set to true. I hope this helps you. /Jan Wasilewski References: [1] https://opendev.org/openstack/kolla-ansible/commit/115c55e1fe55598704ddab8e3... czw., 23 sty 2025 o 12:38 Michał Nasiadka <mnasiadka@gmail.com> napisał(a):
Hello Kamil,
There’s no testing for the multi path case upstream in Kolla-Ansible, and there’s no functionality at the moment to copy multi path.conf to nova_compute container. Feel free to raise a bug in Launchpad ( https://bugs.launchpad.net/kolla-ansible) - but since you’re the best person to fix that in K-A code - we encourage you to contribute that fix upstream.
If you would like to discuss that or need any help with that - we’re happy to help on #openstack-kolla on OFTC IRC
Best regards, Michal
On 23 Jan 2025, at 09:37, Kamil Madac <kamil.madac@gmail.com> wrote:
Hello,
I have a openstack deployed with kolla-ansible 2024.2. I'm trying to configure cinder and nova to use multipath when attaching volumes from the Hitachi SAN array. Without multipathing everything works good. When I enabled multipath in kolla-ansible (enable_multipathd: "yes"), deployment is without errors, volume can be created, but attachment of volume ends up with error in nova_compute container:
... 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] raise exc_type(*result[2]) 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Command: multipath -C dm-0 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Exit code: 1 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stdout: '' 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stderr: '3601817.750747 | /etc/multipath.conf does not exist, blacklisting all devices.\n3601817.750753 | You can run "/sbin/mpathconf --enable" to create\n3601817.750754 | /etc/multipath.conf. See man mpathconf(8) for more details\n3601817.751393 | sdd: error 2 in pathinfo, discarding path\n3601817.751501 | sde: error 2 in pathinfo, discarding path\n3601817.751504 | 360060e8012b142005040b14200000ee2: removing empty pathgroup 0\n3601817.751505 | dm-0: no usable paths found\n'
When I manually added /etc/multipath.conf to nova_compute container, the attachment passed without any issues. I do not want to build special image only because of that single config file.
My question is, how to make kolla-ansible to copy/add /etc/multipath.conf file into nova_compute service? Is there any mechanism for that?
Thanks.
Kamil Madac
Hi Jan, Thanks for the info. I enabled multipathd support too and it works for me. But the multipath.conf file was not propagated to nova_compute containers and I did not find any task/config file which should do it. Maybe I somehow missed it, but even Michal Nasiadka wrote that there is no functionality to propage multipath.conf to nova containers. Can you maybe check your installation if there is really /etc/multipath.conf inside nova_compute containers on your compute nodes? What kolla-ansible version are you using and where exactly in /etc/kolla/config do you have your multipath.conf? It would really help me. On Thu, Jan 23, 2025 at 2:18 PM Jan Wasilewski <finarffin@gmail.com> wrote:
Hi Kamil,
In our cluster, we have enabled multipath support and have a separate container which is responsible for handling multipath [1]. Additionally, we need to remember to keep the multipath.conf configuration file inside /etc/kolla/config, and based on that, our configuration is propagated to nova_compute and multipathd containers. Please also check if nova.conf inside /etc/kolla/config contains the parameter volume_use_multipath set to true. I hope this helps you. /Jan Wasilewski
References: [1] https://opendev.org/openstack/kolla-ansible/commit/115c55e1fe55598704ddab8e3...
czw., 23 sty 2025 o 12:38 Michał Nasiadka <mnasiadka@gmail.com> napisał(a):
Hello Kamil,
There’s no testing for the multi path case upstream in Kolla-Ansible, and there’s no functionality at the moment to copy multi path.conf to nova_compute container. Feel free to raise a bug in Launchpad ( https://bugs.launchpad.net/kolla-ansible) - but since you’re the best person to fix that in K-A code - we encourage you to contribute that fix upstream.
If you would like to discuss that or need any help with that - we’re happy to help on #openstack-kolla on OFTC IRC
Best regards, Michal
On 23 Jan 2025, at 09:37, Kamil Madac <kamil.madac@gmail.com> wrote:
Hello,
I have a openstack deployed with kolla-ansible 2024.2. I'm trying to configure cinder and nova to use multipath when attaching volumes from the Hitachi SAN array. Without multipathing everything works good. When I enabled multipath in kolla-ansible (enable_multipathd: "yes"), deployment is without errors, volume can be created, but attachment of volume ends up with error in nova_compute container:
... 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] raise exc_type(*result[2]) 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Command: multipath -C dm-0 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Exit code: 1 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stdout: '' 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stderr: '3601817.750747 | /etc/multipath.conf does not exist, blacklisting all devices.\n3601817.750753 | You can run "/sbin/mpathconf --enable" to create\n3601817.750754 | /etc/multipath.conf. See man mpathconf(8) for more details\n3601817.751393 | sdd: error 2 in pathinfo, discarding path\n3601817.751501 | sde: error 2 in pathinfo, discarding path\n3601817.751504 | 360060e8012b142005040b14200000ee2: removing empty pathgroup 0\n3601817.751505 | dm-0: no usable paths found\n'
When I manually added /etc/multipath.conf to nova_compute container, the attachment passed without any issues. I do not want to build special image only because of that single config file.
My question is, how to make kolla-ansible to copy/add /etc/multipath.conf file into nova_compute service? Is there any mechanism for that?
Thanks.
Kamil Madac
-- Kamil Madac <https://kmadac.github.io/>
Hi Michal, thanks for your response. I patched it locally by adding kolla container config: roles/nova-cell/templates/nova-compute.json.j2 { "source": "{{ container_config_directory }}/multipath.conf", "dest": "/etc/multipath.conf", "owner": "root", "perm": "0600" } and adding task to roles/nova-cell/tasks/config.yml - name: Copying over multipath.conf vars: service: "{{ nova_cell_services['nova-compute'] }}" become: true template: src: "{{ item }}" dest: "{{ node_config_directory }}/nova-compute/multipath.conf" mode: "0660" with_first_found: - "{{ node_custom_config }}/multipath.conf" - "multipath.conf.j2" when: service | service_enabled_and_mapped_to_host notify: - Restart nova-compute container this did the trick and attachment works now. I will try to contribute to upstream. On Thu, Jan 23, 2025 at 12:38 PM Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hello Kamil,
There’s no testing for the multi path case upstream in Kolla-Ansible, and there’s no functionality at the moment to copy multi path.conf to nova_compute container. Feel free to raise a bug in Launchpad ( https://bugs.launchpad.net/kolla-ansible) - but since you’re the best person to fix that in K-A code - we encourage you to contribute that fix upstream.
If you would like to discuss that or need any help with that - we’re happy to help on #openstack-kolla on OFTC IRC
Best regards, Michal
On 23 Jan 2025, at 09:37, Kamil Madac <kamil.madac@gmail.com> wrote:
Hello,
I have a openstack deployed with kolla-ansible 2024.2. I'm trying to configure cinder and nova to use multipath when attaching volumes from the Hitachi SAN array. Without multipathing everything works good. When I enabled multipath in kolla-ansible (enable_multipathd: "yes"), deployment is without errors, volume can be created, but attachment of volume ends up with error in nova_compute container:
... 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] raise exc_type(*result[2]) 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Command: multipath -C dm-0 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Exit code: 1 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stdout: '' 2025-01-22 12:16:02.962 2 ERROR nova.virt.block_device [instance: 714bd6f8-49e5-406f-b608-b7aaded2e728] Stderr: '3601817.750747 | /etc/multipath.conf does not exist, blacklisting all devices.\n3601817.750753 | You can run "/sbin/mpathconf --enable" to create\n3601817.750754 | /etc/multipath.conf. See man mpathconf(8) for more details\n3601817.751393 | sdd: error 2 in pathinfo, discarding path\n3601817.751501 | sde: error 2 in pathinfo, discarding path\n3601817.751504 | 360060e8012b142005040b14200000ee2: removing empty pathgroup 0\n3601817.751505 | dm-0: no usable paths found\n'
When I manually added /etc/multipath.conf to nova_compute container, the attachment passed without any issues. I do not want to build special image only because of that single config file.
My question is, how to make kolla-ansible to copy/add /etc/multipath.conf file into nova_compute service? Is there any mechanism for that?
Thanks.
Kamil Madac
-- Kamil Madac <https://kmadac.github.io/>
participants (3)
-
Jan Wasilewski
-
Kamil Madac
-
Michał Nasiadka