Hi everyone, I have a fresh Openstack installation and i need to use LXC to launch instances I made the deployment manually, currently i have one controller node, one compute node, one block storage node and one object storage node. I already made the necessary configurations in the nova.conf file as appears on the official documentation site compute_driver = libvirt.LibvirtDriver [libvirt]virt_type = lxc and i have installed the nova-compute-lxc package and the nbd-server package. i created a new ubuntu 18 image for the container with the following command: openstack image create --public --container-format=bare --disk-format=raw --file bionic-server-cloudimg-amd64.tar.gz ubuntu18 and the flavor: openstack flavor create --ram 2048 --vcpus 1 --disk 5 lxc.test then when i try to launch an instance: openstack server create --flavor lxc.test --image ubuntu18 --nic net-id=d0f6cc99-565b-484f-a5ab-fcb1c3af456d --security-group 1160755e-1312-43c6-8ebb-b12b92d35e51 --key-name mykey lxc6 I get the following Error cat /var/log/nova/nova-*.log | grep 269d2480-4477-4382-b09c-9c686ab7a17c 2019-04-23 17:59:26.846 3949 INFO nova.osapi_compute.wsgi.server [req-1dbeb432-67f0-4305-b91c-6548dde49bee a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 - default default] 10.8.9.54 "GET /v2.1/servers/269d2480-4477-4382-b09c-9c686ab7a17c HTTP/1.1" status: 200 len: 1737 time: 0.6656921 2019-04-23 17:59:33.076 3944 INFO nova.api.openstack.compute.server_external_events [req-206e8ace-90e3-4a95-8274-4a9091c174c0 c2657fa5083f4d5c932d53d6bf094604 18337c70d397424f8895cd4f1ae7450e - default default] Creating event network-changed:818d6adc-252e-4d47-9275-009cff55c716 for instance 269d2480-4477-4382-b09c-9c686ab7a17c on compute 2019-04-23 17:59:37.663 3926 *ERROR* nova.scheduler.utils [req-abd32ef2-3a19-4154-a7da-893f3d2b2aef a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 - default default] [instance: 269d2480-4477-4382-b09c-9c686ab7a17c] Error from last host: compute (node compute.cujae.edu.cu): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1841, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2121, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 269d2480-4477-4382-b09c-9c686ab7a17c was re-scheduled: mount() takes exactly 4 arguments (3 given)\n'] Is this error a bug or something fixable?? Has anyone accomplished this kind of environment or used lxc without problems?
Hi everyone,
I have a fresh Openstack installation and i need to use LXC to launch instances I made the deployment manually, currently i have one controller node, one compute node, one block storage node and one object storage node.
I already made the necessary configurations in the nova.conf file as appears on the official documentation site
compute_driver = libvirt.LibvirtDriver [libvirt]virt_type = lxc
and i have installed the nova-compute-lxc package and the nbd-server package.
i created a new ubuntu 18 image for the container with the following command:
openstack image create --public --container-format=bare --disk-format=raw --file bionic-server-cloudimg-amd64.tar.gz ubuntu18
and the flavor:
openstack flavor create --ram 2048 --vcpus 1 --disk 5 lxc.test
then when i try to launch an instance:
openstack server create --flavor lxc.test --image ubuntu18 --nic net-id=d0f6cc99-565b-484f-a5ab-fcb1c3af456d --security-group 1160755e-1312-43c6-8ebb-b12b92d35e51 --key-name mykey lxc6
I get the following Error
cat /var/log/nova/nova-*.log | grep 269d2480-4477-4382-b09c-9c686ab7a17c
2019-04-23 17:59:26.846 3949 INFO nova.osapi_compute.wsgi.server [req-1dbeb432-67f0-4305-b91c-6548dde49bee a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 - default default] 10.8.9.54 "GET /v2.1/servers/269d2480-4477-4382-b09c-9c686ab7a17c HTTP/1.1" status: 200 len: 1737 time: 0.6656921
2019-04-23 17:59:33.076 3944 INFO nova.api.openstack.compute.server_external_events [req-206e8ace-90e3-4a95-8274-4a9091c174c0 c2657fa5083f4d5c932d53d6bf094604 18337c70d397424f8895cd4f1ae7450e - default default] Creating event network-changed:818d6adc-252e-4d47-9275-009cff55c716 for instance 269d2480-4477-4382-b09c-9c686ab7a17c on compute
2019-04-23 17:59:37.663 3926 *ERROR* nova.scheduler.utils [req-abd32ef2-3a19-4154-a7da-893f3d2b2aef a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 - default default] [instance: 269d2480-4477-4382-b09c-9c686ab7a17c] Error from last host: compute (node compute.cujae.edu.cu): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1841, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2121, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 269d2480-4477-4382-b09c-9c686ab7a17c was re-scheduled: mount() takes exactly 4 arguments (3 given)\n']
On Tue, 2019-04-23 at 14:29 -0400, Alejandro Ruiz Bermejo wrote: the libvirt lxc dirver is not tested upstream in the gate so this could be a latent bug. its hard to tell but it might be this mount function https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0... being called here https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0... where only 3 args are passed instead of the 4 that are needed the options arg is missing. but if thats the issue its been broken for years. since https://github.com/openstack/nova/commit/7ad72b092201f87530674a245e2904c6401...
Is this error a bug or something fixable?? Has anyone accomplished this kind of environment or used lxc without problems?
On 4/23/2019 2:04 PM, Sean Mooney wrote:
2019-04-23 17:59:37.663 3926*ERROR* nova.scheduler.utils [req-abd32ef2-3a19-4154-a7da-893f3d2b2aef a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 - default default] [instance: 269d2480-4477-4382-b09c-9c686ab7a17c] Error from last host: compute (node compute.cujae.edu.cu): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1841, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2121, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 269d2480-4477-4382-b09c-9c686ab7a17c was re-scheduled: mount() takes exactly 4 arguments (3 given)\n'] the libvirt lxc dirver is not tested upstream in the gate so this could be a latent bug. its hard to tell but it might be this mount function https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0...
being called here https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0... where only 3 args are passed instead of the 4 that are needed the options arg is missing. but if thats the issue its been broken for years. sincehttps://github.com/openstack/nova/commit/7ad72b092201f87530674a245e2904c6401...
Yup it's a latent bug. Feel free to report a bug and post a patch, but you're likely building your house on sand because as noted the libvirt+lxc driver doesn't get integration testing. I used to try to make a CI job work for that configuration but it was extremely flaky and it was never sorted out what the underlying issues were (and no one cared enough to take it up). -- Thanks, Matt
On Tue, 2019-04-23 at 16:50 -0500, Matt Riedemann wrote:
On 4/23/2019 2:04 PM, Sean Mooney wrote:
2019-04-23 17:59:37.663 3926*ERROR* nova.scheduler.utils [req-abd32ef2-3a19-4154-a7da-893f3d2b2aef a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 - default default] [instance: 269d2480-4477-4382-b09c-9c686ab7a17c] Error from last host: compute (node compute.cujae.edu.cu): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1841, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2121, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 269d2480-4477-4382-b09c-9c686ab7a17c was re-scheduled: mount() takes exactly 4 arguments (3 given)\n']
the libvirt lxc dirver is not tested upstream in the gate so this could be a latent bug. its hard to tell but it might be this mount function https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0...
being called here
https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0...
where only 3 args are passed instead of the 4 that are needed the options arg is missing. but if thats the issue its been broken for years. sincehttps://github.com/openstack/nova/commit/7ad72b092201f87530674a245e2904c6401...
Yup it's a latent bug. Feel free to report a bug and post a patch, but you're likely building your house on sand because as noted the libvirt+lxc driver doesn't get integration testing. i think the lxc backend was original a focus of canonical but they pivoted to nova lxd instead. if you want lxc container using nova-lxd it probably better tested but its also not in the gate. I used to try to make a CI job work for that configuration but it was extremely flaky and it was never sorted out what the underlying issues were (and no one cared enough to take it up). i can proably try to fix this issue and if i figure out how to deploy and test it with devstack i might be able to get a gate job working but is this something peopel want fixed? from what i can tell this looks like its been busted for 2 years so it seams like this does not get used by anyone who is active upstream at least.
the fix iteself whoudl eb relitivly simple but the fact this was not caguht by any of your unit tests indicate we likely dont have any for this code or it not asserting the mocks are called correctly. i might try following https://docs.openstack.org/devstack/latest/guides/lxc.html but that likely is jsut as well maintined and tested...
On Tue, 2019-04-23 at 23:27 +0100, Sean Mooney wrote:
On Tue, 2019-04-23 at 16:50 -0500, Matt Riedemann wrote:
On 4/23/2019 2:04 PM, Sean Mooney wrote:
2019-04-23 17:59:37.663 3926*ERROR* nova.scheduler.utils [req-abd32ef2-3a19-4154-a7da-893f3d2b2aef a16c6ef0319b4643a4ec8e56a1d025cb 59065d8f970b467aa94ef7b35f1edab5 - default default] [instance: 269d2480-4477-4382-b09c-9c686ab7a17c] Error from last host: compute (node compute.cujae.edu.cu): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1841, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2121, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 269d2480-4477-4382-b09c-9c686ab7a17c was re-scheduled: mount() takes exactly 4 arguments (3 given)\n']
the libvirt lxc dirver is not tested upstream in the gate so this could be a latent bug. its hard to tell but it might be this mount function https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0...
being called here
https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0...
where only 3 args are passed instead of the 4 that are needed the options arg is missing. but if thats the issue its been broken for years. sincehttps://github.com/openstack/nova/commit/7ad72b092201f87530674a245e2904c6401...
Yup it's a latent bug. Feel free to report a bug and post a patch, but you're likely building your house on sand because as noted the libvirt+lxc driver doesn't get integration testing.
i think the lxc backend was original a focus of canonical but they pivoted to nova lxd instead. if you want lxc container using nova-lxd it probably better tested but its also not in the gate.
I used to try to make a CI job work for that configuration but it was extremely flaky and it was never sorted out what the underlying issues were (and no one cared enough to take it up).
i can proably try to fix this issue and if i figure out how to deploy and test it with devstack i might be able to get a gate job working but is this something peopel want fixed? from what i can tell this looks like its been busted for 2 years so it seams like this does not get used by anyone who is active upstream at least.
the fix iteself whoudl eb relitivly simple but the fact this was not caguht by any of your unit tests indicate we likely dont have any for this code or it not asserting the mocks are called correctly.
i might try following https://docs.openstack.org/devstack/latest/guides/lxc.html but that likely is jsut as well maintined and tested... oh thats devstack in lxc but looking at https://docs.openstack.org/ocata/config-reference/compute/hypervisor-lxc.htm... its faily simple i just need to get a valid lxc compatibale image to use with it once deployed.
participants (3)
-
Alejandro Ruiz Bermejo
-
Matt Riedemann
-
Sean Mooney