[Ussuri] [Zun] Container deployment problem with Zun "There are not enough hosts available"
From sudo systemctl status zun-compute i see: Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 49, in wrapped_f Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 206, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 247, in get Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise value Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 200, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/compute/compute_node_tracker.py",
Hello, I have an Openstack cluster, with basic services and functionality working based on ussuri release. I am trying to install the Zun service to be able to deploy containers, following [controller] -- https://docs.openstack.org/zun/ussuri/install/controller-install.html and [compute] -- https://docs.openstack.org/zun/ussuri/install/compute-install.html I used the git branch based on ussuri for all components. I veryfined kuryr-libnetwork operation issuing from the compute node # docker network create --driver kuryr --ipam-driver kuryr --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net and seeing the network created successfully, etc. I am not very sure about the zun.conf file. What is the "endpoint_type = internalURL" parameter? Do I need to change internalURL? line 350, in _update_to_placement Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task context, node_rp_uuid, name=compute_node.hostname) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 846, in get_provider_tree_and_ensure_r Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 628, in _ensure_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 514, in _create_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task global_request_id=context.global_id) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 225, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task headers=headers, logger=LOG) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.request(url, 'POST', **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.session.request(url, method, **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise exceptions.from_response(resp, method, url) *Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task keystoneauth1.exceptions.http.Conflict: Conflict (HTTP 409) (Request-ID: req-9a158c41-a485-4937-99e7-e38cdce7fded)* What is this problem? any advice? I used the default configuration values ([keystone_auth] and [keystone_authtoken]) values based on the configuration from the above links. Aslo from the controller *openstack appcontainer service list* +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | Id | Host | Binary | State | Disabled | Disabled Reason | Updated At | Availability Zone | +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | 1 | compute5 | zun-compute | up | False | None | 2021-11-05T14:39:01.000000 | nova | +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ *openstack appcontainer host show compute5* +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | uuid | ee3a5b44-8ffa-463e-939d-0c61868a596f | | links | [{'href': ' http://controller:9517/v1/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'self'}, {'href': ' http://controller:9517/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'bookmark'}] | | hostname | compute5 | | mem_total | 7975 | | mem_used | 0 | | total_containers | 1 | | cpus | 10 | | cpu_used | 0.0 | | architecture | x86_64 | | os_type | linux | | os | Ubuntu 18.04.6 LTS | | kernel_version | 4.15.0-161-generic | | labels | {} | | disk_total | 63 | | disk_used | 0 | | disk_quota_supported | False | | runtimes | ['io.containerd.runc.v2', 'io.containerd.runtime.v1.linux', 'runc'] | | enable_cpu_pinning | False | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ seems to work fine. However when i issue e.g., openstack appcontainer run --name container --net network=$NET_ID cirros ping 8.8.8.8 i get the error: | status_reason | There are not enough hosts available. Any ideas? One final thing is that I did see in the Horizon dashboard the container tab, to be able to deploy containers from horizon. Is there an extra configuration for this? sorry for the long mail. best, Pavlos
Hi, It looks zun-compute wants to create a resource provider in placement, but placement return a 409 response. I would suggest you to check placement's logs. My best guess is the resource provider with the same name is already created so placement returned 409. If this is a case, simply remove those resources and restart zun-compute service should resolve the problem. Best regards, Hongbin At 2021-11-05 22:43:38, "Pavlos Basaras" <pbasaras@gmail.com> wrote: Hello, I have an Openstack cluster, with basic services and functionality working based on ussuri release. I am trying to install the Zun service to be able to deploy containers, following [controller] -- https://docs.openstack.org/zun/ussuri/install/controller-install.html and [compute] -- https://docs.openstack.org/zun/ussuri/install/compute-install.html I used the git branch based on ussuri for all components. I veryfined kuryr-libnetwork operation issuing from the compute node # docker network create --driver kuryr --ipam-driver kuryr --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net and seeing the network created successfully, etc. I am not very sure about the zun.conf file. What is the "endpoint_type = internalURL" parameter? Do I need to change internalURL? From sudo systemctl status zun-compute i see: Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 49, in wrapped_f Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 206, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 247, in get Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise value Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 200, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/compute/compute_node_tracker.py", line 350, in _update_to_placement Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task context, node_rp_uuid, name=compute_node.hostname) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 846, in get_provider_tree_and_ensure_r Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 628, in _ensure_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 514, in _create_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task global_request_id=context.global_id) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 225, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task headers=headers, logger=LOG) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.request(url, 'POST', **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.session.request(url, method, **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise exceptions.from_response(resp, method, url) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task keystoneauth1.exceptions.http.Conflict: Conflict (HTTP 409) (Request-ID: req-9a158c41-a485-4937-99e7-e38cdce7fded) What is this problem? any advice? I used the default configuration values ([keystone_auth] and [keystone_authtoken]) values based on the configuration from the above links. Aslo from the controller openstack appcontainer service list +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | Id | Host | Binary | State | Disabled | Disabled Reason | Updated At | Availability Zone | +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | 1 | compute5 | zun-compute | up | False | None | 2021-11-05T14:39:01.000000 | nova | +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ openstack appcontainer host show compute5 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | uuid | ee3a5b44-8ffa-463e-939d-0c61868a596f | | links | [{'href': 'http://controller:9517/v1/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'self'}, {'href': 'http://controller:9517/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'bookmark'}] | | hostname | compute5 | | mem_total | 7975 | | mem_used | 0 | | total_containers | 1 | | cpus | 10 | | cpu_used | 0.0 | | architecture | x86_64 | | os_type | linux | | os | Ubuntu 18.04.6 LTS | | kernel_version | 4.15.0-161-generic | | labels | {} | | disk_total | 63 | | disk_used | 0 | | disk_quota_supported | False | | runtimes | ['io.containerd.runc.v2', 'io.containerd.runtime.v1.linux', 'runc'] | | enable_cpu_pinning | False | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ seems to work fine. However when i issue e.g., openstack appcontainer run --name container --net network=$NET_ID cirros ping 8.8.8.8 i get the error: | status_reason | There are not enough hosts available. Any ideas? One final thing is that I did see in the Horizon dashboard the container tab, to be able to deploy containers from horizon. Is there an extra configuration for this? sorry for the long mail. best, Pavlos
Hello, you are right, that fixed the problem. In order to solve the problem i revisited https://docs.openstack.org/placement/ussuri/install/verify.html I executed: openstack resource provider list Then removed the host that i use for containers, restarted the zun-compute at the host and works perfectly. Thank you very much for your input. One more thing, I don't see at the horizon dashboard the tab for the containers (i just see the nova compute related tab). Is there any additional configuration for this? btw, from https://docs.openstack.org/placement/ussuri/install/verify.html, i used pip3 install osc-placement (instead of pip install..) all the best Pavlos. On Sat, Nov 6, 2021 at 10:24 AM Hongbin Lu <kira034@163.com> wrote:
Hi,
It looks zun-compute wants to create a resource provider in placement, but placement return a 409 response. I would suggest you to check placement's logs. My best guess is the resource provider with the same name is already created so placement returned 409. If this is a case, simply remove those resources and restart zun-compute service should resolve the problem.
Best regards, Hongbin
At 2021-11-05 22:43:38, "Pavlos Basaras" <pbasaras@gmail.com> wrote:
Hello,
I have an Openstack cluster, with basic services and functionality working based on ussuri release.
I am trying to install the Zun service to be able to deploy containers, following
[controller] -- https://docs.openstack.org/zun/ussuri/install/controller-install.html and [compute] -- https://docs.openstack.org/zun/ussuri/install/compute-install.html
I used the git branch based on ussuri for all components.
I veryfined kuryr-libnetwork operation issuing from the compute node # docker network create --driver kuryr --ipam-driver kuryr --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net
and seeing the network created successfully, etc.
I am not very sure about the zun.conf file. What is the "endpoint_type = internalURL" parameter? Do I need to change internalURL?
From sudo systemctl status zun-compute i see: Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 49, in wrapped_f Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 206, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 247, in get Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise value Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 200, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/compute/compute_node_tracker.py", line 350, in _update_to_placement Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task context, node_rp_uuid, name=compute_node.hostname) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 846, in get_provider_tree_and_ensure_r Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 628, in _ensure_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 514, in _create_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task global_request_id=context.global_id) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 225, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task headers=headers, logger=LOG) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.request(url, 'POST', **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.session.request(url, method, **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise exceptions.from_response(resp, method, url) *Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task keystoneauth1.exceptions.http.Conflict: Conflict (HTTP 409) (Request-ID: req-9a158c41-a485-4937-99e7-e38cdce7fded)*
What is this problem? any advice? I used the default configuration values ([keystone_auth] and [keystone_authtoken]) values based on the configuration from the above links.
Aslo from the controller
*openstack appcontainer service list* +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | Id | Host | Binary | State | Disabled | Disabled Reason | Updated At | Availability Zone |
+----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | 1 | compute5 | zun-compute | up | False | None | 2021-11-05T14:39:01.000000 | nova |
+----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+
*openstack appcontainer host show compute5* +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value
|
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | uuid | ee3a5b44-8ffa-463e-939d-0c61868a596f
| | links | [{'href': ' http://controller:9517/v1/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'self'}, {'href': ' http://controller:9517/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'bookmark'}] | | hostname | compute5
| | mem_total | 7975
| | mem_used | 0
| | total_containers | 1
| | cpus | 10
| | cpu_used | 0.0
| | architecture | x86_64
| | os_type | linux
| | os | Ubuntu 18.04.6 LTS
| | kernel_version | 4.15.0-161-generic
| | labels | {}
| | disk_total | 63
| | disk_used | 0
| | disk_quota_supported | False
| | runtimes | ['io.containerd.runc.v2', 'io.containerd.runtime.v1.linux', 'runc']
| | enable_cpu_pinning | False
|
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
seems to work fine.
However when i issue e.g., openstack appcontainer run --name container --net network=$NET_ID cirros ping 8.8.8.8
i get the error: | status_reason | There are not enough hosts available.
Any ideas?
One final thing is that I did see in the Horizon dashboard the container tab, to be able to deploy containers from horizon. Is there an extra configuration for this?
sorry for the long mail.
best, Pavlos
Sorry for late reply. I missed this email. This is the Zun UI installation guide: https://docs.openstack.org/zun-ui/latest/install/index.html#manual-installat... . About your suggest of pip3 installation, I will add a note to clarify that. Thanks for pointing it out. Best regards, Hongbin At 2021-11-08 19:02:50, "Pavlos Basaras" <pbasaras@gmail.com> wrote: Hello, you are right, that fixed the problem. In order to solve the problem i revisited https://docs.openstack.org/placement/ussuri/install/verify.html I executed: openstack resource provider list Then removed the host that i use for containers, restarted the zun-compute at the host and works perfectly. Thank you very much for your input. One more thing, I don't see at the horizon dashboard the tab for the containers (i just see the nova compute related tab). Is there any additional configuration for this? btw, from https://docs.openstack.org/placement/ussuri/install/verify.html, i used pip3 install osc-placement (instead of pip install..) all the best Pavlos. On Sat, Nov 6, 2021 at 10:24 AM Hongbin Lu <kira034@163.com> wrote: Hi, It looks zun-compute wants to create a resource provider in placement, but placement return a 409 response. I would suggest you to check placement's logs. My best guess is the resource provider with the same name is already created so placement returned 409. If this is a case, simply remove those resources and restart zun-compute service should resolve the problem. Best regards, Hongbin At 2021-11-05 22:43:38, "Pavlos Basaras" <pbasaras@gmail.com> wrote: Hello, I have an Openstack cluster, with basic services and functionality working based on ussuri release. I am trying to install the Zun service to be able to deploy containers, following [controller] -- https://docs.openstack.org/zun/ussuri/install/controller-install.html and [compute] -- https://docs.openstack.org/zun/ussuri/install/compute-install.html I used the git branch based on ussuri for all components. I veryfined kuryr-libnetwork operation issuing from the compute node # docker network create --driver kuryr --ipam-driver kuryr --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net and seeing the network created successfully, etc. I am not very sure about the zun.conf file. What is the "endpoint_type = internalURL" parameter? Do I need to change internalURL? From sudo systemctl status zun-compute i see: Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 49, in wrapped_f Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 206, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 247, in get Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise value Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 200, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/compute/compute_node_tracker.py", line 350, in _update_to_placement Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task context, node_rp_uuid, name=compute_node.hostname) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 846, in get_provider_tree_and_ensure_r Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 628, in _ensure_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 514, in _create_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task global_request_id=context.global_id) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 225, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task headers=headers, logger=LOG) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.request(url, 'POST', **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.session.request(url, method, **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise exceptions.from_response(resp, method, url) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task keystoneauth1.exceptions.http.Conflict: Conflict (HTTP 409) (Request-ID: req-9a158c41-a485-4937-99e7-e38cdce7fded) What is this problem? any advice? I used the default configuration values ([keystone_auth] and [keystone_authtoken]) values based on the configuration from the above links. Aslo from the controller openstack appcontainer service list +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | Id | Host | Binary | State | Disabled | Disabled Reason | Updated At | Availability Zone | +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | 1 | compute5 | zun-compute | up | False | None | 2021-11-05T14:39:01.000000 | nova | +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ openstack appcontainer host show compute5 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | uuid | ee3a5b44-8ffa-463e-939d-0c61868a596f | | links | [{'href': 'http://controller:9517/v1/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'self'}, {'href': 'http://controller:9517/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'bookmark'}] | | hostname | compute5 | | mem_total | 7975 | | mem_used | 0 | | total_containers | 1 | | cpus | 10 | | cpu_used | 0.0 | | architecture | x86_64 | | os_type | linux | | os | Ubuntu 18.04.6 LTS | | kernel_version | 4.15.0-161-generic | | labels | {} | | disk_total | 63 | | disk_used | 0 | | disk_quota_supported | False | | runtimes | ['io.containerd.runc.v2', 'io.containerd.runtime.v1.linux', 'runc'] | | enable_cpu_pinning | False | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ seems to work fine. However when i issue e.g., openstack appcontainer run --name container --net network=$NET_ID cirros ping 8.8.8.8 i get the error: | status_reason | There are not enough hosts available. Any ideas? One final thing is that I did see in the Horizon dashboard the container tab, to be able to deploy containers from horizon. Is there an extra configuration for this? sorry for the long mail. best, Pavlos
Hello, no worries, million thanks for the support and the link for the UI. One more thing to point out is that in ( https://docs.openstack.org/zun/xena/install/compute-install.html) at step 9, the curl command is fixed to amd64 architecture --since i have an aarch64 system i fetched the appropriate version from https://github.com/containernetworking/plugins/releases/, and worked fine. best, Pavlos. On Thu, Nov 11, 2021 at 9:00 AM Hongbin Lu <kira034@163.com> wrote:
Sorry for late reply. I missed this email.
This is the Zun UI installation guide: https://docs.openstack.org/zun-ui/latest/install/index.html#manual-installat... . About your suggest of pip3 installation, I will add a note to clarify that. Thanks for pointing it out.
Best regards, Hongbin
At 2021-11-08 19:02:50, "Pavlos Basaras" <pbasaras@gmail.com> wrote:
Hello,
you are right, that fixed the problem.
In order to solve the problem i revisited https://docs.openstack.org/placement/ussuri/install/verify.html I executed: openstack resource provider list Then removed the host that i use for containers, restarted the zun-compute at the host and works perfectly. Thank you very much for your input.
One more thing, I don't see at the horizon dashboard the tab for the containers (i just see the nova compute related tab). Is there any additional configuration for this?
btw, from https://docs.openstack.org/placement/ussuri/install/verify.html, i used pip3 install osc-placement (instead of pip install..)
all the best Pavlos.
On Sat, Nov 6, 2021 at 10:24 AM Hongbin Lu <kira034@163.com> wrote:
Hi,
It looks zun-compute wants to create a resource provider in placement, but placement return a 409 response. I would suggest you to check placement's logs. My best guess is the resource provider with the same name is already created so placement returned 409. If this is a case, simply remove those resources and restart zun-compute service should resolve the problem.
Best regards, Hongbin
At 2021-11-05 22:43:38, "Pavlos Basaras" <pbasaras@gmail.com> wrote:
Hello,
I have an Openstack cluster, with basic services and functionality working based on ussuri release.
I am trying to install the Zun service to be able to deploy containers, following
[controller] -- https://docs.openstack.org/zun/ussuri/install/controller-install.html and [compute] -- https://docs.openstack.org/zun/ussuri/install/compute-install.html
I used the git branch based on ussuri for all components.
I veryfined kuryr-libnetwork operation issuing from the compute node # docker network create --driver kuryr --ipam-driver kuryr --subnet 10.10.0.0/16 --gateway=10.10.0.1 test_net
and seeing the network created successfully, etc.
I am not very sure about the zun.conf file. What is the "endpoint_type = internalURL" parameter? Do I need to change internalURL?
From sudo systemctl status zun-compute i see: Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 49, in wrapped_f Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 206, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 247, in get Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2]) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise value Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 200, in call Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/compute/compute_node_tracker.py", line 350, in _update_to_placement Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task context, node_rp_uuid, name=compute_node.hostname) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 846, in get_provider_tree_and_ensure_r Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 628, in _ensure_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task parent_provider_uuid=parent_provider_uuid) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 514, in _create_resource_provider Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task global_request_id=context.global_id) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/local/lib/python3.6/dist-packages/zun/scheduler/client/report.py", line 225, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task headers=headers, logger=LOG) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.request(url, 'POST', **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task return self.session.request(url, method, **kwargs) Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task raise exceptions.from_response(resp, method, url) *Nov 05 14:34:58 compute5 zun-compute[8962]: 2021-11-05 14:34:58.828 8962 ERROR oslo_service.periodic_task keystoneauth1.exceptions.http.Conflict: Conflict (HTTP 409) (Request-ID: req-9a158c41-a485-4937-99e7-e38cdce7fded)*
What is this problem? any advice? I used the default configuration values ([keystone_auth] and [keystone_authtoken]) values based on the configuration from the above links.
Aslo from the controller
*openstack appcontainer service list* +----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | Id | Host | Binary | State | Disabled | Disabled Reason | Updated At | Availability Zone |
+----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+ | 1 | compute5 | zun-compute | up | False | None | 2021-11-05T14:39:01.000000 | nova |
+----+----------+-------------+-------+----------+-----------------+----------------------------+-------------------+
*openstack appcontainer host show compute5* +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value
|
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | uuid | ee3a5b44-8ffa-463e-939d-0c61868a596f
| | links | [{'href': ' http://controller:9517/v1/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'self'}, {'href': ' http://controller:9517/hosts/ee3a5b44-8ffa-463e-939d-0c61868a596f', 'rel': 'bookmark'}] | | hostname | compute5
| | mem_total | 7975
| | mem_used | 0
| | total_containers | 1
| | cpus | 10
| | cpu_used | 0.0
| | architecture | x86_64
| | os_type | linux
| | os | Ubuntu 18.04.6 LTS
| | kernel_version | 4.15.0-161-generic
| | labels | {}
| | disk_total | 63
| | disk_used | 0
| | disk_quota_supported | False
| | runtimes | ['io.containerd.runc.v2', 'io.containerd.runtime.v1.linux', 'runc']
| | enable_cpu_pinning | False
|
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
seems to work fine.
However when i issue e.g., openstack appcontainer run --name container --net network=$NET_ID cirros ping 8.8.8.8
i get the error: | status_reason | There are not enough hosts available.
Any ideas?
One final thing is that I did see in the Horizon dashboard the container tab, to be able to deploy containers from horizon. Is there an extra configuration for this?
sorry for the long mail.
best, Pavlos
participants (2)
-
Hongbin Lu
-
Pavlos Basaras