March 2023 - openstack-discuss - lists.openstack.org

[manila] ha for share server
by garcetto 06 Mar '23

06 Mar '23

good afternoon, is it possible to have HA for share server vm in openstack? i mean, the vm that is created on every tenant and used as nfs server. thank you

2 1

[manila] share server with multiple tenant networks
by garcetto 06 Mar '23

06 Mar '23

good afternoon, i am trying to add a second share server or share to existing share server on different network inside same tenant, actually i have: tenant-net-01 (with the share server), ok working. tenant-net-02 (tried to add a share network, but the error says " create: Could not find an existing share server or allocate one on the share network provided. You may use a different share network, or verify the network details in the share network and retry your request. If this doesn't work, contact your administrator to troubleshoot issues with your network. " any clue or doc i can read? thank you

2 1

[ironic] PTL Availability
by Jay Faulkner 06 Mar '23

06 Mar '23

Hi all, Just a heads up -- I'll be out of town and mostly out of IRC from Wednesday, 3/8 to Monday 3/13. If there are emergent issues that need to be addressed urgently, please send an email directly to me and I can have a look. Alternatively, I have a large amount of trust in our delegated release managers and former Ironic PTLs and support them if they need to take action in my stead. - Jay Faulkenr

1 0

[openstack-helm] Get rid of cephfs and rbd provisioners
by Vladimir Kozhukalov 06 Mar '23

06 Mar '23

Hi everyone, I would like to suggest getting rid of cephfs and rbd provisioners. They have been retired and have not been maintained for about 2.5 years now [1]. I believe the CSI approach is what all users rely on nowadays and we can safely remove them. The trigger for this suggestion is that we are currently experiencing issues while trying to switch cephfs provisioner to Ubuntu Focal and fixing this is just wasting time. [2] Stephen spent some time debugging the issues and can give more details if needed. What do you think? [1] https://github.com/kubernetes-retired/external-storage/tree/master/ceph [2] https://review.opendev.org/c/openstack/openstack-helm-infra/+/872976 -- Best regards, Kozhukalov Vladimir

3 3

Paying for Openstack support
by Albert Braden 06 Mar '23

06 Mar '23

I have a question for the operators here. Is anyone paying for Openstack support and getting good value for your money? Can you contact someone for help with an issue, and get a useful response in a reasonable time? If you have an emergency, can you get help quickly? If so, I would like to hear about your experience. Who are you getting good support from? Do they support your operating system too? If not, where do you get your OS support, and how good is it? If you work for a company that provides openstack and/or Linux support, you are welcome to send me a sales pitch, but my goal is to hear from operators.

5 5

[neutron] CI meeting Tuesday 7.03.2023 cancelled
by Slawek Kaplonski 06 Mar '23

06 Mar '23

Hi, Due to some internal event which I have tomorrow in the same time as our CI meeting I will not be able to run the CI meeting. I don't see any really serious issues in our CI this week and after discussing that with Rodolfo we decided to cancel this week's CI meeting. See You on the meeting next week. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[cinder] Error when creating backups from iscsi volume
by Rishat Azizov 06 Mar '23

06 Mar '23

Hello! We have an error with creating backups from iscsi volume. Usually, this happens with large backups over 100GB. 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server [req-f6619913-6f96-4226-8d75-2da3fca722f1 23de1b92e7674cf59486f07ac75b886b a7585b47d1f143e9839c49b4e3bbe1b4 - - -] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: multipath -f 3624a93705842cfae35d7483200015ec6 Exit code: 1 Stdout: '' Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath device\n' 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 890, in wrapper 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 410, in create_backup 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err)) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server raise self.value 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 399, in create_backup 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 493, in _run_backup 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server ignore_errors=True) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 1066, in _detach_device 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server force=force, ignore_errors=ignore_errors) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 141, in trace_logging_wrapper 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 880, in disconnect_volume 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server is_disconnect_call=True) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 942, in _cleanup_connection 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server self._linuxscsi.flush_multipath_device(multipath_name) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/initiator/linuxscsi.py", line 382, in flush_multipath_device 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server root_helper=self._root_helper) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in _execute 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server raise exc_type(*result[2]) 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Command: multipath -f 3624a93705842cfae35d7483200015ec6 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Exit code: 1 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stdout: '' 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Stderr: 'Feb 16 00:22:45 | 3624a93705842cfae35d7483200015ec6 is not a multipath device\n' 2023-02-16 00:23:25.298 1920 ERROR oslo_messaging.rpc.server Could you please help with this error?

2 1

[neutron] Bug deputy report (week 09, starting on Feb-27-2022)
by Lajos Katona 06 Mar '23

06 Mar '23

Hi neutrinos, Here are the bugs reported between February 27 and March 05. *First the unassigned bugs:* * https://bugs.launchpad.net/neutron/+bug/2009043 neutron-l3-agent restart some random ha routers get wrong state *High prio bugs* * https://bugs.launchpad.net/neutron/+bug/2008712 Security group rule deleted by cascade (because its remote group had been deleted) is not deleted in the backend * https://bugs.launchpad.net/neutron/+bug/2008947 Investigate test_restart_rpc_on_sighup_multiple_workers failure * https://bugs.launchpad.net/neutron/+bug/2009055 Performance issue when creating lots of ports * https://bugs.launchpad.net/neutron/+bug/2009215 [OVS] Error during OVS agent start *Medium or lower:* * https://bugs.launchpad.net/neutron/+bug/2008858 Call the api and do not return for a long time * https://bugs.launchpad.net/neutron/+bug/2008912 "_validate_create_network_callback" failing with 'NoneType' object has no attribute 'qos_policy_id' * https://bugs.launchpad.net/neutron/+bug/2008943 OVN DB Sync utility cannot find NB DB Port Group * https://bugs.launchpad.net/neutron/+bug/2009053 OVN: default stateless SG blocks metadata traffic * https://bugs.launchpad.net/neutron/+bug/2009221 [OVS] Custom ethertype traffic is not coming into the VM *One incomplete:* * https://bugs.launchpad.net/neutron/+bug/2008808 Duplicate packet when ping to external with out floating ip *Already merged / released bugs:* * https://bugs.launchpad.net/neutron/+bug/2008695 Remove any LB HM references from the external_ids upon deleting an HM * https://bugs.launchpad.net/neutron/+bug/2008767 [sqlalchemy-20][vnpaas] SQL execution without transaction in progress Have a bugless week :-) Lajos (lajoskatona)

1 0

[kolla] Weekly meeting on 8th March cancelled
by Michał Nasiadka 06 Mar '23

06 Mar '23

Hola Koalas, Weekly meeting on Wed this week is cancelled - I’m off on vacation. See you next week! Best regards, Michal

1 0

[Openstack Trove] Fails to create DB container - "context deadline exceeded", even though images can be pulled manually
by ddorra＠t-online.de 05 Mar '23

05 Mar '23

Hi, my Trove service installed into a Openstack Victoria fails in starting DB instances: 2023-03-05 12:06:17.172 1015 INFO trove.guestagent.datastore.mysql_common.service [-] Starting docker container, image: mysql:5.7.29 2023-03-05 12:06:17.174 1015 WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service [-] Failed to start mysql: docker.errors.APIError: 500 Server Error: Internal Server Error ("Get "https://registry-1.docker.io/v2/": context deadline exceeded") It sounds that it's unable to connect to Docker in order to download the image. However, after logging in to the DB instance I am able to pull docker images manually, or even a cloudinit action can successfully pull images during instance creation... root@t1:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE mysql 5.7.29 5d9483f9a7b2 2 years ago 455MB root@t1:~# <mailto:root@t1:~#> The cloudinit I need to adjust the default GW in oder to reach internet. I hope that is early enough in the boot process and doesn't hamper the guestagent. Any hint what I can do? ============= Trove cloudinit root@voscontrol:~# cat /etc/trove/cloudinit/mysql.cloudinit #cloud-config write_files: - content: | ... path: /etc/ssh/sshd_config - content: | ... path: /root/.bash_aliases runcmd: - [ ip, route, delete, default, via, 10.0.0.79 ] - [ ip, route, add, default, via, 10.0.0.62 ] - [ service, ssh, restart ] - [ docker, pull, "mysql:5.7.29" ] root@voscontrol:~# ================ guest config root@t1:/etc/trove# cat ./conf.d/guest_info.conf [DEFAULT] guest_id=291070bc-671d-437a-a68c-9cee840e614c datastore_manager=mysql datastore_version=5.7.29 tenant_id=606b291caeab4dc2bb1072e2e43b082a instance_rpc_encr_key=EEKenGHSe4exOOU8a0ivV2aR8FwOr2yw root@t1:/etc/trove# cat ./conf.d/trove-guestagent.conf [DEFAULT] log_file = trove-guestagent.log log_dir = /var/log/trove/ ignore_users = os_admin control_exchange = trove #transport_url = rabbit://openstack:pass123@192.168.100.79:5672/ transport_url = rabbit://openstack:pass123@10.0.0.79:5672/ rpc_backend = rabbit command_process_timeout = 60 use_syslog = False debug = True [service_credentials] #auth_url = http://192.168.100.79:5000/v3 auth_url = http://10.0.0.79:5000/v3 region_name = RegionOne project_name = service password = pass123 project_domain_name = Default user_domain_name = Default username = trove root@t1:/etc/trove# ================ /var/log/trove/trove-guestagent.log ... 2023-03-05 12:06:17.162 1015 DEBUG oslo_concurrency.processutils [-] CMD "sudo mkdir -p /var/run/mysqld" returned: 0 in 0.009s execute /opt/guest-agent-venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 2023-03-05 12:06:17.162 1015 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): sudo chown -R 1001:1001 /var/run/mysqld execute /opt/guest-agent-venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 2023-03-05 12:06:17.171 1015 DEBUG oslo_concurrency.processutils [-] CMD "sudo chown -R 1001:1001 /var/run/mysqld" returned: 0 in 0.009s execute /opt/guest-agent-venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 2023-03-05 12:06:17.172 1015 INFO trove.guestagent.datastore.mysql_common.service [-] Starting docker container, image: mysql:5.7.29 2023-03-05 12:06:17.174 1015 WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service [-] Failed to start mysql: docker.errors.APIError: 500 Server Error: Internal Server Error ("Get "https://registry-1.docker.io/v2/": context deadline exceeded") 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service Traceback (most recent call last): 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service response.raise_for_status() 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/database/json 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service Traceback (most recent call last): 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/utils/docker.py", line 58, in start_container 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service container = client.containers.get(name) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 887, in get 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service resp = self.client.api.inspect_container(container_id) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service return f(self, resource_id, *args, **kwargs) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 771, in inspect_container 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service self._get(self._url("/containers/{0}/json", container)), True 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 265, in _result 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service self._raise_for_status(response) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service Traceback (most recent call last): 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service response.raise_for_status() 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/create?name=database 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service Traceback (most recent call last): 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 810, in run 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service detach=detach, **kwargs) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 868, in create 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service resp = self.client.api.create_container(**create_kwargs) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 430, in create_container 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service return self.create_container_from_config(config, name) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 441, in create_container_from_config 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service return self._result(res, True) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 265, in _result 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service self._raise_for_status(response) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: mysql:5.7.29") 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service Traceback (most recent call last): 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service response.raise_for_status() 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=5.7.29&fromImage=mysql 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service Traceback (most recent call last): 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/mysql_common/service.py", line 612, in start_db 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service command=command 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/utils/docker.py", line 73, in start_container 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service command=command 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 812, in run 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service self.client.images.pull(image, platform=platform) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/images.py", line 445, in pull 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service repository, tag=tag, stream=True, **kwargs 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/image.py", line 415, in pull 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service self._raise_for_status(response) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service docker.errors.APIError: 500 Server Error: Internal Server Error ("Get "https://registry-1.docker.io/v2/": context deadline exceeded") 2023-03-05 12:06:32.540 1015 ERROR trove.guestagent.datastore.mysql_common.service 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager [-] Failed to prepare datastore: Failed to start mysql: trove.common.exception.TroveError: Failed to start mysql 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager Traceback (most recent call last): 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager response.raise_for_status() 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/database/json 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager Traceback (most recent call last): 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/utils/docker.py", line 58, in start_container 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager container = client.containers.get(name) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 887, in get 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager resp = self.client.api.inspect_container(container_id) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager return f(self, resource_id, *args, **kwargs) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 771, in inspect_container 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager self._get(self._url("/containers/{0}/json", container)), True 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 265, in _result 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager self._raise_for_status(response) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager Traceback (most recent call last): 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager response.raise_for_status() 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/create?name=database 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager Traceback (most recent call last): 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 810, in run 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager detach=detach, **kwargs) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 868, in create 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager resp = self.client.api.create_container(**create_kwargs) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 430, in create_container 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager return self.create_container_from_config(config, name) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 441, in create_container_from_config 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager return self._result(res, True) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 265, in _result 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager self._raise_for_status(response) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: mysql:5.7.29") 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager Traceback (most recent call last): 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager response.raise_for_status() 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=5.7.29&fromImage=mysql 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager Traceback (most recent call last): 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/mysql_common/service.py", line 612, in start_db 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager command=command 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/utils/docker.py", line 73, in start_container 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager command=command 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 812, in run 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager self.client.images.pull(image, platform=platform) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/images.py", line 445, in pull 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager repository, tag=tag, stream=True, **kwargs 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/image.py", line 415, in pull 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager self._raise_for_status(response) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager docker.errors.APIError: 500 Server Error: Internal Server Error ("Get "https://registry-1.docker.io/v2/": context deadline exceeded") 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager Traceback (most recent call last): 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/manager.py", line 223, in _prepare 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager cluster_config, snapshot, ds_version=ds_version) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager result = f(*args, **kwargs) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/mysql_common/manager.py", line 96, in do_prepare 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager self.app.start_db(ds_version=ds_version, command=command) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/mysql_common/service.py", line 620, in start_db 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager raise exception.TroveError(_("Failed to start mysql")) 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager trove.common.exception.TroveError: Failed to start mysql 2023-03-05 12:06:32.546 1015 ERROR trove.guestagent.datastore.manager 2023-03-05 12:06:32.549 1015 INFO trove.guestagent.datastore.manager [-] Ending datastore prepare for 'mysql'. 2023-03-05 12:06:32.550 1015 INFO trove.guestagent.datastore.service [-] Set final status to failed to spawn. 2023-03-05 12:06:32.550 1015 DEBUG trove.guestagent.datastore.service [-] Casting set_status message to conductor (status is 'failed to spawn'). set_status /opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/service.py:144 2023-03-05 12:06:32.550 1015 DEBUG trove.conductor.api [-] Making async call to cast heartbeat for instance: 291070bc-671d-437a-a68c-9cee840e614c heartbeat /opt/guest-agent-venv/lib/python3.6/site-packages/trove/conductor/api.py:73 2023-03-05 12:06:32.553 1015 DEBUG trove.guestagent.datastore.service [-] Successfully cast set_status. set_status /opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/service.py:151 2023-03-05 12:06:32.553 1015 DEBUG trove.conductor.api [-] Making async call to cast error notification notify_exc_info /opt/guest-agent-venv/lib/python3.6/site-packages/trove/conductor/api.py:115 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server [-] Exception during message handling: trove.common.exception.TroveError: Failed to start mysql 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server response.raise_for_status() 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/database/json 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/utils/docker.py", line 58, in start_container 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server container = client.containers.get(name) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 887, in get 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server resp = self.client.api.inspect_container(container_id) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server return f(self, resource_id, *args, **kwargs) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 771, in inspect_container 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server self._get(self._url("/containers/{0}/json", container)), True 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 265, in _result 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server self._raise_for_status(response) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server response.raise_for_status() 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/create?name=database 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 810, in run 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server detach=detach, **kwargs) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 868, in create 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server resp = self.client.api.create_container(**create_kwargs) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 430, in create_container 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server return self.create_container_from_config(config, name) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/container.py", line 441, in create_container_from_config 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server return self._result(res, True) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 265, in _result 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server self._raise_for_status(response) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: mysql:5.7.29") 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 259, in _raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server response.raise_for_status() 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise HTTPError(http_error_msg, response=self) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=5.7.29&fromImage=mysql 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/mysql_common/service.py", line 612, in start_db 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server command=command 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/utils/docker.py", line 73, in start_container 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server command=command 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/containers.py", line 812, in run 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server self.client.images.pull(image, platform=platform) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/models/images.py", line 445, in pull 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server repository, tag=tag, stream=True, **kwargs 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/image.py", line 415, in pull 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server self._raise_for_status(response) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/api/client.py", line 261, in _raise_for_status 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise create_api_error_from_http_exception(e) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise cls(e, response=response, explanation=explanation) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server docker.errors.APIError: 500 Server Error: Internal Server Error ("Get "https://registry-1.docker.io/v2/": context deadline exceeded") 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/manager.py", line 207, in prepare 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server ds_version=ds_version) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/manager.py", line 223, in _prepare 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server cluster_config, snapshot, ds_version=ds_version) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/mysql_common/manager.py", line 96, in do_prepare 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server self.app.start_db(ds_version=ds_version, command=command) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server File "/opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/mysql_common/service.py", line 620, in start_db 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server raise exception.TroveError(_("Failed to start mysql")) 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server trove.common.exception.TroveError: Failed to start mysql 2023-03-05 12:06:32.557 1015 ERROR oslo_messaging.rpc.server 2023-03-05 12:06:58.303 1015 DEBUG trove.guestagent.datastore.manager [-] Getting file system stats for '/var/lib/mysql' get_filesystem_stats /opt/guest-agent-venv/lib/python3.6/site-packages/trove/guestagent/datastore/manager.py:368 2023-03-05 12:07:16.911 1015 DEBUG oslo_service.periodic_task [-] Running periodic task Manager.update_status run_periodic_tasks /opt/guest-agent-venv/lib/python3.6/site-packages/oslo_service/periodic_task.py:211 2023-03-05 12:07:16.911 1015 INFO trove.guestagent.datastore.manager [-] Database service is not installed, skip status check 2023-03-05 12:08:16.918 1015 DEBUG oslo_service.periodic_task [-] Running periodic task Manager.update_status run_periodic_tasks /opt/guest-agent-venv/lib/python3.6/site-packages/oslo_service/periodic_task.py:211 2023-03-05 12:08:16.919 1015 INFO trove.guestagent.datastore.manager [-] Database service is not installed, skip status check root@t1:/var/log/trove#

1 0