[ops][victoria][trove] Problems running Trove
All; I've had a number of issues getting Trove to run for us, though I've worked through most of them (lots of network problems...). I now have the Nova instance starting, and it reaches out to the cluster's RabbitMQ instance. I'm getting these errors logged in /var/log/trove/trove-guestagent.log inside the Nova instance: ERROR trove.guestagent.volume '/dev/vdb' was not formatted.: oslo.concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: sudo dumpe2fs /dev/vdb Exit code: 1 Stdout: "Couldn't find valid filesystem superblock.\n" Stderror: 'dumpe2fs: Bad magic number in super-block while trying to open /dev/vdb\n' Farther down, I get this: WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") There are a number of similar errors, all indicating that it can't get the Docker files from somewhere. I'm not sure if the first is an actual issue. Wouldn't the current Trove setup expect it's data storage add-on volume to be blank initially? The second suggests that I still have a network issue, most likely routing, as the Nova instance thinks it has 2 routes to the internet, but only one works. Any thoughts would be appreciated. Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
Hi, I think filesystem error is ignorable as trove guest agent first check the existence of filesystem in attached block device. The second error looks like its enable to find container image in docker hub. Could you please check the name of datastore and version. Currently only mysql mariadb and postgresql are supported. Review below link this will be helpful. https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=15 - Ammad On Tue, Feb 23, 2021 at 6:13 AM <DHilsbos@performair.com> wrote:
All;
I've had a number of issues getting Trove to run for us, though I've worked through most of them (lots of network problems...).
I now have the Nova instance starting, and it reaches out to the cluster's RabbitMQ instance.
I'm getting these errors logged in /var/log/trove/trove-guestagent.log inside the Nova instance: ERROR trove.guestagent.volume '/dev/vdb' was not formatted.: oslo.concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: sudo dumpe2fs /dev/vdb Exit code: 1 Stdout: "Couldn't find valid filesystem superblock.\n" Stderror: 'dumpe2fs: Bad magic number in super-block while trying to open /dev/vdb\n'
Farther down, I get this: WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database")
There are a number of similar errors, all indicating that it can't get the Docker files from somewhere.
I'm not sure if the first is an actual issue. Wouldn't the current Trove setup expect it's data storage add-on volume to be blank initially?
The second suggests that I still have a network issue, most likely routing, as the Nova instance thinks it has 2 routes to the internet, but only one works.
Any thoughts would be appreciated.
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-- Regards,
Syed Ammad Ali
Hi Dominic, Ammad is right, you can ignore the "dumpe2fs" command error, trove-guestagent just raises the error when the data storage is initialized for the first time. The warning message "No such container: database" could also be ignored, trove-guestagent is checking if the database container is already running before creating a new one. You need to scroll down to look for the real error, maybe you can paste the trove-guestagent log so we could help you. --- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes) On Tue, Feb 23, 2021 at 5:53 PM Ammad Syed <syedammad83@gmail.com> wrote:
Hi,
I think filesystem error is ignorable as trove guest agent first check the existence of filesystem in attached block device.
The second error looks like its enable to find container image in docker hub. Could you please check the name of datastore and version.
Currently only mysql mariadb and postgresql are supported.
Review below link this will be helpful.
https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=15
- Ammad
On Tue, Feb 23, 2021 at 6:13 AM <DHilsbos@performair.com> wrote:
All;
I've had a number of issues getting Trove to run for us, though I've worked through most of them (lots of network problems...).
I now have the Nova instance starting, and it reaches out to the cluster's RabbitMQ instance.
I'm getting these errors logged in /var/log/trove/trove-guestagent.log inside the Nova instance: ERROR trove.guestagent.volume '/dev/vdb' was not formatted.: oslo.concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: sudo dumpe2fs /dev/vdb Exit code: 1 Stdout: "Couldn't find valid filesystem superblock.\n" Stderror: 'dumpe2fs: Bad magic number in super-block while trying to open /dev/vdb\n'
Farther down, I get this: WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database")
There are a number of similar errors, all indicating that it can't get the Docker files from somewhere.
I'm not sure if the first is an actual issue. Wouldn't the current Trove setup expect it's data storage add-on volume to be blank initially?
The second suggests that I still have a network issue, most likely routing, as the Nova instance thinks it has 2 routes to the internet, but only one works.
Any thoughts would be appreciated.
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-- Regards,
Syed Ammad Ali
All; I'm sorry for not responding earlier. The fix, I believe was to provide DNS servers to the instance. Yes, that was a head-slapping moment. I had Trove working correctly for a while, but now it's back to same basic issue; with "Dynamic backoff interval looping call 'trove.common.utils.build_polling_task.<locals>.poll_and_check' failed: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 876.31 seconds" being reported by trove-taskmanager. To respond to Ammad's suggestion; we're using CentOS 8, so are following the instructions here: https://www.server-world.info/en/note?os=CentOS_8&p=openstack_victoria3&f=15. Note that neither the CentOS 8 version, not the Ubuntu 20.04 version mention the need for a management network. At present, I'm frustrated with the inability to access the Nova instance to observe the logs in real-time. I've tried adding commands to the end of the cloud-init file (temporarily) to set the root user password, and cloud -user password. Still can't login to the console for the Nova instance. At present, I'm resorting to "rescuing" the Nova instance with an image with a known root password, in order to read the logs after the database instances times out. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com From: Lingxian Kong [mailto:anlin.kong@gmail.com] Sent: Tuesday, February 23, 2021 3:03 AM To: Ammad Syed Cc: Dominic Hilsbos; openstack-discuss Subject: Re: [ops][victoria][trove] Problems running Trove Hi Dominic, Ammad is right, you can ignore the "dumpe2fs" command error, trove-guestagent just raises the error when the data storage is initialized for the first time. The warning message "No such container: database" could also be ignored, trove-guestagent is checking if the database container is already running before creating a new one. You need to scroll down to look for the real error, maybe you can paste the trove-guestagent log so we could help you. --- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes) On Tue, Feb 23, 2021 at 5:53 PM Ammad Syed <syedammad83@gmail.com> wrote: Hi, I think filesystem error is ignorable as trove guest agent first check the existence of filesystem in attached block device. The second error looks like its enable to find container image in docker hub. Could you please check the name of datastore and version. Currently only mysql mariadb and postgresql are supported. Review below link this will be helpful. https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=15 - Ammad On Tue, Feb 23, 2021 at 6:13 AM <DHilsbos@performair.com> wrote: All; I've had a number of issues getting Trove to run for us, though I've worked through most of them (lots of network problems...). I now have the Nova instance starting, and it reaches out to the cluster's RabbitMQ instance. I'm getting these errors logged in /var/log/trove/trove-guestagent.log inside the Nova instance: ERROR trove.guestagent.volume '/dev/vdb' was not formatted.: oslo.concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: sudo dumpe2fs /dev/vdb Exit code: 1 Stdout: "Couldn't find valid filesystem superblock.\n" Stderror: 'dumpe2fs: Bad magic number in super-block while trying to open /dev/vdb\n' Farther down, I get this: WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") There are a number of similar errors, all indicating that it can't get the Docker files from somewhere. I'm not sure if the first is an actual issue. Wouldn't the current Trove setup expect it's data storage add-on volume to be blank initially? The second suggests that I still have a network issue, most likely routing, as the Nova instance thinks it has 2 routes to the internet, but only one works. Any thoughts would be appreciated. Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -- Regards, Syed Ammad Ali
Hi Hilsbos, To access the nova instance, you need to define keypair in default section of trove.conf. [Default] ... nova_keypair = trove-key ... Keypair (trove-key) must be created before creating a database instance. openstack keypair create --public-key ~/.ssh/id_rsa.pub trove-key Once the nova instance is deployed. you need to allow port 22 in security group of nova instance. After that use the keypair to ssh nova instance. - Ammad On Thu, Feb 25, 2021 at 1:04 AM <DHilsbos@performair.com> wrote:
All;
I'm sorry for not responding earlier. The fix, I believe was to provide DNS servers to the instance.
Yes, that was a head-slapping moment.
I had Trove working correctly for a while, but now it's back to same basic issue; with "Dynamic backoff interval looping call 'trove.common.utils.build_polling_task.<locals>.poll_and_check' failed: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 876.31 seconds" being reported by trove-taskmanager.
To respond to Ammad's suggestion; we're using CentOS 8, so are following the instructions here: https://www.server-world.info/en/note?os=CentOS_8&p=openstack_victoria3&f=15. Note that neither the CentOS 8 version, not the Ubuntu 20.04 version mention the need for a management network.
At present, I'm frustrated with the inability to access the Nova instance to observe the logs in real-time. I've tried adding commands to the end of the cloud-init file (temporarily) to set the root user password, and cloud -user password. Still can't login to the console for the Nova instance. At present, I'm resorting to "rescuing" the Nova instance with an image with a known root password, in order to read the logs after the database instances times out.
Thank you,
Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
From: Lingxian Kong [mailto:anlin.kong@gmail.com] Sent: Tuesday, February 23, 2021 3:03 AM To: Ammad Syed Cc: Dominic Hilsbos; openstack-discuss Subject: Re: [ops][victoria][trove] Problems running Trove
Hi Dominic,
Ammad is right, you can ignore the "dumpe2fs" command error, trove-guestagent just raises the error when the data storage is initialized for the first time.
The warning message "No such container: database" could also be ignored, trove-guestagent is checking if the database container is already running before creating a new one.
You need to scroll down to look for the real error, maybe you can paste the trove-guestagent log so we could help you.
--- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes)
On Tue, Feb 23, 2021 at 5:53 PM Ammad Syed <syedammad83@gmail.com> wrote: Hi,
I think filesystem error is ignorable as trove guest agent first check the existence of filesystem in attached block device.
The second error looks like its enable to find container image in docker hub. Could you please check the name of datastore and version.
Currently only mysql mariadb and postgresql are supported.
Review below link this will be helpful.
https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=15
- Ammad
On Tue, Feb 23, 2021 at 6:13 AM <DHilsbos@performair.com> wrote: All;
I've had a number of issues getting Trove to run for us, though I've worked through most of them (lots of network problems...).
I now have the Nova instance starting, and it reaches out to the cluster's RabbitMQ instance.
I'm getting these errors logged in /var/log/trove/trove-guestagent.log inside the Nova instance: ERROR trove.guestagent.volume '/dev/vdb' was not formatted.: oslo.concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: sudo dumpe2fs /dev/vdb Exit code: 1 Stdout: "Couldn't find valid filesystem superblock.\n" Stderror: 'dumpe2fs: Bad magic number in super-block while trying to open /dev/vdb\n'
Farther down, I get this: WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database")
There are a number of similar errors, all indicating that it can't get the Docker files from somewhere.
I'm not sure if the first is an actual issue. Wouldn't the current Trove setup expect it's data storage add-on volume to be blank initially?
The second suggests that I still have a network issue, most likely routing, as the Nova instance thinks it has 2 routes to the internet, but only one works.
Any thoughts would be appreciated.
Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-- Regards,
Syed Ammad Ali
-- Regards, Syed Ammad Ali
Ammad; First, I found and fixed the problem. Network basics do still apply, even in a virtualized environment. Suffice it to say, I had made a change in an attempt to support Octavia, that made the OS DHCP server unable to attach to the network intended for use by both Trove and Octavia. I have since corrected the issue, and both Trove and Octavia appear to be working fine. I found the nova_keypair setting, and tried, with no success. With that setting enabled, the nova instance fails to be created, with an error along the lines of 'keypair doesn't exist.' I had created the keypair in advance of restarting the openstack-trove-* services, and prior to creating the database instance. I verified the name several times prior to abandoning that line of investigation. It's possible that project boundaries might have contributed, I don't know. Trove is working now, so I'll revisit that issue in the future. I was disappointed to find that MongoDB is no longer supported in Trove. I'm going to see if I can drum up some interest internally to look at bring support back. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com From: Ammad Syed [mailto:syedammad83@gmail.com] Sent: Wednesday, February 24, 2021 10:12 PM To: Dominic Hilsbos Cc: openstack-discuss; Lingxian Kong Subject: Re: [ops][victoria][trove] Problems running Trove Hi Hilsbos, To access the nova instance, you need to define keypair in default section of trove.conf. [Default] ... nova_keypair = trove-key ... Keypair (trove-key) must be created before creating a database instance. openstack keypair create --public-key ~/.ssh/id_rsa.pub trove-key Once the nova instance is deployed. you need to allow port 22 in security group of nova instance. After that use the keypair to ssh nova instance. - Ammad On Thu, Feb 25, 2021 at 1:04 AM <DHilsbos@performair.com> wrote: All; I'm sorry for not responding earlier. The fix, I believe was to provide DNS servers to the instance. Yes, that was a head-slapping moment. I had Trove working correctly for a while, but now it's back to same basic issue; with "Dynamic backoff interval looping call 'trove.common.utils.build_polling_task.<locals>.poll_and_check' failed: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 876.31 seconds" being reported by trove-taskmanager. To respond to Ammad's suggestion; we're using CentOS 8, so are following the instructions here: https://www.server-world.info/en/note?os=CentOS_8&p=openstack_victoria3&f=15. Note that neither the CentOS 8 version, not the Ubuntu 20.04 version mention the need for a management network. At present, I'm frustrated with the inability to access the Nova instance to observe the logs in real-time. I've tried adding commands to the end of the cloud-init file (temporarily) to set the root user password, and cloud -user password. Still can't login to the console for the Nova instance. At present, I'm resorting to "rescuing" the Nova instance with an image with a known root password, in order to read the logs after the database instances times out. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com From: Lingxian Kong [mailto:anlin.kong@gmail.com] Sent: Tuesday, February 23, 2021 3:03 AM To: Ammad Syed Cc: Dominic Hilsbos; openstack-discuss Subject: Re: [ops][victoria][trove] Problems running Trove Hi Dominic, Ammad is right, you can ignore the "dumpe2fs" command error, trove-guestagent just raises the error when the data storage is initialized for the first time. The warning message "No such container: database" could also be ignored, trove-guestagent is checking if the database container is already running before creating a new one. You need to scroll down to look for the real error, maybe you can paste the trove-guestagent log so we could help you. --- Lingxian Kong Senior Cloud Engineer (Catalyst Cloud) Trove PTL (OpenStack) OpenStack Cloud Provider Co-Lead (Kubernetes) On Tue, Feb 23, 2021 at 5:53 PM Ammad Syed <syedammad83@gmail.com> wrote: Hi, I think filesystem error is ignorable as trove guest agent first check the existence of filesystem in attached block device. The second error looks like its enable to find container image in docker hub. Could you please check the name of datastore and version. Currently only mysql mariadb and postgresql are supported. Review below link this will be helpful. https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=15 - Ammad On Tue, Feb 23, 2021 at 6:13 AM <DHilsbos@performair.com> wrote: All; I've had a number of issues getting Trove to run for us, though I've worked through most of them (lots of network problems...). I now have the Nova instance starting, and it reaches out to the cluster's RabbitMQ instance. I'm getting these errors logged in /var/log/trove/trove-guestagent.log inside the Nova instance: ERROR trove.guestagent.volume '/dev/vdb' was not formatted.: oslo.concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: sudo dumpe2fs /dev/vdb Exit code: 1 Stdout: "Couldn't find valid filesystem superblock.\n" Stderror: 'dumpe2fs: Bad magic number in super-block while trying to open /dev/vdb\n' Farther down, I get this: WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database") There are a number of similar errors, all indicating that it can't get the Docker files from somewhere. I'm not sure if the first is an actual issue. Wouldn't the current Trove setup expect it's data storage add-on volume to be blank initially? The second suggests that I still have a network issue, most likely routing, as the Nova instance thinks it has 2 routes to the internet, but only one works. Any thoughts would be appreciated. Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -- Regards, Syed Ammad Ali -- Regards, Syed Ammad Ali
participants (3)
-
Ammad Syed
-
DHilsbos＠performair.com
-
Lingxian Kong