Open Stack

All;

I'm sorry for not responding earlier.  The fix, I believe was to provide DNS servers to the instance.

Yes, that was a head-slapping moment.

I had Trove working correctly for a while, but now it's back to same basic issue; with "Dynamic backoff interval looping call 'trove.common.utils.build_polling_task.<locals>.poll_and_check' failed: oslo_service.loopingcall.LoopingCallTimeOut: Looping call timed out after 876.31 seconds" being reported by trove-taskmanager.

To respond to Ammad's suggestion; we're using CentOS 8, so are following the instructions here: https://www.server-world.info/en/note?os=CentOS_8&p=openstack_victoria3&f=15.  Note that neither the CentOS 8 version, not the Ubuntu 20.04 version mention the need for a management network.

At present, I'm frustrated with the inability to access the Nova instance to observe the logs in real-time.  I've tried adding commands to the end of the cloud-init file (temporarily) to set the root user password, and cloud -user password.  Still can't login to the console for the Nova instance.  At present, I'm resorting to "rescuing" the Nova instance with an image with a known root password, in order to read the logs after the database instances times out.

Thank you,

Dominic L. Hilsbos, MBA 
Director – Information Technology 
Perform Air International Inc.
DHilsbos at PerformAir.com 
www.PerformAir.com

From: Lingxian Kong [mailto:anlin.kong at gmail.com] 
Sent: Tuesday, February 23, 2021 3:03 AM
To: Ammad Syed
Cc: Dominic Hilsbos; openstack-discuss
Subject: Re: [ops][victoria][trove] Problems running Trove

Hi Dominic,

Ammad is right, you can ignore the "dumpe2fs" command error, trove-guestagent just raises the error when the data storage is initialized for the first time.

The warning message "No such container: database" could also be ignored, trove-guestagent is checking if the database container is already running before creating a new one.

You need to scroll down to look for the real error, maybe you can paste the trove-guestagent log so we could help you.

---
Lingxian Kong
Senior Cloud Engineer (Catalyst Cloud)
Trove PTL (OpenStack)
OpenStack Cloud Provider Co-Lead (Kubernetes)


On Tue, Feb 23, 2021 at 5:53 PM Ammad Syed <syedammad83 at gmail.com> wrote:
Hi,

I think filesystem error is ignorable as trove guest agent first check the existence of filesystem in attached block device.

The second error looks like its enable to find container image in docker hub. Could you please check the name of datastore and version.

Currently only mysql mariadb and postgresql are supported.

Review below link this will be helpful.

https://www.server-world.info/en/note?os=Ubuntu_20.04&p=openstack_victoria4&f=15

- Ammad

On Tue, Feb 23, 2021 at 6:13 AM <DHilsbos at performair.com> wrote:
All;

I've had a number of issues getting Trove to run for us, though I've worked through most of them (lots of network problems...).

I now have the Nova instance starting, and it reaches out to the cluster's RabbitMQ instance.

 I'm getting these errors logged in /var/log/trove/trove-guestagent.log inside the Nova instance:
ERROR trove.guestagent.volume '/dev/vdb' was not formatted.: oslo.concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: sudo dumpe2fs /dev/vdb
Exit code: 1
Stdout: "Couldn't find valid filesystem superblock.\n"
Stderror: 'dumpe2fs: Bad magic number in super-block while trying to open /dev/vdb\n'

Farther down, I get this:
WARNING trove.guestagent.utils.docker [-] Failed to get container database: docker.errors.NotFound: 404 Client Error: Not Found ("No such container: database")

There are a number of similar errors, all indicating that it can't get the Docker files from somewhere.

I'm not sure if the first is an actual issue.  Wouldn't the current Trove setup expect it's data storage add-on volume to be blank initially?

The second suggests that I still have a network issue, most likely routing, as the Nova instance thinks it has 2 routes to the internet, but only one works.

Any thoughts would be appreciated.

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
DHilsbos at PerformAir.com 
www.PerformAir.com


-- 
Regards,


Syed Ammad Ali