[kolla][trove] Best practice to make guest-agent communicate with RPC
Folks, I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service. How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
I found a reference patch but it's not merged yet. - https://review.opendev.org/c/openstack/kolla-ansible/+/863321 On Sat, Jan 6, 2024 at 9:55 AM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible. In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network. This network should be responsible for accessing rabbitmq, swift, keystone and docker registry. For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Thank you for the information. Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net) Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?) Do I need your patch for that or without that patch it’s possible? Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Hi, I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something? # Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Hi, Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321 What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out. On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
HI, yes you are right . the patch [0] depends on the patch[1] In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network. Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf [DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 // if you are using an insecurity docker registry then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon As you mentioned above: Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production. Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1]. [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24 <http://192.168.100.0/24>" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Thank you W Ch, It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla. What is the use of docker registry for trove guest agents? docker_insecure_registries = 10.1.2.49:6400 ? On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24 <http://192.168.100.0/24>" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： > > Folks, > > I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service. > > How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Very helpful, thank you! For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/ What is the use of backup images? On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24 <http://192.168.100.0/24>" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote: > Thank you for the information. > > Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net) > > Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?) > > Do I need your patch for that or without that patch it’s possible? > > > Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? > > On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >> Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible. >> >> In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network. >> >> This network should be responsible for accessing rabbitmq, swift, keystone and docker registry. >> >> For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >> >> thanks. >> >>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>> >>> Folks, >>> >>> I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service. >>> >>> How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance. >>
Thank you W ch, Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol... For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right? On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote:
Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible.
In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network.
This network should be responsible for accessing rabbitmq, swift, keystone and docker registry.
For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m...
thanks.
2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道：
Folks,
I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service.
How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance.
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
> 2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： > > Hi, > > Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321 > > What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out. > > > > On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote: >> Hi, >> >> I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something? >> >> # Trove (DBaaS) >> enable_trove: "yes" >> enable_cinder_backup: "no" >> enable_cinder_backend_lvm: "no" >> docker_disable_ip_forward: false >> trove_mgmt_network: >> name: trove-mgmt-net >> provider_network_type: flat >> provider_physical_network: physnet1 >> external: True >> shared: True >> subnet: >> name: trove-mgmt-subnet >> cidr: "192.168.100.0/24 <http://192.168.100.0/24>" >> gateway_ip: 192.168.100.1 >> allocation_pool_start: 192.168.100.150 >> allocation_pool_end: 192.168.100.199 >> enable_dhcp: yes >> >> On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote: >>> Thank you for the information. >>> >>> Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net) >>> >>> Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?) >>> >>> Do I need your patch for that or without that patch it’s possible? >>> >>> >>> Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? >>> >>> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >>>> Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible. >>>> >>>> In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network. >>>> >>>> This network should be responsible for accessing rabbitmq, swift, keystone and docker registry. >>>> >>>> For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>>> >>>> thanks. >>>> >>>>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>>>> >>>>> Folks, >>>>> >>>>> I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service. >>>>> >>>>> How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance. >>>>
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;) On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com> wrote:
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> wrote:
Thank you for the information.
Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net)
Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?)
Do I need your patch for that or without that patch it’s possible?
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote:
> Hi, please refer to this patch: > https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... > in kolla-ansible. > > In your production, we recommend to create a provider network (vlan > or flat network) to be used as the trove management network. > > This network should be responsible for accessing rabbitmq, swift, > keystone and docker registry. > > For more information about trove management network, please read > this document: > https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... > > thanks. > > 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道： > > Folks, > > I am trying to find a kolla-ansible doc related to how to implement > trove in production with best practices. Especially when a guest-agent > (running inside VM) talks to RPC service. > > How are other folks doing in their environments? In most cases RPC > is an internal service and not visible to any guest VMs. Looking for some > guidance. > > >
Trove only supports swift as backup backend at present.
2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com> 写道：
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;)
On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
> 2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： > > Thank you W Ch, > > It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla. > > What is the use of docker registry for trove guest agents? > > docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> ? > > On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >> HI, yes you are right . the patch [0] depends on the patch[1] >> >> In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network. >> >> Setting mgmt network in kolla for Trove: >> Vim /etc/kolla/config/trove.conf >> >> [DEFAULT] >> management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, >> nova_keypair = trove-key // nova keypair, the same as the octavia nova key >> management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port >> docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> // if you are using an insecurity docker registry >> >> then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon >> >> As you mentioned above: >> >> Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? >> >> This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production. >> >> Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1]. >> >> [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 >> [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >> >>> 2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>> >>> Hi, >>> >>> Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >>> >>> What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out. >>> >>> >>> >>> On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote: >>>> Hi, >>>> >>>> I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something? >>>> >>>> # Trove (DBaaS) >>>> enable_trove: "yes" >>>> enable_cinder_backup: "no" >>>> enable_cinder_backend_lvm: "no" >>>> docker_disable_ip_forward: false >>>> trove_mgmt_network: >>>> name: trove-mgmt-net >>>> provider_network_type: flat >>>> provider_physical_network: physnet1 >>>> external: True >>>> shared: True >>>> subnet: >>>> name: trove-mgmt-subnet >>>> cidr: "192.168.100.0/24 <http://192.168.100.0/24>" >>>> gateway_ip: 192.168.100.1 >>>> allocation_pool_start: 192.168.100.150 >>>> allocation_pool_end: 192.168.100.199 >>>> enable_dhcp: yes >>>> >>>> On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote: >>>>> Thank you for the information. >>>>> >>>>> Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net) >>>>> >>>>> Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?) >>>>> >>>>> Do I need your patch for that or without that patch it’s possible? >>>>> >>>>> >>>>> Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? >>>>> >>>>> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >>>>>> Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible. >>>>>> >>>>>> In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network. >>>>>> >>>>>> This network should be responsible for accessing rabbitmq, swift, keystone and docker registry. >>>>>> >>>>>> For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>>>>> >>>>>> thanks. >>>>>> >>>>>>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>>>>>> >>>>>>> Folks, >>>>>>> >>>>>>> I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service. >>>>>>> >>>>>>> How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance. >>>>>> >>
Oh really? Any plan for ceph or s3 style backup? On Wed, Jan 10, 2024 at 2:18 AM W Ch <wchy1001@gmail.com> wrote:
Trove only supports swift as backup backend at present.
2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com> 写道：
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;)
On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com> wrote:
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi,
I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something?
# Trove (DBaaS) enable_trove: "yes" enable_cinder_backup: "no" enable_cinder_backend_lvm: "no" docker_disable_ip_forward: false trove_mgmt_network: name: trove-mgmt-net provider_network_type: flat provider_physical_network: physnet1 external: True shared: True subnet: name: trove-mgmt-subnet cidr: "192.168.100.0/24" gateway_ip: 192.168.100.1 allocation_pool_start: 192.168.100.150 allocation_pool_end: 192.168.100.199 enable_dhcp: yes
On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> wrote:
> Thank you for the information. > > Let me understand. You are saying I should create trove-mgmt network > which attached to all controller services and same network will attach to > trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net) > > Where do I tell trove to use trove-mgmt network when you spin up > instance? (In what config file?) > > Do I need your patch for that or without that patch it’s possible? > > > Second thing I was thinking why don’t I expose rabbitmq/keystone etc > on public ip using ngnix so trove vm can access those public endpoints.. > what is the problem in this approach ? > > On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote: > >> Hi, please refer to this patch: >> https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... >> in kolla-ansible. >> >> In your production, we recommend to create a provider network (vlan >> or flat network) to be used as the trove management network. >> >> This network should be responsible for accessing rabbitmq, swift, >> keystone and docker registry. >> >> For more information about trove management network, please read >> this document: >> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >> >> thanks. >> >> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道： >> >> Folks, >> >> I am trying to find a kolla-ansible doc related to how to implement >> trove in production with best practices. Especially when a guest-agent >> (running inside VM) talks to RPC service. >> >> How are other folks doing in their environments? In most cases RPC >> is an internal service and not visible to any guest VMs. Looking for some >> guidance. >> >> >>
There is already a patch to support Cinder and Ceph, but it hasn’t been reviewed yet. Satish Patel <satish.txt@gmail.com> 于2024年1月10日周三 20:57写道：
Oh really? Any plan for ceph or s3 style backup?
On Wed, Jan 10, 2024 at 2:18 AM W Ch <wchy1001@gmail.com> wrote:
Trove only supports swift as backup backend at present.
2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com> 写道：
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;)
On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com> wrote:
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote:
HI, yes you are right . the patch [0] depends on the patch[1]
In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network.
Setting mgmt network in kolla for Trove: Vim /etc/kolla/config/trove.conf
[DEFAULT] management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, nova_keypair = trove-key // nova keypair, the same as the octavia nova key management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port docker_insecure_registries = 10.1.2.49:6400 // if you are using an insecurity docker registry
then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon
As you mentioned above:
Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ?
This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production.
Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1].
[0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321
2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道：
Hi,
Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321
What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out.
On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> wrote:
> Hi, > > I have tried following in global.yml and it didn't create any > network. Assuming your patch is missing for that right? Or am I missing > something? > > # Trove (DBaaS) > enable_trove: "yes" > enable_cinder_backup: "no" > enable_cinder_backend_lvm: "no" > docker_disable_ip_forward: false > trove_mgmt_network: > name: trove-mgmt-net > provider_network_type: flat > provider_physical_network: physnet1 > external: True > shared: True > subnet: > name: trove-mgmt-subnet > cidr: "192.168.100.0/24" > gateway_ip: 192.168.100.1 > allocation_pool_start: 192.168.100.150 > allocation_pool_end: 192.168.100.199 > enable_dhcp: yes > > On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> > wrote: > >> Thank you for the information. >> >> Let me understand. You are saying I should create trove-mgmt >> network which attached to all controller services and same network will >> attach to trove instances (guest-agent) right? (Just like Octavia >> lb-mgmt-net) >> >> Where do I tell trove to use trove-mgmt network when you spin up >> instance? (In what config file?) >> >> Do I need your patch for that or without that patch it’s possible? >> >> >> Second thing I was thinking why don’t I expose rabbitmq/keystone >> etc on public ip using ngnix so trove vm can access those public >> endpoints.. what is the problem in this approach ? >> >> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote: >> >>> Hi, please refer to this patch: >>> https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... >>> in kolla-ansible. >>> >>> In your production, we recommend to create a provider network >>> (vlan or flat network) to be used as the trove management network. >>> >>> This network should be responsible for accessing rabbitmq, swift, >>> keystone and docker registry. >>> >>> For more information about trove management network, please read >>> this document: >>> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>> >>> thanks. >>> >>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道： >>> >>> Folks, >>> >>> I am trying to find a kolla-ansible doc related to how to >>> implement trove in production with best practices. Especially when a >>> guest-agent (running inside VM) talks to RPC service. >>> >>> How are other folks doing in their environments? In most cases RPC >>> is an internal service and not visible to any guest VMs. Looking for some >>> guidance. >>> >>> >>>
Can you send me that patch? I might try to test and see. On Wed, Jan 10, 2024 at 9:19 AM chunyang wu <wchy1001@gmail.com> wrote:
There is already a patch to support Cinder and Ceph, but it hasn’t been reviewed yet.
Satish Patel <satish.txt@gmail.com> 于2024年1月10日周三 20:57写道：
Oh really? Any plan for ceph or s3 style backup?
On Wed, Jan 10, 2024 at 2:18 AM W Ch <wchy1001@gmail.com> wrote:
Trove only supports swift as backup backend at present.
2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com> 写道：
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;)
On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com> wrote:
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote:
Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu...
2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W Ch,
It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla.
What is the use of docker registry for trove guest agents?
docker_insecure_registries = 10.1.2.49:6400 ?
On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote:
> HI, yes you are right . the patch [0] depends on the patch[1] > > In your case, you can create a trove manage network like octavia > lb-mgmt-net, and set it as trove mgmt network. > > Setting mgmt network in kolla for Trove: > Vim /etc/kolla/config/trove.conf > > [DEFAULT] > management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // > management network which is the same as the lb-mgmt-net, > nova_keypair = trove-key // nova keypair, the same as the octavia > nova key > management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 > // used for management port > docker_insecure_registries = 10.1.2.49:6400 // if you are > using an insecurity docker registry > > then, reconfigure your trove deployment // kolla-ansible > reconfigure —tags trove, horizon > > As you mentioned above: > > Second thing I was thinking why don’t I expose rabbitmq/keystone etc > on public ip using ngnix so trove vm can access those public endpoints.. > what is the problem in this approach ? > > This is also feasible for trove deployment, but in this situation, > the cloud operator might be hard to ssh into the trove guest vm for > debugging. We don’t recommend to use this approach in production. > > Finally, you don’t need to cherry-pick the above patches. Just set > up the resources manually to achieve the same result as patch[1]. > > [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 > [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321 > > 2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道： > > Hi, > > Didn't realize that entire trove deployment depend on this patch set > - https://review.opendev.org/c/openstack/kolla-ansible/+/863321 > > What stopped us to merge this patch because I am desperately looking > to implement trove on our cloud infra. Let me know if you need help to test > it out. > > > > On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> > wrote: > >> Hi, >> >> I have tried following in global.yml and it didn't create any >> network. Assuming your patch is missing for that right? Or am I missing >> something? >> >> # Trove (DBaaS) >> enable_trove: "yes" >> enable_cinder_backup: "no" >> enable_cinder_backend_lvm: "no" >> docker_disable_ip_forward: false >> trove_mgmt_network: >> name: trove-mgmt-net >> provider_network_type: flat >> provider_physical_network: physnet1 >> external: True >> shared: True >> subnet: >> name: trove-mgmt-subnet >> cidr: "192.168.100.0/24" >> gateway_ip: 192.168.100.1 >> allocation_pool_start: 192.168.100.150 >> allocation_pool_end: 192.168.100.199 >> enable_dhcp: yes >> >> On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> >> wrote: >> >>> Thank you for the information. >>> >>> Let me understand. You are saying I should create trove-mgmt >>> network which attached to all controller services and same network will >>> attach to trove instances (guest-agent) right? (Just like Octavia >>> lb-mgmt-net) >>> >>> Where do I tell trove to use trove-mgmt network when you spin up >>> instance? (In what config file?) >>> >>> Do I need your patch for that or without that patch it’s possible? >>> >>> >>> Second thing I was thinking why don’t I expose rabbitmq/keystone >>> etc on public ip using ngnix so trove vm can access those public >>> endpoints.. what is the problem in this approach ? >>> >>> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote: >>> >>>> Hi, please refer to this patch: >>>> https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... >>>> in kolla-ansible. >>>> >>>> In your production, we recommend to create a provider network >>>> (vlan or flat network) to be used as the trove management network. >>>> >>>> This network should be responsible for accessing rabbitmq, >>>> swift, keystone and docker registry. >>>> >>>> For more information about trove management network, please read >>>> this document: >>>> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>>> >>>> thanks. >>>> >>>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道： >>>> >>>> Folks, >>>> >>>> I am trying to find a kolla-ansible doc related to how to >>>> implement trove in production with best practices. Especially when a >>>> guest-agent (running inside VM) talks to RPC service. >>>> >>>> How are other folks doing in their environments? In most cases >>>> RPC is an internal service and not visible to any guest VMs. Looking for >>>> some guidance. >>>> >>>> >>>> >
Hi W I made good progress. Now my trove instances are stuck in the BUILD state. When I get on to an instance using ssh and I found the following error logs. I can telnet on RabbitMQ ports, which means I don't have any network firewalling issues. To me the error looks like related to the rabbitMQ queue declaration. ERROR oslo_service.service amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'trove' in vhost '/': received 'false' but current is 'true' Here are the full logs: Full logs - https://paste.opendev.org/show/bQdXnEFZ12JzTRTYhZwa/ Any idea what could be wrong? On Wed, Jan 10, 2024 at 10:17 AM Satish Patel <satish.txt@gmail.com> wrote:
Can you send me that patch? I might try to test and see.
On Wed, Jan 10, 2024 at 9:19 AM chunyang wu <wchy1001@gmail.com> wrote:
There is already a patch to support Cinder and Ceph, but it hasn’t been reviewed yet.
Satish Patel <satish.txt@gmail.com> 于2024年1月10日周三 20:57写道：
Oh really? Any plan for ceph or s3 style backup?
On Wed, Jan 10, 2024 at 2:18 AM W Ch <wchy1001@gmail.com> wrote:
Trove only supports swift as backup backend at present.
2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com> 写道：
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;)
On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com> wrote:
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote:
yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string.
What is the use of backup images?
Trove uses this image to backup and restore instance.
2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道：
Very helpful, thank you!
For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/
What is the use of backup images?
On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote:
> Since Victoria release, Trove started to use docker in guest-agent. > By default, guest-agent will pull the image(mysql, mariaDB or postgres ) > from the docker hub to set up the database. > For more information please read: > https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... > However, the backup images needs to be built by yourself. You can > refer to this patch: > https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu... > > 2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道： > > Thank you W Ch, > > It makes sense, I am thinking of creating a dbaas vlan interface and > using a vlan tenant for trove-mgmt-net instead of directly attaching to the > api network of kolla. > > What is the use of docker registry for trove guest agents? > > docker_insecure_registries = 10.1.2.49:6400 ? > > On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote: > >> HI, yes you are right . the patch [0] depends on the patch[1] >> >> In your case, you can create a trove manage network like octavia >> lb-mgmt-net, and set it as trove mgmt network. >> >> Setting mgmt network in kolla for Trove: >> Vim /etc/kolla/config/trove.conf >> >> [DEFAULT] >> management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // >> management network which is the same as the lb-mgmt-net, >> nova_keypair = trove-key // nova keypair, the same as the >> octavia nova key >> management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 >> // used for management port >> docker_insecure_registries = 10.1.2.49:6400 // if you are >> using an insecurity docker registry >> >> then, reconfigure your trove deployment // kolla-ansible >> reconfigure —tags trove, horizon >> >> As you mentioned above: >> >> Second thing I was thinking why don’t I expose rabbitmq/keystone >> etc on public ip using ngnix so trove vm can access those public >> endpoints.. what is the problem in this approach ? >> >> This is also feasible for trove deployment, but in this situation, >> the cloud operator might be hard to ssh into the trove guest vm for >> debugging. We don’t recommend to use this approach in production. >> >> Finally, you don’t need to cherry-pick the above patches. Just set >> up the resources manually to achieve the same result as patch[1]. >> >> [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 >> [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >> >> 2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道： >> >> Hi, >> >> Didn't realize that entire trove deployment depend on this patch >> set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >> >> >> What stopped us to merge this patch because I am desperately >> looking to implement trove on our cloud infra. Let me know if you need help >> to test it out. >> >> >> >> On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> >> wrote: >> >>> Hi, >>> >>> I have tried following in global.yml and it didn't create any >>> network. Assuming your patch is missing for that right? Or am I missing >>> something? >>> >>> # Trove (DBaaS) >>> enable_trove: "yes" >>> enable_cinder_backup: "no" >>> enable_cinder_backend_lvm: "no" >>> docker_disable_ip_forward: false >>> trove_mgmt_network: >>> name: trove-mgmt-net >>> provider_network_type: flat >>> provider_physical_network: physnet1 >>> external: True >>> shared: True >>> subnet: >>> name: trove-mgmt-subnet >>> cidr: "192.168.100.0/24" >>> gateway_ip: 192.168.100.1 >>> allocation_pool_start: 192.168.100.150 >>> allocation_pool_end: 192.168.100.199 >>> enable_dhcp: yes >>> >>> On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com> >>> wrote: >>> >>>> Thank you for the information. >>>> >>>> Let me understand. You are saying I should create trove-mgmt >>>> network which attached to all controller services and same network will >>>> attach to trove instances (guest-agent) right? (Just like Octavia >>>> lb-mgmt-net) >>>> >>>> Where do I tell trove to use trove-mgmt network when you spin up >>>> instance? (In what config file?) >>>> >>>> Do I need your patch for that or without that patch it’s >>>> possible? >>>> >>>> >>>> Second thing I was thinking why don’t I expose rabbitmq/keystone >>>> etc on public ip using ngnix so trove vm can access those public >>>> endpoints.. what is the problem in this approach ? >>>> >>>> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote: >>>> >>>>> Hi, please refer to this patch: >>>>> https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... >>>>> in kolla-ansible. >>>>> >>>>> In your production, we recommend to create a provider network >>>>> (vlan or flat network) to be used as the trove management network. >>>>> >>>>> This network should be responsible for accessing rabbitmq, >>>>> swift, keystone and docker registry. >>>>> >>>>> For more information about trove management network, please read >>>>> this document: >>>>> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>>>> >>>>> thanks. >>>>> >>>>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道： >>>>> >>>>> Folks, >>>>> >>>>> I am trying to find a kolla-ansible doc related to how to >>>>> implement trove in production with best practices. Especially when a >>>>> guest-agent (running inside VM) talks to RPC service. >>>>> >>>>> How are other folks doing in their environments? In most cases >>>>> RPC is an internal service and not visible to any guest VMs. Looking for >>>>> some guidance. >>>>> >>>>> >>>>> >> >
Never mind found this patch - https://review.opendev.org/c/openstack/kolla-ansible/+/904972/4 It works!! # openstack database instance list +-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+ | ID | Name | Datastore | Datastore Version | Status | Operating Status | Public | Addresses | Flavor ID | Size | Role | +-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+ | 3538c3dc-5a2d-49ae- | mydb1 | mysql | 5.7.29 | ACTIVE | HEALTHY | False | [{'address': | 3 | 10 | | | b3cb-bcc3c5927153 | | | | | | | '10.0.0.232', 'type': | | | | | | | | | | | | 'private', 'network': | | | | | | | | | | | | '81ce26ec-1151-4960- | | | | | | | | | | | | b3ca-bf529e6296b2'}] | | | | +-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+ On Wed, Jan 24, 2024 at 11:55 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi W
I made good progress. Now my trove instances are stuck in the BUILD state. When I get on to an instance using ssh and I found the following error logs. I can telnet on RabbitMQ ports, which means I don't have any network firewalling issues. To me the error looks like related to the rabbitMQ queue declaration.
ERROR oslo_service.service amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'trove' in vhost '/': received 'false' but current is 'true'
Here are the full logs: Full logs - https://paste.opendev.org/show/bQdXnEFZ12JzTRTYhZwa/
Any idea what could be wrong?
On Wed, Jan 10, 2024 at 10:17 AM Satish Patel <satish.txt@gmail.com> wrote:
Can you send me that patch? I might try to test and see.
On Wed, Jan 10, 2024 at 9:19 AM chunyang wu <wchy1001@gmail.com> wrote:
There is already a patch to support Cinder and Ceph, but it hasn’t been reviewed yet.
Satish Patel <satish.txt@gmail.com> 于2024年1月10日周三 20:57写道：
Oh really? Any plan for ceph or s3 style backup?
On Wed, Jan 10, 2024 at 2:18 AM W Ch <wchy1001@gmail.com> wrote:
Trove only supports swift as backup backend at present.
2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com> 写道：
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;)
On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com> wrote:
Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service.
2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道：
Thank you W ch,
Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol...
For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right?
On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote:
> yes, you can use this image： > https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... > remember don’t use the image that it’s name contains “dev” string. > > What is the use of backup images? > > > Trove uses this image to backup and restore instance. > > 2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道： > > Very helpful, thank you! > > For my testing environment can I use publicly available images from > here - https://tarballs.opendev.org/openstack/trove/images/ > > What is the use of backup images? > > On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote: > >> Since Victoria release, Trove started to use docker in >> guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or >> postgres ) from the docker hub to set up the database. >> For more information please read: >> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... >> However, the backup images needs to be built by yourself. You can >> refer to this patch: >> https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu... >> >> 2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道： >> >> Thank you W Ch, >> >> It makes sense, I am thinking of creating a dbaas vlan interface >> and using a vlan tenant for trove-mgmt-net instead of directly attaching to >> the api network of kolla. >> >> What is the use of docker registry for trove guest agents? >> >> docker_insecure_registries = 10.1.2.49:6400 ? >> >> On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote: >> >>> HI, yes you are right . the patch [0] depends on the patch[1] >>> >>> In your case, you can create a trove manage network like octavia >>> lb-mgmt-net, and set it as trove mgmt network. >>> >>> Setting mgmt network in kolla for Trove: >>> Vim /etc/kolla/config/trove.conf >>> >>> [DEFAULT] >>> management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // >>> management network which is the same as the lb-mgmt-net, >>> nova_keypair = trove-key // nova keypair, the same as the >>> octavia nova key >>> management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 >>> // used for management port >>> docker_insecure_registries = 10.1.2.49:6400 // if you are >>> using an insecurity docker registry >>> >>> then, reconfigure your trove deployment // kolla-ansible >>> reconfigure —tags trove, horizon >>> >>> As you mentioned above: >>> >>> Second thing I was thinking why don’t I expose rabbitmq/keystone >>> etc on public ip using ngnix so trove vm can access those public >>> endpoints.. what is the problem in this approach ? >>> >>> This is also feasible for trove deployment, but in this >>> situation, the cloud operator might be hard to ssh into the trove guest >>> vm for debugging. We don’t recommend to use this approach in production. >>> >>> Finally, you don’t need to cherry-pick the above patches. Just set >>> up the resources manually to achieve the same result as patch[1]. >>> >>> [0]: >>> https://review.opendev.org/c/openstack/kolla-ansible/+/863521 >>> [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >>> >>> 2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道： >>> >>> Hi, >>> >>> Didn't realize that entire trove deployment depend on this patch >>> set - >>> https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >>> >>> What stopped us to merge this patch because I am desperately >>> looking to implement trove on our cloud infra. Let me know if you need help >>> to test it out. >>> >>> >>> >>> On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I have tried following in global.yml and it didn't create any >>>> network. Assuming your patch is missing for that right? Or am I missing >>>> something? >>>> >>>> # Trove (DBaaS) >>>> enable_trove: "yes" >>>> enable_cinder_backup: "no" >>>> enable_cinder_backend_lvm: "no" >>>> docker_disable_ip_forward: false >>>> trove_mgmt_network: >>>> name: trove-mgmt-net >>>> provider_network_type: flat >>>> provider_physical_network: physnet1 >>>> external: True >>>> shared: True >>>> subnet: >>>> name: trove-mgmt-subnet >>>> cidr: "192.168.100.0/24" >>>> gateway_ip: 192.168.100.1 >>>> allocation_pool_start: 192.168.100.150 >>>> allocation_pool_end: 192.168.100.199 >>>> enable_dhcp: yes >>>> >>>> On Sun, Jan 7, 2024 at 11:33 AM Satish Patel < >>>> satish.txt@gmail.com> wrote: >>>> >>>>> Thank you for the information. >>>>> >>>>> Let me understand. You are saying I should create trove-mgmt >>>>> network which attached to all controller services and same network will >>>>> attach to trove instances (guest-agent) right? (Just like Octavia >>>>> lb-mgmt-net) >>>>> >>>>> Where do I tell trove to use trove-mgmt network when you spin up >>>>> instance? (In what config file?) >>>>> >>>>> Do I need your patch for that or without that patch it’s >>>>> possible? >>>>> >>>>> >>>>> Second thing I was thinking why don’t I expose rabbitmq/keystone >>>>> etc on public ip using ngnix so trove vm can access those public >>>>> endpoints.. what is the problem in this approach ? >>>>> >>>>> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote: >>>>> >>>>>> Hi, please refer to this patch: >>>>>> https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... >>>>>> in kolla-ansible. >>>>>> >>>>>> In your production, we recommend to create a provider network >>>>>> (vlan or flat network) to be used as the trove management network. >>>>>> >>>>>> This network should be responsible for accessing rabbitmq, >>>>>> swift, keystone and docker registry. >>>>>> >>>>>> For more information about trove management network, please >>>>>> read this document: >>>>>> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>>>>> >>>>>> thanks. >>>>>> >>>>>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道： >>>>>> >>>>>> Folks, >>>>>> >>>>>> I am trying to find a kolla-ansible doc related to how to >>>>>> implement trove in production with best practices. Especially when a >>>>>> guest-agent (running inside VM) talks to RPC service. >>>>>> >>>>>> How are other folks doing in their environments? In most cases >>>>>> RPC is an internal service and not visible to any guest VMs. Looking for >>>>>> some guidance. >>>>>> >>>>>> >>>>>> >>> >> >
I have a question related to external networks. What If I want to attach a network to a trove which doesn't have a neutron router attached. When I try public network which doesn't have any neutron router and I got error saying - Subnet d243bf1c-01db-408a-b8f1-757c3e28b4bb is not associated with router. Some of my customers use their own router like Pfsense instead of a neutron router. In that case how do I attach a network to trove database instances? On Thu, Jan 25, 2024 at 12:36 AM Satish Patel <satish.txt@gmail.com> wrote:
Never mind found this patch - https://review.opendev.org/c/openstack/kolla-ansible/+/904972/4
It works!!
# openstack database instance list
+-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+ | ID | Name | Datastore | Datastore Version | Status | Operating Status | Public | Addresses | Flavor ID | Size | Role |
+-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+ | 3538c3dc-5a2d-49ae- | mydb1 | mysql | 5.7.29 | ACTIVE | HEALTHY | False | [{'address': | 3 | 10 | | | b3cb-bcc3c5927153 | | | | | | | '10.0.0.232', 'type': | | | | | | | | | | | | 'private', 'network': | | | | | | | | | | | | '81ce26ec-1151-4960- | | | | | | | | | | | | b3ca-bf529e6296b2'}] | | | |
+-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+
On Wed, Jan 24, 2024 at 11:55 PM Satish Patel <satish.txt@gmail.com> wrote:
Hi W
I made good progress. Now my trove instances are stuck in the BUILD state. When I get on to an instance using ssh and I found the following error logs. I can telnet on RabbitMQ ports, which means I don't have any network firewalling issues. To me the error looks like related to the rabbitMQ queue declaration.
ERROR oslo_service.service amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'trove' in vhost '/': received 'false' but current is 'true'
Here are the full logs: Full logs - https://paste.opendev.org/show/bQdXnEFZ12JzTRTYhZwa/
Any idea what could be wrong?
On Wed, Jan 10, 2024 at 10:17 AM Satish Patel <satish.txt@gmail.com> wrote:
Can you send me that patch? I might try to test and see.
On Wed, Jan 10, 2024 at 9:19 AM chunyang wu <wchy1001@gmail.com> wrote:
There is already a patch to support Cinder and Ceph, but it hasn’t been reviewed yet.
Satish Patel <satish.txt@gmail.com> 于2024年1月10日周三 20:57写道：
Oh really? Any plan for ceph or s3 style backup?
On Wed, Jan 10, 2024 at 2:18 AM W Ch <wchy1001@gmail.com> wrote:
Trove only supports swift as backup backend at present.
2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com> 写道：
We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;)
On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com> wrote:
> Typically, we don’t need to restrict outbound traffic, only need to > restrict inbound traffic. If you still want to restrict outbound traffic, > besides the service you mentioned above, you also need to add rules for > swift service. > > > 2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com> 写道： > > Thank you W ch, > > Question related security group. I saw in patch only icmp & ssh is > allowed in security rules. That should be enough? > https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol... > > > For security reasons I want to restrict outbound traffic from db > instances (vm). In that case only RabbitMQ, Keystone & Docker should be > enough right? > > > On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com> wrote: > >> yes, you can use this image： >> https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... >> remember don’t use the image that it’s name contains “dev” string. >> >> What is the use of backup images? >> >> >> Trove uses this image to backup and restore instance. >> >> 2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com> 写道： >> >> Very helpful, thank you! >> >> For my testing environment can I use publicly available images from >> here - https://tarballs.opendev.org/openstack/trove/images/ >> >> What is the use of backup images? >> >> On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com> wrote: >> >>> Since Victoria release, Trove started to use docker in >>> guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or >>> postgres ) from the docker hub to set up the database. >>> For more information please read: >>> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... >>> However, the backup images needs to be built by yourself. You >>> can refer to this patch: >>> https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu... >>> >>> 2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com> 写道： >>> >>> Thank you W Ch, >>> >>> It makes sense, I am thinking of creating a dbaas vlan interface >>> and using a vlan tenant for trove-mgmt-net instead of directly attaching to >>> the api network of kolla. >>> >>> What is the use of docker registry for trove guest agents? >>> >>> docker_insecure_registries = 10.1.2.49:6400 ? >>> >>> On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com> wrote: >>> >>>> HI, yes you are right . the patch [0] depends on the patch[1] >>>> >>>> In your case, you can create a trove manage network like octavia >>>> lb-mgmt-net, and set it as trove mgmt network. >>>> >>>> Setting mgmt network in kolla for Trove: >>>> Vim /etc/kolla/config/trove.conf >>>> >>>> [DEFAULT] >>>> management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // >>>> management network which is the same as the lb-mgmt-net, >>>> nova_keypair = trove-key // nova keypair, the same as the >>>> octavia nova key >>>> management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 >>>> // used for management port >>>> docker_insecure_registries = 10.1.2.49:6400 // if you are >>>> using an insecurity docker registry >>>> >>>> then, reconfigure your trove deployment // kolla-ansible >>>> reconfigure —tags trove, horizon >>>> >>>> As you mentioned above: >>>> >>>> Second thing I was thinking why don’t I expose rabbitmq/keystone >>>> etc on public ip using ngnix so trove vm can access those public >>>> endpoints.. what is the problem in this approach ? >>>> >>>> This is also feasible for trove deployment, but in this >>>> situation, the cloud operator might be hard to ssh into the trove guest >>>> vm for debugging. We don’t recommend to use this approach in production. >>>> >>>> Finally, you don’t need to cherry-pick the above patches. Just >>>> set up the resources manually to achieve the same result as patch[1]. >>>> >>>> [0]: >>>> https://review.opendev.org/c/openstack/kolla-ansible/+/863521 >>>> [1]: >>>> https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >>>> >>>> 2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com> 写道： >>>> >>>> Hi, >>>> >>>> Didn't realize that entire trove deployment depend on this patch >>>> set - >>>> https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >>>> >>>> What stopped us to merge this patch because I am desperately >>>> looking to implement trove on our cloud infra. Let me know if you need help >>>> to test it out. >>>> >>>> >>>> >>>> On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have tried following in global.yml and it didn't create any >>>>> network. Assuming your patch is missing for that right? Or am I missing >>>>> something? >>>>> >>>>> # Trove (DBaaS) >>>>> enable_trove: "yes" >>>>> enable_cinder_backup: "no" >>>>> enable_cinder_backend_lvm: "no" >>>>> docker_disable_ip_forward: false >>>>> trove_mgmt_network: >>>>> name: trove-mgmt-net >>>>> provider_network_type: flat >>>>> provider_physical_network: physnet1 >>>>> external: True >>>>> shared: True >>>>> subnet: >>>>> name: trove-mgmt-subnet >>>>> cidr: "192.168.100.0/24" >>>>> gateway_ip: 192.168.100.1 >>>>> allocation_pool_start: 192.168.100.150 >>>>> allocation_pool_end: 192.168.100.199 >>>>> enable_dhcp: yes >>>>> >>>>> On Sun, Jan 7, 2024 at 11:33 AM Satish Patel < >>>>> satish.txt@gmail.com> wrote: >>>>> >>>>>> Thank you for the information. >>>>>> >>>>>> Let me understand. You are saying I should create trove-mgmt >>>>>> network which attached to all controller services and same network will >>>>>> attach to trove instances (guest-agent) right? (Just like Octavia >>>>>> lb-mgmt-net) >>>>>> >>>>>> Where do I tell trove to use trove-mgmt network when you spin >>>>>> up instance? (In what config file?) >>>>>> >>>>>> Do I need your patch for that or without that patch it’s >>>>>> possible? >>>>>> >>>>>> >>>>>> Second thing I was thinking why don’t I expose >>>>>> rabbitmq/keystone etc on public ip using ngnix so trove vm can access those >>>>>> public endpoints.. what is the problem in this approach ? >>>>>> >>>>>> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com> wrote: >>>>>> >>>>>>> Hi, please refer to this patch: >>>>>>> https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... >>>>>>> in kolla-ansible. >>>>>>> >>>>>>> In your production, we recommend to create a provider network >>>>>>> (vlan or flat network) to be used as the trove management network. >>>>>>> >>>>>>> This network should be responsible for accessing rabbitmq, >>>>>>> swift, keystone and docker registry. >>>>>>> >>>>>>> For more information about trove management network, please >>>>>>> read this document: >>>>>>> https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>>>>>> >>>>>>> thanks. >>>>>>> >>>>>>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com> 写道： >>>>>>> >>>>>>> Folks, >>>>>>> >>>>>>> I am trying to find a kolla-ansible doc related to how to >>>>>>> implement trove in production with best practices. Especially when a >>>>>>> guest-agent (running inside VM) talks to RPC service. >>>>>>> >>>>>>> How are other folks doing in their environments? In most cases >>>>>>> RPC is an internal service and not visible to any guest VMs. Looking for >>>>>>> some guidance. >>>>>>> >>>>>>> >>>>>>> >>>> >>> >> >
Hi, you can set `enable_access_check` to `false` in your trove.conf like follows: [network] enable_access_check =false
2024年1月25日 13:48，Satish Patel <satish.txt@gmail.com> 写道：
I have a question related to external networks. What If I want to attach a network to a trove which doesn't have a neutron router attached. When I try public network which doesn't have any neutron router and I got error saying - Subnet d243bf1c-01db-408a-b8f1-757c3e28b4bb is not associated with router.
Some of my customers use their own router like Pfsense instead of a neutron router. In that case how do I attach a network to trove database instances?
On Thu, Jan 25, 2024 at 12:36 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Never mind found this patch - https://review.opendev.org/c/openstack/kolla-ansible/+/904972/4
It works!!
# openstack database instance list +-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+ | ID | Name | Datastore | Datastore Version | Status | Operating Status | Public | Addresses | Flavor ID | Size | Role | +-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+ | 3538c3dc-5a2d-49ae- | mydb1 | mysql | 5.7.29 | ACTIVE | HEALTHY | False | [{'address': | 3 | 10 | | | b3cb-bcc3c5927153 | | | | | | | '10.0.0.232', 'type': | | | | | | | | | | | | 'private', 'network': | | | | | | | | | | | | '81ce26ec-1151-4960- | | | | | | | | | | | | b3ca-bf529e6296b2'}] | | | | +-----------------------+-------+-----------+-------------------+--------+------------------+--------+------------------------+-----------+------+------+
On Wed, Jan 24, 2024 at 11:55 PM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Hi W
I made good progress. Now my trove instances are stuck in the BUILD state. When I get on to an instance using ssh and I found the following error logs. I can telnet on RabbitMQ ports, which means I don't have any network firewalling issues. To me the error looks like related to the rabbitMQ queue declaration.
ERROR oslo_service.service amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'trove' in vhost '/': received 'false' but current is 'true'
Here are the full logs: Full logs - https://paste.opendev.org/show/bQdXnEFZ12JzTRTYhZwa/
Any idea what could be wrong?
On Wed, Jan 10, 2024 at 10:17 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Can you send me that patch? I might try to test and see.
On Wed, Jan 10, 2024 at 9:19 AM chunyang wu <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote:
There is already a patch to support Cinder and Ceph, but it hasn’t been reviewed yet.
Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 于2024年1月10日周三 20:57写道：
Oh really? Any plan for ceph or s3 style backup?
On Wed, Jan 10, 2024 at 2:18 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: > Trove only supports swift as backup backend at present. > > >> 2024年1月8日 21:30，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >> >> We don’t use swift. We have ceph storage. Hope it’s not hard requirement for trove ;) >> >> On Mon, Jan 8, 2024 at 12:36 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >>> Typically, we don’t need to restrict outbound traffic, only need to restrict inbound traffic. If you still want to restrict outbound traffic, besides the service you mentioned above, you also need to add rules for swift service. >>> >>> >>>> 2024年1月8日 12:03，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>>> >>>> Thank you W ch, >>>> >>>> Question related security group. I saw in patch only icmp & ssh is allowed in security rules. That should be enough? https://review.opendev.org/c/openstack/kolla-ansible/+/863321/12/ansible/rol... >>>> >>>> For security reasons I want to restrict outbound traffic from db instances (vm). In that case only RabbitMQ, Keystone & Docker should be enough right? >>>> >>>> >>>> On Sun, Jan 7, 2024 at 10:33 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >>>>> yes, you can use this image： https://tarballs.opendev.org/openstack/trove/images/trove-master-guest-ubunt... remember don’t use the image that it’s name contains “dev” string. >>>>> >>>>>> What is the use of backup images? >>>>> >>>>> Trove uses this image to backup and restore instance. >>>>> >>>>>> 2024年1月8日 11:26，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>>>>> >>>>>> Very helpful, thank you! >>>>>> >>>>>> For my testing environment can I use publicly available images from here - https://tarballs.opendev.org/openstack/trove/images/ >>>>>> >>>>>> What is the use of backup images? >>>>>> >>>>>> On Sun, Jan 7, 2024 at 10:15 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >>>>>>> Since Victoria release, Trove started to use docker in guest-agent. By default, guest-agent will pull the image(mysql, mariaDB or postgres ) from the docker hub to set up the database. >>>>>>> For more information please read: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#p... >>>>>>> However, the backup images needs to be built by yourself. You can refer to this patch: https://review.opendev.org/c/openstack/trove/+/886619/22/doc/source/admin/bu... >>>>>>> >>>>>>>> 2024年1月8日 10:59，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>>>>>>> >>>>>>>> Thank you W Ch, >>>>>>>> >>>>>>>> It makes sense, I am thinking of creating a dbaas vlan interface and using a vlan tenant for trove-mgmt-net instead of directly attaching to the api network of kolla. >>>>>>>> >>>>>>>> What is the use of docker registry for trove guest agents? >>>>>>>> >>>>>>>> docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> ? >>>>>>>> >>>>>>>> On Sun, Jan 7, 2024 at 8:39 PM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >>>>>>>>> HI, yes you are right . the patch [0] depends on the patch[1] >>>>>>>>> >>>>>>>>> In your case, you can create a trove manage network like octavia lb-mgmt-net, and set it as trove mgmt network. >>>>>>>>> >>>>>>>>> Setting mgmt network in kolla for Trove: >>>>>>>>> Vim /etc/kolla/config/trove.conf >>>>>>>>> >>>>>>>>> [DEFAULT] >>>>>>>>> management_networks = 7a93bb03-2797-4529-8d6e-27b1699b7eb1 // management network which is the same as the lb-mgmt-net, >>>>>>>>> nova_keypair = trove-key // nova keypair, the same as the octavia nova key >>>>>>>>> management_security_groups = c8c65b8b-17f5-48e5-b094-867a8ecfce32 // used for management port >>>>>>>>> docker_insecure_registries = 10.1.2.49:6400 <http://10.1.2.49:6400/> // if you are using an insecurity docker registry >>>>>>>>> >>>>>>>>> then, reconfigure your trove deployment // kolla-ansible reconfigure —tags trove, horizon >>>>>>>>> >>>>>>>>> As you mentioned above: >>>>>>>>> >>>>>>>>> Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? >>>>>>>>> >>>>>>>>> This is also feasible for trove deployment, but in this situation, the cloud operator might be hard to ssh into the trove guest vm for debugging. We don’t recommend to use this approach in production. >>>>>>>>> >>>>>>>>> Finally, you don’t need to cherry-pick the above patches. Just set up the resources manually to achieve the same result as patch[1]. >>>>>>>>> >>>>>>>>> [0]: https://review.opendev.org/c/openstack/kolla-ansible/+/863521 >>>>>>>>> [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >>>>>>>>> >>>>>>>>>> 2024年1月8日 04:53，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Didn't realize that entire trove deployment depend on this patch set - https://review.opendev.org/c/openstack/kolla-ansible/+/863321 >>>>>>>>>> >>>>>>>>>> What stopped us to merge this patch because I am desperately looking to implement trove on our cloud infra. Let me know if you need help to test it out. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Jan 7, 2024 at 1:05 PM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I have tried following in global.yml and it didn't create any network. Assuming your patch is missing for that right? Or am I missing something? >>>>>>>>>>> >>>>>>>>>>> # Trove (DBaaS) >>>>>>>>>>> enable_trove: "yes" >>>>>>>>>>> enable_cinder_backup: "no" >>>>>>>>>>> enable_cinder_backend_lvm: "no" >>>>>>>>>>> docker_disable_ip_forward: false >>>>>>>>>>> trove_mgmt_network: >>>>>>>>>>> name: trove-mgmt-net >>>>>>>>>>> provider_network_type: flat >>>>>>>>>>> provider_physical_network: physnet1 >>>>>>>>>>> external: True >>>>>>>>>>> shared: True >>>>>>>>>>> subnet: >>>>>>>>>>> name: trove-mgmt-subnet >>>>>>>>>>> cidr: "192.168.100.0/24 <http://192.168.100.0/24>" >>>>>>>>>>> gateway_ip: 192.168.100.1 >>>>>>>>>>> allocation_pool_start: 192.168.100.150 >>>>>>>>>>> allocation_pool_end: 192.168.100.199 >>>>>>>>>>> enable_dhcp: yes >>>>>>>>>>> >>>>>>>>>>> On Sun, Jan 7, 2024 at 11:33 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote: >>>>>>>>>>>> Thank you for the information. >>>>>>>>>>>> >>>>>>>>>>>> Let me understand. You are saying I should create trove-mgmt network which attached to all controller services and same network will attach to trove instances (guest-agent) right? (Just like Octavia lb-mgmt-net) >>>>>>>>>>>> >>>>>>>>>>>> Where do I tell trove to use trove-mgmt network when you spin up instance? (In what config file?) >>>>>>>>>>>> >>>>>>>>>>>> Do I need your patch for that or without that patch it’s possible? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Second thing I was thinking why don’t I expose rabbitmq/keystone etc on public ip using ngnix so trove vm can access those public endpoints.. what is the problem in this approach ? >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Jan 7, 2024 at 5:16 AM W Ch <wchy1001@gmail.com <mailto:wchy1001@gmail.com>> wrote: >>>>>>>>>>>>> Hi, please refer to this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/863521/43/tests/templ... in kolla-ansible. >>>>>>>>>>>>> >>>>>>>>>>>>> In your production, we recommend to create a provider network (vlan or flat network) to be used as the trove management network. >>>>>>>>>>>>> >>>>>>>>>>>>> This network should be responsible for accessing rabbitmq, swift, keystone and docker registry. >>>>>>>>>>>>> >>>>>>>>>>>>> For more information about trove management network, please read this document: https://docs.openstack.org/trove/latest/admin/run_trove_in_production.html#m... >>>>>>>>>>>>> >>>>>>>>>>>>> thanks. >>>>>>>>>>>>> >>>>>>>>>>>>>> 2024年1月6日 22:55，Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> 写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>> Folks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am trying to find a kolla-ansible doc related to how to implement trove in production with best practices. Especially when a guest-agent (running inside VM) talks to RPC service. >>>>>>>>>>>>>> >>>>>>>>>>>>>> How are other folks doing in their environments? In most cases RPC is an internal service and not visible to any guest VMs. Looking for some guidance. >>>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> >
participants (3)
-
chunyang wu
-
Satish Patel
-
W Ch