<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for the details.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The MariaDB/Galera healthcheck occurs on port 9200, which may not be functioning. You can verify that in the /etc/haproxy/haproxy.cfg file. In the Galera container, there is a file, /etc/systemd/system/mariadbcheck.socket,
which has the details, including the “allow” list. Might be worth looking at that to ensure the haproxy node IP is allowed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">James Denton<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Principal Architect<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Rackspace Private Cloud - OpenStack<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">james.denton@rackspace.com</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">jmarcelo.alencar@gmail.com <jmarcelo.alencar@gmail.com><br>
<b>Date: </b>Friday, January 20, 2023 at 9:20 AM<br>
<b>To: </b>James Denton <james.denton@rackspace.com>, openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
<b>Subject: </b>Re: [openstack-ansible] Installing OpenStack with Ansible fails during Keystone playbook on TASK openstack.osa.db_setup<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!<br>
<br>
<br>
Hi James Denton,<br>
<br>
Thanks for your quick response!!!<br>
<br>
So as far as I understand, running "openstack-ansible<br>
setup-openstack.yml" will start a keystone installation TASK that<br>
connects to HAProxy, which in turn sends the connection to the galera<br>
container. The machine targethost01 runs both the containers and<br>
HAProxy. From deploymenthost, there is some connectivity to HAProxy:<br>
<br>
root@deploymenthost:/opt/openstack-ansible/playbooks# telnet 172.29.236.101 3306<br>
Trying 172.29.236.101...<br>
Connected to 172.29.236.101.<br>
Escape character is '^]'.<br>
Connection closed by foreign host.<br>
<br>
It appears that HAProxy is listening, but cannot provide a proper<br>
reply, so the connection closes. Following your suggestion, on<br>
targethost01, HAProxy is running, but complains about no galera<br>
backend:<br>
<br>
root@targethost01:~# systemctl status haproxy.service<br>
● haproxy.service - HAProxy Load Balancer<br>
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled;<br>
vendor preset: enabled)<br>
Active: active (running) since Fri 2023-01-20 11:35:40 -03; 33min ago<br>
Docs: man:haproxy(1)<br>
file:/usr/share/doc/haproxy/configuration.txt.gz<br>
Process: 276870 ExecStartPre=/usr/sbin/haproxy -Ws -f $CONFIG -c<br>
-q $EXTRAOPTS (code=exited, status=0/SUCCESS)<br>
Main PID: 276873 (haproxy)<br>
Tasks: 5 (limit: 8192)<br>
Memory: 13.1M<br>
CPU: 2.165s<br>
CGroup: /system.slice/haproxy.service<br>
<br>
</span><span style="font-size:11.0pt;font-family:"MS Gothic"">├</span><span style="font-size:11.0pt">─276873 /usr/sbin/haproxy -Ws -f<br>
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S<br>
/run/haproxy-master.sock<br>
<br>
└─276875 /usr/sbin/haproxy -Ws -f<br>
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S<br>
/run/haproxy-master.sock<br>
Jan 20 11:35:48 targethost01 haproxy[276875]: Server<br>
nova_console-back/targethost01_nova_api_container-56e92564 is DOWN,<br>
reason: Layer4 connection problem, info: "Conn><br>
Jan 20 11:35:48 targethost01 haproxy[276875]: backend<br>
nova_console-back has no server available!<br>
Jan 20 11:35:49 targethost01 haproxy[276875]: [WARNING] (276875) :<br>
Server placement-back/targethost01_placement_container-90ccebb6 is<br>
DOWN, reason: Layer4 connection ><br>
Jan 20 11:35:49 targethost01 haproxy[276875]: Server<br>
placement-back/targethost01_placement_container-90ccebb6 is DOWN,<br>
reason: Layer4 connection problem, info: "Connec><br>
Jan 20 11:35:49 targethost01 haproxy[276875]: [ALERT] (276875) :<br>
backend 'placement-back' has no server available!<br>
Jan 20 11:35:49 targethost01 haproxy[276875]: backend placement-back<br>
has no server available!<br>
Jan 20 11:35:53 targethost01 haproxy[276875]: [WARNING] (276875) :<br>
Server galera-back/targethost01_galera_container-5aa8474a is DOWN,<br>
reason: Layer4 timeout, check du><br>
Jan 20 11:35:53 targethost01 haproxy[276875]: [ALERT] (276875) :<br>
backend 'galera-back' has no server available!<br>
Jan 20 11:35:53 targethost01 haproxy[276875]: Server<br>
galera-back/targethost01_galera_container-5aa8474a is DOWN, reason:<br>
Layer4 timeout, check duration: 12001ms. 0 act><br>
Jan 20 11:35:53 targethost01 haproxy[276875]: backend galera-back has<br>
no server available!<br>
<br>
It also warns about the other services, but since they are not<br>
installed yet, I believe that it is the expected behavior. But galera<br>
should have a functional backend, right? The container is running:<br>
<br>
root@targethost01:~# lxc-ls<br>
targethost01_cinder_api_container-b7ec9bdd<br>
targethost01_galera_container-5aa8474a<br>
targethost01_glance_container-b3ce5a33<br>
targethost01_heat_api_container-57ec2a00<br>
targethost01_horizon_container-c99d168e<br>
targethost01_keystone_container-76e9b31b<br>
targethost01_memcached_container-8edca03c<br>
targethost01_neutron_server_container-fba7cb77<br>
targethost01_nova_api_container-56e92564<br>
targethost01_placement_container-90ccebb6<br>
targethost01_rabbit_mq_container-2e5c5470<br>
targethost01_repo_container-00531c23<br>
targethost01_utility_container-dc05dc90<br>
targethost01_zookeeper_container-294429e8 ubuntu-22-amd64<br>
root@targethost01:~# lxc-info targethost01_galera_container-5aa8474a<br>
Name: targethost01_galera_container-5aa8474a<br>
State: RUNNING<br>
PID: 102446<br>
IP: 10.0.3.53<br>
IP: 172.29.238.177<br>
Link: 5aa8474a_eth0<br>
TX bytes: 811.30 KiB<br>
RX bytes: 57.49 MiB<br>
Total bytes: 58.28 MiB<br>
Link: 5aa8474a_eth1<br>
TX bytes: 84.35 KiB<br>
RX bytes: 1.06 MiB<br>
Total bytes: 1.14 MiB<br>
<br>
I can establish a connection and the server waits for a password:<br>
<br>
root@targethost01:~# telnet 172.29.238.177 3306<br>
Trying 172.29.238.177...<br>
Connected to 172.29.238.177.<br>
Escape character is '^]'.<br>
u<br>
5.5.5-10.6.10-MariaDB-1:10.6.10+maria~ubu2204-log:8PmS7Y:W'Yn=#6%Vbjmcmysql_native_password<br>
<br>
Any hints?<br>
<br>
Best regards.<br>
<br>
On Fri, Jan 20, 2023 at 11:18 AM James Denton<br>
<james.denton@rackspace.com> wrote:<br>
><br>
> Hi –<br>
><br>
><br>
><br>
> The ansible command to test the DB hits the Galera container directly, while the Ansible playbooks are likely using the VIP managed by HAproxy. I suspect that HAproxy has not started properly or is otherwise not serving traffic directed toward the internal_lb_vip_address.<br>
><br>
><br>
><br>
> My suggestion at the moment is to check out the logs on the haproxy node to see if it’s working properly, and try testing connectivity from the deploy node via 172.29.236.101:3306. The haproxy logs will likely provide some insight here.<br>
><br>
><br>
><br>
> --<br>
><br>
> James Denton<br>
><br>
> Principal Architect<br>
><br>
> Rackspace Private Cloud - OpenStack<br>
><br>
> james.denton@rackspace.com<br>
><br>
><br>
><br>
> From: jmarcelo.alencar@gmail.com <jmarcelo.alencar@gmail.com><br>
> Date: Friday, January 20, 2023 at 6:45 AM<br>
> To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
> Subject: [openstack-ansible] Installing OpenStack with Ansible fails during Keystone playbook on TASK openstack.osa.db_setup<br>
><br>
> CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!<br>
><br>
><br>
> Hello Community,<br>
><br>
> I am trying to create a two machine deployment following Openstack<br>
> Ansible Deployment Guide<br>
> (<a href="https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Fproject-deploy-guide%2Fopenstack-ansible%2Flatest%2F&data=05%7C01%7Cjames.denton%40rackspace.com%7Ca0d5435aeb294d38bbcb08dafaf9ccd7%7C570057f473ef41c8bcbb08db2fc15c2b%7C0%7C0%7C638098248039916228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9Guhh4n3xlExA0biSyHR5iXrxmzrkZyF0xJh2cf8zrk%3D&reserved=0">https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Fproject-deploy-guide%2Fopenstack-ansible%2Flatest%2F&data=05%7C01%7Cjames.denton%40rackspace.com%7Ca0d5435aeb294d38bbcb08dafaf9ccd7%7C570057f473ef41c8bcbb08db2fc15c2b%7C0%7C0%7C638098248039916228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9Guhh4n3xlExA0biSyHR5iXrxmzrkZyF0xJh2cf8zrk%3D&reserved=0</a>).<br>
> The two machines are named targethost01 and targethost02, and I am<br>
> running Ansible from deploymenthost. Every machine has 4-Core CPUs, 8<br>
> GB of RAM, and 240 GB SSD. I am using Ubuntu 22.04.1 LTS.<br>
><br>
> The machine targethost01 has the following network configuration:<br>
><br>
> network:<br>
> version: 2<br>
> ethernets:<br>
> enp5s0:<br>
> dhcp4: true<br>
> enp6s0: {}<br>
> enp7s0: {}<br>
> enp8s0: {}<br>
> enp9s0: {}<br>
> vlans:<br>
> vlan.10:<br>
> id: 10<br>
> link: enp6s0<br>
> addresses: [ ]<br>
> vlan.20:<br>
> id: 20<br>
> link: enp7s0<br>
> addresses: [ ]<br>
> vlan.30:<br>
> id: 30<br>
> link: enp8s0<br>
> addresses: [ ]<br>
> vlan.40:<br>
> id: 40<br>
> link: enp9s0<br>
> addresses: [ ]<br>
> bridges:<br>
> br-mgmt:<br>
> addresses: [ 172.29.236.101/22 ]<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.10<br>
> br-storage:<br>
> addresses: [ 172.29.244.101/22 ]<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.20<br>
> br-vlan:<br>
> addresses: []<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.30<br>
> br-vxlan:<br>
> addresses: [ 172.29.240.101/22 ]<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.40<br>
><br>
><br>
> And targethost02 has the following network configuration:<br>
><br>
><br>
> network:<br>
> version: 2<br>
> ethernets:<br>
> enp5s0:<br>
> dhcp4: true<br>
> enp6s0: {}<br>
> enp7s0: {}<br>
> enp8s0: {}<br>
> enp9s0: {}<br>
> vlans:<br>
> vlan.10:<br>
> id: 10<br>
> link: enp6s0<br>
> addresses: [ ]<br>
> vlan.20:<br>
> id: 20<br>
> link: enp7s0<br>
> addresses: [ ]<br>
> vlan.30:<br>
> id: 30<br>
> link: enp8s0<br>
> addresses: [ ]<br>
> vlan.40:<br>
> id: 40<br>
> link: enp9s0<br>
> addresses: [ ]<br>
> bridges:<br>
> br-mgmt:<br>
> addresses: [ 172.29.236.102/22 ]<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.10<br>
> br-storage:<br>
> addresses: [ 172.29.244.102/22 ]<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.20<br>
> br-vlan:<br>
> addresses: []<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.30<br>
> br-vxlan:<br>
> addresses: [ 172.29.240.102/22 ]<br>
> mtu: 1500<br>
> interfaces:<br>
> - vlan.40<br>
><br>
><br>
> On the deploymenthost, /etc/openstack_deploy/openstack_user_config.yml<br>
> has the following:<br>
><br>
><br>
> ---<br>
> cidr_networks:<br>
> container: 172.29.236.0/22<br>
> tunnel: 172.29.240.0/22<br>
> storage: 172.29.244.0/22<br>
> used_ips:<br>
> - 172.29.236.1<br>
> - "172.29.236.100,172.29.236.200"<br>
> - "172.29.240.100,172.29.240.200"<br>
> - "172.29.244.100,172.29.244.200"<br>
> global_overrides:<br>
> internal_lb_vip_address: 172.29.236.101<br>
> external_lb_vip_address: "{{ bootstrap_host_public_address |<br>
> default(ansible_facts['default_ipv4']['address']) }}"<br>
> management_bridge: "br-mgmt"<br>
> provider_networks:<br>
> - network:<br>
> group_binds:<br>
> - all_containers<br>
> - hosts<br>
> type: "raw"<br>
> container_bridge: "br-mgmt"<br>
> container_interface: "eth1"<br>
> container_type: "veth"<br>
> ip_from_q: "container"<br>
> is_container_address: true<br>
> - network:<br>
> group_binds:<br>
> - glance_api<br>
> - cinder_api<br>
> - cinder_volume<br>
> - nova_compute<br>
> type: "raw"<br>
> container_bridge: "br-storage"<br>
> container_type: "veth"<br>
> container_interface: "eth2"<br>
> container_mtu: "9000"<br>
> ip_from_q: "storage"<br>
> - network:<br>
> group_binds:<br>
> - neutron_linuxbridge_agent<br>
> container_bridge: "br-vxlan"<br>
> container_type: "veth"<br>
> container_interface: "eth10"<br>
> container_mtu: "9000"<br>
> ip_from_q: "tunnel"<br>
> type: "vxlan"<br>
> range: "1:1000"<br>
> net_name: "vxlan"<br>
> - network:<br>
> group_binds:<br>
> - neutron_linuxbridge_agent<br>
> container_bridge: "br-vlan"<br>
> container_type: "veth"<br>
> container_interface: "eth11"<br>
> type: "vlan"<br>
> range: "101:200,301:400"<br>
> net_name: "vlan"<br>
> - network:<br>
> group_binds:<br>
> - neutron_linuxbridge_agent<br>
> container_bridge: "br-vlan"<br>
> container_type: "veth"<br>
> container_interface: "eth12"<br>
> host_bind_override: "eth12"<br>
> type: "flat"<br>
> net_name: "flat"<br>
> shared-infra_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> repo-infra_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> coordination_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> os-infra_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> identity_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> network_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> compute_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> targethost02:<br>
> ip: 172.29.236.102<br>
> storage-infra_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
> storage_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
><br>
><br>
> Also on the deploymenthost, /etc/openstack_deploy/conf.d/haproxy.yml<br>
> has the following:<br>
><br>
><br>
> haproxy_hosts:<br>
> targethost01:<br>
> ip: 172.29.236.101<br>
><br>
><br>
> At the Run Playbooks step of the guide, the following two Ansible<br>
> commands return with unreachable=0 failed=0:<br>
><br>
> # openstack-ansible setup-hosts.yml<br>
> # openstack-ansible setup-infrastructure.yml<br>
><br>
> And verifying the database also returns no error:<br>
><br>
><br>
> root@deploymenthost:/opt/openstack-ansible/playbooks# ansible<br>
> galera_container -m shell \<br>
> -a "mysql -h localhost -e 'show status like \"%wsrep_cluster_%\";'"<br>
> Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e<br>
> @/etc/openstack_deploy/user_variables.yml "<br>
> [WARNING]: Unable to parse /etc/openstack_deploy/inventory.ini as an<br>
> inventory source<br>
> targethost01_galera_container-5aa8474a | CHANGED | rc=0 >><br>
> Variable_name Value<br>
> wsrep_cluster_weight 1<br>
> wsrep_cluster_capabilities<br>
> wsrep_cluster_conf_id 1<br>
> wsrep_cluster_size 1<br>
> wsrep_cluster_state_uuid e7a0c332-97fe-11ed-b0d4-26b30049826d<br>
> wsrep_cluster_status Primary<br>
><br>
><br>
> But when I execute openstack-ansible setup-openstack.yml, I get this:<br>
><br>
><br>
> TASK [os_keystone : Fact for apache module mod_auth_openidc to be installed] ***<br>
> ok: [targethost01_keystone_container-76e9b31b]<br>
> TASK [include_role : openstack.osa.db_setup] ***********************************<br>
> TASK [openstack.osa.db_setup : Create database for service] ********************<br>
> failed: [targethost01_keystone_container-76e9b31b -><br>
> targethost01_utility_container-dc05dc90(172.29.238.59)] (item=None) =><br>
> {"censored": "the output has been hidden due to the fact that 'no_log:<br>
> true' was specified for this result", "changed": false}<br>
> fatal: [targethost01_keystone_container-76e9b31b -> {{<br>
> _oslodb_setup_host }}]: FAILED! => {"censored": "the output has been<br>
> hidden due to the fact that 'no_log: true' was specified for this<br>
> result", "changed": false}<br>
> PLAY RECAP *********************************************************************<br>
> targethost01_keystone_container-76e9b31b : ok=33 changed=0<br>
> unreachable=0 failed=1 skipped=8 rescued=0 ignored=0<br>
> targethost01_utility_container-dc05dc90 : ok=3 changed=0<br>
> unreachable=0 failed=0 skipped=0 rescued=0 ignored=0<br>
> EXIT NOTICE [Playbook execution failure] **************************************<br>
> ===============================================================================<br>
><br>
><br>
> First, how can I disable the "censored" warning? I wonder if the<br>
> uncensored running could give me more clues. Second, it appears to be<br>
> a problem creating the database (keystone db sync?) How can I test the<br>
> database execution inside the LXC containers? I tried to log into one<br>
> of the containers and ping the hosts IP and it works, so they have<br>
> connectivity. I set up the passwords with:<br>
><br>
> # cd /opt/openstack-ansible<br>
> # ./scripts/pw-token-gen.py --file /etc/openstack_deploy/user_secrets.yml<br>
><br>
><br>
> Any help?<br>
><br>
> Best Regards.<br>
><br>
><br>
><br>
><br>
> --<br>
> __________________________________<br>
><br>
> João Marcelo Uchôa de Alencar<br>
> jmarcelo.alencar(at)gmail.com<br>
> __________________________________<br>
<br>
<br>
<br>
--<br>
__________________________________<br>
<br>
João Marcelo Uchôa de Alencar<br>
jmarcelo.alencar(at)gmail.com<br>
__________________________________<o:p></o:p></span></p>
</div>
</div>
</body>
</html>