[openstack-ansible] Installing OpenStack with Ansible fails during Keystone playbook on TASK openstack.osa.db_setup
jmarcelo.alencar at gmail.com
jmarcelo.alencar at gmail.com
Fri Jan 20 15:19:48 UTC 2023
Hi James Denton,
Thanks for your quick response!!!
So as far as I understand, running "openstack-ansible
setup-openstack.yml" will start a keystone installation TASK that
connects to HAProxy, which in turn sends the connection to the galera
container. The machine targethost01 runs both the containers and
HAProxy. From deploymenthost, there is some connectivity to HAProxy:
root at deploymenthost:/opt/openstack-ansible/playbooks# telnet 172.29.236.101 3306
Trying 172.29.236.101...
Connected to 172.29.236.101.
Escape character is '^]'.
Connection closed by foreign host.
It appears that HAProxy is listening, but cannot provide a proper
reply, so the connection closes. Following your suggestion, on
targethost01, HAProxy is running, but complains about no galera
backend:
root at targethost01:~# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled;
vendor preset: enabled)
Active: active (running) since Fri 2023-01-20 11:35:40 -03; 33min ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 276870 ExecStartPre=/usr/sbin/haproxy -Ws -f $CONFIG -c
-q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 276873 (haproxy)
Tasks: 5 (limit: 8192)
Memory: 13.1M
CPU: 2.165s
CGroup: /system.slice/haproxy.service
├─276873 /usr/sbin/haproxy -Ws -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S
/run/haproxy-master.sock
└─276875 /usr/sbin/haproxy -Ws -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S
/run/haproxy-master.sock
Jan 20 11:35:48 targethost01 haproxy[276875]: Server
nova_console-back/targethost01_nova_api_container-56e92564 is DOWN,
reason: Layer4 connection problem, info: "Conn>
Jan 20 11:35:48 targethost01 haproxy[276875]: backend
nova_console-back has no server available!
Jan 20 11:35:49 targethost01 haproxy[276875]: [WARNING] (276875) :
Server placement-back/targethost01_placement_container-90ccebb6 is
DOWN, reason: Layer4 connection >
Jan 20 11:35:49 targethost01 haproxy[276875]: Server
placement-back/targethost01_placement_container-90ccebb6 is DOWN,
reason: Layer4 connection problem, info: "Connec>
Jan 20 11:35:49 targethost01 haproxy[276875]: [ALERT] (276875) :
backend 'placement-back' has no server available!
Jan 20 11:35:49 targethost01 haproxy[276875]: backend placement-back
has no server available!
Jan 20 11:35:53 targethost01 haproxy[276875]: [WARNING] (276875) :
Server galera-back/targethost01_galera_container-5aa8474a is DOWN,
reason: Layer4 timeout, check du>
Jan 20 11:35:53 targethost01 haproxy[276875]: [ALERT] (276875) :
backend 'galera-back' has no server available!
Jan 20 11:35:53 targethost01 haproxy[276875]: Server
galera-back/targethost01_galera_container-5aa8474a is DOWN, reason:
Layer4 timeout, check duration: 12001ms. 0 act>
Jan 20 11:35:53 targethost01 haproxy[276875]: backend galera-back has
no server available!
It also warns about the other services, but since they are not
installed yet, I believe that it is the expected behavior. But galera
should have a functional backend, right? The container is running:
root at targethost01:~# lxc-ls
targethost01_cinder_api_container-b7ec9bdd
targethost01_galera_container-5aa8474a
targethost01_glance_container-b3ce5a33
targethost01_heat_api_container-57ec2a00
targethost01_horizon_container-c99d168e
targethost01_keystone_container-76e9b31b
targethost01_memcached_container-8edca03c
targethost01_neutron_server_container-fba7cb77
targethost01_nova_api_container-56e92564
targethost01_placement_container-90ccebb6
targethost01_rabbit_mq_container-2e5c5470
targethost01_repo_container-00531c23
targethost01_utility_container-dc05dc90
targethost01_zookeeper_container-294429e8 ubuntu-22-amd64
root at targethost01:~# lxc-info targethost01_galera_container-5aa8474a
Name: targethost01_galera_container-5aa8474a
State: RUNNING
PID: 102446
IP: 10.0.3.53
IP: 172.29.238.177
Link: 5aa8474a_eth0
TX bytes: 811.30 KiB
RX bytes: 57.49 MiB
Total bytes: 58.28 MiB
Link: 5aa8474a_eth1
TX bytes: 84.35 KiB
RX bytes: 1.06 MiB
Total bytes: 1.14 MiB
I can establish a connection and the server waits for a password:
root at targethost01:~# telnet 172.29.238.177 3306
Trying 172.29.238.177...
Connected to 172.29.238.177.
Escape character is '^]'.
u
5.5.5-10.6.10-MariaDB-1:10.6.10+maria~ubu2204-log:8PmS7Y:W'Yn=#6%Vbjmcmysql_native_password
Any hints?
Best regards.
On Fri, Jan 20, 2023 at 11:18 AM James Denton
<james.denton at rackspace.com> wrote:
>
> Hi –
>
>
>
> The ansible command to test the DB hits the Galera container directly, while the Ansible playbooks are likely using the VIP managed by HAproxy. I suspect that HAproxy has not started properly or is otherwise not serving traffic directed toward the internal_lb_vip_address.
>
>
>
> My suggestion at the moment is to check out the logs on the haproxy node to see if it’s working properly, and try testing connectivity from the deploy node via 172.29.236.101:3306. The haproxy logs will likely provide some insight here.
>
>
>
> --
>
> James Denton
>
> Principal Architect
>
> Rackspace Private Cloud - OpenStack
>
> james.denton at rackspace.com
>
>
>
> From: jmarcelo.alencar at gmail.com <jmarcelo.alencar at gmail.com>
> Date: Friday, January 20, 2023 at 6:45 AM
> To: openstack-discuss at lists.openstack.org <openstack-discuss at lists.openstack.org>
> Subject: [openstack-ansible] Installing OpenStack with Ansible fails during Keystone playbook on TASK openstack.osa.db_setup
>
> CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!
>
>
> Hello Community,
>
> I am trying to create a two machine deployment following Openstack
> Ansible Deployment Guide
> (https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.openstack.org%2Fproject-deploy-guide%2Fopenstack-ansible%2Flatest%2F&data=05%7C01%7Cjames.denton%40rackspace.com%7C2030b246126f4b053abd08dafae42aba%7C570057f473ef41c8bcbb08db2fc15c2b%7C0%7C0%7C638098155124685217%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jBqnF439N%2BD4e05ZoWzz11rMrtu1gxA7fxYStBnRXnw%3D&reserved=0).
> The two machines are named targethost01 and targethost02, and I am
> running Ansible from deploymenthost. Every machine has 4-Core CPUs, 8
> GB of RAM, and 240 GB SSD. I am using Ubuntu 22.04.1 LTS.
>
> The machine targethost01 has the following network configuration:
>
> network:
> version: 2
> ethernets:
> enp5s0:
> dhcp4: true
> enp6s0: {}
> enp7s0: {}
> enp8s0: {}
> enp9s0: {}
> vlans:
> vlan.10:
> id: 10
> link: enp6s0
> addresses: [ ]
> vlan.20:
> id: 20
> link: enp7s0
> addresses: [ ]
> vlan.30:
> id: 30
> link: enp8s0
> addresses: [ ]
> vlan.40:
> id: 40
> link: enp9s0
> addresses: [ ]
> bridges:
> br-mgmt:
> addresses: [ 172.29.236.101/22 ]
> mtu: 1500
> interfaces:
> - vlan.10
> br-storage:
> addresses: [ 172.29.244.101/22 ]
> mtu: 1500
> interfaces:
> - vlan.20
> br-vlan:
> addresses: []
> mtu: 1500
> interfaces:
> - vlan.30
> br-vxlan:
> addresses: [ 172.29.240.101/22 ]
> mtu: 1500
> interfaces:
> - vlan.40
>
>
> And targethost02 has the following network configuration:
>
>
> network:
> version: 2
> ethernets:
> enp5s0:
> dhcp4: true
> enp6s0: {}
> enp7s0: {}
> enp8s0: {}
> enp9s0: {}
> vlans:
> vlan.10:
> id: 10
> link: enp6s0
> addresses: [ ]
> vlan.20:
> id: 20
> link: enp7s0
> addresses: [ ]
> vlan.30:
> id: 30
> link: enp8s0
> addresses: [ ]
> vlan.40:
> id: 40
> link: enp9s0
> addresses: [ ]
> bridges:
> br-mgmt:
> addresses: [ 172.29.236.102/22 ]
> mtu: 1500
> interfaces:
> - vlan.10
> br-storage:
> addresses: [ 172.29.244.102/22 ]
> mtu: 1500
> interfaces:
> - vlan.20
> br-vlan:
> addresses: []
> mtu: 1500
> interfaces:
> - vlan.30
> br-vxlan:
> addresses: [ 172.29.240.102/22 ]
> mtu: 1500
> interfaces:
> - vlan.40
>
>
> On the deploymenthost, /etc/openstack_deploy/openstack_user_config.yml
> has the following:
>
>
> ---
> cidr_networks:
> container: 172.29.236.0/22
> tunnel: 172.29.240.0/22
> storage: 172.29.244.0/22
> used_ips:
> - 172.29.236.1
> - "172.29.236.100,172.29.236.200"
> - "172.29.240.100,172.29.240.200"
> - "172.29.244.100,172.29.244.200"
> global_overrides:
> internal_lb_vip_address: 172.29.236.101
> external_lb_vip_address: "{{ bootstrap_host_public_address |
> default(ansible_facts['default_ipv4']['address']) }}"
> management_bridge: "br-mgmt"
> provider_networks:
> - network:
> group_binds:
> - all_containers
> - hosts
> type: "raw"
> container_bridge: "br-mgmt"
> container_interface: "eth1"
> container_type: "veth"
> ip_from_q: "container"
> is_container_address: true
> - network:
> group_binds:
> - glance_api
> - cinder_api
> - cinder_volume
> - nova_compute
> type: "raw"
> container_bridge: "br-storage"
> container_type: "veth"
> container_interface: "eth2"
> container_mtu: "9000"
> ip_from_q: "storage"
> - network:
> group_binds:
> - neutron_linuxbridge_agent
> container_bridge: "br-vxlan"
> container_type: "veth"
> container_interface: "eth10"
> container_mtu: "9000"
> ip_from_q: "tunnel"
> type: "vxlan"
> range: "1:1000"
> net_name: "vxlan"
> - network:
> group_binds:
> - neutron_linuxbridge_agent
> container_bridge: "br-vlan"
> container_type: "veth"
> container_interface: "eth11"
> type: "vlan"
> range: "101:200,301:400"
> net_name: "vlan"
> - network:
> group_binds:
> - neutron_linuxbridge_agent
> container_bridge: "br-vlan"
> container_type: "veth"
> container_interface: "eth12"
> host_bind_override: "eth12"
> type: "flat"
> net_name: "flat"
> shared-infra_hosts:
> targethost01:
> ip: 172.29.236.101
> repo-infra_hosts:
> targethost01:
> ip: 172.29.236.101
> coordination_hosts:
> targethost01:
> ip: 172.29.236.101
> os-infra_hosts:
> targethost01:
> ip: 172.29.236.101
> identity_hosts:
> targethost01:
> ip: 172.29.236.101
> network_hosts:
> targethost01:
> ip: 172.29.236.101
> compute_hosts:
> targethost01:
> ip: 172.29.236.101
> targethost02:
> ip: 172.29.236.102
> storage-infra_hosts:
> targethost01:
> ip: 172.29.236.101
> storage_hosts:
> targethost01:
> ip: 172.29.236.101
>
>
> Also on the deploymenthost, /etc/openstack_deploy/conf.d/haproxy.yml
> has the following:
>
>
> haproxy_hosts:
> targethost01:
> ip: 172.29.236.101
>
>
> At the Run Playbooks step of the guide, the following two Ansible
> commands return with unreachable=0 failed=0:
>
> # openstack-ansible setup-hosts.yml
> # openstack-ansible setup-infrastructure.yml
>
> And verifying the database also returns no error:
>
>
> root at deploymenthost:/opt/openstack-ansible/playbooks# ansible
> galera_container -m shell \
> -a "mysql -h localhost -e 'show status like \"%wsrep_cluster_%\";'"
> Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e
> @/etc/openstack_deploy/user_variables.yml "
> [WARNING]: Unable to parse /etc/openstack_deploy/inventory.ini as an
> inventory source
> targethost01_galera_container-5aa8474a | CHANGED | rc=0 >>
> Variable_name Value
> wsrep_cluster_weight 1
> wsrep_cluster_capabilities
> wsrep_cluster_conf_id 1
> wsrep_cluster_size 1
> wsrep_cluster_state_uuid e7a0c332-97fe-11ed-b0d4-26b30049826d
> wsrep_cluster_status Primary
>
>
> But when I execute openstack-ansible setup-openstack.yml, I get this:
>
>
> TASK [os_keystone : Fact for apache module mod_auth_openidc to be installed] ***
> ok: [targethost01_keystone_container-76e9b31b]
> TASK [include_role : openstack.osa.db_setup] ***********************************
> TASK [openstack.osa.db_setup : Create database for service] ********************
> failed: [targethost01_keystone_container-76e9b31b ->
> targethost01_utility_container-dc05dc90(172.29.238.59)] (item=None) =>
> {"censored": "the output has been hidden due to the fact that 'no_log:
> true' was specified for this result", "changed": false}
> fatal: [targethost01_keystone_container-76e9b31b -> {{
> _oslodb_setup_host }}]: FAILED! => {"censored": "the output has been
> hidden due to the fact that 'no_log: true' was specified for this
> result", "changed": false}
> PLAY RECAP *********************************************************************
> targethost01_keystone_container-76e9b31b : ok=33 changed=0
> unreachable=0 failed=1 skipped=8 rescued=0 ignored=0
> targethost01_utility_container-dc05dc90 : ok=3 changed=0
> unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
> EXIT NOTICE [Playbook execution failure] **************************************
> ===============================================================================
>
>
> First, how can I disable the "censored" warning? I wonder if the
> uncensored running could give me more clues. Second, it appears to be
> a problem creating the database (keystone db sync?) How can I test the
> database execution inside the LXC containers? I tried to log into one
> of the containers and ping the hosts IP and it works, so they have
> connectivity. I set up the passwords with:
>
> # cd /opt/openstack-ansible
> # ./scripts/pw-token-gen.py --file /etc/openstack_deploy/user_secrets.yml
>
>
> Any help?
>
> Best Regards.
>
>
>
>
> --
> __________________________________
>
> João Marcelo Uchôa de Alencar
> jmarcelo.alencar(at)gmail.com
> __________________________________
--
__________________________________
João Marcelo Uchôa de Alencar
jmarcelo.alencar(at)gmail.com
__________________________________
More information about the openstack-discuss
mailing list