<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1688025678;
mso-list-type:hybrid;
mso-list-template-ids:-2088367100 -1754785022 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:2076394006;
mso-list-type:hybrid;
mso-list-template-ids:930931748 1675396942 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">>> </span><span style="font-size:11.0pt">If I am using bonding on the infra nodes, should the haproxy_keepalived_external_interface be the device name (enp1s0) or bond0?<br>
<br>
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This will likely be the bond0 interface and not the individual bond member. However, the interface defined here will ultimately depend on the networking of that host, and should be an external facing one (i.e.
the interface with the default gateway).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In many environments, you’ll have something like this (or using 2 bonds, but same idea):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo2"><span style="font-size:11.0pt">Bond0 (192.168.100.5/24 gw 192.168.100.1)<o:p></o:p></span></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo2"><span style="font-size:11.0pt">Em49<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo2"><span style="font-size:11.0pt">Em50<o:p></o:p></span></li></ul>
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo2"><span style="font-size:11.0pt">Br-mgmt (172.29.236.5/22)<o:p></o:p></span></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo2"><span style="font-size:11.0pt">Bond0.236<o:p></o:p></span></li></ul>
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo2"><span style="font-size:11.0pt">Br-vxlan (172.29.240.5/22)<o:p></o:p></span></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo2"><span style="font-size:11.0pt">Bond0.240<o:p></o:p></span></li></ul>
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo2"><span style="font-size:11.0pt">Br-storage (172.29.244.5/22)<o:p></o:p></span></li><ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo2"><span style="font-size:11.0pt">Bond0.244<o:p></o:p></span></li></ul>
</ul>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In this example, bond0 has the management IP 192.168.100.5 and br-mgmt is the “container” bridge with an IP configured from the ‘container’ network (see cidr_networks in openstack_user_config.yml). FYI: LXC
containers will automatically be assigned IPs from the ‘container’ network outside of the ‘used_ips’ range(s). The infra host will communicate with the containers via this br-mgmt interface.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m using FQDNs for the VIPs, which are specified in openstack_user_config.yml here:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">global_overrides:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> internal_lb_vip_address: internalapi.openstack.rackspace.lab<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> external_lb_vip_address: publicapi.openstack.rackspace.lab<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">To avoid DNS resolution issues internally (or rather, to ensure the IP is configured in the config files and not the domain name) I’ll override with the IP and hard set the preferred interface(s):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">haproxy_keepalived_external_vip_cidr: "192.168.100.10/32"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">haproxy_keepalived_internal_vip_cidr: "172.29.236.10/32"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">haproxy_keepalived_external_interface: bond0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">haproxy_keepalived_internal_interface: br-mgmt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">haproxy_bind_external_lb_vip_address: 192.168.100.10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">haproxy_bind_internal_lb_vip_address: 172.29.236.10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">With the above configuration, keepalived will manage two VIPs - one external and one internal, and endpoints will have the FQDN rather than IP.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">>> </span><span style="font-size:11.0pt">Curl shows "503 Service Unavailable No server is available to handle this request”<br>
<br>
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hard to say without seeing logs why this is happening, but I will assume that keepalived is having issues binding the IP to the interface. You might find the reason in syslog or ‘journalctl -xe -f -u keepalived’.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">>> </span><span style="font-size:11.0pt">Running "systemctl status var-www-repo.mount” gives an output of “Unit var-www-repo.mount could not be found."<br>
<br>
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">You might try running ‘umount /var/www/repo’ and re-run the repo-install.yml playbook (or setup-infrastructure.yml).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hope that helps!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">James Denton<br>
Rackspace Private Cloud</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Father Vlasie <fv@spots.edu><br>
<b>Date: </b>Tuesday, August 16, 2022 at 4:31 PM<br>
<b>To: </b>James Denton <james.denton@rackspace.com><br>
<b>Cc: </b>openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
<b>Subject: </b>Re: [openstack-ansible] [yoga] utility_container failure<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt">CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!<br>
<br>
<br>
Hello,<br>
<br>
Thank you very much for the reply!<br>
<br>
haproxy and keepalived both show status active on infra1 (my primary node).<br>
<br>
Curl shows "503 Service Unavailable No server is available to handle this request”<br>
<br>
(Also the URL is http not https….)<br>
<br>
If I am using bonding on the infra nodes, should the haproxy_keepalived_external_interface be the device name (enp1s0) or bond0?<br>
<br>
Earlier in the output I find the following error (showing for all 3 infra nodes):<br>
<br>
------------<br>
<br>
TASK [systemd_mount : Set the state of the mount] *****************************************************************************************************************************************<br>
fatal: [infra3_repo_container-7ca5db88]: FAILED! => {"changed": false, "cmd": "systemctl reload-or-restart $(systemd-escape -p --suffix=\"mount\" \"/var/www/repo\")", "delta": "0:00:00.022275", "end": "2022-08-16 14:16:34.926861", "msg": "non-zero return code",
"rc": 1, "start": "2022-08-16 14:16:34.904586", "stderr": "Job for var-www-repo.mount failed.\nSee \"systemctl status var-www-repo.mount\" and \"journalctl -xe\" for details.", "stderr_lines": ["Job for var-www-repo.mount failed.", "See \"systemctl status
var-www-repo.mount\" and \"journalctl -xe\" for details."], "stdout": "", "stdout_lines": []}<br>
<br>
——————<br>
<br>
Running "systemctl status var-www-repo.mount” gives an output of “Unit var-www-repo.mount could not be found."<br>
<br>
Thank you again!<br>
<br>
Father Vlasie<br>
<br>
> On Aug 16, 2022, at 6:32 AM, James Denton <james.denton@rackspace.com> wrote:<br>
><br>
> Hello,<br>
><br>
> That error means the repo server at 192.168.3.9:8181 is unavailable. The repo server sits behind haproxy, which should be listening on 192.168.3.9 port 8181 on the active (primary) node. You can verify this by issuing a ‘curl -v
<a href="https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2F192.168.3.9%3A8181%2F%25E2%2580%2599&data=05%7C01%7Cjames.denton%40rackspace.com%7C17cc3373086d47b3321408da7fceb08a%7C570057f473ef41c8bcbb08db2fc15c2b%7C0%7C0%7C637962823085464366%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PSsQW%2FW9N5JXapmMal%2FrsUm%2B8IYkDFTxwqxaH3K7tbA%3D&reserved=0">
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2F192.168.3.9%3A8181%2F%25E2%2580%2599&data=05%7C01%7Cjames.denton%40rackspace.com%7C17cc3373086d47b3321408da7fceb08a%7C570057f473ef41c8bcbb08db2fc15c2b%7C0%7C0%7C637962823085464366%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PSsQW%2FW9N5JXapmMal%2FrsUm%2B8IYkDFTxwqxaH3K7tbA%3D&reserved=0</a>.
You might check the haproxy service status and/or keepalived status to ensure they are operating properly. If the IP cannot be bound to the correct interface, keepalive may not start.<br>
><br>
> James Denton<br>
> Rackspace Private Cloud<br>
><br>
> From: Father Vlasie <fv@spots.edu><br>
> Date: Tuesday, August 16, 2022 at 7:38 AM<br>
> To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
> Subject: [openstack-ansible] [yoga] utility_container failure<br>
><br>
> CAUTION: This message originated externally, please use caution when clicking on links or opening attachments!<br>
><br>
><br>
> Hello everyone,<br>
><br>
> I have happily progressed to the second step of running the playbooks, namely "openstack-ansible setup-infrastructure.yml"<br>
><br>
> Everything looks good except for just one error which is mystifying me:<br>
><br>
> ----------------<br>
><br>
> TASK [Get list of repo packages] **********************************************************************************************************************************************************<br>
> fatal: [infra1_utility_container-5ec32cb5]: FAILED! => {"changed": false, "content": "", "elapsed": 30, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error timed out>", "redirected": false, "status": -1, "url": "<a href="https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.3.9%3A8181%2Fconstraints%2Fupper_constraints_cached.txt&data=05%7C01%7Cjames.denton%40rackspace.com%7C17cc3373086d47b3321408da7fceb08a%7C570057f473ef41c8bcbb08db2fc15c2b%7C0%7C0%7C637962823085620584%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XhtQdDC8GpQuJXGbQxhlBkUOS3krH%2F9d%2FxU9hWB1Cts%3D&reserved=0">https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.3.9%3A8181%2Fconstraints%2Fupper_constraints_cached.txt&data=05%7C01%7Cjames.denton%40rackspace.com%7C17cc3373086d47b3321408da7fceb08a%7C570057f473ef41c8bcbb08db2fc15c2b%7C0%7C0%7C637962823085620584%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XhtQdDC8GpQuJXGbQxhlBkUOS3krH%2F9d%2FxU9hWB1Cts%3D&reserved=0</a>"}<br>
><br>
> ----------------<br>
><br>
> 192.168.3.9 is the IP listed in user_variables.yml under haproxy_keepalived_internal_vip_cidr<br>
><br>
> Any help or pointers would be very much appreciated!<br>
><br>
> Thank you,<br>
><br>
> Father Vlasie<br>
><o:p></o:p></span></p>
</div>
</div>
</body>
</html>