<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }div.foxdiv20140516182626710390 { }body { font-size: 10.5pt; font-family: 微软雅黑; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>Hi Marica:</div><div><span style="background-color: rgba(0, 0, 0, 0);"> I have test your RA script for 2 days. After so many times of </span><span style="background-color: rgba(0, 0, 0, 0);">attempts i</span><span style="background-color: rgba(0, 0, 0, 0);">t finally works well :)</span></div><div><span style="background-color: rgba(0, 0, 0, 0);"> I changed serveral settings(</span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">resource-stickiness,RA script,neutron l3-agent setting etc.)</span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> in my envirment ,so I can't tell which is the key point of my changes.</span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> I have attach my RA script here ,hope to help anybody else with same problems,</span><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;">and I have commented for all my changes.</span></div><div><span style="font-size: 10.5pt; line-height: 1.5; background-color: rgba(0, 0, 0, 0);"> In my RA script I had made a change to yours "</span><span style="font-size: 10.5pt; line-height: 1.5; background-color: rgba(0, 0, 0, 0);">restore old hostname" section:</span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> The origin section is :</span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">hostname Network01</span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> I changed it to : </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">hostname $(cat /etc/sysconfig/network | grep HOSTNAME | awk -F "=" '{print $2}') </span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> So both nodes can use one RA script.</span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"><br></span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> </span><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;">Aleita's </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">method is good because when l3-agent start on whicherver node,the l3-agent id which hosting-router is the same because the node's hostname is the same,</span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> in this example script,it's network-controller. We can use </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">neutron l3-agent-list-hosting-router $ext-router-id to check it.It will always be like:</span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">+--------------------------------------+--------------------+----------------+-------+ </span></div><span style="background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0);">| id | host | admin_state_up | alive |
<br></span><span style="background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0);">+--------------------------------------+--------------------+----------------+-------+
<br></span><span style="background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0);">| XXXXXXXX-xxxxxx-xxxxx-xxxxx-xxxxxxxxxxxx| network-controller | True | :-) |
<br></span><span style="background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0);">+--------------------------------------+--------------------+----------------+-------+</span><div> So when one node goes down,other node's l3-agent can take the same l3-agent id as own.</div><div><br></div><div> And my thought before your mail was: remove the l3-agent from hosting router and add the backup l3-agent to hosting router,something like:</div><div><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;"> #=============================================================================================</span></div><div><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;"> </span><span style="font-size: 10.5pt; line-height: 1.5; background-color: rgba(0, 0, 0, 0);">down_l3_agent_ID=$(/usr/bin/neutron agent-list | grep 'L3 agent' | awk '$7!="'`hostname`'" {print $2}') </span></div><span style="background-color: rgba(0, 0, 0, 0);">
<br> back_l3_agent_ID=$(/usr/bin/neutron agent-list | grep 'L3 agent' | awk '$7=="'`hostname`'" {print $2}')</span><div><br></div><div> <span style="background-color: rgba(0, 0, 0, 0);">for r in $(/usr/bin/neutron router-list-on-l3-agent $down_l3_agent_ID | awk 'NR>3 && NF>1{print $2}');
<br></span><span style="background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">do /usr/bin/neutron l3-agent-router-remove $down_l3_agent_ID $r && /usr/bin/neutron l3-agent-router-add $back_l3_agent_ID $r; </span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">done</span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> </span><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;">#=============================================================================================</span></div><div><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;"><br></span></div><div><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;"> I think it will work as well,but your method is better I think :) </span></div><div><span style="font-size: 10.5pt; line-height: 1.5; background-color: window;"> So thank you very much!</span></div><div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"><br></span></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;"> btw: I have change </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">OCF_RESKEY_agent_config_default to </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 10.5pt; line-height: 1.5;">OCF_RESKEY_plugin_config_default, otherwise we can't set the pacemaker as </span><span style="font-size: 10.5pt; line-height: 1.5; font-family: Helvetica; background-color: window;">high-availability-guide like :</span></div><div><span style="font-size: 10.5pt; line-height: 1.5; font-family: Helvetica; background-color: window;"> </span><span style="font-family: Helvetica; font-size: 10.5pt; line-height: 1.5; background-color: window;">primitive p_neutron-l3-agent ocf:openstack:neutron-agent-l3 \ </span></div><span style="font-family: Helvetica;"> params config="/etc/neutron/neutron.conf" \
<br> plugin_config="/etc/neutron/l3_agent.ini" \
<br> op monitor interval="30s" timeout="30s" </span><div><font face="Helvetica"><br></font></div><div><font face="Helvetica"> these changes are based on :</font><span style="font-family: Helvetica; background-color: rgba(0, 0, 0, 0);"></span><span style="font-family: Helvetica; background-color: rgba(0, 0, 0, 0);"></span> <a href="https://bugs.launchpad.net/openstack-manuals/+bug/1252131" style="font-family: Helvetica;">https://bugs.launchpad.net/openstack-manuals/+bug/1252131</a><span style="font-family: Helvetica; background-color: rgba(0, 0, 0, 0);"> </span></div><div><font face="Helvetica"><br></font></div><div><font face="Helvetica"> <br></font><div><br></div>
<div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span>Walter Xu</span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:walterxj@gmail.com">walterxj</a></div><div><b>Date:</b> 2014-05-15 09:51</div><div><b>To:</b> <a href="mailto:marica.antonacci@gmail.com">Marica Antonacci</a></div><div><b>Subject:</b> Re: Re: [Openstack] pacemaker would be wrong when both node have same hostname</div></div></div><div><div class="FoxDiv20140516182626710390">
<div class="">Hi Marica,<br> When I use "crm node standby",it seems work,but I think by that way,the virtual router still resident on the former node because when I poweroff this node, the VM instance can not access the external net.<br> I'll test again carefully,After testing I'll feed back to you.<br> Thank you again for your help.</div><div> </div><hr style="WIDTH: 210px; HEIGHT: 1px" color="#b5c4df" size="1" align="left"><div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt">walterxj</div></span></div><blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:marica.antonacci@gmail.com">Marica Antonacci</a></div><div><b>Date:</b> 2014-05-14 22:21</div><div><b>To:</b> <a href="mailto:walterxj@gmail.com">xu Walter</a></div><div><b>Subject:</b> Re: [Openstack] pacemaker would be wrong when both node have same hostname</div></div></div><div><div class=""><div class="FoxDIV_20140515090833187" style="font-size: 10.5pt;"><div foxmail_class="mail_content_body" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi Walter,<div><br></div><div>we are using it in our production havana environment. We have tested it both using "crm node standby” and “crm resource migrate g_network <node2>” and turning off the node network interfaces and shutting down the node, etc..</div><div><br></div><div>Have you modified our script using the correct hostnames for the two different nodes?</div><div><br></div><div>Cheers,</div><div>Marica </div><div><br><div><div>Il giorno 14/mag/2014, alle ore 16:08, xu Walter <<a href="mailto:walterxj@gmail.com">walterxj@gmail.com</a>> ha scritto:</div><br class="Apple-interchange-newline">
<div class=""><blockquote type="cite" style="margin-top: 0px;"><div dir="ltr">Hi Marica:<div> Thanks for your script,but it seems not work for me.I want to know how did you test it? Just use "crm node standby" or shutdown the node physically?</div><div class="gmail_extra">
<br><br><div class="gmail_quote">2014-05-14 19:49 GMT+08:00 Marica Antonacci <span dir="ltr"><<a href="mailto:marica.antonacci@gmail.com" target="_blank">marica.antonacci@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">
<div style="word-wrap:break-word">Hi,<div><br></div><div>in attachment you can find our modified resource agent…we have noticed that the network namespaces (router and dhcp) are automatically re-created on the new node when the resource manager migrates the network controller on the other physical node (we have grouped all the services related to the network node).</div>
<div><br></div><div>Please, note that the attached script contains also other patches wrt to the RA available at <a href="https://raw.githubusercontent.com/madkiss/openstack-resource-agents/master/ocf/neutron-agent-l3" target="_blank">https://raw.githubusercontent.com/madkiss/openstack-resource-agents/master/ocf/neutron-agent-l3</a> because we found some issues with the resource agent parameters and the port used to check the established connection with the server; moreover we have added the start/stop operations for the neutron-plugin-openvswitch-agent since there is no available RA at the moment for this service. </div>
<div><br></div><div>Cheers, </div><div>Marica</div><div>---------------------------</div>dhcp117:~ marica$ diff -w -b neutron-agent-l3 neutron-agent-l3.1<br>20c20<br>< # OCF_RESKEY_agent_config<br>---<br>> # OCF_RESKEY_plugin_config<br>
37c37<br>< OCF_RESKEY_agent_config_default="/etc/neutron/l3_agent.ini"<br>---<br>> OCF_RESKEY_plugin_config_default="/etc/neutron/l3_agent.ini"<br>40c40<br>< OCF_RESKEY_neutron_server_port_default="5672"<br>
---<br>> OCF_RESKEY_neutron_server_port_default="9696"<br>44c44<br>< : ${OCF_RESKEY_agent_config=${OCF_RESKEY_agent_config_default}}<br>---<br>> : ${OCF_RESKEY_plugin_config=${OCF_RESKEY_plugin_config_default}}<br>
98c98<br>< <parameter name="agent_config" unique="0" required="0"><br>---<br>> <parameter name="plugin config" unique="0" required="0"><br>103c103<br>
< <content type="string" default="${OCF_RESKEY_agent_config_default}" /><br>---<br>> <content type="string" default="${OCF_RESKEY_plugin_config_default}" /><br>241,247d240<br>
< # Aleita<br>< # change hostname<br>< hostname network-controller<br>< <br>< #Marant: temporary patch - restart neutron-plugin-openvswitch-agent<br>< service neutron-plugin-openvswitch-agent start <br>
< <br>251c244<br>< --config-file=$OCF_RESKEY_agent_config --log-file=/var/log/neutron/l3-agent.log $OCF_RESKEY_additional_parameters"' >> \<br>---<br>> --config-file=$OCF_RESKEY_plugin_config --log-file=/var/log/neutron/l3-agent.log $OCF_RESKEY_additional_parameters"' >> \<br>
320,325d312<br>< # Aleita<br>< # restore old hostname<br>< hostname node1<br>< <br>< #Marant: <br>< service neutron-plugin-openvswitch-agent stop<br><br><div><br></div><div></div></div>
<br><div style="word-wrap:break-word"><div></div><div> </div><div><div><br></div><div><br><div><div>Il giorno 14/mag/2014, alle ore 13:04, walterxj <<a href="mailto:walterxj@gmail.com" target="_blank">walterxj@gmail.com</a>> ha scritto:</div>
<br><blockquote type="cite" style="margin-top: 0px;"><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div>Hi Marica:<br> Can you give me your modified RA to me?And is that with your RA you configure the pacemaker just like the guide?I mean I notice that when the former network node down,the l3-agent of the back node must bind the router which belong to the former node.</div>
<div> </div><hr size="1" align="left" style="width:210px;min-height:1px"><div><span><div style="margin:10px;font-family:verdana;font-size:10pt">walterxj</div></span></div><blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;">
<div> </div><div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0cm 0cm"><div style="padding:8px;font-size:12px;font-family:tahoma;background-color:rgb(239,239,239)">
<div><b>From:</b> <a href="mailto:marica.antonacci@gmail.com" target="_blank">Marica Antonacci</a></div><div><b>Date:</b> 2014-05-14 18:55</div><div><b>To:</b> <a href="mailto:walterxj@gmail.com" target="_blank">walterxj</a></div>
<div><b>CC:</b> <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack</a></div><div><b>Subject:</b> Re: [Openstack] pacemaker would be wrong when both node have same hostname</div></div></div><div>
<div style="font-size:10.5pt"><div style="word-wrap:break-word">Hi all,<div><br></div><div>we are currently using pacemaker to manage 2 network nodes (node1, node2) and we have modified the neutron L3 agent RA in order to dynamically change the hostname of the active network node: start() function sets the hostname “network-controller" to be used by the scheduler; the stop() function restores the old hostname (“node1” or “node2”). It seems to work, yet it’s a rude patch :) A more general solution that exploits neutron functionalities would be very appreciated!</div>
<div><br></div><div>Best,</div><div>Marica </div><div><br><div><div>Il giorno 14/mag/2014, alle ore 12:34, walterxj <<a href="mailto:walterxj@gmail.com" target="_blank">walterxj@gmail.com</a>> ha scritto:</div>
<br><div><blockquote type="cite" style="margin-top: 0px;"><div style="line-height:1.5;font-size:10.5pt;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div><span></span>hi:</div><div> the high-availability-guide (<a rel="nofollow" href="http://docs.openstack.org/high-availability-guide/content/ch-network.html" style="font-size:10.5pt;line-height:1.5;background-color:window" target="_blank">http://<u></u>docs.openstack.<u></u>org/high-<u></u>availability-<u></u>guide/content/<u></u>ch-network.<u></u>html</a>) says that Both nodes should have the same hostname since the Networking scheduler will be aware of one node, for example a virtual router attached to a single L3 node.</div>
<div><br></div><div> <span style="font-size:10.5pt;line-height:1.5;background-color:window">But when I test it on two servers with same hostname,after installing corosync and pacemaker service on them(with no resource configured),the crm_mon output goes into endless loop.And in the log of corosync,there are so many messages like:May 09 22:25:40 [2149] TEST crmd: warning: crm_get_peer: Node 'TEST' and 'TEST' share the same cluster nodeid: 1678901258.After this I set diffrent nodeid in /etc/corosync/</span><u></u><span style="font-size:10.5pt;line-height:1.5;background-color:window">corosync.</span><u></u><span style="font-size:10.5pt;line-height:1.5;background-color:window">conf of each test node,but it didn't help.</span></div>
So,I set diffrent hostname for each server,and then configure pacemaker just like the manual except the hostname,the neutron-dhcp-agent and neutron-<u></u>metadata-<u></u>agent works well,but neutron-l3-agent not(VM instance can't not access the external net,further more the gateway of the VM instance can't be accessed either).<br>
After two days checking,finally I found that we can use "netron l3-agent-<u></u>router-<u></u>remove network1_l3_agentid external-routeid" and "netron l3-agent-router-add network2_l3_agentid external-routeid" to let the backup l3-agent to work when the former network node is down.(assume the two node's names are network1 and network2)<u></u>,alternatively,<u></u>we can update the mysql table routerl3agentbi<u></u>ndings in neutron base directly.If it make sense,I think we can change the scrip neutron-agent-l3 , in it's neutron_<u></u>l3_agent_<u></u>start() function,only need few lines to make it work well.<div>
<br></div><hr size="1" align="left" style="width:210px;min-height:1px"><div><span>Walter Xu</span></div>_______________________________________________<br>Mailing list:<span> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to :<span> </span><a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>Unsubscribe :<span> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></div>
</blockquote></div></div></div></div></div></div></blockquote></div></blockquote></div><br></div></div></div><br></blockquote></div><br></div></div>
</blockquote></div>
</div><br></div></div></div></div></div></blockquote></div></div></blockquote>
</div></div></body></html>