<div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace">Hrmmm it isn't going so well:</div><div class="gmail_default" style="font-family:'courier new',monospace"><br></div>
<div class="gmail_default"><div class="gmail_default"><font face="courier new, monospace">root@test1# ip a s dev eth0</font></div><div class="gmail_default"><font face="courier new, monospace">2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000</font></div>
<div class="gmail_default"><font face="courier new, monospace"> link/ether 00:25:90:10:00:78 brd ff:ff:ff:ff:ff:ff</font></div><div class="gmail_default"><font face="courier new, monospace"> inet <a href="http://10.21.0.1/16">10.21.0.1/16</a> brd 10.21.255.255 scope global eth0</font></div>
<div class="gmail_default"><font face="courier new, monospace"> inet <a href="http://10.21.1.1/16">10.21.1.1/16</a> brd 10.21.255.255 scope global secondary eth0</font></div><div class="gmail_default"><font face="courier new, monospace"> inet <a href="http://10.21.21.1/16">10.21.21.1/16</a> scope global secondary eth0</font></div>
<div class="gmail_default"><font face="courier new, monospace"> inet6 fe80::225:90ff:fe10:78/64 scope link </font></div><div class="gmail_default"><font face="courier new, monospace"> valid_lft forever preferred_lft forever</font></div>
<div class="gmail_default"><font face="courier new, monospace"><br></font></div><div class="gmail_default"><font face="courier new, monospace"><br></font></div><div class="gmail_default"><font face="courier new, monospace"><div class="gmail_default">
root@test1# ipvsadm -L -n</div><div class="gmail_default">IP Virtual Server version 1.2.1 (size=4096)</div><div class="gmail_default">Prot LocalAddress:Port Scheduler Flags</div><div class="gmail_default"> -> RemoteAddress:Port Forward Weight ActiveConn InActConn</div>
<div class="gmail_default">TCP <a href="http://10.21.21.1:5000">10.21.21.1:5000</a> wlc persistent 600</div><div class="gmail_default"> -> <a href="http://10.21.0.1:5000">10.21.0.1:5000</a> Masq 100 0 1 </div>
<div class="gmail_default"> -> <a href="http://10.21.0.2:5000">10.21.0.2:5000</a> Masq 100 0 0 </div><div class="gmail_default">TCP <a href="http://10.21.21.1:35357">10.21.21.1:35357</a> wlc persistent 600</div>
<div class="gmail_default"> -> <a href="http://10.21.0.1:35357">10.21.0.1:35357</a> Masq 100 0 0 </div><div class="gmail_default"> -> <a href="http://10.21.0.2:35357">10.21.0.2:35357</a> Masq 100 0 0</div>
<div class="gmail_default"><br></div><div class="gmail_default"><div class="gmail_default">root@test1# iptables -L -v -tnat</div><div class="gmail_default">Chain PREROUTING (policy ACCEPT 283 packets, 24902 bytes)</div><div class="gmail_default">
pkts bytes target prot opt in out source destination </div><div class="gmail_default"><br></div><div class="gmail_default">Chain INPUT (policy ACCEPT 253 packets, 15256 bytes)</div><div class="gmail_default">
pkts bytes target prot opt in out source destination </div><div class="gmail_default"><br></div><div class="gmail_default">Chain OUTPUT (policy ACCEPT 509 packets, 37182 bytes)</div><div class="gmail_default">
pkts bytes target prot opt in out source destination </div><div class="gmail_default"><br></div><div class="gmail_default">Chain POSTROUTING (policy ACCEPT 196 packets, 12010 bytes)</div>
<div class="gmail_default"> pkts bytes target prot opt in out source destination </div><div class="gmail_default"> 277 16700 MASQUERADE all -- any eth0 anywhere anywhere</div>
<div class="gmail_default"><br></div><div class="gmail_default">root@test1:~# export OS_AUTH_URL="<a href="http://10.21.21.1:5000/v2.0/">http://10.21.21.1:5000/v2.0/</a>"<br></div><div class="gmail_default"><div class="gmail_default">
root@test1:~# keystone user-list</div><div class="gmail_default">No handlers could be found for logger "keystoneclient.client"</div><div class="gmail_default">Unable to communicate with identity service: [Errno 113] No route to host. (HTTP 400)</div>
<div class="gmail_default"><br></div><div class="gmail_default"><br></div><div class="gmail_default" style>I still have some debugging to do with tcpdump, but I thought I would post my initial results.</div></div></div></font></div>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Feb 15, 2013 at 2:56 PM, Sébastien Han <span dir="ltr"><<a href="mailto:han.sebastien@gmail.com" target="_blank">han.sebastien@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Well if you follow my article, you will get LVS-NAT running. It's fairly easy, no funky stuff. Yes you will probably need the postrouting rule, as usual :). Let me know how it goes ;)</div>
<div class="gmail_extra">
<br clear="all"><div><div>--</div>Regards,<div>Sébastien Han.</div></div><div><div class="h5">
<br><br><div class="gmail_quote">On Fri, Feb 15, 2013 at 8:51 PM, Samuel Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I<div style="font-family:'courier new',monospace;display:inline" class="gmail_default"> didn't give NAT a shot because it didn't seem as well documented.</div><div><div style="font-family:'courier new',monospace;display:inline" class="gmail_default">
<br></div></div><div><div style="font-family:'courier new',monospace;display:inline" class="gmail_default">I will give NAT a shot. Will I need to enable to iptables and add a rule to the nat table? None of the documentation mentioned that but every time I have ever done NAT I had to setup a rule like... iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE</div>
</div><div><div style="font-family:'courier new',monospace;display:inline" class="gmail_default"><br></div></div><div><div style="font-family:'courier new',monospace;display:inline" class="gmail_default">
Thanks for helping me with this.</div>
</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Feb 15, 2013 at 2:07 PM, Sébastien Han <span dir="ltr"><<a href="mailto:han.sebastien@gmail.com" target="_blank">han.sebastien@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ok but why direct routing instead of NAT? If the public IPs are _only_<br>
on LVS there is no point to use LVS-DR.<br>
<br>
LVS has the public IPs and redirects to the private IPs, this _must_ work.<br>
<br>
Did you try NAT? Or at least can you give it a shot?<br>
--<br>
Regards,<br>
Sébastien Han.<br>
<div><div><br>
<br>
On Fri, Feb 15, 2013 at 3:55 PM, Samuel Winchenbach <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>> wrote:<br>
> Sure... I have undone these settings but I saved a copy:<br>
><br>
> two hosts:<br>
> test1 eth0: <a href="http://10.21.0.1/16" target="_blank">10.21.0.1/16</a> eth1: 130.x.x.x/24<br>
> test2 eth0: <a href="http://10.21.0.2/16" target="_blank">10.21.0.2/16</a> eth1: 130.x.x.x/24<br>
><br>
> VIP: 10.21.21.1 (just for testing, later I would add a 130.x.x.x/24 VIP for<br>
> public APIs<br>
><br>
> k<br>
> eystone is bound to 10.21.0.1 on test1 and 10.21.0.2 on test2<br>
><br>
><br>
><br>
> in /etc/sysctl.conf:<br>
> net.ipv4.conf.all.arp_ignore = 1<br>
> net.ipv4.conf.eth0.arp_ignore = 1<br>
> net.ipv4.conf.all.arp_announce = 2<br>
> net.ipv4.conf.eth0.arp_announce = 2<br>
><br>
> root# sysctl -p<br>
><br>
> in /etc/sysctl.conf:<br>
><br>
> checktimeout=<br>
> 3<br>
><br>
><br>
> checkinterval=<br>
> 5<br>
><br>
><br>
> autoreload=<br>
> yes<br>
><br>
><br>
> logfile="/var/log/ldirectord.log"<br>
><br>
> quiescent=no<br>
><br>
> virtual=<a href="http://10.21.21.1:5000" target="_blank">10.21.21.1:5000</a><br>
><br>
> real=10.2<br>
> 1<br>
> .0.1:5000 gate<br>
><br>
> real=10.2<br>
> 1<br>
> .0.2:5000 gate<br>
><br>
> scheduler=<br>
> w<br>
> rr<br>
> protocol=tcp<br>
> checktype=connect<br>
> checkport=5000<br>
><br>
> virtual=<a href="http://10.21.21.1" target="_blank">10.21.21.1</a>:<br>
> 35357<br>
><br>
> real=10.2<br>
> 1<br>
> .0.1:<br>
> 35357<br>
> gate<br>
><br>
> real=10.2<br>
> 1<br>
> .0.2:<br>
> 35357<br>
> gate<br>
><br>
> scheduler=<br>
> w<br>
> rr<br>
> protocol=tcp<br>
> checktype=connect<br>
> checkport=35357<br>
><br>
><br>
> crm shell:<br>
><br>
><br>
> primitive<br>
> p_openstack_<br>
> ip ocf:heartbeat:IPaddr2 \<br>
><br>
><br>
> op monitor interval="60" timeout="20" \<br>
><br>
><br>
> params ip=<br>
> "10.21.21.1<br>
> "<br>
> cidr_netmask="<br>
> 16<br>
> "<br>
> lvs_support="true"<br>
><br>
> p<br>
> rimitive<br>
> p_openstack_ip_lo<br>
> ocf:heartbeat:IPaddr2 \<br>
><br>
><br>
> op monitor interval="60" timeout="20" \<br>
><br>
><br>
> params ip="<br>
> 10.21.21.1<br>
> " nic="lo"<br>
> cidr_netmask="32"<br>
><br>
> primitive<br>
> p_openstack_<br>
> lvs ocf:heartbeat:ldirectord \<br>
><br>
><br>
> op monitor interval="20" timeout="10"<br>
><br>
> group<br>
> g_openstack_<br>
> ip<br>
> _<br>
> lvs<br>
> p_openstack_<br>
> ip<br>
> p_openstack_<br>
> lvs<br>
><br>
> clone<br>
> c_openstack_ip_lo<br>
><br>
> p_openstack_ip_lo<br>
> meta interleave="true"<br>
><br>
> colocation<br>
> co_openstack_lo_never_lvs<br>
> -inf: c<br>
> _openstack_ip_lo<br>
><br>
> g_openstack_ip_lvs<br>
><br>
> Thanks for taking a look at this.<br>
><br>
> Sam<br>
><br>
><br>
><br>
><br>
> On Fri, Feb 15, 2013 at 3:54 AM, Sébastien Han <<a href="mailto:han.sebastien@gmail.com" target="_blank">han.sebastien@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Hum I don't see the problem, it's possible to load-balance VIPs with LVS,<br>
>> there are just IPs... Can I see your conf?<br>
>><br>
>> --<br>
>> Regards,<br>
>> Sébastien Han.<br>
>><br>
>><br>
>> On Thu, Feb 14, 2013 at 8:34 PM, Samuel Winchenbach <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>><br>
>> wrote:<br>
>>><br>
>>> W<br>
>>> ell, I think I will have to go with one ip per service and forget about<br>
>>> load balancing. It seems as though with LVS routing requests internally<br>
>>> through the VIP is difficult (impossible?) at least with LVS-DR. It seems<br>
>>> like a shame not to be able to distribute the work among the controller<br>
>>> nodes.<br>
>>><br>
>>><br>
>>> On Thu, Feb 14, 2013 at 9:50 AM, Samuel Winchenbach <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> Hi Sébastien,<br>
>>>><br>
>>>> I have two hosts with public interfaces with a number (~8) compute nodes<br>
>>>> behind them. I am trying to set the two public nodes in for HA and load<br>
>>>> balancing, I plan to run all the openstack services on these two nodes in<br>
>>>> Active/Active where possible. I currently have MySQL and RabbitMQ setup in<br>
>>>> pacemaker with a drbd backend.<br>
>>>><br>
>>>> That is a quick summary. If there is anything else I can answer about<br>
>>>> my setup please let me know.<br>
>>>><br>
>>>> Thanks,<br>
>>>> Sam<br>
>>>><br>
>>>><br>
>>>> On Thu, Feb 14, 2013 at 9:26 AM, Sébastien Han <<a href="mailto:han.sebastien@gmail.com" target="_blank">han.sebastien@gmail.com</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>> Well I don't know your setup, if you use LB for API service or if you<br>
>>>>> use an active/passive pacemaker but at the end it's not that much IPs I<br>
>>>>> guess. I dare to say that Keepalived sounds outdated to me...<br>
>>>>><br>
>>>>> If you use pacemaker and want to have the same IP for all the resources<br>
>>>>> simply create a resource group with all the openstack service inside it<br>
>>>>> (it's ugly but if it's what you want :)). Give me more info about your setup<br>
>>>>> and we can go further in the discussion :).<br>
>>>>><br>
>>>>> --<br>
>>>>> Regards,<br>
>>>>> Sébastien Han.<br>
>>>>><br>
>>>>><br>
>>>>> On Thu, Feb 14, 2013 at 3:15 PM, Samuel Winchenbach<br>
>>>>> <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>> wrote:<br>
>>>>>><br>
>>>>>> T<br>
>>>>>> he only real problem is that it would consume a lot of IP addresses<br>
>>>>>> when exposing the public interfaces. I _think_ I may have the solution in<br>
>>>>>> your blog actually:<br>
>>>>>> <a href="http://www.sebastien-han.fr/blog/2012/10/19/highly-available-lvs/" target="_blank">http://www.sebastien-han.fr/blog/2012/10/19/highly-available-lvs/</a><br>
>>>>>> and<br>
>>>>>> <a href="http://clusterlabs.org/wiki/Using_ldirectord" target="_blank">http://clusterlabs.org/wiki/Using_ldirectord</a><br>
>>>>>><br>
>>>>>> I am trying to weigh the pros and cons of this method vs<br>
>>>>>> keepalived/haproxy and just biting the bullet and using one IP per service.<br>
>>>>>><br>
>>>>>><br>
>>>>>> On Thu, Feb 14, 2013 at 4:17 AM, Sébastien Han<br>
>>>>>> <<a href="mailto:han.sebastien@gmail.com" target="_blank">han.sebastien@gmail.com</a>> wrote:<br>
>>>>>>><br>
>>>>>>> What's the problem to have one IP on service pool basis?<br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> Regards,<br>
>>>>>>> Sébastien Han.<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Wed, Feb 13, 2013 at 8:45 PM, Samuel Winchenbach<br>
>>>>>>> <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>> wrote:<br>
>>>>>>>><br>
>>>>>>>> What if the VIP is created on a different host than keystone is<br>
>>>>>>>> started on? It seems like you either need to set net.ipv4.ip_nonlocal_bind<br>
>>>>>>>> = 1 or create a colocation in pacemaker (which would either require all<br>
>>>>>>>> services to be on the same host, or have an ip-per-service).<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> On Wed, Feb 13, 2013 at 2:28 PM, Razique Mahroua<br>
>>>>>>>> <<a href="mailto:razique.mahroua@gmail.com" target="_blank">razique.mahroua@gmail.com</a>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> There we go<br>
>>>>>>>>> <a href="https://review.openstack.org/#/c/21581/" target="_blank">https://review.openstack.org/#/c/21581/</a><br>
>>>>>>>>><br>
>>>>>>>>> Razique Mahroua - Nuage & Co<br>
>>>>>>>>> <a href="mailto:razique.mahroua@gmail.com" target="_blank">razique.mahroua@gmail.com</a><br>
>>>>>>>>> Tel : <a href="tel:%2B33%209%2072%2037%2094%2015" value="+33972379415" target="_blank">+33 9 72 37 94 15</a><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> Le 13 févr. 2013 à 20:15, Razique Mahroua<br>
>>>>>>>>> <<a href="mailto:razique.mahroua@gmail.com" target="_blank">razique.mahroua@gmail.com</a>> a écrit :<br>
>>>>>>>>><br>
>>>>>>>>> I'm currently updating that part of the documentation - indeed it<br>
>>>>>>>>> states that two IPs are used, but in fact, you end up with only one VIP for<br>
>>>>>>>>> the API service.<br>
>>>>>>>>> I'll send the patch tonight<br>
>>>>>>>>><br>
>>>>>>>>> Razique Mahroua - Nuage & Co<br>
>>>>>>>>> <a href="mailto:razique.mahroua@gmail.com" target="_blank">razique.mahroua@gmail.com</a><br>
>>>>>>>>> Tel : <a href="tel:%2B33%209%2072%2037%2094%2015" value="+33972379415" target="_blank">+33 9 72 37 94 15</a><br>
>>>>>>>>><br>
>>>>>>>>> <NUAGECO-LOGO-Fblan_petit.jpg><br>
>>>>>>>>><br>
>>>>>>>>> Le 13 févr. 2013 à 20:05, Samuel Winchenbach <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>> a<br>
>>>>>>>>> écrit :<br>
>>>>>>>>><br>
>>>>>>>>> In that documentation it looks like each openstack service gets it<br>
>>>>>>>>> own IP (keystone is being assigned 192.168.42.103 and glance is getting<br>
>>>>>>>>> 192.168.42.104).<br>
>>>>>>>>><br>
>>>>>>>>> I might be missing something too because in the section titled<br>
>>>>>>>>> "Configure the VIP" it create a primitive called "p_api-ip" (or p_ip_api if<br>
>>>>>>>>> you read the text above it) and then in "Adding Keystone resource to<br>
>>>>>>>>> Pacemaker" it creates a group with "p_ip_keystone"???<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> Stranger yet, "Configuring OpenStack Services to use High Available<br>
>>>>>>>>> Glance API" says: "For Nova, for example, if your Glance API service IP<br>
>>>>>>>>> address is 192.168.42.104 as in the configuration explained here, you would<br>
>>>>>>>>> use the following line in your nova.conf file : glance_api_servers =<br>
>>>>>>>>> 192.168.42.103" But, in the step before it set: "registry_host =<br>
>>>>>>>>> 192.168.42.104"?<br>
>>>>>>>>><br>
>>>>>>>>> So I am not sure which ip you would connect to here...<br>
>>>>>>>>><br>
>>>>>>>>> Sam<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> On Wed, Feb 13, 2013 at 1:29 PM, JuanFra Rodriguez Cardoso<br>
>>>>>>>>> <<a href="mailto:juanfra.rodriguez.cardoso@gmail.com" target="_blank">juanfra.rodriguez.cardoso@gmail.com</a>> wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>> Hi Samuel:<br>
>>>>>>>>>><br>
>>>>>>>>>> Yes, it's possible with pacemaker. Look at<br>
>>>>>>>>>> <a href="http://docs.openstack.org/trunk/openstack-ha/content/ch-intro.html" target="_blank">http://docs.openstack.org/trunk/openstack-ha/content/ch-intro.html</a>.<br>
>>>>>>>>>><br>
>>>>>>>>>> Regards,<br>
>>>>>>>>>> JuanFra<br>
>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> 2013/2/13 Samuel Winchenbach <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>><br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hi All,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> I currently have a HA OpenStack cluster running where the<br>
>>>>>>>>>>> OpenStack services are kept alive with a combination of haproxy and<br>
>>>>>>>>>>> keepalived.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Is it possible to configure pacemaker so that all the OpenStack<br>
>>>>>>>>>>> services are served by the same IP? With keepalived I have a virtual ip<br>
>>>>>>>>>>> that can move from server to server and haproxy sends the request to a<br>
>>>>>>>>>>> machine that has a "live" service. This allows one (public) ip to handle<br>
>>>>>>>>>>> all incoming requests. I believe it is the combination of VRRP/IPVS that<br>
>>>>>>>>>>> allows this.<br>
>>>>>>>>>>><br>
>>>>>>>>>>><br>
>>>>>>>>>>> Is it possible to do something similar with pacemaker? I really<br>
>>>>>>>>>>> don't want to have an IP for each service, and I don't want to make it a<br>
>>>>>>>>>>> requirement that all OpenStack services must be running on the same server.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Thanks... I hope this question is clear, I feel like I sort of<br>
>>>>>>>>>>> butchered the wording a bit.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Sam<br>
>>>>>>>>>>><br>
>>>>>>>>>>> _______________________________________________<br>
>>>>>>>>>>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>>>>>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>>>>>>>>>>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>>>>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>>>>>>>>>>><br>
>>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> _______________________________________________<br>
>>>>>>>>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>>>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>>>>>>>>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> _______________________________________________<br>
>>>>>>>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>>>>>>>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>>>>>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>>>>>>>><br>
>>>>>>><br>
>>>>>><br>
>>>>><br>
>>>><br>
>>><br>
>><br>
><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>