<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 11/27/2012 09:52 PM, Gary Kotton
wrote:<br>
</div>
<blockquote cite="mid:50B4C59F.7010002@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
On 11/27/2012 03:05 PM, gong yong sheng wrote:
<blockquote cite="mid:50B4BAA1.70900@linux.vnet.ibm.com"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 11/27/2012 07:50 PM, Gary Kotton
wrote:<br>
</div>
<blockquote cite="mid:50B4A907.8090505@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
On 11/27/2012 06:06 AM, Mark McClain wrote:
<blockquote
cite="mid:3CD1FD21-B1CB-4EBD-BF8C-FCC4E79505E9@dreamhost.com"
type="cite">
<pre wrap="">All-
I wanted to continue the discussion from the today's meeting about the L3 agents. The two proposed solutions take different approaches, so I think we first should agree on what we're trying to solve: scaling or availability or both.
Nachi and Yong call their proposal "scheduler", but really it is really a horizontal scale out model. Scheduling is the means they've chosen to distribute the load. While their solution scales out horizontally, it does not address fully availability. Gary's proposal fronts the l3 services with a load balancing like service. It addresses availability by using an active/standby setup, but does not cover what happens when vertical scaling maxes out due to too many tenant networks and/or routers to fit on a physical node.
I think the answer is to do both by incorporating a combination of the two proposals. The L3 and DHCP agents are different enough that we may not be able to find a universal solution and that's is ok.
Lastly, deployers have different SLAs and may even have different SLAs for different tenants, so we need to make sure we have a foundation for vendors and deployers to meet their varying SLAs.
Thoughts?</pre>
</blockquote>
<br>
It would be great if we could have a universal solution. I
feel that due to the different roles of the services this will
be very challenging to achieve. I'll try and explain in more
detail below.<br>
<br>
<b>DHCP agents</b>:<br>
At the moment each DHCP agent is able to allocate the IP
address for a specific mac address. Each agent has this
information as it is received from the notifications from the
Quantum service. The problems with the DHCP agent are as
follows (please feel free to add or remove):<br>
i. For each networking providing DHCP services (currently only
implemented by dnsmasq in Quantum) a dnsmasq process is
created by the agent. This is problematic when the number of
networks is large.<br>
ii. When interacting with Nova firewall rules are created to
enable the traffic to arrive from the DHCP server to the VM.
This is problematic if the DHCP agent terminates and the VM
wishes to renew a IP.<br>
<br>
Originally I suggested that we use a load balancer to
distribute the traffic amongst the DHCP agents. Sadly this is
not relevant for two reasons:<br>
i. HA proxy does not have UDP support. This would have enabled
a virtual IP address for the DHCP server => no changes to
the nova rules. The load balancer would have detected if
agents were down and redirect traffic to agents that are up.<br>
ii. It does not address point #ii above. I suggested to have a
flag or configuration variable for each agent that indicate a
list of networks that the agent can service. This will enable
the agent to limit the resources that can be consumed on a
specific host. Naturally the devil is in the details on how
one can go about this if it is relevant.<br>
<br>
I think that if we had the "supported list of networks"
configurable for the DHCP agents then the vendor can deploy as
many DHCP agents as she/he wishes. I would prefer that this
information is not on the Quantum service but locally on the
agents. This will offer a solution for scale and high
availability of DHCP resources.<br>
</blockquote>
Why do we need supported list of networks on dhcp agents? Do u
want some agents have the same network data, right? If this is
the case, we need to modify our dhcp agents. we will create as
many as dhcp server ports on one network. (This is not too bad)<br>
My design will use scheduler to distribute networks among dhcp
agents.</blockquote>
<br>
At the moment all of the dhcp agents receive all of the updates. I
do not see why we need the quantum service to indicate which agent
runs where. This will change the manner in which the dhcp agents
work.<br>
</blockquote>
No. currently, we can run only one dhcp agent since we are using a
topic queue for notification.<br>
And one problem with multiple agents serving the same ip is:<br>
we will have more than one agents want to update the ip's leasetime
now and than.<br>
<blockquote cite="mid:50B4C59F.7010002@redhat.com" type="cite"> We
will certainly want more than one agent to support more than one
network. These agents will certain be on different hosts for HA.<br>
<br>
<blockquote cite="mid:50B4BAA1.70900@linux.vnet.ibm.com"
type="cite"> So if agents have configured "supported list of
networks", we can schedule these networks to these agents. But
now, it is a list of agents which can host one network at the
same time. In addition to the 'supported list of networks', we
can support 'supported list of tenants' scheduling hint
configured on agents. With tenants hint, quantum scheduler will
schedule the networks of a given list tenants to this agent.<br>
<br>
<blockquote cite="mid:50B4A907.8090505@redhat.com" type="cite">
<br>
Only problem is the ensuring that the DHCP traffic gets to the
VM :). I do not think that it is feasible to update the hosts
each time with a rule for a new DHCP agent that is added. One
option to consider to to rewrite the source IP of the traffic
sent from the DHCP agent. This is essentially what is done by
a load balancer.<br>
</blockquote>
For this problem, Dan said we will replace the dhcp
anti-spoofing with our own quantum implementation. I think this
is a chance.<br>
So if user wants to use the HA configuration of dhcp server,
quantum version must be used, otherwise, he cannot configure HA
(i.e. more than one l3 agent serves one network), he can live
with simple server side scheduling (scheduling network to only
one dhcp agent)<br>
</blockquote>
<br>
I am not really sure that I understand the scheduling. I interpret
scheduling as being something dynamic. From what I think I
understand is that it is statically managed by the user/admin. <br>
</blockquote>
The nova-scheduler is scheduling according to compute host's
dynamic capabilities and request spec. we can do it too. we will
have how much load on each agents, agents' configuration and input
to decide how resources are scheduled.<br>
<blockquote cite="mid:50B4C59F.7010002@redhat.com" type="cite">
<blockquote cite="mid:50B4BAA1.70900@linux.vnet.ibm.com"
type="cite">
<blockquote cite="mid:50B4A907.8090505@redhat.com" type="cite">
<br>
<b>L3 agents</b>:<br>
Problems here are:<br>
i. HA - what if a L3 agent goes down.<br>
ii. Scale - how can we deploy a number of l3 agents<br>
iii. Amount of firewall rules<br>
<br>
In the first case if the L3 agent goes down then someone
accessing a floating IP will be unable to access that IP. This
is something that is critical for anyone running a cloud.<br>
<br>
I have thought about a number of options but each has its
shortcomings:<br>
i. L3 agents to run VRRP. This will enable l3 agents to work
in an active backup pair. This requires a number of changes to
the agent. Each agent will have the same configuration
enabling them to treat inbound and outbound traffic.<br>
</blockquote>
I like this way, But I don't how to implement. If it works, we
can have same scheduling ways:<br>
on l3 agents, we configure it to support a list of routers or
routers of a list of tenants.<br>
we will use same scheduling algorithm as we use for dhcp agents<br>
<blockquote cite="mid:50B4A907.8090505@redhat.com" type="cite">
ii. Transparent load balancing - HA proxy does not support
this<br>
iii. Having the agents rewrite the destination MAC address of
the default GW (the l3 agent). This solves outbound traffic
but inbound is problematic<br>
iv. Running l3 agents on each host could ensure that the
traffic generated on those hosts has floating IP's. This would
require us to change the implementation of the l3 agents to
only build firewall rules for devices on the HOST. <br>
</blockquote>
I don't like multi-host way.<br>
</blockquote>
<br>
Is this when it runs on the same host as the VM's?<br>
</blockquote>
in multi-host, dhcp agent and l3 agent running on a host serves Vms
on the same compute host. <br>
<blockquote cite="mid:50B4C59F.7010002@redhat.com" type="cite"> <br>
<blockquote cite="mid:50B4BAA1.70900@linux.vnet.ibm.com"
type="cite">
<blockquote cite="mid:50B4A907.8090505@redhat.com" type="cite">
<br>
None of the above deal with the firewall rules. This is
something that can be addressed in a similar way to the DHCP
agent with the L3 agents specifically indicating which routers
it will support (this is already implemented when namespaces
are not supported)<br>
<br>
Thanks<br>
Gary<br>
<br>
</blockquote>
Thanks<br>
Yong Sheng Gong<br>
<blockquote cite="mid:50B4A907.8090505@redhat.com" type="cite">
<blockquote
cite="mid:3CD1FD21-B1CB-4EBD-BF8C-FCC4E79505E9@dreamhost.com"
type="cite">
<pre wrap="">mark
_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>