[openstack-dev] [Quantum] continuing todays discussion about the l3 agents

Yi Yang (yiya) yiya at cisco.com
Fri Dec 7 05:41:08 UTC 2012


I think Soheil brings up a very interesting point -- the deployment flexibility. IMHO, there are 3 questions/topics been discussed in this thread regarding the DHCP operations: scalability, high availability and flexibility.

1. Scalability.

Traditionally, a user doesn't need to configure a DHCP server per vlan -- A single dhcp server can serve multiple vlans with using DHCP relays. Nowadays, as DHCP server also supports clients from different VRFs, overlapping of IPs is not an issue. This scales-up approach works well for a decent size network.

Without DHCP relays (at least not yet) in quantum, the dhcp server has to be present in each vlan. A single DHCP server with multiple interfaces  (one per vlan/network) should have worked if no overlapping IP. This could be the case if openstack is deployed as a private cloud in enterprise data center. However, for a multi tenant environment, openstack has to work with overlapping addresses, requiring a separate address space per tenant at least. To make it even simpler, openstack adopts one-address-space-per-vlan/network approach, at the cost of one-dhcp-process-per-vlan/network).

Logically, quantum dhcp uses a scale-out model. But implementation wise, it's a scale-up model, as there is only one dhcp agent. To close the gap,  openstack has to support multiple dhcp agents. The question is how? 2 approaches have been discussed: LB or scheduler. One big difference between them is in-band or out-of-band. 

For an in-band solution like LB, all the dhcp packets have to go to the LB. There are a couple of concerns for this:
A. hot-spot -- depending on the scale of the cloud, a hot-spot around LB may be created.
B. Scalability of LB -- To receive DHCP packets from all the vlans/networks, the LB has to be present in all of them -- this seems indicating the LB is going to have a huge number of virtual interfaces. 
C. Security -- As a rendezvous point, the security policy for the LB has to be designed very carefully to maintain isolation between tenants.

IMHO, a scheduler based solution would avoid these issues.

2. High Availability

With multiple DHCP agents, the HA of whole system will be improved. On the other hand, the HA per vlan/network still needs to be addressed . One way to solve this is to enable 2 dhcp process per vlan, with split scope -- Unfortunately, dnsmasq doesn't support dhcp failover, and adding it into dnsmasq is out of scope of openstack, although there are DHCP servers supporting dhcp failover -- As Soheil mentioned below, microsoft supports it, as well as cisco :-)  The question is how to support them in quantum? -- This actually brings up the next question -- flexibility.

3. Flexibility

We have different plugins for network, linuxbridge, ovs, cisco and etc. Why can't we adopt the plugin approach for dhcp? Instead of working with dhcp agents directly, the scheduler can work with dhcp-plugins. Actually, even the scheduler itself could be a plugin -- quantum-server can communicate with scheduler-dhcp-plugin, linux-dhcp-plugin, and let the plugin decides how to work with ***-plugin-dhcp-agents.

Yi



> From: Mark McClain <mark.mcclain at dreamhost.com>
> Date: Mon, Nov 26, 2012 at 8:06 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> All-
> 
> I wanted to continue the discussion from the today's meeting about the L3 agents.  The two proposed solutions take different approaches, so I think we first should agree on what we're trying to solve: scaling or availability or both.
> 
> Nachi and Yong call their proposal "scheduler", but really it is really a horizontal scale out model.  Scheduling is the means they've chosen to distribute the load.  While their solution scales out horizontally, it does not address fully availability.  Gary's proposal fronts the l3 services with a load balancing like service.  It addresses availability by using an active/standby setup, but does not cover what happens when vertical scaling maxes out due to too many tenant networks and/or routers to fit on a physical node.
> 
> I think the answer is to do both by incorporating a combination of the two proposals.  The L3 and DHCP agents are different enough that we may not be able to find a universal solution and that's is ok.
> 
> Lastly, deployers have different SLAs and may even have different SLAs for different tenants, so we need to make sure we have a foundation for vendors and deployers to meet their varying SLAs.
> 
> Thoughts?
> 
> mark
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ----------
> From: Gary Kotton <gkotton at redhat.com>
> Date: Tue, Nov 27, 2012 at 3:50 AM
> To: openstack-dev at lists.openstack.org
> 
> 
> On 11/27/2012 06:06 AM, Mark McClain wrote:
>> All-
>> 
>> I wanted to continue the discussion from the today's meeting about the L3 agents.  The two proposed solutions take different approaches, so I think we first should agree on what we're trying to solve: scaling or availability or both.
>> 
>> Nachi and Yong call their proposal "scheduler", but really it is really a horizontal scale out model.  Scheduling is the means they've chosen to distribute the load.  While their solution scales out horizontally, it does not address fully availability.  Gary's proposal fronts the l3 services with a load balancing like service.  It addresses availability by using an active/standby setup, but does not cover what happens when vertical scaling maxes out due to too many tenant networks and/or routers to fit on a physical node.
>> 
>> I think the answer is to do both by incorporating a combination of the two proposals.  The L3 and DHCP agents are different enough that we may not be able to find a universal solution and that's is ok.
>> 
>> Lastly, deployers have different SLAs and may even have different SLAs for different tenants, so we need to make sure we have a foundation for vendors and deployers to meet their varying SLAs.
>> 
>> Thoughts?
>> 
> 
> It would be great if we could have a universal solution. I feel that due to the different roles of the services this will be very challenging to achieve. I'll try and explain in more detail below.
> 
> DHCP agents:
> At the moment each DHCP agent is able to allocate the IP address for a specific mac address. Each agent has this information as it is received from the notifications from the Quantum service. The problems with the DHCP agent are as follows (please feel free to add or remove):
> i. For each networking providing DHCP services (currently only implemented by dnsmasq in Quantum) a dnsmasq process is created by the agent. This is problematic when the number of networks is large.
> ii. When interacting with Nova firewall rules are created to enable the traffic to arrive from the DHCP server to the VM. This is problematic if the DHCP agent terminates and the VM wishes to renew a IP.
> 
> Originally I suggested that we use a load balancer to distribute the traffic amongst the DHCP agents. Sadly this is not relevant for two reasons:
> i. HA proxy does not have UDP support. This would have enabled a virtual IP address for the DHCP server => no changes to the nova rules. The load balancer would have detected if agents were down and redirect traffic to agents that are up.
> ii. It does not address point #ii above. I suggested to have a flag or configuration variable for each agent that indicate a list of networks that the agent can service.  This will enable the agent to limit the resources that can be consumed on a specific host. Naturally the devil is in the details on how one can go about this if it is relevant.
> 
> I think that if we had the "supported list of networks" configurable for the DHCP agents then the vendor can deploy as many DHCP agents as she/he wishes. I would prefer that this information is not on the Quantum service but locally on the agents. This will offer a solution for scale and high availability of DHCP resources.
> 
> Only problem is the ensuring that the DHCP traffic gets to the VM :). I do not think that it is feasible to update the hosts each time with a rule for a new DHCP agent that is added. One option to consider to to rewrite the source IP of the traffic sent from the DHCP agent. This is essentially what is done by a load balancer.
> 
> L3 agents:
> Problems here are:
> i. HA - what if a L3 agent goes down.
> ii. Scale - how can we deploy a number of l3 agents
> iii. Amount of firewall rules
> 
> In the first case if the L3 agent goes down then someone accessing a floating IP will be unable to access that IP. This is something that is critical for anyone running a cloud.
> 
> I have thought about a number of options but each has its shortcomings:
> i. L3 agents to run VRRP. This will enable l3 agents to work in an active backup pair. This requires a number of changes to the agent. Each agent will have the same configuration enabling them to treat inbound and outbound traffic.
> ii. Transparent load balancing - HA proxy does not support this
> iii. Having the agents rewrite the destination MAC address of the default GW (the l3 agent). This solves outbound traffic but inbound is problematic
> iv. Running l3 agents on each host could ensure that the traffic generated on those hosts has floating IP's. This would require us to change the implementation of the l3 agents to only build firewall rules for devices on the HOST. 
> 
> None of the above deal with the firewall rules. This is something that can be addressed in a similar way to the DHCP agent with the L3 agents specifically indicating which routers it will support (this is already implemented when namespaces are not supported)
> 
> Thanks
> Gary
> 
> 
>> mark
>> _______________________________________________
>> OpenStack-dev mailing list
>> 
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Tue, Nov 27, 2012 at 5:05 AM
> To: openstack-dev at lists.openstack.org
> 
> 
> On 11/27/2012 07:50 PM, Gary Kotton wrote:
>> On 11/27/2012 06:06 AM, Mark McClain wrote:
>>> All-
>>> 
>>> I wanted to continue the discussion from the today's meeting about the L3 agents.  The two proposed solutions take different approaches, so I think we first should agree on what we're trying to solve: scaling or availability or both.
>>> 
>>> Nachi and Yong call their proposal "scheduler", but really it is really a horizontal scale out model.  Scheduling is the means they've chosen to distribute the load.  While their solution scales out horizontally, it does not address fully availability.  Gary's proposal fronts the l3 services with a load balancing like service.  It addresses availability by using an active/standby setup, but does not cover what happens when vertical scaling maxes out due to too many tenant networks and/or routers to fit on a physical node.
>>> 
>>> I think the answer is to do both by incorporating a combination of the two proposals.  The L3 and DHCP agents are different enough that we may not be able to find a universal solution and that's is ok.
>>> 
>>> Lastly, deployers have different SLAs and may even have different SLAs for different tenants, so we need to make sure we have a foundation for vendors and deployers to meet their varying SLAs.
>>> 
>>> Thoughts?
>>> 
>> 
>> It would be great if we could have a universal solution. I feel that due to the different roles of the services this will be very challenging to achieve. I'll try and explain in more detail below.
>> 
>> DHCP agents:
>> At the moment each DHCP agent is able to allocate the IP address for a specific mac address. Each agent has this information as it is received from the notifications from the Quantum service. The problems with the DHCP agent are as follows (please feel free to add or remove):
>> i. For each networking providing DHCP services (currently only implemented by dnsmasq in Quantum) a dnsmasq process is created by the agent. This is problematic when the number of networks is large.
>> ii. When interacting with Nova firewall rules are created to enable the traffic to arrive from the DHCP server to the VM. This is problematic if the DHCP agent terminates and the VM wishes to renew a IP.
>> 
>> Originally I suggested that we use a load balancer to distribute the traffic amongst the DHCP agents. Sadly this is not relevant for two reasons:
>> i. HA proxy does not have UDP support. This would have enabled a virtual IP address for the DHCP server => no changes to the nova rules. The load balancer would have detected if agents were down and redirect traffic to agents that are up.
>> ii. It does not address point #ii above. I suggested to have a flag or configuration variable for each agent that indicate a list of networks that the agent can service.  This will enable the agent to limit the resources that can be consumed on a specific host. Naturally the devil is in the details on how one can go about this if it is relevant.
>> 
>> I think that if we had the "supported list of networks" configurable for the DHCP agents then the vendor can deploy as many DHCP agents as she/he wishes. I would prefer that this information is not on the Quantum service but locally on the agents. This will offer a solution for scale and high availability of DHCP resources.
> Why do we need supported list of networks on dhcp agents? Do u want some agents have the same network data, right? If this is the case, we need to modify our dhcp agents. we will create as many as dhcp server ports on one network. (This is not too bad)
> My design will use scheduler to distribute networks among dhcp agents.  So if agents have configured "supported list of networks", we can schedule these networks to these agents. But now, it is a list of agents which can host one network at the same time. In addition to the 'supported list of networks', we can support 'supported list of tenants' scheduling hint configured on agents. With tenants hint, quantum scheduler will schedule the networks of a given list tenants to this agent.
> 
> 
>> 
>> Only problem is the ensuring that the DHCP traffic gets to the VM :). I do not think that it is feasible to update the hosts each time with a rule for a new DHCP agent that is added. One option to consider to to rewrite the source IP of the traffic sent from the DHCP agent. This is essentially what is done by a load balancer.
> For this problem, Dan said we will replace the dhcp anti-spoofing with our own quantum implementation. I think this is a chance.
> So if user wants to use the HA configuration of dhcp server, quantum version must be used, otherwise, he cannot configure HA (i.e. more than one l3 agent serves one network), he can live with simple server side scheduling (scheduling network to only one dhcp agent)
> 
>> 
>> L3 agents:
>> Problems here are:
>> i. HA - what if a L3 agent goes down.
>> ii. Scale - how can we deploy a number of l3 agents
>> iii. Amount of firewall rules
>> 
>> In the first case if the L3 agent goes down then someone accessing a floating IP will be unable to access that IP. This is something that is critical for anyone running a cloud.
>> 
>> I have thought about a number of options but each has its shortcomings:
>> i. L3 agents to run VRRP. This will enable l3 agents to work in an active backup pair. This requires a number of changes to the agent. Each agent will have the same configuration enabling them to treat inbound and outbound traffic.
> I like this way, But I don't how to implement. If it works, we can have same scheduling ways:
> on l3 agents, we configure it to support a list of routers or routers of a list of tenants.
> we will use same scheduling algorithm  as we use for dhcp agents
> 
>> ii. Transparent load balancing - HA proxy does not support this
>> iii. Having the agents rewrite the destination MAC address of the default GW (the l3 agent). This solves outbound traffic but inbound is problematic
>> iv. Running l3 agents on each host could ensure that the traffic generated on those hosts has floating IP's. This would require us to change the implementation of the l3 agents to only build firewall rules for devices on the HOST. 
> I don't like multi-host way.
> 
>> 
>> None of the above deal with the firewall rules. This is something that can be addressed in a similar way to the DHCP agent with the L3 agents specifically indicating which routers it will support (this is already implemented when namespaces are not supported)
>> 
>> Thanks
>> Gary
>> 
> Thanks
> Yong Sheng Gong
> 
>>> mark
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> 
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> 
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: Gary Kotton <gkotton at redhat.com>
> Date: Tue, Nov 27, 2012 at 5:52 AM
> To: openstack-dev at lists.openstack.org
> 
> 
> On 11/27/2012 03:05 PM, gong yong sheng wrote:
>> On 11/27/2012 07:50 PM, Gary Kotton wrote:
>>> On 11/27/2012 06:06 AM, Mark McClain wrote:
>>>> All-
>>>> 
>>>> I wanted to continue the discussion from the today's meeting about the L3 agents.  The two proposed solutions take different approaches, so I think we first should agree on what we're trying to solve: scaling or availability or both.
>>>> 
>>>> Nachi and Yong call their proposal "scheduler", but really it is really a horizontal scale out model.  Scheduling is the means they've chosen to distribute the load.  While their solution scales out horizontally, it does not address fully availability.  Gary's proposal fronts the l3 services with a load balancing like service.  It addresses availability by using an active/standby setup, but does not cover what happens when vertical scaling maxes out due to too many tenant networks and/or routers to fit on a physical node.
>>>> 
>>>> I think the answer is to do both by incorporating a combination of the two proposals.  The L3 and DHCP agents are different enough that we may not be able to find a universal solution and that's is ok.
>>>> 
>>>> Lastly, deployers have different SLAs and may even have different SLAs for different tenants, so we need to make sure we have a foundation for vendors and deployers to meet their varying SLAs.
>>>> 
>>>> Thoughts?
>>>> 
>>> 
>>> It would be great if we could have a universal solution. I feel that due to the different roles of the services this will be very challenging to achieve. I'll try and explain in more detail below.
>>> 
>>> DHCP agents:
>>> At the moment each DHCP agent is able to allocate the IP address for a specific mac address. Each agent has this information as it is received from the notifications from the Quantum service. The problems with the DHCP agent are as follows (please feel free to add or remove):
>>> i. For each networking providing DHCP services (currently only implemented by dnsmasq in Quantum) a dnsmasq process is created by the agent. This is problematic when the number of networks is large.
>>> ii. When interacting with Nova firewall rules are created to enable the traffic to arrive from the DHCP server to the VM. This is problematic if the DHCP agent terminates and the VM wishes to renew a IP.
>>> 
>>> Originally I suggested that we use a load balancer to distribute the traffic amongst the DHCP agents. Sadly this is not relevant for two reasons:
>>> i. HA proxy does not have UDP support. This would have enabled a virtual IP address for the DHCP server => no changes to the nova rules. The load balancer would have detected if agents were down and redirect traffic to agents that are up.
>>> ii. It does not address point #ii above. I suggested to have a flag or configuration variable for each agent that indicate a list of networks that the agent can service.  This will enable the agent to limit the resources that can be consumed on a specific host. Naturally the devil is in the details on how one can go about this if it is relevant.
>>> 
>>> I think that if we had the "supported list of networks" configurable for the DHCP agents then the vendor can deploy as many DHCP agents as she/he wishes. I would prefer that this information is not on the Quantum service but locally on the agents. This will offer a solution for scale and high availability of DHCP resources.
>> Why do we need supported list of networks on dhcp agents? Do u want some agents have the same network data, right? If this is the case, we need to modify our dhcp agents. we will create as many as dhcp server ports on one network. (This is not too bad)
>> My design will use scheduler to distribute networks among dhcp agents.
> 
> At the moment all of the dhcp agents receive all of the updates. I do not see why we need the quantum service to indicate which agent runs where. This will change the manner in which the dhcp agents work.
> We will certainly want more than one agent to support more than one network. These agents will certain be on different hosts for HA.
> 
> 
>>   So if agents have configured "supported list of networks", we can schedule these networks to these agents. But now, it is a list of agents which can host one network at the same time. In addition to the 'supported list of networks', we can support 'supported list of tenants' scheduling hint configured on agents. With tenants hint, quantum scheduler will schedule the networks of a given list tenants to this agent.
>> 
>>> 
>>> Only problem is the ensuring that the DHCP traffic gets to the VM :). I do not think that it is feasible to update the hosts each time with a rule for a new DHCP agent that is added. One option to consider to to rewrite the source IP of the traffic sent from the DHCP agent. This is essentially what is done by a load balancer.
>> For this problem, Dan said we will replace the dhcp anti-spoofing with our own quantum implementation. I think this is a chance.
>> So if user wants to use the HA configuration of dhcp server, quantum version must be used, otherwise, he cannot configure HA (i.e. more than one l3 agent serves one network), he can live with simple server side scheduling (scheduling network to only one dhcp agent)
> 
> I am not really sure that I understand the scheduling. I interpret scheduling as being something dynamic. From what I think I understand is that it is statically managed by the user/admin. 
>>> 
>>> L3 agents:
>>> Problems here are:
>>> i. HA - what if a L3 agent goes down.
>>> ii. Scale - how can we deploy a number of l3 agents
>>> iii. Amount of firewall rules
>>> 
>>> In the first case if the L3 agent goes down then someone accessing a floating IP will be unable to access that IP. This is something that is critical for anyone running a cloud.
>>> 
>>> I have thought about a number of options but each has its shortcomings:
>>> i. L3 agents to run VRRP. This will enable l3 agents to work in an active backup pair. This requires a number of changes to the agent. Each agent will have the same configuration enabling them to treat inbound and outbound traffic.
>> I like this way, But I don't how to implement. If it works, we can have same scheduling ways:
>> on l3 agents, we configure it to support a list of routers or routers of a list of tenants.
>> we will use same scheduling algorithm  as we use for dhcp agents
>>> ii. Transparent load balancing - HA proxy does not support this
>>> iii. Having the agents rewrite the destination MAC address of the default GW (the l3 agent). This solves outbound traffic but inbound is problematic
>>> iv. Running l3 agents on each host could ensure that the traffic generated on those hosts has floating IP's. This would require us to change the implementation of the l3 agents to only build firewall rules for devices on the HOST. 
>> I don't like multi-host way.
> 
> Is this when it runs on the same host as the VM's?
> 
> 
>>> 
>>> None of the above deal with the firewall rules. This is something that can be addressed in a similar way to the DHCP agent with the L3 agents specifically indicating which routers it will support (this is already implemented when namespaces are not supported)
>>> 
>>> Thanks
>>> Gary
>>> 
>> Thanks
>> Yong Sheng Gong
>>>> mark
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> 
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> 
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> 
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: Vinay Bannai <vbannai at gmail.com>
> Date: Tue, Nov 27, 2012 at 2:38 PM
> To: gkotton at redhat.com, OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> Gary,
> 
> Given a lack of a standard way of dealing with active/standby DHCP server, I agree that LB does provide an interesting allure to piggy back on top of that. 
> I am all for discussing this further but I would also like to explore if a more simpler solution exists. 
>  
> Please see my comments and questions in-line. 
> 
> Vinay
> 
> On Tue, Nov 27, 2012 at 3:50 AM, Gary Kotton <gkotton at redhat.com> wrote:
> On 11/27/2012 06:06 AM, Mark McClain wrote:
>> All-
>> 
>> I wanted to continue the discussion from the today's meeting about the L3 agents.  The two proposed solutions take different approaches, so I think we first should agree on what we're trying to solve: scaling or availability or both.
>> 
>> Nachi and Yong call their proposal "scheduler", but really it is really a horizontal scale out model.  Scheduling is the means they've chosen to distribute the load.  While their solution scales out horizontally, it does not address fully availability.  Gary's proposal fronts the l3 services with a load balancing like service.  It addresses availability by using an active/standby setup, but does not cover what happens when vertical scaling maxes out due to too many tenant networks and/or routers to fit on a physical node.
>> 
>> I think the answer is to do both by incorporating a combination of the two proposals.  The L3 and DHCP agents are different enough that we may not be able to find a universal solution and that's is ok.
>> 
>> Lastly, deployers have different SLAs and may even have different SLAs for different tenants, so we need to make sure we have a foundation for vendors and deployers to meet their varying SLAs.
>> 
>> Thoughts?
>> 
> 
> It would be great if we could have a universal solution. I feel that due to the different roles of the services this will be very challenging to achieve. I'll try and explain in more detail below.
> 
> DHCP agents:
> At the moment each DHCP agent is able to allocate the IP address for a specific mac address. Each agent has this information as it is received from the notifications from the Quantum service. The problems with the DHCP agent are as follows (please feel free to add or remove):
> i. For each networking providing DHCP services (currently only implemented by dnsmasq in Quantum) a dnsmasq process is created by the agent. This is problematic when the number of networks is large.
> 
> I am not sure that would be a problem. Can we quantify the impact of handling large networks? 
>  
> ii. When interacting with Nova firewall rules are created to enable the traffic to arrive from the DHCP server to the VM. This is problematic if the DHCP agent terminates and the VM wishes to renew a IP.
> 
> I saw that in a previous post, Dan also mentioned this. I am a little curious as why the Nova firewall rules need to be tweaked. Isn't the VM connected to the bridge (in case of a VLAN)? Or is it only for the case where GRE or other non-L2 based overlays are used?
>  
> 
> Originally I suggested that we use a load balancer to distribute the traffic amongst the DHCP agents. Sadly this is not relevant for two reasons:
> i. HA proxy does not have UDP support. This would have enabled a virtual IP address for the DHCP server => no changes to the nova rules. The load balancer would have detected if agents were down and redirect traffic to agents that are up.
> ii. It does not address point #ii above. I suggested to have a flag or configuration variable for each agent that indicate a list of networks that the agent can service.  This will enable the agent to limit the resources that can be consumed on a specific host. Naturally the devil is in the details on how one can go about this if it is relevant.
> 
> I think that if we had the "supported list of networks" configurable for the DHCP agents then the vendor can deploy as many DHCP agents as she/he wishes. I would prefer that this information is not on the Quantum service but locally on the agents. This will offer a solution for scale and high availability of DHCP resources.
> 
> Only problem is the ensuring that the DHCP traffic gets to the VM :). I do not think that it is feasible to update the hosts each time with a rule for a new DHCP agent that is added. One option to consider to to rewrite the source IP of the traffic sent from the DHCP agent. This is essentially what is done by a load balancer.
> 
> I am not clear again as to why the host needs to be updated for each new DHCP agent. 
>  
> 
> L3 agents:
> Problems here are:
> i. HA - what if a L3 agent goes down.
> ii. Scale - how can we deploy a number of l3 agents
> iii. Amount of firewall rules
> 
> In the first case if the L3 agent goes down then someone accessing a floating IP will be unable to access that IP. This is something that is critical for anyone running a cloud.
> 
> I have thought about a number of options but each has its shortcomings:
> i. L3 agents to run VRRP. This will enable l3 agents to work in an active backup pair. This requires a number of changes to the agent. Each agent will have the same configuration enabling them to treat inbound and outbound traffic.
> ii. Transparent load balancing - HA proxy does not support this
> iii. Having the agents rewrite the destination MAC address of the default GW (the l3 agent). This solves outbound traffic but inbound is problematic
> iv. Running l3 agents on each host could ensure that the traffic generated on those hosts has floating IP's. This would require us to change the implementation of the l3 agents to only build firewall rules for devices on the HOST. 
> 
> None of the above deal with the firewall rules. This is something that can be addressed in a similar way to the DHCP agent with the L3 agents specifically indicating which routers it will support (this is already implemented when namespaces are not supported)
> 
> Thanks
> Gary
> 
> 
>> mark
>> _______________________________________________
>> OpenStack-dev mailing list
>> 
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> -- 
> Vinay Bannai
> Email: vbannai at gmail.com
> Google Voice: 415 938 7576
> 
> 
> ----------
> From: Nachi Ueno <nachi at nttmcl.com>
> Date: Tue, Nov 27, 2012 at 2:59 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> Hi folks
> 
> IMO, quantum should follow the nova scheduler architecture. 
> Because the scheduler problem is common in nova, quantum, cinder ( may be glance ).
> We have done tons of discussion about scheduling staff in nova-scheduler. 
> so  why not start to archive nova-network compatible scheduling , 
> such as we are working on floating ips or security groups? 
> 
> Scheduling have also role of distribute load for resource efficient way. 
> nova-scheduler has pluggable architecture and some driver implement different scheduling 
> strategies.
> There are also several availability strategites ( N-Act, M-Sby, Act-Act, Act-hot standby)  and methods (VRRP or simple vip based failover) based on service requirement.
> These strategies could be implemented as a one nova-scheduler driver.
> 
> ( IMO, in future, nova-scheduler should be one core Project )
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Tue, Nov 27, 2012 at 3:33 PM
> To: gkotton at redhat.com, OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> On 11/27/2012 09:52 PM, Gary Kotton wrote:
>> On 11/27/2012 03:05 PM, gong yong sheng wrote:
>>> On 11/27/2012 07:50 PM, Gary Kotton wrote:
>>>> On 11/27/2012 06:06 AM, Mark McClain wrote:
>>>>> All-
>>>>> 
>>>>> I wanted to continue the discussion from the today's meeting about the L3 agents.  The two proposed solutions take different approaches, so I think we first should agree on what we're trying to solve: scaling or availability or both.
>>>>> 
>>>>> Nachi and Yong call their proposal "scheduler", but really it is really a horizontal scale out model.  Scheduling is the means they've chosen to distribute the load.  While their solution scales out horizontally, it does not address fully availability.  Gary's proposal fronts the l3 services with a load balancing like service.  It addresses availability by using an active/standby setup, but does not cover what happens when vertical scaling maxes out due to too many tenant networks and/or routers to fit on a physical node.
>>>>> 
>>>>> I think the answer is to do both by incorporating a combination of the two proposals.  The L3 and DHCP agents are different enough that we may not be able to find a universal solution and that's is ok.
>>>>> 
>>>>> Lastly, deployers have different SLAs and may even have different SLAs for different tenants, so we need to make sure we have a foundation for vendors and deployers to meet their varying SLAs.
>>>>> 
>>>>> Thoughts?
>>>>> 
>>>> 
>>>> It would be great if we could have a universal solution. I feel that due to the different roles of the services this will be very challenging to achieve. I'll try and explain in more detail below.
>>>> 
>>>> DHCP agents:
>>>> At the moment each DHCP agent is able to allocate the IP address for a specific mac address. Each agent has this information as it is received from the notifications from the Quantum service. The problems with the DHCP agent are as follows (please feel free to add or remove):
>>>> i. For each networking providing DHCP services (currently only implemented by dnsmasq in Quantum) a dnsmasq process is created by the agent. This is problematic when the number of networks is large.
>>>> ii. When interacting with Nova firewall rules are created to enable the traffic to arrive from the DHCP server to the VM. This is problematic if the DHCP agent terminates and the VM wishes to renew a IP.
>>>> 
>>>> Originally I suggested that we use a load balancer to distribute the traffic amongst the DHCP agents. Sadly this is not relevant for two reasons:
>>>> i. HA proxy does not have UDP support. This would have enabled a virtual IP address for the DHCP server => no changes to the nova rules. The load balancer would have detected if agents were down and redirect traffic to agents that are up.
>>>> ii. It does not address point #ii above. I suggested to have a flag or configuration variable for each agent that indicate a list of networks that the agent can service.  This will enable the agent to limit the resources that can be consumed on a specific host. Naturally the devil is in the details on how one can go about this if it is relevant.
>>>> 
>>>> I think that if we had the "supported list of networks" configurable for the DHCP agents then the vendor can deploy as many DHCP agents as she/he wishes. I would prefer that this information is not on the Quantum service but locally on the agents. This will offer a solution for scale and high availability of DHCP resources.
>>> Why do we need supported list of networks on dhcp agents? Do u want some agents have the same network data, right? If this is the case, we need to modify our dhcp agents. we will create as many as dhcp server ports on one network. (This is not too bad)
>>> My design will use scheduler to distribute networks among dhcp agents.
>> 
>> At the moment all of the dhcp agents receive all of the updates. I do not see why we need the quantum service to indicate which agent runs where. This will change the manner in which the dhcp agents work.
> No. currently, we can run only one dhcp agent  since we are using a topic queue for notification.
> And one problem with multiple agents serving the same ip is:
> we will have more than one agents want to update the ip's leasetime now and than.
> 
>> We will certainly want more than one agent to support more than one network. These agents will certain be on different hosts for HA.
>> 
>>>   So if agents have configured "supported list of networks", we can schedule these networks to these agents. But now, it is a list of agents which can host one network at the same time. In addition to the 'supported list of networks', we can support 'supported list of tenants' scheduling hint configured on agents. With tenants hint, quantum scheduler will schedule the networks of a given list tenants to this agent.
>>> 
>>>> 
>>>> Only problem is the ensuring that the DHCP traffic gets to the VM :). I do not think that it is feasible to update the hosts each time with a rule for a new DHCP agent that is added. One option to consider to to rewrite the source IP of the traffic sent from the DHCP agent. This is essentially what is done by a load balancer.
>>> For this problem, Dan said we will replace the dhcp anti-spoofing with our own quantum implementation. I think this is a chance.
>>> So if user wants to use the HA configuration of dhcp server, quantum version must be used, otherwise, he cannot configure HA (i.e. more than one l3 agent serves one network), he can live with simple server side scheduling (scheduling network to only one dhcp agent)
>> 
>> I am not really sure that I understand the scheduling. I interpret scheduling as being something dynamic. From what I think I understand is that it is statically managed by the user/admin. 
> The nova-scheduler  is scheduling according to compute host's dynamic capabilities and request spec. we can do it too. we will have how much load on each agents, agents' configuration and input to decide how resources are scheduled.
> 
>>>> 
>>>> L3 agents:
>>>> Problems here are:
>>>> i. HA - what if a L3 agent goes down.
>>>> ii. Scale - how can we deploy a number of l3 agents
>>>> iii. Amount of firewall rules
>>>> 
>>>> In the first case if the L3 agent goes down then someone accessing a floating IP will be unable to access that IP. This is something that is critical for anyone running a cloud.
>>>> 
>>>> I have thought about a number of options but each has its shortcomings:
>>>> i. L3 agents to run VRRP. This will enable l3 agents to work in an active backup pair. This requires a number of changes to the agent. Each agent will have the same configuration enabling them to treat inbound and outbound traffic.
>>> I like this way, But I don't how to implement. If it works, we can have same scheduling ways:
>>> on l3 agents, we configure it to support a list of routers or routers of a list of tenants.
>>> we will use same scheduling algorithm  as we use for dhcp agents
>>>> ii. Transparent load balancing - HA proxy does not support this
>>>> iii. Having the agents rewrite the destination MAC address of the default GW (the l3 agent). This solves outbound traffic but inbound is problematic
>>>> iv. Running l3 agents on each host could ensure that the traffic generated on those hosts has floating IP's. This would require us to change the implementation of the l3 agents to only build firewall rules for devices on the HOST. 
>>> I don't like multi-host way.
>> 
>> Is this when it runs on the same host as the VM's?
> in multi-host, dhcp agent and l3 agent running on a host serves Vms on the same compute host. 
>> 
>>>> 
>>>> None of the above deal with the firewall rules. This is something that can be addressed in a similar way to the DHCP agent with the L3 agents specifically indicating which routers it will support (this is already implemented when namespaces are not supported)
>>>> 
>>>> Thanks
>>>> Gary
>>>> 
>>> Thanks
>>> Yong Sheng Gong
>>>>> mark
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> 
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> 
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> 
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> 
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: Mark McClain <mark.mcclain at dreamhost.com>
> Date: Tue, Nov 27, 2012 at 4:11 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> 
> On Nov 27, 2012, at 6:33 PM, gong yong sheng <gongysh at linux.vnet.ibm.com> wrote:
> 
> Just wanted to clarify two items:
> 
> >
> >>
> >> At the moment all of the dhcp agents receive all of the updates. I do not see why we need the quantum service to indicate which agent runs where. This will change the manner in which the dhcp agents work.
> > No. currently, we can run only one dhcp agent  since we are using a topic queue for notification.
> 
> You are correct.  There is a bug in the underlying Oslo RPC implementation that sets the topic and queue names to be same value.  I didn't get a clear explanation of this problem until today and will have to figure out a fix to oslo.
> 
> > And one problem with multiple agents serving the same ip is:
> > we will have more than one agents want to update the ip's leasetime now and than.
> 
> This is not a problem.  The DHCP protocol was designed for multiple servers on a network.  When a client accepts a lease, the server that offered the accepted lease will be the only process attempting to update the lease for that port.  The other DHCP instances will not do anything, so there won't be any chance for a conflict.  Also, when a client renews it sends a unicast message to that previous DHCP server and so there will only be one writer in this scenario too.  Additionally, we don't have to worry about conflicting assignments because the dhcp agents use the same static allocations from the Quantum database.
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Wed, Nov 28, 2012 at 5:03 AM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> On 11/28/2012 08:11 AM, Mark McClain wrote:
> On Nov 27, 2012, at 6:33 PM, gong yong sheng <gongysh at linux.vnet.ibm.com> wrote:
> 
> Just wanted to clarify two items:
> 
> At the moment all of the dhcp agents receive all of the updates. I do not see why we need the quantum service to indicate which agent runs where. This will change the manner in which the dhcp agents work.
> No. currently, we can run only one dhcp agent  since we are using a topic queue for notification.
> You are correct.  There is a bug in the underlying Oslo RPC implementation that sets the topic and queue names to be same value.  I didn't get a clear explanation of this problem until today and will have to figure out a fix to oslo.
> 
> And one problem with multiple agents serving the same ip is:
> we will have more than one agents want to update the ip's leasetime now and than.
> This is not a problem.  The DHCP protocol was designed for multiple servers on a network.  When a client accepts a lease, the server that offered the accepted lease will be the only process attempting to update the lease for that port.  The other DHCP instances will not do anything, so there won't be any chance for a conflict.  Also, when a client renews it sends a unicast message to that previous DHCP server and so there will only be one writer in this scenario too.  Additionally, we don't have to worry about conflicting assignments because the dhcp agents use the same static allocations from the Quantum database.
> I mean dhcp agent is trying to update leasetime to quantum server. If we have more than one dhcp agents, this will cause confusion.
>     def update_lease(self, network_id, ip_address, time_remaining):
>         try:
>             self.plugin_rpc.update_lease_expiration(network_id, ip_address,
>                                                     time_remaining)
>         except:
>             self.needs_resync = True
>             LOG.exception(_('Unable to update lease'))
> I think it is our dhcp agent's defect. Why does our dhcp agent need the lease time? all the IPs are managed in our quantum server, there is not need for dynamic ip management in dhcp server managed by dhcp agent.
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Wed, Nov 28, 2012 at 5:07 AM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> On 11/28/2012 06:59 AM, Nachi Ueno wrote:
>> Hi folks
>> 
>> IMO, quantum should follow the nova scheduler architecture. 
>> Because the scheduler problem is common in nova, quantum, cinder ( may be glance ).
>> We have done tons of discussion about scheduling staff in nova-scheduler. 
>> so  why not start to archive nova-network compatible scheduling , 
>> such as we are working on floating ips or security groups? 
>> 
>> Scheduling have also role of distribute load for resource efficient way. 
>> nova-scheduler has pluggable architecture and some driver implement different scheduling 
>> strategies.
>> There are also several availability strategites ( N-Act, M-Sby, Act-Act, Act-hot standby)  and methods (VRRP or simple vip based failover) based on service requirement.
>> These strategies could be implemented as a one nova-scheduler driver.
>> 
> Where is such kind of drivers which implement several availability strategites?
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: Mark McClain <mark.mcclain at dreamhost.com>
> Date: Wed, Nov 28, 2012 at 8:03 AM
> To: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Cc: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> 
> On Nov 28, 2012, at 8:03 AM, gong yong sheng <gongysh at linux.vnet.ibm.com> wrote:
> 
> > On 11/28/2012 08:11 AM, Mark McClain wrote:
> >> On Nov 27, 2012, at 6:33 PM, gong yong sheng <gongysh at linux.vnet.ibm.com> wrote:
> >>
> >> Just wanted to clarify two items:
> >>
> >>>> At the moment all of the dhcp agents receive all of the updates. I do not see why we need the quantum service to indicate which agent runs where. This will change the manner in which the dhcp agents work.
> >>> No. currently, we can run only one dhcp agent  since we are using a topic queue for notification.
> >> You are correct.  There is a bug in the underlying Oslo RPC implementation that sets the topic and queue names to be same value.  I didn't get a clear explanation of this problem until today and will have to figure out a fix to oslo.
> >>
> >>> And one problem with multiple agents serving the same ip is:
> >>> we will have more than one agents want to update the ip's leasetime now and than.
> >> This is not a problem.  The DHCP protocol was designed for multiple servers on a network.  When a client accepts a lease, the server that offered the accepted lease will be the only process attempting to update the lease for that port.  The other DHCP instances will not do anything, so there won't be any chance for a conflict.  Also, when a client renews it sends a unicast message to that previous DHCP server and so there will only be one writer in this scenario too.  Additionally, we don't have to worry about conflicting assignments because the dhcp agents use the same static allocations from the Quantum database.
> > I mean dhcp agent is trying to update leasetime to quantum server. If we have more than one dhcp agents, this will cause confusion.
> >    def update_lease(self, network_id, ip_address, time_remaining):
> >        try:
> >            self.plugin_rpc.update_lease_expiration(network_id, ip_address,
> >                                                    time_remaining)
> >        except:
> >            self.needs_resync = True
> >            LOG.exception(_('Unable to update lease'))
> > I think it is our dhcp agent's defect. Why does our dhcp agent need the lease time? all the IPs are managed in our quantum server, there is not need for dynamic ip management in dhcp server managed by dhcp agent.
> 
> There cannot be confusion.  The dhcp client selects only one server to accept a lease, so only one agent will update this field at a time. (See RFC2131 section 4.3.2 for protocol specifics).  The dnsmasq allocation database is static in Quantum's setup, so the lease renewal needs to propagate to the Quantum Server.  The Quantum server then uses the lease time to avoid allocating IP addresses before the lease has expired.  In Quantum, we add an additional restriction that expired allocations are not reclaimed until the associated port has been deleted as well.
> 
> ----------
> From: Vinay Bannai <vbannai at gmail.com>
> Date: Fri, Nov 30, 2012 at 3:49 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> Gary and Mark,
> 
> You brought up the issue of scaling horizontally and vertically in your earlier email. In the case of horizontal scaling, I would agree that it would have to be based on the "scheduler" approach proposed by Gong and Nachi. 
> 
> On the issue of vertical scaling (I am using the DHCP redundancy as an example), I think it would be good to base our discussions on the various methods that have been discussed and do pro/con analysis in terms of scale, performance and other such metrics. 
> 
> - Split scope DHCP (two or more servers split the IP address and there is no overlap)
>   pros: simple
>   cons: wastes IP addresses,
> 
> - Active/Standby model (might have run VRRP or hearbeats to dictate who is active)
>   pros: load evenly shared
>   cons: needs shared knowledge of address assignments, 
>             need hearbeats or VRRP to keep track of failovers
> 
> - LB method (use load balancer to fan out to multiple dhcp servers)
>   pros: scales very well 
>   cons: the lb becomes the single point of failure,
>            the lease assignments needs to be shared between the dhcp servers
> 
> I see that the DHCP agent and the quantum server communicate using RPC. Is the plan to leave it alone or migrate it towards something like AMQP based server in the future when the "scheduler" stuff is implemented. 
> 
> Vinay
> 
> ----------
> From: Bhandaru, Malini K <malini.k.bhandaru at intel.com>
> Date: Fri, Nov 30, 2012 at 4:52 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> Vinay, good summary of the options and their pros-cons!
> 
>  
> 
> The simple solution does not waste IP addresses, same we have 2 servers and the first exhausts its address pool, then the second server is the only one with addresses left, we it will get used.  A negative is that if one of the servers were to go down, we have lost access to half the address space .. but if and when the server comes back, all shall be good. In the meantime, guess we would need to monitor the dhcp agents along with all other services/servers to see if they are alive.
> 
>  
> 
> - Split scope DHCP (two or more servers split the IP address and there is no overlap)
> 
>   pros: simple
> 
>   cons: wastes IP addresses,
> 
>  
> 
>  
> 
> From: Vinay Bannai [mailto:vbannai at gmail.com] 
> Sent: Friday, November 30, 2012 3:49 PM
> To: OpenStack Development Mailing List
> Subject: Re: [openstack-dev] [Quantum] continuing todays discussion about the l3 agents
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Fri, Nov 30, 2012 at 5:31 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> Cc: Vinay Bannai <vbannai at gmail.com>
> 
> 
> On 12/01/2012 07:49 AM, Vinay Bannai wrote:
>> Gary and Mark,
>> 
>> You brought up the issue of scaling horizontally and vertically in your earlier email. In the case of horizontal scaling, I would agree that it would have to be based on the "scheduler" approach proposed by Gong and Nachi. 
>> 
>> On the issue of vertical scaling (I am using the DHCP redundancy as an example), I think it would be good to base our discussions on the various methods that have been discussed and do pro/con analysis in terms of scale, performance and other such metrics. 
>> 
>> - Split scope DHCP (two or more servers split the IP address and there is no overlap)
>>   pros: simple
>>   cons: wastes IP addresses,
>> 
>> - Active/Standby model (might have run VRRP or hearbeats to dictate who is active)
>>   pros: load evenly shared
>>   cons: needs shared knowledge of address assignments, 
>>             need hearbeats or VRRP to keep track of failovers
> another one is the IP address waste. we need one VIP, and 2+ more address for VRRP servers. ( we can use dhcp server's ip if we don't want to do load balancing behind the VRRP servers)
> another one is it will make system complicated.
> 
>> 
>> - LB method (use load balancer to fan out to multiple dhcp servers)
>>   pros: scales very well 
>>   cons: the lb becomes the single point of failure,
>>            the lease assignments needs to be shared between the dhcp servers
>> 
> LB method will also wast ip address. First we at lease need a VIP address. then we will need more dhcp servers running for one network.
> If we need to VRRP the VIP, we will need 2+ more addresses.
> another one is it will make system complicated.
> 
>> I see that the DHCP agent and the quantum server communicate using RPC. Is the plan to leave it alone or migrate it towards something like AMQP based server in the future when the "scheduler" stuff is implemented. 
> I am not very clear your point. But current RPC is on AMQP.
> 
> ----------
> From: Vinay Bannai <vbannai at gmail.com>
> Date: Sun, Dec 2, 2012 at 6:52 AM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> Yes we will have to monitor the dhcp agents. 
> 
> To scale "vertically" (as Mark put it), we need to adopt some of these redundancy and failover techniques. We may end up having to provide all the  options to the users and let them pick at the time of creation of their networks. 
> 
> Vinay
> 
> ----------
> From: Gary Kotton <gkotton at redhat.com>
> Date: Sun, Dec 2, 2012 at 7:16 AM
> To: openstack-dev at lists.openstack.org
> 
> 
> On 12/01/2012 03:31 AM, gong yong sheng wrote:
>> On 12/01/2012 07:49 AM, Vinay Bannai wrote:
>>> Gary and Mark,
>>> 
>>> You brought up the issue of scaling horizontally and vertically in your earlier email. In the case of horizontal scaling, I would agree that it would have to be based on the "scheduler" approach proposed by Gong and Nachi. 
> 
> I am not sure that I understand the need for a scheduler when it comes to the DHCP agent.  In my opinion this is unnecessary overhead and it is not necessarily required. 
> 
> Last week Mark addressed the problem with all of the DHCP agents all listening on the same message queue. In theory we are able to run more than one DHCP agents in parallel. This offers HA at the expense of an IP per DHCP agent per subnet. 
> 
> I think that for the DHCP agents we need to look into enabling the DHCP agents to treat specific networks. This can be done in a very rudimentary way - have a configuration variable for the DHCP agent indicating a list of networks to be treated by the agent. A orchestration tool can just configure the network ID's and launch the service - then we will have scalable and highly available DHCP service. I would prefer not to have to add this into the Quantum API as it just complicates things.
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: Vinay Bannai <vbannai at gmail.com>
> Date: Sun, Dec 2, 2012 at 10:29 AM
> To: gkotton at redhat.com, OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> My understanding of the "scheduler" approach based on what I read on the ML's is to have a mechanism where the DHCP agents can land on different nodes. For example, just like we have compute hosts in nova, we have a bunch of DHCP capable hosts (and L3 capable hosts etc) that can be selected to host the network service for a tenant when the network/subnet is created. The process of selecting the host to run the service is based on a "scheduler". This allows a graceful horizontal scaling. This approach is similar to what nova does. You have a bunch of hosts capable of providing a network service and the "scheduler" picks them based on filters and other tunable knobs. I think you already know this:-). I  was spelling it out so that you can see where I am coming from. 
> 
> Either way we look at it, I think it will be helpful if we decoupled the horizontal (scaling to multiple nodes) and vertical scaling (redundancy and failover). One should not imply the other. In your last paragraph, you mention "orchestration tool" and dhcp agents configured to handle specific networks. I have not been able to wrap my head around this completely but it appears to b ea different variant of the "scheduler" approach where it is configured manually. Is my understanding correct? Or if you don't mind, can you elaborate further on that idea. 
> 
> Thanks
> Vinay
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Sun, Dec 2, 2012 at 6:16 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> Cc: Vinay Bannai <vbannai at gmail.com>, gkotton at redhat.com
> 
> 
> On 12/03/2012 02:29 AM, Vinay Bannai wrote:
>> My understanding of the "scheduler" approach based on what I read on the ML's is to have a mechanism where the DHCP agents can land on different nodes. For example, just like we have compute hosts in nova, we have a bunch of DHCP capable hosts (and L3 capable hosts etc) that can be selected to host the network service for a tenant when the network/subnet is created. The process of selecting the host to run the service is based on a "scheduler". This allows a graceful horizontal scaling. This approach is similar to what nova does. You have a bunch of hosts capable of providing a network service and the "scheduler" picks them based on filters and other tunable knobs. I think you already know this:-). I  was spelling it out so that you can see where I am coming from. 
> If we don't want all dhcp agents to host the data of all the networks,
> My Idea is:
> 1. let quantum server have the ability to know all about the dhcp agents. for example we can have quantum agents-list to show all the agents running in the quantum deloyment,
> and the network they are hosting.
> 2. let admin user have the ability to config the dhcp agents what networks they should host.   For example, quantum dhcpagent-update dhcpagent1 --networks network1 network2 network3. or quantum net-create network1 --dhcpagents agent1 agent2. And if admin user does not specify which agent to host which network, we can let scheduler to decide automatically
> So for scale vertically:
> we can specify much agents host some same networks
> So for scale horizontally:
> we can add as many as dhcp agents. quantum scheduler will distribute new networks automatically or admin user can specify.
> 
> For us to run multiple dhcp agents, we need to make sure our dhcp anti spoofing work.
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Sun, Dec 2, 2012 at 6:21 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> Cc: Vinay Bannai <vbannai at gmail.com>, gkotton at redhat.com
> 
> 
> On 12/03/2012 02:29 AM, Vinay Bannai wrote:
>> My understanding of the "scheduler" approach based on what I read on the ML's is to have a mechanism where the DHCP agents can land on different nodes. For example, just like we have compute hosts in nova, we have a bunch of DHCP capable hosts (and L3 capable hosts etc) that can be selected to host the network service for a tenant when the network/subnet is created. The process of selecting the host to run the service is based on a "scheduler". This allows a graceful horizontal scaling. This approach is similar to what nova does. You have a bunch of hosts capable of providing a network service and the "scheduler" picks them based on filters and other tunable knobs. I think you already know this:-). I  was spelling it out so that you can see where I am coming from. 
> If we don't want all dhcp agents to host the data of all the networks,
> My Idea is:
> 1. let quantum server have the ability to know about all  dhcp agents. for example we can have quantum agents-list to show all the agents running in the quantum deloyment,
> and the network they are hosting. (I think this command is needed anyway)
> 2. let admin user have the ability to config the dhcp agents what networks they should host.   For example, quantum dhcpagent-update dhcpagent1 --networks network1 network2 network3. or quantum net-create|udpate network1 --dhcpagents agent1 agent2. And if admin user does not specify which agent to host which network, we can let scheduler to decide automatically
> So for scale vertically:
> we can specify much agents to host some same networks
> 
> So for scale horizontally:
> we can add as many as dhcp agents. quantum scheduler will distribute new networks automatically or admin user can specify.
> 
> For us to run multiple dhcp agents, we need to make sure our dhcp anti spoofing work.
>> 
> 
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Sun, Dec 2, 2012 at 6:57 PM
> To: gkotton at redhat.com, OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> 
> 
> On 12/02/2012 11:16 PM, Gary Kotton wrote:
>> On 12/01/2012 03:31 AM, gong yong sheng wrote:
>>> On 12/01/2012 07:49 AM, Vinay Bannai wrote:
>>>> Gary and Mark,
>>>> 
>>>> You brought up the issue of scaling horizontally and vertically in your earlier email. In the case of horizontal scaling, I would agree that it would have to be based on the "scheduler" approach proposed by Gong and Nachi. 
>> 
>> I am not sure that I understand the need for a scheduler when it comes to the DHCP agent.  In my opinion this is unnecessary overhead and it is not necessarily required. 
>> 
>> Last week Mark addressed the problem with all of the DHCP agents all listening on the same message queue. In theory we are able to run more than one DHCP agents in parallel. This offers HA at the expense of an IP per DHCP agent per subnet. 
>> 
>> I think that for the DHCP agents we need to look into enabling the DHCP agents to treat specific networks. This can be done in a very rudimentary way - have a configuration variable for the DHCP agent indicating a list of networks to be treated by the agent. A orchestration tool can just configure the network ID's and launch the service - then we will have scalable and highly available DHCP service. I would prefer not to have to add this into the Quantum API as it just complicates things.
>> 
>> 
>> 
> I think we should not write configure item on the side of dhcp agents. we should allow to add dynamically dhcp agent and then configure it through quantum server.
> zero configuration (except the quantum server and message queue configuration) on dhcp agents will help a lot.
> We just start a new dhcp agent, and then config it through quantum API (maybe extension) or schedule networks auto to newly added dhcp agents.
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ----------
> From: Gary Kotton <gkotton at redhat.com>
> Date: Mon, Dec 3, 2012 at 1:32 AM
> To: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Cc: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>, Vinay Bannai <vbannai at gmail.com>
> 
> 
> On 12/03/2012 04:16 AM, gong yong sheng wrote:
>> On 12/03/2012 02:29 AM, Vinay Bannai wrote:
>>> My understanding of the "scheduler" approach based on what I read on the ML's is to have a mechanism where the DHCP agents can land on different nodes. For example, just like we have compute hosts in nova, we have a bunch of DHCP capable hosts (and L3 capable hosts etc) that can be selected to host the network service for a tenant when the network/subnet is created. The process of selecting the host to run the service is based on a "scheduler". This allows a graceful horizontal scaling. This approach is similar to what nova does. You have a bunch of hosts capable of providing a network service and the "scheduler" picks them based on filters and other tunable knobs. I think you already know this:-). I  was spelling it out so that you can see where I am coming from. 
>> If we don't want all dhcp agents to host the data of all the networks,
>> My Idea is:
>> 1. let quantum server have the ability to know all about the dhcp agents. for example we can have quantum agents-list to show all the agents running in the quantum deloyment,
>> and the network they are hosting.
> 
> ok, this may be useful for debugging. will this display and have the status of the dhcp agent, say for example i deploy and agent and it has an exception do to a bug? 
> 
>> 2. let admin user have the ability to config the dhcp agents what networks they should host.   For example, quantum dhcpagent-update dhcpagent1 --networks network1 network2 network3. or quantum net-create network1 --dhcpagents agent1 agent2. And if admin user does not specify which agent to host which network, we can let scheduler to decide automatically
> 
> this is exactly what i am suggesting, except we do not need to change the quantum api to provide this. the agent can receive this as an input parameter. in principle we agree on what needs to be done, but the question is how.
> 
> 
>> So for scale vertically:
>> we can specify much agents host some same networks
>> So for scale horizontally:
>> we can add as many as dhcp agents. quantum scheduler will distribute new networks automatically or admin user can specify.
> 
> i have a number of problems with a quantum "scheduler". the first being a single point of failure. the second being the fact that it needs to be aware of the state and load of a dhcp agent. how will the scheduler provide for HA?
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Mon, Dec 3, 2012 at 1:53 AM
> To: gkotton at redhat.com
> Cc: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>, Vinay Bannai <vbannai at gmail.com>
> 
> 
> On 12/03/2012 05:32 PM, Gary Kotton wrote:
>> On 12/03/2012 04:16 AM, gong yong sheng wrote:
>>> On 12/03/2012 02:29 AM, Vinay Bannai wrote:
>>>> My understanding of the "scheduler" approach based on what I read on the ML's is to have a mechanism where the DHCP agents can land on different nodes. For example, just like we have compute hosts in nova, we have a bunch of DHCP capable hosts (and L3 capable hosts etc) that can be selected to host the network service for a tenant when the network/subnet is created. The process of selecting the host to run the service is based on a "scheduler". This allows a graceful horizontal scaling. This approach is similar to what nova does. You have a bunch of hosts capable of providing a network service and the "scheduler" picks them based on filters and other tunable knobs. I think you already know this:-). I  was spelling it out so that you can see where I am coming from. 
>>> If we don't want all dhcp agents to host the data of all the networks,
>>> My Idea is:
>>> 1. let quantum server have the ability to know all about the dhcp agents. for example we can have quantum agents-list to show all the agents running in the quantum deloyment,
>>> and the network they are hosting.
>> 
>> ok, this may be useful for debugging. will this display and have the status of the dhcp agent, say for example i deploy and agent and it has an exception do to a bug? nn
> I think it is good for management, not just good for debugging. If the agent cannot start, the quantum agents-list will not have it in a :) status. If it has exception and is still running, we can have it report its status with exception. But regarding the log,  I think we should leave it to upper management tools. For example, admin user can start the agent with integrated log facility.
> 
>> 
>>> 2. let admin user have the ability to config the dhcp agents what networks they should host.   For example, quantum dhcpagent-update dhcpagent1 --networks network1 network2 network3. or quantum net-create network1 --dhcpagents agent1 agent2. And if admin user does not specify which agent to host which network, we can let scheduler to decide automatically
>> 
>> this is exactly what i am suggesting, except we do not need to change the quantum api to provide this. the agent can receive this as an input parameter. in principle we agree on what needs to be done, but the question is how.
> I am suggesting to control agents from quantum server. we can use quantum cli or API to command which agents host which networks. Of course admin user also can configure the dhcp agents to accept only some networks data. We can support both.
> We can add quantum api ( maybe by extension) and store the networks and dhcp agents mapping in db. then notify related dhcp agents with related data.
> 
>> 
>>> So for scale vertically:
>>> we can specify much agents host some same networks
>>> So for scale horizontally:
>>> we can add as many as dhcp agents. quantum scheduler will distribute new networks automatically or admin user can specify.
>> 
>> i have a number of problems with a quantum "scheduler". the first being a single point of failure. the second being the fact that it needs to be aware of the state and load of a dhcp agent. how will the scheduler provide for HA?
> What do you mean by single point of failure? In fact we don't need to run it in a separate binary at all. It is just a module of quantum-server. 
> state and load of a dhcp agent is in db. If you are meaning the quantum-server's HA, I don't think we already have a good one by now. I think we should break the current quantum-server into two parts:
> 1. one part is for just API (rest): for which we can use multiple process just like other nova api servers. and operator will also be able to use load balancer to HA it.
> 2. anyone part is for incoming queue message processing. We can run multiple this part nodes, this is the AMQP feature to scale out.
> 
> ----------
> From: Gary Kotton <gkotton at redhat.com>
> Date: Mon, Dec 3, 2012 at 2:19 AM
> To: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Cc: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>, Vinay Bannai <vbannai at gmail.com>
> 
> 
> On 12/03/2012 11:53 AM, gong yong sheng wrote:
>> On 12/03/2012 05:32 PM, Gary Kotton wrote:
>>> On 12/03/2012 04:16 AM, gong yong sheng wrote:
>>>> On 12/03/2012 02:29 AM, Vinay Bannai wrote:
>>>>> My understanding of the "scheduler" approach based on what I read on the ML's is to have a mechanism where the DHCP agents can land on different nodes. For example, just like we have compute hosts in nova, we have a bunch of DHCP capable hosts (and L3 capable hosts etc) that can be selected to host the network service for a tenant when the network/subnet is created. The process of selecting the host to run the service is based on a "scheduler". This allows a graceful horizontal scaling. This approach is similar to what nova does. You have a bunch of hosts capable of providing a network service and the "scheduler" picks them based on filters and other tunable knobs. I think you             already know this:-). I  was spelling it out so that you can see where I am coming from. 
>>>> If we don't want all dhcp agents to host the data of all the networks,
>>>> My Idea is:
>>>> 1. let quantum server have the ability to know all about the dhcp agents. for example we can have quantum agents-list to show all the agents running in the quantum deloyment,
>>>> and the network they are hosting.
>>> 
>>> ok, this may be useful for debugging. will this display and have the status of the dhcp agent, say for example i deploy and agent and it has an exception do to a bug? nn
>> I think it is good for management, not just good for debugging. If the agent cannot start, the quantum agents-list will not have it in a :) status. If it has exception and is still running, we can have it report its status with exception. But regarding the log,  I think we should leave it to upper management tools. For example, admin user can start the agent with integrated log facility.
> 
> Visibility is always good. 
>>>> 2. let admin user have the ability to config the dhcp agents what networks they should host.   For example, quantum dhcpagent-update dhcpagent1 --networks network1 network2 network3. or quantum net-create network1 --dhcpagents agent1 agent2. And if admin user does not specify which agent to host which network, we can let scheduler to decide automatically
>>> 
>>> this is exactly what i am suggesting, except we do not need to change the quantum api to provide this. the agent can receive this as an input parameter. in principle we agree on what needs to be done, but the question is how.
>> I am suggesting to control agents from quantum server. we can use quantum cli or API to command which agents host which networks. Of course admin user also can configure the dhcp agents to accept only some networks data. We can support both.
>> We can add quantum api ( maybe by extension) and store the networks and dhcp agents mapping in db. then notify related dhcp agents with related data.
> 
> I personally do not like this approach and I think that the agents should be independent of the service (as much as possible). At the moment the DHCP agent gets its configuration via the notifications. I have a number of questions:
> 1. do you plan to continue to use the same mechanism of implement direct communication with the specific agents?
> 2. just because a compute node/network node loses connectivity with the quantum service does not mean that it loses network connectivity (and vice versa). How will we deal with issues like this?
> 3. Can you please clarify more on the extension. will this be on the subnet (that is where there is a flag for dhcp status)
> 
> ----------
> From: gong yong sheng <gongysh at linux.vnet.ibm.com>
> Date: Mon, Dec 3, 2012 at 3:42 AM
> To: gkotton at redhat.com
> Cc: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>, Vinay Bannai <vbannai at gmail.com>
> 
> 
> On 12/03/2012 06:19 PM, Gary Kotton wrote:
>> On 12/03/2012 11:53 AM, gong yong sheng wrote:
>>> On 12/03/2012 05:32 PM, Gary Kotton wrote:
>>>> On 12/03/2012 04:16 AM, gong yong sheng wrote:
>>>>> On 12/03/2012 02:29 AM, Vinay Bannai wrote:
>>>>>> My understanding of the "scheduler" approach based on what I read on the ML's is to have a mechanism where the DHCP agents can land on different nodes. For example, just like we have compute hosts in nova, we have a bunch of DHCP capable hosts (and L3 capable hosts etc)               that can be selected to host the network service for a tenant when the network/subnet is created. The process of selecting the host to run the service is based on a "scheduler". This allows a graceful horizontal scaling. This approach is similar to what nova does. You have a bunch of hosts capable of providing a network service and the "scheduler" picks them based on filters and other tunable knobs. I think you already know this:-). I  was spelling it out so that you can see where I am coming from. 
>>>>> If we don't want all dhcp agents to host the data of all the networks,
>>>>> My Idea is:
>>>>> 1. let quantum server have the ability to know all about the dhcp agents. for example we can have quantum agents-list to show all the agents running in the quantum deloyment,
>>>>> and the network they are hosting.
>>>> 
>>>> ok, this may be useful for debugging. will this display and have the status of the dhcp agent, say for example i deploy and agent and it has an exception do to a bug? nn
>>> I think it is good for management, not just good for debugging. If the agent cannot start, the quantum agents-list will not have it in a :) status. If it has exception and is still running, we can have it report its status with exception. But regarding the log,  I think we should leave it to upper management tools. For example, admin user can start the agent with integrated log facility.
>> 
>> Visibility is always good. 
>>>>> 2. let admin user have the ability to config the dhcp agents what networks they should host.   For example, quantum dhcpagent-update dhcpagent1 --networks network1 network2 network3. or quantum net-create network1 --dhcpagents agent1 agent2. And if admin user does not specify which agent to host which network, we can let scheduler to decide automatically
>>>> 
>>>> this is exactly what i am suggesting, except we do not need to change the quantum api to provide this. the agent can receive this as an input parameter. in principle we agree on what needs to be done, but the question is how.
>>> I am suggesting to control agents from quantum server. we can use quantum cli or API to command which agents host which networks. Of course admin user also can configure the dhcp agents to accept only some networks data. We can support both.
>>> We can add quantum api ( maybe by extension) and store the networks and dhcp agents mapping in db. then notify related dhcp agents with related data.
>> 
>> I personally do not like this approach and I think that the agents should be independent of the service (as much as possible). At the moment the DHCP agent gets its configuration via the notifications. I have a number of questions:
> Agent's configuration is all in quantum.conf and dhcp.ini file.  the network data of dhcp server is from quantum server by message notification.
> 
>> 1. do you plan to continue to use the same mechanism of implement direct communication with the specific agents?
> I am not quite following you here. But I think we can use the similar AMQP message way to do this.  dhcp agents listening on their specific queue, such as dhcp.hosta, dhcp.hostb, etc.
> we will send message to these queues according to which networks are hosted by which agents. Maybe, there are some ones hating host (which is kind of physical meaning). But I think it is just a name.
> 
>> 2. just because a compute node/network node loses connectivity with the quantum service does not mean that it loses network connectivity (and vice versa). How will we deal with issues like this?
> I don't understand what u mean here. Can u give an example?
> 
>> 3. Can you please clarify more on the extension. will this be on the subnet (that is where there is a flag for dhcp status)
> Extension is just like l3, quota and provider network extension we have. I think it will extend the network model since our dhcp agent is starting a dnsmasq server per network.
> of course, we can have a try to a smaller granule subnet
> 
> ----------
> From: Soheil Eizadi <seizadi at infoblox.com>
> Date: Mon, Dec 3, 2012 at 10:12 AM
> To: "openstack-dev at lists.openstack.org" <openstack-dev at lists.openstack.org>
> 
> 
> I am commenting on this thread since it is an area of interest, I am new
> to this list and only get the Digest version right now. If there is a link
> to the Wiki that documents the use cases for this feature that would be
> great, I am new to OpenStack development and might not have the full
> picture of what you are trying to implement.
> 
> There was discussion of FO, but I did not see the draft IETF draft from
> 2003 referenced:
> http://tools.ietf.org/html/draft-ietf-dhc-failover-12
> I don't think there is any ongoing effort to standardize a real RFC. We
> don't see a market demand for interoperable solutions, probably why there
> is not an RFC, but this RFC draft is a good guideline if you are going to
> design FO. Vendors that have commercial FO, have implemented extension to
> the draft RFC. Microsoft documented their extensions here:
> http://msdn.microsoft.com/en-us/library/hh880579(v=prot.20).aspx
> 
> 
> From this thread I see a requirement for HA and in the thread there was a
> discussion of VRRP and FO, I read the writeup as mutually exclusive
> options. Depending on the requirements for HA you might need both. The use
> case for VRRP type HA active/passive provides local survivability while
> the active/active FO solution could be used by customers that desire DR.
> In some use cases we have customers use both of them.
> 
> 
> I think the proposal on this thread to have a flexible agent registration
> approach that can accommodate how dhcp agents are mapped to networks, e.g.
> One to one, many to one, and one to many is great. It provides the
> flexibility depending on the Service Provider use cases.
> 
> I did not see a discussion of following topics for dhcp agents, not sure
> if it is in the use case, but of interest to the work I am researching:
> - Ability for DHCP as a Plugin, rather than built-in as core service.
> - Ability to support Hybrid Cloud, in this environment the DHCP Service
> would need to coordinate IP Address between private and public cloud, with
> a stretch network. (DHCP Plugin is a component of this solution.)
> 
> -Soheil




More information about the OpenStack-dev mailing list