<div dir="ltr"><div>><span style="font-size:12.8000001907349px">The only thing this discussion has convinced me of is that allowing users to </span><span style="font-size:12.8000001907349px">change the fixed IP address on a neutron port leads to a bad user-experience.</span></div><div><br></div><div>Not as bad as having to delete a port and create another one on the same network just to change addresses though...</div><div><br></div>><span style="font-size:12.8000001907349px">Even with an 8-minute renew time you're talking up to a 7-minute blackout (87.5% </span><span style="font-size:12.8000001907349px">of lease time before using broadcast).</span><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">I suggested 240 seconds renewal time, which is up to 4 minutes of connectivity outage. This doesn't have anything to do with lease time and unicast DHCP will work because the spoof rules allow DHCP client traffic before restricting to specific IPs. </span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px"> </span><span style="font-size:12.8000001907349px">Most would have rebooted long before then, true? Cattle not pets, right?</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">Only in an ideal world that I haven't encountered with customer deployments. Many enterprise deployments end up bringing pets along where reboots aren't always free. The time taken to relaunch programs and restore state can end up being 10 minutes+ if it's something like a VDI deployment or dev environment where someone spends a lot of time working on one VM.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px">Changing the lease time is just papering-over the real bug - neutron doesn't </span><span style="font-size:12.8000001907349px">support seamless changes in IP addresses on ports, since it totally relies on </span><span style="font-size:12.8000001907349px">the dhcp configuration settings a deployer has chosen.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">It doesn't need to be seamless, but it certainly shouldn't be useless. Connectivity interruptions can be expected with IP changes (e.g. I've seen changes in elastic IPs on EC2 can interrupt connectivity to an instance for up to 2 minutes), but an entire day of downtime is awful. </span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">One of the things I'm getting at is that a deployer shouldn't be choosing such high lease times and we are encouraging it with a high default. You are arguing for infrequent renewals to work around excessive logging, which is just an implementation problem that should be addressed with a patch to your logging collector (de-duplication) or to dnsmasq (don't log renewals).</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">></span><span style="font-size:12.8000001907349px">Documenting a VM </span><span style="font-size:12.8000001907349px">reboot is necessary, or even deprecating this (you won't like that) are sounding </span><span style="font-size:12.8000001907349px">better to me by the minute.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">If this is an approach you really want to go with, then we should at least be consistent and deprecate the extra dhcp options extension (or at least the ability to update ports' dhcp options). Updating subnet attributes like gateway_ip, dns_nameserves, and host_routes should be thrown out as well. All of these things depend on the DHCP server to deliver updated information and are hindered by renewal times. Why discriminate against IP updates on a port? A failure to receive many of those other types of changes could result in just as severe of a connection disruption.</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><br></div><div>In summary, the information the DHCP server gives to clients is not static. Unless we eliminate updates to everything in the Neutron API that results in different DHCP lease information, my suggestion is that we include a new option for the renewal interval and have the default set <5 minutes. We can leave the lease default to 1 day so the amount of time a DHCP server can be offline without impacting running clients can stay the same.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 30, 2015 at 8:00 AM, Brian Haley <span dir="ltr"><<a href="mailto:brian.haley@hp.com" target="_blank">brian.haley@hp.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Kevin,<br>
<br>
The only thing this discussion has convinced me of is that allowing users to<br>
change the fixed IP address on a neutron port leads to a bad user-experience.<br>
Even with an 8-minute renew time you're talking up to a 7-minute blackout (87.5%<br>
of lease time before using broadcast). This is time that customers are paying<br>
for. Most would have rebooted long before then, true? Cattle not pets, right?<br>
<br>
Changing the lease time is just papering-over the real bug - neutron doesn't<br>
support seamless changes in IP addresses on ports, since it totally relies on<br>
the dhcp configuration settings a deployer has chosen. Bickering over the lease<br>
time doesn't fix that non-deterministic recovery for the VM. Documenting a VM<br>
reboot is necessary, or even deprecating this (you won't like that) are sounding<br>
better to me by the minute.<br>
<br>
Is there anyone else that has used, or has customers using, this part of the<br>
neutron API? Can they share their experiences?<br>
<br>
-Brian<br>
<div><div class="h5"><br>
<br>
On 01/30/2015 07:26 AM, Kevin Benton wrote:<br>
>>But they will if we document it well, which is what Salvatore suggested.<br>
><br>
> I don't think this is a good approach, and it's a big part of why I started this<br>
> thread. Most of the deployers/operators I have worked with only read the bare<br>
> minimum documentation to get a Neutron deployment working and they only adjust<br>
> the settings necessary for basic functionality.<br>
><br>
> We have an overwhelming amount of configuration options and adding a note<br>
> specifying that a particular setting for DHCP leases has been optimized to<br>
> reduce logging at the cost of long downtimes during port IP address updates is a<br>
> waste of time and effort on our part.<br>
><br>
>>I think the current default value is also more indicative of something<br>
> you'd find in your house, or at work - i.e. stable networks.<br>
><br>
> Tenants don't care what the DHCP lease time is or that it matches what they<br>
> would see from a home router. They only care about connectivity.<br>
><br>
>>One solution is to disallow this operation.<br>
><br>
> I want this feature to be useful in deployments by default, not strip it<br>
> away. You can probably do this with /etc/neutron/policy.json without a code<br>
> change if you wanted to block it in a deployment like yours where you have such<br>
> a high lease time.<br>
><br>
>>Perhaps letting the user set it, but allow the admin to set the valid range<br>
> for min/max? And if they don't specify they get the default?<br>
><br>
> Tenants wouldn't have any reason to adjust this default. They would be even less<br>
> likely than the operator to know about this weird relationship between a DHCP<br>
> setting and the amount of time they lose connectivity after updating their<br>
> ports' IPs.<br>
><br>
>>It impacts anyone that hasn't changed from the default since July 2013 and later<br>
> (Havana), since if they don't notice, they might get bitten by it.<br>
><br>
> Keep in mind that what I am suggesting with the lease-renewal-time would be<br>
> separate from the lease expiration time. The only difference that an operator<br>
> would see on upgrade (if using the defaults) is increased DHCP traffic and more<br>
> logs to syslog from dnsmasq. The lease time would still be the same so the<br>
> downtime windows for DHCP agents would be maintained. That is much less of an<br>
> impact than many of the non-config changes we make between cycles.<br>
><br>
> To clarify, even with an option for dhcp-renewal-time I am proposing, you are<br>
> still opposed to setting it to anything low because of logging and the ~24 bps<br>
> background DHCP traffic per VM?<br>
><br>
> On Thu, Jan 29, 2015 at 7:11 PM, Brian Haley <<a href="mailto:brian.haley@hp.com">brian.haley@hp.com</a><br>
</div></div><div><div class="h5">> <mailto:<a href="mailto:brian.haley@hp.com">brian.haley@hp.com</a>>> wrote:<br>
><br>
> On 01/29/2015 05:28 PM, Kevin Benton wrote:<br>
> >>How is Neutron breaking this? If I move a port on my physical switch to a<br>
> > different subnet, can you still communicate with the host sitting on it?<br>
> > Probably not since it has a view of the world (next-hop router) that no longer<br>
> > exists, and the network won't route packets for it's old IP address to the new<br>
> > location. It has to wait for it's current DHCP lease to tick down to the point<br>
> > where it will use broadcast to get a new one, after which point it will work.<br>
> ><br>
> > That's not just moving to a different subnet. That's moving to a different<br>
> > broadcast domain. Neutron supports multiple subnets per network (broadcast<br>
> > domain). An address on either subnet will work. The router has two interfaces<br>
> > into the network, one on each subnet.[2]<br>
> ><br>
> ><br>
> >>Does it work on Windows VMs too? People run those in clouds too. The point is<br>
> > that if we don't know if all the DHCP clients will support it then it's a<br>
> > non-starter since there's no way to tell from the server side.<br>
> ><br>
> > It appears they do.[1] Even for clients that don't, the worst case scenario is<br>
> > just that they are stuck where we are now.<br>
> ><br>
> >>"... then the deployer can adjust the value upwards...", hmm, can they adjust it<br>
> > downwards as well? :)<br>
> ><br>
> > Yes, but most people doing initial openstack deployments don't and wouldn't<br>
> > think to without understanding the intricacies of the security groups filtering<br>
> > in Neutron.<br>
><br>
> But they will if we document it well, which is what Salvatore suggested.<br>
><br>
> >>I'm glad you're willing to "boil the ocean" to try and get the default changed,<br>
> > but is all this really worth it when all you have to do is edit the config file<br>
> > in your deployment? That's why the value is there in the first place.<br>
> ><br>
> > The default value is basically incompatible with port IP changes. We shouldn't<br>
> > be shipping defaults that lead to half-broken functionality. What I'm<br>
> > understanding is that the current default value is to workaround shortcomings in<br>
> > dnsmasq. This is an example of implementation details leaking out and leading to<br>
> > bad UX.<br>
><br>
> I think the current default value is also more indicative of something you'd<br>
> find in your house, or at work - i.e. stable networks.<br>
><br>
> I had another thought on this Kevin, hoping that we could come to some<br>
> resolution, because sure, shipping broken functionality isn't great. But here's<br>
> the rub - how do we make a change in a fixed IP work in *all* deployments?<br>
> Since the end-user can't set this value, they'll run into this problem in my<br>
> deployment, or any other that has some not-very-short lease time. One solution<br>
> is to disallow this operation. The other is to fix neutron to make this work<br>
> better (I don't know what that involves, but there's bound to be a way).<br>
> Perhaps letting the user set it, but allow the admin to set the valid range for<br>
> min/max? And if they don't specify they get the default?<br>
><br>
> > If we had an option to configure how often iptables rules were refreshed to<br>
> > match their security group, there is no way we would have a default of 12 hours.<br>
> > This is essentially the same level of connectivity interruption, it just happens<br>
> > to be a narrow use case so it hasn't been getting any attention.<br>
> ><br>
> > To flip your question around, why do you care if the default is lower? You<br>
> > already adjust it beyond the 1 day default in your deployment, so how would a<br>
> > different default impact you?<br>
><br>
> It impacts anyone that hasn't changed from the default since July 2013 and later<br>
> (Havana), since if they don't notice, they might get bitten by it.<br>
><br>
> -Brian<br>
><br>
><br>
> ><br>
> > 1. <a href="http://support.microsoft.com/kb/121005" target="_blank">http://support.microsoft.com/kb/121005</a><br>
> > 2. Similar to using the "secondary" keyword on Cisco devices. Or just the "ip<br>
> > addr add" command on linux.<br>
> ><br>
> > On Thu, Jan 29, 2015 at 1:34 PM, Brian Haley <<a href="mailto:brian.haley@hp.com">brian.haley@hp.com</a> <mailto:<a href="mailto:brian.haley@hp.com">brian.haley@hp.com</a>><br>
</div></div><div><div class="h5">> > <mailto:<a href="mailto:brian.haley@hp.com">brian.haley@hp.com</a> <mailto:<a href="mailto:brian.haley@hp.com">brian.haley@hp.com</a>>>> wrote:<br>
> ><br>
> > On 01/29/2015 03:55 AM, Kevin Benton wrote:<br>
> > >>Why would users want to change an active port's IP address anyway?<br>
> > ><br>
> > > Re-addressing. It's not common, but the entire reason I brought this<br>
> up is<br>
> > > because a user was moving an instance to another subnet on the same<br>
> network and<br>
> > > stranded one of their VMs.<br>
> > ><br>
> > >> I worry about setting a default config value to handle a very<br>
> unusual use case.<br>
> > ><br>
> > > Changing a static lease is something that works on normal networks<br>
> so I don't<br>
> > > think we should break it in Neutron without a really good reason.<br>
> ><br>
> > How is Neutron breaking this? If I move a port on my physical switch to a<br>
> > different subnet, can you still communicate with the host sitting on it?<br>
> > Probably not since it has a view of the world (next-hop router) that<br>
> no longer<br>
> > exists, and the network won't route packets for it's old IP address to<br>
> the new<br>
> > location. It has to wait for it's current DHCP lease to tick down to<br>
> the point<br>
> > where it will use broadcast to get a new one, after which point it<br>
> will work.<br>
> ><br>
> > > Right now, the big reason to keep a high lease time that I agree<br>
> with is that it<br>
> > > buys operators lots of dnsmasq downtime without affecting running<br>
> clients. To<br>
> > > get the best of both worlds we can set DHCP option 58 (a.k.a<br>
> dhcp-renewal-time<br>
> > > or T1) to 240 seconds. Then the lease time can be left to be<br>
> something large<br>
> > > like 10 days to allow for tons of DHCP server downtime without<br>
> affecting running<br>
> > > clients.<br>
> > ><br>
> > > There are two issues with this approach. First, some simple dhcp<br>
> clients don't<br>
> > > honor that dhcp option (e.g. the one with Cirros), but it works with<br>
> dhclient so<br>
> > > it should work on CentOS, Fedora, etc (I verified it works on<br>
> Ubuntu). This<br>
> > > isn't a big deal because the worst case is what we have already<br>
> (half of the<br>
> > > lease time). The second issue is that dnsmasq hardcodes that option,<br>
> so a patch<br>
> > > would be required to allow it to be specified in the options file. I<br>
> am happy to<br>
> > > submit the patch required there so that isn't a big deal either.<br>
> ><br>
> > Does it work on Windows VMs too? People run those in clouds too. The<br>
> point is<br>
> > that if we don't know if all the DHCP clients will support it then it's a<br>
> > non-starter since there's no way to tell from the server side.<br>
> ><br>
> > > If we implement that fix, the remaining issue is Brian's other<br>
> comment about too<br>
> > > much DHCP traffic. I've been doing some packet captures and the standard<br>
> > > request/reply for a renewal is 2 unicast packets totaling about 725<br>
> bytes.<br>
> > > Assuming 10,000 VMs renewing every 240 seconds, there will be an<br>
> average of 242<br>
> > > kbps background traffic across the entire network. Even at a density<br>
> of 50 VMs,<br>
> > > that's only 1.2 kbps per compute node. If that's still too much,<br>
> then the<br>
> > > deployer can adjust the value upwards, but that's hardly a reason to<br>
> have a high<br>
> > > default.<br>
> ><br>
> > "... then the deployer can adjust the value upwards...", hmm, can they<br>
> adjust it<br>
> > downwards as well? :)<br>
> ><br>
> > > That just leaves the logging problem. Since we require a change to<br>
> dnsmasq<br>
> > > anyway, perhaps we could also request an option to suppress logs<br>
> from renewals?<br>
> > > If that's not adequate, I think 2 log entries per vm every 240<br>
> seconds is really<br>
> > > only a concern for operators with large clouds and they should have the<br>
> > > knowledge required to change a config file anyway. ;-)<br>
> ><br>
> > I'm glad you're willing to "boil the ocean" to try and get the default<br>
> changed,<br>
> > but is all this really worth it when all you have to do is edit the<br>
> config file<br>
> > in your deployment? That's why the value is there in the first place.<br>
> ><br>
> > Sorry, I'm still unconvinced we need to do anything more than document<br>
> this.<br>
> ><br>
> > -Brian<br>
> ><br>
> ><br>
> ><br>
> > __________________________________________________________________________<br>
> > OpenStack Development Mailing List (not for usage questions)<br>
> > Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Kevin Benton<br>
> ><br>
> ><br>
> > __________________________________________________________________________<br>
> > OpenStack Development Mailing List (not for usage questions)<br>
> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</div></div>> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<div class="HOEnZb"><div class="h5">> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Kevin Benton<br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>