[Openstack] DHCP problem in grizzly
Thomas Kärgel
kaergel at b1-systems.de
Thu May 23 07:35:07 UTC 2013
Hi,
the bug report is open and can be viewed at:
http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2013q2/007212.html
I used some of your logs in order to add more details to my description.
Hopefully someone can help us solving this issue.
Best regards
thomas
Am 22.05.2013 17:05, schrieb Thomas Kärgel:
> Hi Marco,
>
> thanks for the interesting information. I think we can be sure now that
> this is a bug in dnsmasq. I'll go ahead and report this bug to the
> dnsmasq-developers tomorrow morning. Hopefully they can fix this issue soon.
> I'll report the details when the bugreport is open. Maybe they want more
> infos from us...
>
> Best regards
> Thomas
>
> Am 22.05.2013 14:33, schrieb Marco Colombo:
>> Hi Thomas,
>> i have upgrade dnsmasq to version 2.66, but the problem persist.
>> If you need more log/information, i'm able to reproduce the problem.
>>
>> Thanks
>> Best Regards
>>
>>
>> 2013/5/22 Thomas Kärgel <kaergel at b1-systems.de
>> <mailto:kaergel at b1-systems.de>>
>>
>> Hi Marco,
>>
>> can you also reproduce it?
>>
>> If so i would suggest to give dnsmasq 2.66 a try. Since i'm not able to
>> reproduce this behavior on the environment i'm woking on it will take
>> weeks to get to an really conclusive result.
>>
>> Best regards
>> thomas
>>
>>
>> Am 22.05.2013 09:55, schrieb Marco Colombo:
>> > Hi All,
>> > i have the same problem with Grizzly, Ubuntu 12.04 and Dnsmasq 2.59
>> > "killall dnsmasq && service quantum-dhcp-agent restart" fixes the
>> > problem temporarily
>> >
>> > Best Regards,
>> > Marco
>> >
>> >
>> >
>> >
>> > 2013/5/22 Thomas Kärgel <kaergel at b1-systems.de
>> <mailto:kaergel at b1-systems.de>
>> > <mailto:kaergel at b1-systems.de <mailto:kaergel at b1-systems.de>>>
>> >
>> > Hi Johanna,
>> >
>> > sure, I'll report my results, but this can take a while. The
>> last time
>> > this issue occurred on my environment is May 10th. It occurs quite
>> > sporadic and i don't know how to manually reproduce this issue
>> yet (i
>> > have to be very careful with my experiments since the
>> environment is
>> > running productive already). Maybe you should also give
>> dnsmasq 2.66 a
>> > try, because you wrote that you can manually reproduce this
>> behavior.
>> >
>> >
>> > Best Regards,
>> > Thomas
>> >
>> >
>> > Am 20.05.2013 12:25, schrieb Heinonen, Johanna (NSN - FI/Espoo):
>> > > Hi,
>> > >
>> > > You are right. Manually doing kill -HUP <PID> has just the
>> effect
>> > you mentioned. In the syslog you can see
>> > >
>> > > May 20 13:09:41 grizzly-236 dnsmasq[6170]: cleared cache
>> > > May 20 13:09:41 grizzly-236 dnsmasq-dhcp[6170]: read
>> > /var/lib/quantum/dhcp/d5879bbb-ada6-4323-a7b1-87b5db244513/host
>> > > May 20 13:09:41 grizzly-236 dnsmasq-dhcp[6170]: read
>> > /var/lib/quantum/dhcp/d5879bbb-ada6-4323-a7b1-87b5db244513/opts
>> > >
>> > >
>> > > But the problem stays. Only 'service quantum-dhcp-agent restart'
>> > fixes the problem.
>> > > If you try the newer version of the qnsmasq, I'd be
>> interested to
>> > hear the results.
>> > >
>> > > BR
>> > > Johanna
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: ext Thomas Kärgel [mailto:kaergel at b1-systems.de
>> <mailto:kaergel at b1-systems.de>
>> > <mailto:kaergel at b1-systems.de <mailto:kaergel at b1-systems.de>>]
>> > > Sent: Monday, May 20, 2013 11:37 AM
>> > > To: Heinonen, Johanna (NSN - FI/Espoo)
>> > > Cc: openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > > Subject: Re: [Openstack] DHCP problem in grizzly
>> > >
>> > > Hi,
>> > >
>> > > thx for the interesting info, Johanna: dnsmasq version is
>> also 2.59 on
>> > > my environment. Can you can confirm that manually sigHUPing all
>> > running
>> > > dnsmasq processes has no effect?
>> > > (dnsmasq claims to have reread the configs in syslog, but still
>> > refuses
>> > > to deliver the new addresses.)
>> > > Maybe updating dnsmasq to a more recent release would solve our
>> > problem.
>> > > Current stable is 2.66. I'll give it a try when i get back
>> to office
>> > > tomorrow.
>> > >
>> > > Best regards
>> > >
>> > > Thomas
>> > >
>> > >
>> > > Am 20.05.2013 10:21, schrieb Heinonen, Johanna (NSN - FI/Espoo):
>> > >> Hi Thomas,
>> > >>
>> > >> I am using Ubuntu12.04 and the dnsmasq version is 2.59
>> > >>
>> > >> BR
>> > >> Johanna
>> > >>
>> > >>
>> > >> -----Original Message-----
>> > >> From: Openstack [mailto:openstack-bounces+johanna.heinonen
>> <mailto:openstack-bounces%2Bjohanna.heinonen>
>> > <mailto:openstack-bounces%2Bjohanna.heinonen
>> <mailto:openstack-bounces%252Bjohanna.heinonen>>=nsn.com at lists.launchpad.net
>> <mailto:nsn.com at lists.launchpad.net>
>> > <mailto:nsn.com at lists.launchpad.net
>> <mailto:nsn.com at lists.launchpad.net>>] On Behalf Of ext Thomas Kärgel
>> > >> Sent: Monday, May 20, 2013 10:09 AM
>> > >> To: openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > >> Subject: Re: [Openstack] DHCP problem in grizzly
>> > >>
>> > >> Hi Johanna,
>> > >>
>> > >> I'm facing the same behavior on a Folsom-installation on
>> SLES11SP2. I
>> > >> noticed that new hosts have correct entries in the dnsmasq
>> > config-files.
>> > >> The dnsmasq processes get a HUP signal by DHCP-Agent, but
>> simply
>> > refuse
>> > >> to deliver the new address. Instead dnsmasq logs claim "no
>> address
>> > >> available". Exactly like in your description.
>> > >> What distrubtion and exact dnsmasq-versions are in use on your
>> > environment?
>> > >> I assume dnsmasq is not rereading its configs correctly on
>> signal
>> > HUP.
>> > >> dnsmasq logs claim it has reread configs, but it still does not
>> > deliver
>> > >> the new adresses in host-file.
>> > >> Manually trying to sigHUP dnsmasq had no effect. The only
>> way to
>> > get out
>> > >> of this state seems to be restarting DHCP-agent.
>> > >>
>> > >> Best regards
>> > >> Thomas
>> > >>
>> > >> Am 20.05.2013 08:51, schrieb Heinonen, Johanna (NSN -
>> FI/Espoo):
>> > >>> Hi,
>> > >>>
>> > >>> I have installed grizzly with quantum and ovs-plugin. It
>> seems that
>> > >>> grizzly allocates the third address of each subnet for
>> dhcp. (In
>> > folsom
>> > >>> it was the second address). This means that the VMs will get
>> > addresses
>> > >>> .2, .4, .5, .
>> > >>>
>> > >>> In my setup the first VM always boots fine and gets the
>> address
>> > x.x.x.2.
>> > >>> This can be seen in the syslog:
>> > >>>
>> > >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>> > >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:2d:0d:e0
>> > >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>> > >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>> > >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>> > >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>> > >>> May 20 08:29:23 grizzly-236 dnsmasq-dhcp[2190]:
>> > DHCPACK(tapdbcef145-f5)
>> > >>> 10.20.30.2 fa:16:3e:2d:0d:e0 10-20-30-2
>> > >>>
>> > >>> The problem comes when I start the second VM. Nova shows
>> that the
>> > >>> x.x.x.4 is allocated
>> > >>>
>> > >>> root at grizzly-236:~# nova list
>> > >>>
>> >
>> +--------------------------------------+----------+--------+------------------------+
>> > >>> | ID | Name | Status |
>> > >>> Networks |
>> > >>>
>> >
>> +--------------------------------------+----------+--------+------------------------+
>> > >>> | c112ccbb-5039-4d05-b414-b53a1eafa2d8 | q-test | ACTIVE |
>> > >>> tenant1-net=10.20.30.2 |
>> > >>> | 4f26975c-995d-403d-88b0-e7bbf189baad | q-test-2 | ACTIVE |
>> > >>> tenant1-net=10.20.30.4 |
>> > >>>
>> >
>> +--------------------------------------+----------+--------+------------------------+
>> > >>>
>> > >>> But from syslog I see that the answer to the DHCPDISCOVER
>> is "no
>> > address
>> > >>> available"
>> > >>>
>> > >>> May 20 08:33:34 grizzly-236 dnsmasq-dhcp[2190]:
>> > >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:fc:1f:9a
>> no address
>> > >>> available
>> > >>> May 20 08:33:52 grizzly-236 dnsmasq-dhcp[2190]:
>> > >>> DHCPDISCOVER(tapdbcef145-f5) 10.0.2.15 fa:16:3e:fc:1f:9a
>> no address
>> > >>> available
>> > >>>
>> > >>> When I restart the quantum-dhcp-server the problem disappears.
>> > This can
>> > >>> be seen from the syslog:
>> > >>>
>> > >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: started,
>> version 2.59
>> > >>> cachesize 150
>> > >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: compile time
>> options:
>> > IPv6
>> > >>> GNU-getopt DBus i18n DHCP TFTP conntrack IDN
>> > >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: warning: no
>> upstream
>> > servers
>> > >>> configured
>> > >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: DHCP, static
>> > leases only
>> > >>> on 10.20.30.0, lease time 2m
>> > >>> May 20 09:01:40 grizzly-236 dnsmasq[7235]: cleared cache
>> > >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: read
>> > >>>
>> /var/lib/quantum/dhcp/e6f27330-be41-478c-b4d2-49ed4ce0af00/host
>> > >>> May 20 09:01:40 grizzly-236 dnsmasq-dhcp[7235]: read
>> > >>>
>> /var/lib/quantum/dhcp/e6f27330-be41-478c-b4d2-49ed4ce0af00/opts
>> > >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>> > >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>> > >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>> > DHCPNAK(tapdbcef145-f5)
>> > >>> 10.20.30.2 fa:16:3e:2d:0d:e0 lease not found
>> > >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>> > >>> DHCPDISCOVER(tapdbcef145-f5) fa:16:3e:2d:0d:e0
>> > >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>> > >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>> > >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>> > >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.2 fa:16:3e:2d:0d:e0
>> > >>> May 20 09:02:19 grizzly-236 dnsmasq-dhcp[7235]:
>> > DHCPACK(tapdbcef145-f5)
>> > >>> 10.20.30.2 fa:16:3e:2d:0d:e0 10-20-30-2
>> > >>>
>> > >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>> > >>> DHCPDISCOVER(tapdbcef145-f5) fa:16:3e:fc:1f:9a*
>> > >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>> > >>> DHCPOFFER(tapdbcef145-f5) 10.20.30.4 fa:16:3e:fc:1f:9a*
>> > >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>> > >>> DHCPREQUEST(tapdbcef145-f5) 10.20.30.4 fa:16:3e:fc:1f:9a*
>> > >>> *May 20 09:03:39 grizzly-236 dnsmasq-dhcp[7235]:
>> > DHCPACK(tapdbcef145-f5)
>> > >>> 10.20.30.4 fa:16:3e:fc:1f:9a 10-20-30-4*
>> > >>>
>> > >>>
>> > >>> What could be the problem? Have you seen similar behavior? If
>> > yes, how
>> > >>> did you fix this?
>> > >>>
>> > >>> Best regards,
>> > >>> Johanna
>> > >>>
>> > >>>
>> > >>>
>> > >>> _______________________________________________
>> > >>> Mailing list: https://launchpad.net/~openstack
>> > >>> Post to : openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > >>> Unsubscribe : https://launchpad.net/~openstack
>> > >>> More help : https://help.launchpad.net/ListHelp
>> > >>>
>> > >>
>> > >>
>> > >
>> > >
>> >
>> >
>> > --
>> > Thomas Kärgel
>> > Linux Consultant
>> > Mail: kaergel at b1-systems.de <mailto:kaergel at b1-systems.de>
>> <mailto:kaergel at b1-systems.de <mailto:kaergel at b1-systems.de>>
>> > B1 Systems GmbH
>> > Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
>> > GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG:
>> Ingolstadt,HRB 3537
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>
>> > <mailto:openstack at lists.launchpad.net
>> <mailto:openstack at lists.launchpad.net>>
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>> >
>> >
>> >
>> >
>> > --
>> > Marco Colombo
>>
>>
>> --
>> Thomas Kärgel
>> Linux Consultant
>> Tel.: +49 172 2037945
>> Mail: kaergel at b1-systems.de <mailto:kaergel at b1-systems.de>
>> B1 Systems GmbH
>> Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
>> GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
>>
>>
>>
>>
>> --
>> Marco Colombo
>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
--
Thomas Kärgel
Linux Consultant
Mail: kaergel at b1-systems.de
B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130523/3d2c92f0/attachment.sig>
More information about the Openstack
mailing list