Open Stack

On 08/09/14 00:06, George Shuklin wrote:
> Yep, it is bug.
>
> I report it, and one guy close in (hate you, devstack), but now it
> seems be in the process of fixing. But there is an 25% chance it will
> be ported to icehouse and 5% change for havana.
>
> This bug happens if you got more than one network node. If routers are
> in database, but not present on network node running metering agent,
> it will fail to do anything.
>
> We manage to scramble patch to work with problem, but it is rather
> drastic - we skip all errors in that place of code and continue.
>
> If you want I cant post patch, but you gonna need to rebuild neutron
> package (rather annoying process).
>
> Here my bugreport:
> https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1286209


Thanks George, although that's not quite the same behaviour we're seeing
it's one that is going to hit us when we switch on the second L3 node
(currently we're using Pacemaker but it's too slow to re-schedule the
routers during failover).  Having seen your bug report I don't think
we'll do any changes in that respect till after our upgrade to Icehouse
at least.

In our particular case, we have just one (running) L3 agent, and it
appears fine when we add metering labels right up to one of our special
tenants/routers, and somehow the iptables rules are not being applied
correctly.

My biggest trouble with this is reproducing it in our test environment -
so far:
- create a bunch of tenants, routers and networks, and attach instances
to the networks
- start neutron-metering-agent
- add metering label for each tenant (one per tenant)
- add 2 x metering rules for each metering label, one for ingress and
one for egress, both to/from 0.0.0.0/0

In most cases this works fine, and I can see samples in ceilometer plus
traffic is forwarded correctly.
In some cases, I can get ICMP to the instances but no other traffic
(e.g. http or ssh).

When there is a problem, the metering agent log gets the error listed below.

I guess I'd better open a bug report at least, if noone else is seeing
this.  I was kind of hoping someone might tell me I'm an idiot and doing
it wrong :)

>
> On 09/07/2014 10:06 AM, Xav Paice wrote:
>> Hi,
>>
>> A quick query if anyone is using the neutron metering agent to measure
>> traffic in/out of routers:
>>
>> When adding the metering label and rules, on some, but not all, of our
>> routers, I get a traceback in the metering_agent.log pointing at
>> "TRACE neutron.openstack.common.rpc.amqp TypeError: cannot concatenate
>> 'str' and 'NoneType' objects".
>>
>> After that error, the router no longer passes traffic other than ping
>> until we stop the metering agent, and restart the L3 agent.
>>
>> This doesn't seem to affect routers with only one subnet, and only one
>> router in the tenant - but I may be completely misunderstanding the
>> whole thing here.  It appears as soon as we get a router with more than
>> one network (plus gateway) we get trouble.
>>
>> We're using Havana on Trusty, and the current Cloud Archive packages.
>>
>> The rules are a simple egress and ingress, 0.0.0.0/0, both on the same
>> meter-label.
>>
>> Anyone had similar experiences, or got tips for diagnosing this?
>>
>> I will be reading the code over the next few days :)
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators