Open Stack

Sylvain,

Something like this would require no marking:

# iptables -N test2
# iptables -N test3
# iptables -A test3
# iptables -A test2 -d 9.9.9.9/32 -j RETURN
# iptables -A test2 -d 10.10.10.10/32 -j RETURN
# iptables -A test2 -j test3
# iptables -A OUTPUT -j test2

# ping -I eth0 -r 9.9.9.9
PING 9.9.9.9 (9.9.9.9) from 16.1.1.40 eth0: 56(84) bytes of data.
^C
--- 9.9.9.9 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1007ms

# iptables-save -c | grep test
:test2 - [0:0]
:test3 - [0:0]
[3198:403274] -A OUTPUT -j test2
[2:168] -A test2 -d 9.9.9.9/32 -j RETURN
[0:0] -A test2 -d 10.10.10.10/32 -j RETURN
[3196:403106] -A test2 -j test3
[3196:403106] -A test3

# iptables -L test2 -v -x -n
Chain test2 (1 references)
    pkts      bytes target     prot opt in     out     source
destination
       2      168 RETURN     all  --  *      *       0.0.0.0/0
9.9.9.9
       0        0 RETURN     all  --  *      *       0.0.0.0/0
10.10.10.10
    3182   401554 test3      all  --  *      *       0.0.0.0/0            0.0.0.0/0

> iptables -L test3 -v -x -n
Chain test3 (1 references)
    pkts      bytes target     prot opt in     out     source
destination
    3182   401554            all  --  *      *       0.0.0.0/0            0.0.0.0/0

And I seems similar to your cut/paste from below.

Thoughts?

-Brian

On 07/22/2013 03:55 AM, Sylvain Afchain wrote:
> Hi Brian,
> 
> Thanks for your reply.
> 
>> 1. This isn't something a tenant should be able to do, so should be admin-only,
>> correct?
> 
> Correct.
> 
>> 2. I think it would be useful for an admin to be able to add metering rules for
>> all tenants with a single command.  This gets back to wanting to pre-seed an ini
>> file with a set of subnets, then add/subtract from it later without restarting
>> the daemon.
> 
> I agree with you, could be a future enhancement.
> 
>> 3. I think it would be better if you didn't mark the packets, for performance
>> reasons.  If you were marking them on input to be matched by something on output
>> I'd feel different, but for just counting bytes we should be able to do it
>> another way.  I can get back to you next week on figuring this out.
> 
> Ok, I'll take a look too.
> 
> Thanks.
> 
> Sylvain.
> 
> ----- Original Message -----
> From: "Brian Haley" <brian.haley at hp.com>
> To: "Sylvain Afchain" <sylvain.afchain at enovance.com>
> Cc: openstack-dev at lists.openstack.org
> Sent: Friday, July 19, 2013 11:47:41 PM
> Subject: Re: Change in openstack/neutron[master]: Add method to get iptables traffic counters
> 
> Hi Sylvain,
> 
> Sorry for the slow reply, I'll have to look closer next week, but I did have
> some comments.
> 
> 1. This isn't something a tenant should be able to do, so should be admin-only,
> correct?
> 
> 2. I think it would be useful for an admin to be able to add metering rules for
> all tenants with a single command.  This gets back to wanting to pre-seed an ini
> file with a set of subnets, then add/subtract from it later without restarting
> the daemon.
> 
> 3. I think it would be better if you didn't mark the packets, for performance
> reasons.  If you were marking them on input to be matched by something on output
> I'd feel different, but for just counting bytes we should be able to do it
> another way.  I can get back to you next week on figuring this out.
> 
> Thanks,
> 
> -Brian
> 
> On 07/18/2013 04:29 AM, Sylvain Afchain wrote:
>> Hi Brian,
>>
>> For iptables rules, see below
>>
>> Yes the only way to setup metering labels/rules is the neutronclient. I agree with you about the future
>> enhancement.
>>
>> Regards,
>>
>> Sylvain
>>
>> ----- Original Message -----
>> From: "Brian Haley" <brian.haley at hp.com>
>> To: "Sylvain Afchain" <sylvain.afchain at enovance.com>
>> Cc: openstack-dev at lists.openstack.org
>> Sent: Thursday, July 18, 2013 4:58:26 AM
>> Subject: Re: Change in openstack/neutron[master]: Add method to get iptables traffic counters
>>
>>> Hi Sylvain,
>>>
>>> I think I've caught-up with all your reviews, but I still did have some
>>> questions on the iptables rules, below.
>>>
>>> One other question, and maybe it's simply a future enhancement, but is the only
>>> way to setup these meters using neutronclient?  I think being able to specify
>>> these in an .ini file would be good as well, which is something I'd want to do
>>> as a provider, such that they're always there, and actually not visible to the
>>> tenant.
>>>
>>> On 07/11/2013 10:04 AM, Sylvain Afchain wrote:
>>>> Hi Brian,
>>>>
>>>> You're right It could be easier with your approach to get and keep the traffic counters.
>>>>
>>>> I will add a new method to get the details of traffic counters of a chain.
>>>> https://review.openstack.org/35624
>>>>
>>>> Thoughts?
>>>>
>>>> Regards,
>>>>
>>>> Sylvain.
>>>>
>>>> ----- Original Message -----
>>>> From: "Sylvain Afchain" <sylvain.afchain at enovance.com>
>>>> To: "Brian Haley" <brian.haley at hp.com>
>>>> Cc: openstack-dev at lists.openstack.org
>>>> Sent: Thursday, July 11, 2013 11:19:39 AM
>>>> Subject: Re: Change in openstack/neutron[master]: Add method to get iptables traffic counters
>>>>
>>>> Hi Brian,
>>>>
>>>> First thanks for the reviews and your detailed email.
>>>>
>>>> Second I will update the blueprint specs. as soon as possible, but for example it will look like that:
>>>>
>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>     pkts      bytes target     prot opt in     out     source               destination         
>>>>        55       245 metering-r-aef1456343  all  --  *      *       0.0.0.0/0            0.0.0.0/0   /* jump to rules the label aef1456343 */        
>>>>        55       245 metering-r-badf566782  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
>>
>>> So are these two used to separate out what you don't want to count from what you
>>> want to count?  Seems the jump to the r-aef1456343 will filter, then the
>>> r-badf566782 will count per-subnet?  I'm just trying to understand why you're
>>> splitting the two up.
>>
>> No here, there are two rules only because there are two labels. In the chain of each 
>> label you will find the label's rules. 
>>
>>>> Chain metering-l-aef1456343 (1 references)	/* the chain for the label aef1456343 */
>>>>     pkts      bytes target     prot opt in     out     source               destination         
>>>>
>>>> Chain metering-l-badf566782 (1 references)	/* the chain for the label badf566782 */
>>>>     pkts      bytes target     prot opt in     out     source               destination      
>>
>>> These two chains aren't really doing anything, and I believe their packet/byte
>>> counts would be duplicated in the calling rules, correct?  If that's the case I
>>> don't see the reason to jump to them.  Our performance person always reminds me
>>> when I increase path length by doing things like this, so removing unnecessary
>>> things is one of my goals.  Of course we're doing more work here to count
>>> things, but that needs to be done.
>>
>> I recently change this(according to your remarks on iptables accounting), so now there is a 
>> rule which is used to count the traffic for a label. A mark is added one this rule to be 
>> sure to not count it twice. You can check the metering iptables drivers.
>> https://review.openstack.org/#/c/36813/
>>
>>>> Chain metering-r-aef1456343 (1 references)
>>>>     pkts      bytes target     prot opt in     out     source               destination         
>>>>        20     100 RETURN     all  --  *      *       0.0.0.0/0           !10.0.0.0/24      /* don't want to count this traffic */   
>>>>        0        0 RETURN     all  --  *      *       0.0.0.0/0           !20.0.0.0/24      /* don't want to count this traffic */   
>>>>        25      145 metering-l-aef1456343  all  --  *      *       0.0.0.0/0            0.0.0.0/0	/* count the remaining traffic */     
>>
>>> This has excluded (!) certain subnets, then allowed anything else to pass through.
>>
>> Correct, I don't want to count the traffic in/out for these network and I want to count anything else.
>>
>>>> Chain metering-r-badf566782 (1 references)
>>>>     pkts      bytes target     prot opt in     out     source               destination         
>>>>        0        0 metering-l-badf56678  all  --  *      *       0.0.0.0/0            30.0.0.0/24 /* want to count only this */
>>
>>> This is only counting packets going to a certain subnet.
>>
>> Correct.
>>
>>> I'm still trying to get all these changes running in devstack, so sorry for all
>>> the questions.
>>
>> No problem, and thanks for your reviews.
>>
>> Sylvain.
>>
>>> Thanks,
>>
>>> -Brian
>>
>>>
>>> Of course the in/out interfaces will be set in order to get the ingress or the egress traffic.
>>>
>>> I agree with you I could add a single rule to the chain of the label and get the traffic of the first entry, though I found this approach less generic. 
>>> I mean, to be forced to add a rule at the top of a chain to get its traffic. My approach is I don't want the counters of a specific rule but I want to count
>>> the traffic going through the chain.
>>>
>>> Thoughts?
>>>
>>> Regards,
>>>
>>> Sylvain.
>>>
>>> ----- Original Message -----
>>> From: "Brian Haley" <brian.haley at hp.com>
>>> To: "sylvain afchain" <sylvain.afchain at enovance.com>
>>> Cc: openstack-dev at lists.openstack.org
>>> Sent: Thursday, July 11, 2013 2:30:24 AM
>>> Subject: Re: Change in openstack/neutron[master]: Add method to get iptables traffic counters
>>>
>>> On 07/08/2013 01:10 PM, Sylvain Afchain (Code Review) wrote:
>>>> Sylvain Afchain has posted comments on this change.
>>>>
>>>> Change subject: Add method to get iptables traffic counters
>>> <snip>
>>>> --
>>>> To view, visit https://review.openstack.org/35624
>>>
>>> Hi Sylvain,
>>>
>>> Instead of trying to ask questions directly in the review itself (since it will mess-up formatting) I'll just send this to you and the list since I had some questions on the traffic counter changes you've been doing.
>>>
>>> First, thanks for working on this, it's definitely something I'm interested in, and I'm trying to review all your changes.
>>>
>>> Second, do you have more than just the short description from the blueprint for how the iptables chains/rules will look like when created?  I'm still a little confused with this change (above) and how it's matching chains to get packet/byte statistics.  I'm thinking it can be done within a single chain so that you can do an 'iptables -L $chain' call to get just what you need, instead of parsing the entire table.
>>>
>>> For example, I did something similar in Nova (out of tree), and it used a single chain per-VM, such that you could get it's statistics with a single iptables call like:
>>>
>>> (sorry if this wraps)
>>> $ sudo iptables -t mangle -L nova-meter-output-91 -n -v -x [-Z]
>>> Chain nova-meter-output-91 (1 references)
>>>     pkts      bytes target     prot opt in     out     source               destination         
>>>   805210 247931149            all  --  *      *       0.0.0.0/0            0.0.0.0/0        /* inst-91 packets transmitted total */ 
>>>    15510   964648             all  --  *      *       0.0.0.0/0            x.y.0.0/16
>>>    21282  3075403             all  --  *      *       0.0.0.0/0            x.z.0.0/16
>>>    [...]
>>>
>>> None of the rules in the chain has a jump target, so they simply count packets/bytes as they pass through, and the chain is called from a single location based on IP address, so in iptables-save format it looks like this:
>>>
>>> -A nova-meter-output -s $my_ip/32 -i bridge1 -j nova-meter-output-91 
>>> -A nova-meter-output-91 -m comment --comment "inst-91 packets transmitted total"
>>> -A nova-meter-output-91 -d x.y.0.0/16
>>> -A nova-meter-output-91 -d x.z.0.0/16
>>> [...]
>>>
>>> Obviously with Neutron, and doing this at the router egress, things change, but I think it could still be done in a single OUTPUT chain in the filter table.
>>>
>>> Thoughts?
>>>
>>> -Brian
>>>
>>
>