Hello Brian, Thanks for the info. I looked into the code for the metering agent and discovered the following: 1. The metering agent on compute nodes doesn't get notified about DVR routers running on the node. 2. For DVR routers it tries to meter the rfp- interface(for floating ips) on the snat- namespace. This is wrong. The rfp- interface lies on the qrouter- namespace on compute nodes. Also, there is a qg- interface on the snat- namespace on network nodes (for performing NAT) which should be metered too. 3. There is a race condition whereby the metering agent is notified about a router before its namespaces are created. The agent ends up not adding the metering rules for those namespaces and this leads to unrecorded traffic. I addressed those issues in a change: https://review.openstack.org/#/c/621165/. Any feedback is appreciated. Best regards, Alex On 26/10/2018 21:49, Brian Haley wrote:
On 10/25/2018 08:06 AM, Alexandru Sorodoc wrote:
Hello,
I'm trying to set up metering for neutron in Pike. I tested it with a centralized router and it works, but when I try with a distributed router it doesn't record any usage samples. I have one compute node and one network node and I've created an instance with a floating ip.
The metering agent isn't very well maintained, and I don't see any open bugs similar to this issue. The only thing I can remember is this abandoned change regarding traffic counters for DVR routers - https://review.openstack.org/#/c/486493/ but there was no follow-on from the author.
The best thing to do would be to try and reproduce it on the master branch (or Rocky) and file a bug.
I think this is because the router is running on network1. Why is it running on network1 and why does it seem that the l3 agent on compute1 does the actual routing?
The compute node will do all the routing when a floating IP is associated, the router on network1 is for default snat traffic when there is no floating IP and the instance tries to communicate out the external network.
-Brian
openstack router show public-router2 +-------------------------+----------------------------------------------------+
| Field | Value | +-------------------------+----------------------------------------------------+
| admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2018-10-05T12:07:32Z | | description | | | distributed | True | | external_gateway_info | {"network_id": "b96473ce- | | | 94f6-464f-a703-5285fb8ff3d3", "enable_snat": true, | | | "external_fixed_ips": [{"subnet_id": | | | "6c08c3d9-7df1-4bec-b847-19f80b9d1764", | | | "ip_address": "192.168.252.102"}]} | | flavor_id | None | | ha | False | | id | 37c1794b-58d1-4d0d-b34b-944ca411b86b | | name | public-router2 | | project_id | fe203109e67f4e39b066c9529f9fc35d | | revision_number | 5 | | routes | | | status | ACTIVE | | tags | | | updated_at | 2018-10-05T12:09:36Z | +-------------------------+----------------------------------------------------+
openstack network agent list +-----------+------------+-----------+-------------------+-------+-------+--------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +-----------+------------+-----------+-------------------+-------+-------+--------------+
| 14b9ea75- | L3 agent | compute1. | nova | :-) | UP | neutron-l3-a | | 1dc1-4e37 | | localdoma | | | | gent | | -a2b0-508 | | in | | | | | | 3d336916d | | | | | | | | 26139ec1- | Metering | compute1. | None | :-) | UP | neutron- | | f4f9-4bb3 | agent | localdoma | | | | metering- | | -aebb-c35 | | in | | | | agent | | 3a36ed79c | | | | | | | | 2a54971f- | DHCP agent | network1. | nova | :-) | UP | neutron- | | 9849-4ed2 | | localdoma | | | | dhcp-agent | | -b009-00e | | in | | | | | | 45eb4d255 | | | | | | | | 443c0b49- | Open | compute1. | None | :-) | UP | neutron- | | 4484-44d2 | vSwitch | localdoma | | | | openvswitch- | | -a704-32a | agent | in | | | | agent | | 92ffe6982 | | | | | | | | 5d00a219 | L3 agent | network1. | nova | :-) | UP | neutron-vpn- | | -abce- | | localdoma | | | | agent | | 48ca- | | in | | | | | | ba1e-d962 | | | | | | | | 01bd7de3 | | | | | | | | bc3458b4 | Open | network1. | None | :-) | UP | neutron- | | -250e- | vSwitch | localdoma | | | | openvswitch- | | 4adf-90e0 | agent | in | | | | agent | | -110a1a7f | | | | | | | | 6ccb | | | | | | | | c29f9da8- | Metering | network1. | None | :-) | UP | neutron- | | ca58-4a11 | agent | localdoma | | | | metering- | | -b500-a25 | | in | | | | agent | | 3f820808e | | | | | | | | cdce667d- | Metadata | network1. | None | :-) | UP | neutron- | | faa4 | agent | localdoma | | | | metadata- | | -49ed- | | in | | | | agent | | 83ee-e0e5 | | | | | | | | a352d482 | | | | | | | | cf5ae104- | Metadata | compute1. | None | :-) | UP | neutron- | | 49d7-4c85 | agent | localdoma | | | | metadata- | | -a252-cc5 | | in | | | | agent | | 9a9a12789 | | | | | | | +-----------+------------+-----------+-------------------+-------+-------+--------------+
If I check the node on which my distributed router is running it tells me that it's running on the network node:
neutron l3-agent-list-hosting-router 37c1794b-58d1-4d0d-b34b-944ca411b86b +--------------------------------------+----------------------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state | +--------------------------------------+----------------------+----------------+-------+----------+
| 5d00a219-abce-48ca-ba1e-d96201bd7de3 | network1.localdomain | True | :-) | | +--------------------------------------+----------------------+----------------+-------+----------+
If I check the iptable rules for the router on the compute and network nodes by running:
ip netns exec qrouter-37c1794b-58d1-4d0d-b34b-944ca411b86b iptables -nv -L
I see that compute1 records the traffic while network1 doesn't. Also, I did some debugging and found out that the metering agent on compute1 receives an empty list of routers when querying the routers that it should monitor.
Source:
https://github.com/openstack/neutron/blob/stable/pike/neutron/services/meter...
https://github.com/openstack/neutron/blob/stable/pike/neutron/db/metering/me...
I think this is because the router is running on network1. Why is it running on network1 and why does it seem that the l3 agent on compute1 does the actual routing?
Thanks, Alex
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack