<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span class="x_elementToProof ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Dear folks,</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<div class="x_elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br class="ContentPasted0">
</div>
<div class="x_elementToProof x_ContentPasted0 ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Recently, we met some neutron networking problems in our envrionment, openstack version is zed, and kolla-ansible as the deployment tool.<br class="ContentPasted0">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br class="ContentPasted0">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div>
<ol data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}">
<li style="list-style-type:"1. ""><span class="x_ContentPasted1 ContentPasted0">Neutron BGP agent doesn't advertise the floating IPs to the BGP peer, in case of the floating IPs were served for port forwarding, but the floating IPs attached to VM/Container
were advertised correctly. so the question is <b class="ContentPasted0">this scenario supported by BGP agent</b>, if not when will it be supported, is it in the plan?</span></li><li style="list-style-type:"2. ""><span class="x_ContentPasted1 x_ContentPasted7 ContentPasted0">iptable rules restoring error in l3-agent and openvswitch-agent (A bug was reported in launchpad:
<a href="https://bugs.launchpad.net/neutron/+bug/2024976" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-linkindex="0" class="ContentPasted0">
https://bugs.launchpad.net/neutron/+bug/2024976</a>)<br class="ContentPasted0">
</span>
<div class="x__Entity x__EType_OWALinkPreview x__EId_OWALinkPreview x__EReadonly_1">
<div class="x_LPBorder272423" style="margin-top:16px;margin-bottom:16px;max-width:800px;min-width:424px">
<table style="padding:12px 36px 12px 12px;width:100%;border-width:1px;border-style:solid;border-color:rgb(200,200,200);border-radius:2px">
<tbody>
<tr style="border-spacing:0px" valign="top">
<td style="width:100%">
<div style="font-size:21px;font-weight:300;margin-right:8px;font-family:"wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;margin-bottom:12px">
<a href="https://bugs.launchpad.net/neutron/+bug/2024976" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable" data-loopstyle="linkonly" style="text-decoration:none" data-linkindex="1" class="ContentPasted0">Bug #2024976 “iptable rules restoring
error in l3-agent and open...” : Bugs : neutron</a></div>
<div style="font-size: 14px; max-height: 100px; font-family: "wf_segoe-ui_normal", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px; margin-right: 8px; overflow: hidden; color: rgb(102, 102, 102);" class="ContentPasted0">
Openstack version: zed/stable OS version: Ubuntu 22.04.2 LTS Kernel version: 5.15.0-75-generic #82-Ubuntu Deployment: kolla-ansible iptable rules restoring error in l3-agent and openvswitch-agent: openvswitch-agnet log: 2023-06-23 15:54:58.616 7 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: [...</div>
<div style="font-size: 14px; font-family: "wf_segoe-ui_normal", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; color: rgb(166, 166, 166);" class="ContentPasted0">
bugs.launchpad.net</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div>
<div><br class="ContentPasted0">
</div>
</div>
</li></ol>
<div><b class="ContentPasted0"><span class="ContentPasted0"> </span>openvswitch-agnet log:</b></div>
<div><br class="ContentPasted0">
</div>
<div class="x_ContentPasted2">
<blockquote style="margin-top:0;margin-bottom:0">
<blockquote style="margin-top:0;margin-bottom:0" class="ContentPasted0">2023-06-23 15:54:58.616 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-4440bce1-8c07-4243-ac1b-2566b406a30a - - - - - -] Error while processing VIF ports:
neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager
<div class="x_ContentPasted2 ContentPasted0">*filter</div>
<div class="x_ContentPasted2 ContentPasted0">:FORWARD - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:INPUT - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:OUTPUT - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-filter-top - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-FORWARD - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-INPUT - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-OUTPUT - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-local - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-sg-chain - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-sg-fallback - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">-I FORWARD 1 -j neutron-filter-top</div>
<div class="x_ContentPasted2 ContentPasted0">-I FORWARD 2 -j neutron-openvswi-FORWARD</div>
<div class="x_ContentPasted2 ContentPasted0">-I INPUT 1 -j neutron-openvswi-INPUT</div>
<div class="x_ContentPasted2 ContentPasted0">-I OUTPUT 1 -j neutron-filter-top</div>
<div class="x_ContentPasted2 ContentPasted0">-I OUTPUT 2 -j neutron-openvswi-OUTPUT</div>
<div class="x_ContentPasted2 ContentPasted0">-I neutron-filter-top 1 -j neutron-openvswi-local</div>
<div class="x_ContentPasted2 ContentPasted0">-I neutron-openvswi-FORWARD 1 -m physdev --physdev-out tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT</div>
<div class="x_ContentPasted2 ContentPasted0">-I neutron-openvswi-FORWARD 2 -m physdev --physdev-in tap2fcacaf9-9d --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT</div>
<div class="x_ContentPasted2 ContentPasted0">-I neutron-openvswi-FORWARD 3 -m physdev --physdev-out tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT</div>
<div class="x_ContentPasted2 ContentPasted0">-I neutron-openvswi-FORWARD 4 -m physdev --physdev-in tap8c64cce3-ea --physdev-is-bridged -m comment --comment "Accept all packets when port is trusted." -j ACCEPT</div>
<div class="x_ContentPasted2 ContentPasted0">-I neutron-openvswi-sg-chain 1 -j ACCEPT</div>
<div class="x_ContentPasted2 ContentPasted0">-I neutron-openvswi-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP</div>
<div class="x_ContentPasted2 ContentPasted0">COMMIT</div>
<div class="x_ContentPasted2 ContentPasted0"># Completed by iptables_manager</div>
<div class="x_ContentPasted2 ContentPasted0"># Generated by iptables_manager</div>
<div class="x_ContentPasted2 ContentPasted0">*raw</div>
<div class="x_ContentPasted2 ContentPasted0">:OUTPUT - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:PREROUTING - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-OUTPUT - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">:neutron-openvswi-PREROUTING - [0:0]</div>
<div class="x_ContentPasted2 ContentPasted0">-I OUTPUT 1 -j neutron-openvswi-OUTPUT</div>
<div class="x_ContentPasted2 ContentPasted0">-I PREROUTING 1 -j neutron-openvswi-PREROUTING</div>
<div class="x_ContentPasted2 ContentPasted0">COMMIT</div>
<div class="x_ContentPasted2 ContentPasted0"># Completed by iptables_manager</div>
<div class="x_ContentPasted2 ContentPasted0">; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables):
<b class="ContentPasted0">Couldn't load match `</b><span style="color: rgb(237, 92, 87);"><b class="ContentPasted0">physdev</b></span><b class="ContentPasted0">':No such file or directory</b></div>
<div><br class="x_ContentPasted2 ContentPasted0">
</div>
<div class="x_ContentPasted2 ContentPasted0">Error occurred at line: 19</div>
Try `iptables-restore -h' or 'iptables-restore --help' for more information.<br class="ContentPasted0">
</blockquote>
</blockquote>
</div>
</div>
<div><br class="ContentPasted0">
</div>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b class="ContentPasted0"><span class="ContentPasted0"> </span>l3-agent log:</b></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br class="ContentPasted0">
</div>
<blockquote style="margin-top:0;margin-bottom:0">
<blockquote style="margin-top:0;margin-bottom:0">
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 ContentPasted0">2023-06-23 16:15:49.545 33 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Cmd: ['ip', 'netns', 'exec', 'qrouter-0f0e60d0-bf51-4361-901b-4b998201b44b',
'iptables-restore', '-n']; Stdin: # Generated by iptables_manager</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 ContentPasted0">
<div class="x_ContentPasted3 ContentPasted0">*filter</div>
<div class="x_ContentPasted3 ContentPasted0">:FORWARD - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:INPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:OUTPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-filter-top - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-FORWARD - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-INPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-OUTPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-local - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-scope - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">-I FORWARD 1 -j neutron-filter-top</div>
<div class="x_ContentPasted3 ContentPasted0">-I FORWARD 2 -j neutron-l3-agent-FORWARD</div>
<div class="x_ContentPasted3 ContentPasted0">-I INPUT 1 -j neutron-l3-agent-INPUT</div>
<div class="x_ContentPasted3 ContentPasted0">-I OUTPUT 1 -j neutron-filter-top</div>
<div class="x_ContentPasted3 ContentPasted0">-I OUTPUT 2 -j neutron-l3-agent-OUTPUT</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-filter-top 1 -j neutron-l3-agent-local</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-FORWARD 1 -j neutron-l3-agent-scope</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-scope 1 -m mark --mark 0x1/0xffff -j DROP</div>
<div class="x_ContentPasted3 ContentPasted0">COMMIT</div>
<div class="x_ContentPasted3 ContentPasted0"># Completed by iptables_manager</div>
<div class="x_ContentPasted3 ContentPasted0"># Generated by iptables_manager</div>
<div class="x_ContentPasted3 ContentPasted0">*mangle</div>
<div class="x_ContentPasted3 ContentPasted0">:FORWARD - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:INPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:OUTPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:POSTROUTING - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:PREROUTING - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-FORWARD - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-INPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-OUTPUT - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-POSTROUTING - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-PREROUTING - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-float-snat - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-floatingip - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-mark - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">:neutron-l3-agent-scope - [0:0]</div>
<div class="x_ContentPasted3 ContentPasted0">-I FORWARD 1 -j neutron-l3-agent-FORWARD</div>
<div class="x_ContentPasted3 ContentPasted0">-I INPUT 1 -j neutron-l3-agent-INPUT</div>
<div class="x_ContentPasted3 ContentPasted0">-I OUTPUT 1 -j neutron-l3-agent-OUTPUT</div>
<div class="x_ContentPasted3 ContentPasted0">-I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING</div>
<div class="x_ContentPasted3 ContentPasted0">-I PREROUTING 1 -j neutron-l3-agent-PREROUTING</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-mark</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-PREROUTING 2 -j neutron-l3-agent-scope</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-PREROUTING 3 -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-PREROUTING 4 -j neutron-l3-agent-floatingip</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-PREROUTING 5 -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff</div>
<div class="x_ContentPasted3 ContentPasted0">-I neutron-l3-agent-float-snat 1 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000</div>
COMMIT
<div class="x_ContentPasted4 ContentPasted0"># Completed by iptables_manager</div>
<div class="x_ContentPasted4 ContentPasted0"># Generated by iptables_manager</div>
<div class="x_ContentPasted4 ContentPasted0">*nat</div>
<div class="x_ContentPasted4 ContentPasted0">:OUTPUT - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:POSTROUTING - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:PREROUTING - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-l3-agent-OUTPUT - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-l3-agent-POSTROUTING - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-l3-agent-PREROUTING - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-l3-agent-float-snat - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-l3-agent-snat - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-postrouting-bottom - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">-I OUTPUT 1 -j neutron-l3-agent-OUTPUT</div>
<div class="x_ContentPasted4 ContentPasted0">-I POSTROUTING 1 -j neutron-l3-agent-POSTROUTING</div>
<div class="x_ContentPasted4 ContentPasted0">-I POSTROUTING 2 -j neutron-postrouting-bottom</div>
<div class="x_ContentPasted4 ContentPasted0">-I PREROUTING 1 -j neutron-l3-agent-PREROUTING</div>
<div class="x_ContentPasted4 ContentPasted0">-I neutron-l3-agent-POSTROUTING 1 ! -o rfp-0f0e60d0-b -m conntrack ! --ctstate DNAT -j ACCEPT</div>
<div class="x_ContentPasted4 ContentPasted0">-I neutron-l3-agent-PREROUTING 1 -d 137.175.31.207/32 -i rfp-0f0e60d0-b -j DNAT --to-destination 10.10.0.246</div>
<div class="x_ContentPasted4 ContentPasted0">-I neutron-l3-agent-float-snat 1 -s 10.10.0.246/32 -j SNAT --to-source 137.175.31.207 --random-fully</div>
<div class="x_ContentPasted4 ContentPasted0">-I neutron-l3-agent-snat 1 -j neutron-l3-agent-float-snat</div>
<div class="x_ContentPasted4 ContentPasted0">-I neutron-postrouting-bottom 1 -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat</div>
<div class="x_ContentPasted4 ContentPasted0">COMMIT</div>
<div class="x_ContentPasted4 ContentPasted0"># Completed by iptables_manager</div>
<div class="x_ContentPasted4 ContentPasted0"># Generated by iptables_manager</div>
<div class="x_ContentPasted4 ContentPasted0">*raw</div>
<div class="x_ContentPasted4 ContentPasted0">:OUTPUT - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:PREROUTING - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-l3-agent-OUTPUT - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">:neutron-l3-agent-PREROUTING - [0:0]</div>
<div class="x_ContentPasted4 ContentPasted0">-I OUTPUT 1 -j neutron-l3-agent-OUTPUT</div>
<div class="x_ContentPasted4 ContentPasted0">-I PREROUTING 1 -j neutron-l3-agent-PREROUTING</div>
<div class="x_ContentPasted4 ContentPasted0">COMMIT</div>
<div class="x_ContentPasted4 ContentPasted0"># Completed by iptables_manager</div>
<div class="x_ContentPasted4 ContentPasted0">; Stdout: ; Stderr: iptables-restore v1.8.7 (nf_tables):
<b class="ContentPasted0">Couldn't load match `</b><span style="color: rgb(237, 92, 87);"><b class="ContentPasted0">mark</b></span><b class="ContentPasted0">':No such file or directory</b></div>
<div><br class="x_ContentPasted4 ContentPasted0">
</div>
</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 ContentPasted0">Error occurred at line: 19</span></div>
</blockquote>
</blockquote>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br class="ContentPasted0">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 ContentPasted0"><span class="ContentPasted0"> </span>And we check the system the x_tables kernel module were loaded:</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4"><br class="ContentPasted0">
</span></div>
<blockquote style="margin-top:0;margin-bottom:0">
<blockquote style="margin-top:0;margin-bottom:0">
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6 ContentPasted0"># lsmod | grep x_tables<br class="x_ContentPasted6 ContentPasted0">
</span><span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6" style="color: rgb(237, 92, 87);"><b class="ContentPasted0">x_tables</b></span><span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6 ContentPasted0">
53248 12 xt_conntrack,nft_compat,xt_tcpudp,</span><span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6" style="color: rgb(237, 92, 87);"><b class="ContentPasted0">xt_physdev</b></span><span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6 ContentPasted0">,xt_nat,xt_comment,ip6_tables,xt_connmark,xt_CT,ip_tables,xt_REDIRECT,</span><span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6" style="color: rgb(237, 92, 87);"><b class="ContentPasted0">xt_mark</b></span><span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6"><b><br class="ContentPasted0">
</b></span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted6"><br class="ContentPasted0">
</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted5 ContentPasted0">(neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*mark.so"
<div class="x_ContentPasted5 ContentPasted0">./lib/x86_64-linux-gnu/xtables/libxt_connmark.so</div>
<div class="x_ContentPasted5 ContentPasted0">./lib/x86_64-linux-gnu/xtables/libxt_mark.so</div>
<div class="x_ContentPasted5 ContentPasted0">./lib/x86_64-linux-gnu/xtables/libebt_mark.so</div>
<div class="x_ContentPasted5 ContentPasted0">(neutron-l3-agent)[neutron@compute06 usr]$ find . -name "*physdev.so"</div>
</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted5 ContentPasted0">./lib/x86_64-linux-gnu/xtables/libxt_physdev.so<br class="ContentPasted0">
</span></div>
</blockquote>
</blockquote>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br class="ContentPasted0">
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted5 ContentPasted0">Does someone have ever met the problems what is the solution the resovle them. Thanks in advance<br class="ContentPasted0">
</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted5"><br class="ContentPasted0">
</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted5"><br class="ContentPasted0">
</span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span class="x_ContentPasted3 x_elementToProof x_ContentPasted4 x_ContentPasted5">
<div>
<p style="background-color: white;"><span><i><span class="x_x_x_ContentPasted0 x_x_ContentPasted0 x_ContentPasted8 ContentPasted0" style="font-family: Calibri, sans-serif; color: black; background-color: white;">Sincerely</span></i></span><span><i><span class="x_x_x_ContentPasted0 x_x_ContentPasted0 x_ContentPasted8 ContentPasted0" style="color: black; background-color: white;" lang="zh-CN">，</span></i></span><span style="color: black;"></span></p>
</div>
<div>
<p style="background-color: white;"><span><span class="x_x_x_ContentPasted0 x_x_ContentPasted0 x_ContentPasted8 ContentPasted0" style="font-family: Calibri, sans-serif; color: black; background-color: white;"> </span></span><span style="color: black;"></span></p>
</div>
<p style="font-family: 宋体; margin: 0px; background-color: white;"><span><span class="x_x_x_ContentPasted0 x_x_ContentPasted0 x_ContentPasted8 ContentPasted0" style="font-family: Calibri, sans-serif; color: black; background-color: white;">Bryan</span></span></p>
</span></div>
<br>
</div>
</body>
</html>