Open Stack

Right, my bad, Nova only did this a long time ago with nova-network network
types... Surely it's handled by Neutron nowadays. In my Fuel 9.0-based lab
neutron-openvswitch-agent service is managing the rules. I can erase some
VM rules, restart the agent (while nova-compute is stopped - to ensure it
does not do anything) and they reappear. That's how normal OpenStack
installation works:

root at node-3:~# iptables -S | grep 54321
-A neutron-openvswi-i7eeecb33-1 -p tcp -m tcp --dport 54321 -j RETURN
root at node-3:~# service nova-compute stop
nova-compute stop/waiting
root at node-3:~# iptables -D neutron-openvswi-i7eeecb33-1 -p tcp -m tcp
--dport 54321 -j RETURN
root at node-3:~# restart neutron-openvswitch-agent
neutron-openvswitch-agent start/running, process 12541
root at node-3:~# iptables -S | grep 54321 # rule not recreated yet
root at node-3:~# iptables -S | grep 54321 # a few moments later it's there
-A neutron-openvswi-i7eeecb33-1 -p tcp -m tcp --dport 54321 -j RETURN

There are of course also default iptables rules which always get applied
for every VM (by the same service). My firewall driver is the same:
/etc/neutron/plugins/ml2/openvswitch_agent.ini: firewall_driver =
neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

On Wed, May 24, 2017 at 12:25 AM, duhongwei <duhongwei at qiniu.com> wrote:

>
> Dmitry Sutyagin also replied, quote:
>
> *Afaik, iptables are set by Nova, and the driver is set via
> firewall_driver option in nova.conf*
>
> After some investigation, I am thinking both Nova and Neutron provide
> security_group implementation, while Nova operates on vm granularity and
> Neuton operates on port.
>
> In my openstack environment, I see *firewall_driver =
> nova.virt.firewall.NoopFirewallDriver* in nova.conf and *firewall_driver
> = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver*
> in ml2_conf.ini. So it seems I'm using Neutron's security_group
> implementation.
>
> Back to my question, I tried to name qbr, qvo, qvb as qbr1234567890, ...
> and, container side veth as veth0, qbr side veth as tap1234567890. As a
> result, no iptables rules added, therefore packets from veth0 are dropped
> on qbr1234567890.
>
> So, how would Neutron know that it has to add iptables rules for a device,
> by device name prefix or what? What can I do to let it happen?
>
> Regards,
> Dastan
>
> ------------------ Original ------------------
> *From: * "Kevin Benton"<kevin at benton.pub>;
> *Date: * Wed, May 24, 2017 05:21 AM
> *To: * "duhongwei"<duhongwei at qiniu.com>;
> *Cc: * "openstack"<openstack at lists.openstack.org>; "Vallachorum
> Tyranorum"<ardeleandanflorin at gmail.com>;
> *Subject: * Re: [Openstack] How to utilize Neutron independently with
> veths
>
> Neutron sets up the iptables rules if you have security groups enabled and
> the agent firewall is set to iptables_hybrid or
> neutron.agent.linux.iptables_firewall:OVSHybridIptablesFirewallDriver .
>
> What are you naming your vNIC? The iptables rules setup by the agent match
> specifically on 'tap' + port UUID prefix. So if the bridge is qvb1234567890
> then the vnic you plug into it needs to be named tap1234567890 .
>
> On Tue, May 23, 2017 at 12:15 AM, duhongwei <duhongwei at qiniu.com> wrote:
>
>>
>> Thanks Kevin! I've made a big step forward!
>>
>> Till now, I've successfully connect *vNIC* directly into *br-int *without
>> *qbr*, *qvo*, and *qvb*. And, it works well.
>>
>> However, following your scripts (connect *vNIC* into *qbr*, then connect
>> *qbr* into *br-int*) exposes another problem. In this scenario, *qbr*
>> won't forward packets from *vNIC* to *br-int* (packets seem to be
>> dropped on *qbr*).
>>
>> After some troubleshooting, it turns out to be *iptables* who drops
>> packets on *qbr*. Reviewing the *FORWARD* chain in *filter* table,
>> packets come from *vNIC* won't match any rule of *neutron-filter-top*
>> and *neutron-openvswi-FORWARD* so that the default policy *DROP* applies.
>>
>> So, after setting up all these *qbr*, *qvo*, *qvb*, *vNIC*, it seems
>> there're still some *iptables rules* missed. Question is,
>>
>> Who's adding this iptables rules? (Nova or Neutron?) How can I make it
>> happen?
>>
>> Regards,
>> Dastan
>>
>>
>> ------------------ Original ------------------
>> *From: * "Kevin Benton"<kevin at benton.pub>;
>> *Date: * Mon, May 22, 2017 10:47 PM
>> *To: * "duhongwei"<duhongwei at qiniu.com>;
>> *Cc: * "openstack"<openstack at lists.openstack.org>; "Vallachorum
>> Tyranorum"<ardeleandanflorin at gmail.com>;
>> *Subject: * Re: [Openstack] How to utilize Neutron independently with
>> veths
>>
>> Yes, the only thing that needs to use the correct MAC is whatever is
>> actually sending traffic.
>>
>> On May 21, 2017 22:06, "duhongwei" <duhongwei at qiniu.com> wrote:
>>
>>>
>>> Thanks for your patient, Kevin.
>>>
>>> So *qvo *could be any veth whose mac address doesn't matter, but *veth/tap
>>> *must have exact the same mac address as *port*, otherwise it will be
>>> anti-spoofed.
>>>
>>> *qvo*'s attributes (external-ids) tell neutron which logical *port*
>>> *qvo* is connecting, so neutron knows how to add flows to ovs *br-int *and
>>> *br-tun*.
>>>
>>> Am I correct?
>>>
>>> Regards,
>>> Dastan
>>>
>>> ------------------ Original ------------------
>>> *From: * "Kevin Benton"<kevin at benton.pub>;
>>> *Date: * Sat, May 20, 2017 03:26 AM
>>> *To: * "duhongwei"<duhongwei at qiniu.com>;
>>> *Cc: * "openstack"<openstack at lists.openstack.org>; "Vallachorum
>>> Tyranorum"<ardeleandanflorin at gmail.com>;
>>> *Subject: * Re: [Openstack] How to utilize Neutron independently with
>>> veths
>>>
>>> >After all these, we create *veth/tap* (as vm/containers vNIC) and
>>> plugin it into *qbr* then we're able to talk with other vms/containers
>>> on the same network through *veth/tap*, am I understanding it right?
>>>
>>> Yes, this last step of creating a veth/tap is missing from my script
>>> because I didn't need actual dataplane communication for the tests I was
>>> doing.
>>>
>>> >1) isn't it necessary that *veth/tap*'s mac address same as neutron
>>> *port*'s mac address?
>>>
>>> Yeah, if you attach something to qbr to behave like the VM interface,
>>> you will need it to be using the mac address of the neutron port, or else
>>> the neutron anti-spoofing rules will prevent it from communicating.
>>>
>>>
>>> >2) after we plug *qvo* into ovs *br-int*, neutron just automatically
>>> add flows into ovs bridge?
>>>
>>> Yes, the agent will receive to the new port event from ovs, retrieve
>>> port details from the server and then setup the flows.
>>>
>>> On Fri, May 19, 2017 at 12:09 AM, duhongwei <duhongwei at qiniu.com> wrote:
>>>
>>>>
>>>> This script seems easy and cool!
>>>>
>>>> So first we have to create a logical neutron *port*, then create *qbr*,
>>>> *qvo* and *qvb*, and plug *qvb* into *qbr*, finally plug *qvo* into
>>>> ovs *br-int*. After all these, we create *veth/tap* (as vm/containers
>>>> vNIC) and plugin it into *qbr* then we're able to talk with other
>>>> vms/containers on the same network through *veth/tap*, am I
>>>> understanding it right?
>>>>
>>>> Questions,
>>>>
>>>> 1) isn't it necessary that *veth/tap*'s mac address same as neutron
>>>> *port*'s mac address?
>>>> 2) after we plug *qvo* into ovs *br-int*, neutron just automatically
>>>> add flows into ovs bridge?
>>>>
>>>> Regards,
>>>> Dastan
>>>>
>>>> ------------------ Original ------------------
>>>> *From: * "Kevin Benton"<kevin at benton.pub>;
>>>> *Date: * Sat, May 13, 2017 07:46 AM
>>>> *To: * "duhongwei"<duhongwei at qiniu.com>;
>>>> *Cc: * "openstack"<openstack at lists.openstack.org>; "Vallachorum
>>>> Tyranorum"<ardeleandanflorin at gmail.com>;
>>>> *Subject: * Re: [Openstack] How to utilize Neutron independently with
>>>> veths
>>>>
>>>> Nova is only responsible for creating the interface and plugging it
>>>> into the OVS bridge. It's the neutron agent (or alternative neutron backend
>>>> like OVN) responsible for setting up all of the flows.
>>>>
>>>> Here is a hacky script that I had used to create and delete a bunch of
>>>> ports like Nova would that you can probably start with:
>>>> http://paste.openstack.org/show/609478/
>>>>
>>>> On Fri, May 12, 2017 at 4:25 AM, duhongwei <duhongwei at qiniu.com> wrote:
>>>>
>>>>>
>>>>> Thanks Kevin!
>>>>>
>>>>> I'll dig into neutron.agent.linux.interface to see how it works.
>>>>> Before that, would you give me any previews about what steps should be
>>>>> taken to add a veth to a existed Neutron network?
>>>>>
>>>>> Furthermore, is it Neutron who add a veth to ovs bridge or is it the
>>>>> Neutron caller? (such as Nova)
>>>>>
>>>>> Who's adding flows to ovs bridge? Neutron or caller?
>>>>>
>>>>> Regards,
>>>>> Dastan
>>>>>
>>>>> ------------------ Original ------------------
>>>>> *From: * "Kevin Benton"<kevin at benton.pub>;
>>>>> *Date: * Fri, May 12, 2017 10:45 AM
>>>>> *To: * "duhongwei"<duhongwei at qiniu.com>;
>>>>> *Cc: * "openstack"<openstack at lists.openstack.org>; "Vallachorum
>>>>> Tyranorum"<ardeleandanflorin at gmail.com>;
>>>>> *Subject: * Re: [Openstack] How to utilize Neutron independently with
>>>>> veths
>>>>>
>>>>> You want to look in neutron.agent.linux.interface to see how things
>>>>> are plugged into OVS. That's the module used by the L3 agent to plug into
>>>>> OVS/linux bridge/etc.
>>>>>
>>>>> There is a well defined interface name format corresponding to the
>>>>> port ID and the port ID, Mac address, and a couple of other things I can't
>>>>> recall are set in ovsdb to help the agent identify the port as something it
>>>>> should care about.
>>>>>
>>>>> On May 9, 2017 04:49, "duhongwei" <duhongwei at qiniu.com> wrote:
>>>>>
>>>>>>
>>>>>> Thanks Dan!
>>>>>>
>>>>>> I have to write a customized CNI plugin for our product, so it's
>>>>>> better if I know more operation details about how to interact with Neutron
>>>>>> manually (consider myself as Nova). Therefore Kuryr is not my best option
>>>>>> right now, but it's cool!
>>>>>>
>>>>>> Regards,
>>>>>> Dastan
>>>>>>
>>>>>> ------------------ Original ------------------
>>>>>> *From: * "Vallachorum Tyranorum"<ardeleandanflorin at gmail.com>;
>>>>>> *Date: * Tue, May 9, 2017 04:08 PM
>>>>>> *To: * "duhongwei"<duhongwei at qiniu.com>; "openstack"<
>>>>>> openstack at lists.openstack.org>;
>>>>>> *Subject: * Re: [Openstack] How to utilize Neutron independently
>>>>>> with veths
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Please take a look at Kuryr <https://wiki.openstack.org/wiki/Kuryr>.
>>>>>> Maybe this is what you are looking for.
>>>>>>
>>>>>> Dan.
>>>>>>
>>>>>> On Tue, May 9, 2017 at 10:17 AM duhongwei <duhongwei at qiniu.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I'm new to OpenStack and currently interested in the Neutron part of
>>>>>>> it. What I'm seeking is some advice about how to utilize Neutron
>>>>>>> independently, to build a virtual network, for Docker containers maybe.
>>>>>>>
>>>>>>> Suppose I've already got Neutron and Keystone installed on
>>>>>>> controller node and compute nodes. I guess the following steps are required
>>>>>>> to test a virtual network.
>>>>>>>
>>>>>>> 1) create a *network*
>>>>>>> 2) create a *subnet*
>>>>>>> 3) create two pairs of veths (each pair represents a vm)
>>>>>>> *for each pair of them*:
>>>>>>> 4) create a *port *for one end of the veth pair (passing veth's mac
>>>>>>> address as a parameter)
>>>>>>> 5) attach another end of the veth pair to ovs bridge
>>>>>>> 6) ping from one veth pair to another
>>>>>>>
>>>>>>> The above is my general idea, don't know if it is correct and don't
>>>>>>> know the operation details either.
>>>>>>> Expecting your suggestions, any links are appreciated.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Dastan
>>>>>>> _______________________________________________
>>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>>> -bin/mailman/listinfo/openstack
>>>>>>> Post to     : openstack at lists.openstack.org
>>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>>> -bin/mailman/listinfo/openstack
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: http://lists.openstack.org/cgi
>>>>>> -bin/mailman/listinfo/openstack
>>>>>> Post to     : openstack at lists.openstack.org
>>>>>> Unsubscribe : http://lists.openstack.org/cgi
>>>>>> -bin/mailman/listinfo/openstack
>>>>>>
>>>>>>
>>>>
>>>
>


-- 
Yours sincerely,
Dmitrii Sutiagin
Cloud Operations Engineer,
Mirantis, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170524/11da0246/attachment.html>