[Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR

Florian Engelmann florian.engelmann at everyware.ch
Thu Oct 25 08:22:17 UTC 2018


Or could I create lb-mgmt-net as VxLAN and connect the control nodes to 
this VxLAN? How to do something like that?

Am 10/25/18 um 10:03 AM schrieb Florian Engelmann:
> Hmm - so right now I can't see any routed option because:
> 
> The gateway connected to the VLAN provider networks (bond1 on the 
> network nodes) is not able to route any traffic to my control nodes in 
> the spine-leaf layer3 backend network.
> 
> And right now there is no br-ex at all nor any "streched" L2 domain 
> connecting all compute nodes.
> 
> 
> So the only solution I can think of right now is to create an overlay 
> VxLAN in the spine-leaf backend network, connect all compute and control 
> nodes to this overlay L2 network, create a OVS bridge connected to that 
> network on the compute nodes and allow the Amphorae to get an IPin this 
> network as well.
> Not to forget about DHCP... so the network nodes will need this bridge 
> as well.
> 
> Am 10/24/18 um 10:01 PM schrieb Erik McCormick:
>>
>>
>> On Wed, Oct 24, 2018, 3:33 PM Engelmann Florian 
>> <florian.engelmann at everyware.ch 
>> <mailto:florian.engelmann at everyware.ch>> wrote:
>>
>>     On the network nodes we've got a dedicated interface to deploy VLANs
>>     (like the provider network for internet access). What about creating
>>     another VLAN on the network nodes, give that bridge a IP which is
>>     part of the subnet of lb-mgmt-net and start the octavia worker,
>>     healthmanager and controller on the network nodes binding to that IP?
>>
>> The problem with that is you can't out an IP in the vlan interface and 
>> also use it as an OVS bridge, so the Octavia processes would have 
>> nothing to bind to.
>>
>>
>>     
>> ------------------------------------------------------------------------
>>     *From:* Erik McCormick <emccormick at cirrusseven.com
>>     <mailto:emccormick at cirrusseven.com>>
>>     *Sent:* Wednesday, October 24, 2018 6:18 PM
>>     *To:* Engelmann Florian
>>     *Cc:* openstack-operators
>>     *Subject:* Re: [Openstack-operators] [octavia][rocky] Octavia and
>>     VxLAN without DVR
>>
>>
>>     On Wed, Oct 24, 2018, 12:02 PM Florian Engelmann
>>     <florian.engelmann at everyware.ch
>>     <mailto:florian.engelmann at everyware.ch>> wrote:
>>
>>         Am 10/24/18 um 2:08 PM schrieb Erik McCormick:
>>          >
>>          >
>>          > On Wed, Oct 24, 2018, 3:14 AM Florian Engelmann
>>          > <florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>
>>         <mailto:florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>>>
>>          > wrote:
>>          >
>>          >     Ohoh - thank you for your empathy :)
>>          >     And those great details about how to setup this mgmt 
>> network.
>>          >     I will try to do so this afternoon but solving that
>>         routing "puzzle"
>>          >     (virtual network to control nodes) I will need our
>>         network guys to help
>>          >     me out...
>>          >
>>          >     But I will need to tell all Amphorae a static route to
>>         the gateway that
>>          >     is routing to the control nodes?
>>          >
>>          >
>>          > Just set the default gateway when you create the neutron
>>         subnet. No need
>>          > for excess static routes. The route on the other connection
>>         won't
>>          > interfere with it as it lives in a namespace.
>>
>>
>>         My compute nodes have no br-ex and there is no L2 domain spread
>>         over all
>>         compute nodes. As far as I understood lb-mgmt-net is a provider
>>         network
>>         and has to be flat or VLAN and will need a "physical" gateway
>>         (as there
>>         is no virtual router).
>>         So the question - is it possible to get octavia up and running
>>         without a
>>         br-ex (L2 domain spread over all compute nodes) on the compute
>>         nodes?
>>
>>
>>     Sorry, I only meant it was *like* br-ex on your network nodes. You
>>     don't need that on your computes.
>>
>>     The router here would be whatever does routing in your physical
>>     network. Setting the gateway in the neutron subnet simply adds that
>>     to the DHCP information sent to the amphorae.
>>
>>     This does bring up another thingI forgot  though. You'll probably
>>     want to add the management network / bridge to your network nodes or
>>     wherever you run the DHCP agents. When you create the subnet, be
>>     sure to leave some space in the address scope for the physical
>>     devices with static IPs.
>>
>>     As for multiple L2 domains, I can't think of a way to go about that
>>     for the lb-mgmt network. It's a single network with a single subnet.
>>     Perhaps you could limit load balancers to an AZ in a single rack?
>>     Seems not very HA friendly.
>>
>>
>>
>>          >
>>          >
>>          >
>>          >     Am 10/23/18 um 6:57 PM schrieb Erik McCormick:
>>          >      > So in your other email you said asked if there was a
>>         guide for
>>          >      > deploying it with Kolla ansible...
>>          >      >
>>          >      > Oh boy. No there's not. I don't know if you've seen my
>>         recent
>>          >     mails on
>>          >      > Octavia, but I am going through this deployment
>>         process with
>>          >      > kolla-ansible right now and it is lacking in a few 
>> areas.
>>          >      >
>>          >      > If you plan to use different CA certificates for
>>         client and server in
>>          >      > Octavia, you'll need to add that into the playbook.
>>         Presently it only
>>          >      > copies over ca_01.pem, cacert.key, and client.pem and
>>         uses them for
>>          >      > everything. I was completely unable to make it work
>>         with only one CA
>>          >      > as I got some SSL errors. It passes gate though, so I
>>         aasume it must
>>          >      > work? I dunno.
>>          >      >
>>          >      > Networking comments and a really messy kolla-ansible /
>>         octavia
>>          >     how-to below...
>>          >      >
>>          >      > On Tue, Oct 23, 2018 at 10:09 AM Florian Engelmann
>>          >      > <florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>
>>          >     <mailto:florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>>> wrote:
>>          >      >>
>>          >      >> Am 10/23/18 um 3:20 PM schrieb Erik McCormick:
>>          >      >>> On Tue, Oct 23, 2018 at 7:53 AM Florian Engelmann
>>          >      >>> <florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>
>>          >     <mailto:florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>>> wrote:
>>          >      >>>>
>>          >      >>>> Hi,
>>          >      >>>>
>>          >      >>>> We did test Octavia with Pike (DVR deployment) and
>>         everything was
>>          >      >>>> working right our of the box. We changed our
>>         underlay network to a
>>          >      >>>> Layer3 spine-leaf network now and did not deploy
>>         DVR as we
>>          >     don't wanted
>>          >      >>>> to have that much cables in a rack.
>>          >      >>>>
>>          >      >>>> Octavia is not working right now as the lb-mgmt-net
>>         does not
>>          >     exist on
>>          >      >>>> the compute nodes nor does a br-ex.
>>          >      >>>>
>>          >      >>>> The control nodes running
>>          >      >>>>
>>          >      >>>> octavia_worker
>>          >      >>>> octavia_housekeeping
>>          >      >>>> octavia_health_manager
>>          >      >>>> octavia_api
>>          >      >>>>
>>          >     Amphorae-VMs, z.b.
>>          >
>>          >     lb-mgmt-net 172.16.0.0/16 <http://172.16.0.0/16>
>>         <http://172.16.0.0/16> default GW
>>          >      >>>> and as far as I understood octavia_worker,
>>          >     octavia_housekeeping and
>>          >      >>>> octavia_health_manager have to talk to the amphora
>>         instances.
>>          >     But the
>>          >      >>>> control nodes are spread over three different
>>         leafs. So each
>>          >     control
>>          >      >>>> node in a different L2 domain.
>>          >      >>>>
>>          >      >>>> So the question is how to deploy a lb-mgmt-net
>>         network in our
>>          >     setup?
>>          >      >>>>
>>          >      >>>> - Compute nodes have no "stretched" L2 domain
>>          >      >>>> - Control nodes, compute nodes and network nodes
>>         are in L3
>>          >     networks like
>>          >      >>>> api, storage, ...
>>          >      >>>> - Only network nodes are connected to a L2 domain
>>         (with a
>>          >     separated NIC)
>>          >      >>>> providing the "public" network
>>          >      >>>>
>>          >      >>> You'll need to add a new bridge to your compute
>>         nodes and create a
>>          >      >>> provider network associated with that bridge. In my
>>         setup this is
>>          >      >>> simply a flat network tied to a tagged interface. In
>>         your case it
>>          >      >>> probably makes more sense to make a new VNI and
>>         create a vxlan
>>          >      >>> provider network. The routing in your switches
>>         should handle
>>          >     the rest.
>>          >      >>
>>          >      >> Ok that's what I try right now. But I don't get how
>>         to setup
>>          >     something
>>          >      >> like a VxLAN provider Network. I thought only vlan
>>         and flat is
>>          >     supported
>>          >      >> as provider network? I guess it is not possible to
>>         use the tunnel
>>          >      >> interface that is used for tenant networks?
>>          >      >> So I have to create a separated VxLAN on the control
>>         and compute
>>          >     nodes like:
>>          >      >>
>>          >      >> # ip link add vxoctavia type vxlan id 42 dstport 4790
>>         group
>>          >     239.1.1.1
>>          >      >> dev vlan3535 ttl 5
>>          >      >> # ip addr add 172.16.1.11/20 <http://172.16.1.11/20>
>>         <http://172.16.1.11/20> dev vxoctavia
>>          >      >> # ip link set vxoctavia up
>>          >      >>
>>          >      >> and use it like a flat provider network, true?
>>          >      >>
>>          >      > This is a fine way of doing things, but it's only half
>>         the battle.
>>          >      > You'll need to add a bridge on the compute nodes and
>>         bind it to that
>>          >      > new interface. Something like this if you're using
>>         openvswitch:
>>          >      >
>>          >      > docker exec openvswitch_db
>>          >      > /usr/local/bin/kolla_ensure_openvswitch_configured
>>         br-mgmt vxoctavia
>>          >      >
>>          >      > Also you'll want to remove the IP address from that
>>         interface as it's
>>          >      > going to be a bridge. Think of it like your public
>>         (br-ex) interface
>>          >      > on your network nodes.
>>          >      >
>>          >      >  From there you'll need to update the bridge mappings
>>         via kolla
>>          >      > overrides. This would usually be in
>>         /etc/kolla/config/neutron. Create
>>          >      > a subdirectory for your compute inventory group and
>>         create an
>>          >      > ml2_conf.ini there. So you'd end up with something 
>> like:
>>          >      >
>>          >      > [root at kolla-deploy ~]# cat
>>          >     /etc/kolla/config/neutron/compute/ml2_conf.ini
>>          >      > [ml2_type_flat]
>>          >      > flat_networks = mgmt-net
>>          >      >
>>          >      > [ovs]
>>          >      > bridge_mappings = mgmt-net:br-mgmt
>>          >      >
>>          >      > run kolla-ansible --tags neutron reconfigure to push
>>         out the new
>>          >      > configs. Note that there is a bug where the neutron
>>         containers
>>          >     may not
>>          >      > restart after the change, so you'll probably need to
>>         do a 'docker
>>          >      > container restart neutron_openvswitch_agent' on each
>>         compute node.
>>          >      >
>>          >      > At this point, you'll need to create the provider
>>         network in the
>>          >     admin
>>          >      > project like:
>>          >      >
>>          >      > openstack network create --provider-network-type flat
>>          >      > --provider-physical-network mgmt-net lb-mgmt-net
>>          >      >
>>          >      > And then create a normal subnet attached to this
>>         network with some
>>          >      > largeish address scope. I wouldn't use 172.16.0.0/16
>>         <http://172.16.0.0/16>
>>          >     <http://172.16.0.0/16> because docker
>>          >      > uses that by default. I'm not sure if it matters since
>>         the network
>>          >      > traffic will be isolated on a bridge, but it makes me
>>         paranoid so I
>>          >      > avoided it.
>>          >      >
>>          >      > For your controllers, I think you can just let 
>> everything
>>          >     function off
>>          >      > your api interface since you're routing in your
>>         spines. Set up a
>>          >      > gateway somewhere from that lb-mgmt network and save
>>         yourself the
>>          >      > complication of adding an interface to your
>>         controllers. If you
>>          >     choose
>>          >      > to use a separate interface on your controllers,
>>         you'll need to make
>>          >      > sure this patch is in your kolla-ansible install or
>>         cherry pick it.
>>          >      >
>>          >      >
>>          >
>>         
>> https://github.com/openstack/kolla-ansible/commit/0b6e401c4fdb9aa4ff87d0bfd4b25c91b86e0d60#diff-6c871f6865aecf0057a5b5f677ae7d59 
>>
>>          >      >
>>          >      > I don't think that's been backported at all, so unless
>>         you're running
>>          >      > off master you'll need to go get it.
>>          >      >
>>          >      >  From here on out, the regular Octavia instruction
>>         should serve you.
>>          >      > Create a flavor, Create a security group, and capture
>>         their UUIDs
>>          >      > along with the UUID of the provider network you made.
>>         Override
>>          >     them in
>>          >      > globals.yml with:
>>          >      >
>>          >      > octavia_amp_boot_network_list: <uuid>
>>          >      > octavia_amp_secgroup_list: <uuid>
>>          >      > octavia_amp_flavor_id: <uuid>
>>          >      >
>>          >      > This is all from my scattered notes and bad memory.
>>         Hopefully it
>>          >     makes
>>          >      > sense. Corrections welcome.
>>          >      >
>>          >      > -Erik
>>          >      >
>>          >      >
>>          >      >
>>          >      >>
>>          >      >>
>>          >      >>>
>>          >      >>> -Erik
>>          >      >>>>
>>          >      >>>> All the best,
>>          >      >>>> Florian
>>          >      >>>> _______________________________________________
>>          >      >>>> OpenStack-operators mailing list
>>          >      >>>> OpenStack-operators at lists.openstack.org
>>         <mailto:OpenStack-operators at lists.openstack.org>
>>          >     <mailto:OpenStack-operators at lists.openstack.org
>>         <mailto:OpenStack-operators at lists.openstack.org>>
>>          >      >>>>
>>          >
>>         
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>          >
>>          >     --
>>          >
>>          >     EveryWare AG
>>          >     Florian Engelmann
>>          >     Systems Engineer
>>          >     Zurlindenstrasse 52a
>>          >     CH-8003 Zürich
>>          >
>>          >     tel: +41 44 466 60 00
>>          >     fax: +41 44 466 60 10
>>          >     mail: mailto:florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>
>>          >     <mailto:florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>>
>>          >     web: http://www.everyware.ch
>>          >
>>
>>         --
>>         EveryWare AG
>>         Florian Engelmann
>>         Systems Engineer
>>         Zurlindenstrasse 52a
>>         CH-8003 Zürich
>>
>>         tel: +41 44 466 60 00
>>         fax: +41 44 466 60 10
>>         mail: mailto:florian.engelmann at everyware.ch
>>         <mailto:florian.engelmann at everyware.ch>
>>         web: http://www.everyware.ch
>>
> 
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 

-- 

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelmann at everyware.ch
web: http://www.everyware.ch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5210 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20181025/b033f0ac/attachment.bin>


More information about the OpenStack-operators mailing list