[Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR
Fox, Kevin M
Kevin.Fox at pnnl.gov
Thu Oct 25 15:31:15 UTC 2018
Would it make sense to move the control plane for this piece into the cluster? (vm in a mangement tenant?)
Thanks,
Kevin
________________________________________
From: Florian Engelmann [florian.engelmann at everyware.ch]
Sent: Thursday, October 25, 2018 7:39 AM
To: openstack-operators at lists.openstack.org
Subject: Re: [Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR
It looks like devstack implemented some o-hm0 interface to connect the
physical control host to a VxLAN.
In our case there is no VxLAN at the control nodes nor is OVS.
Is it a option to deploy those Octavia services needing this conenction
to the compute or network nodes and use o-hm0?
Am 10/25/18 um 10:22 AM schrieb Florian Engelmann:
> Or could I create lb-mgmt-net as VxLAN and connect the control nodes to
> this VxLAN? How to do something like that?
>
> Am 10/25/18 um 10:03 AM schrieb Florian Engelmann:
>> Hmm - so right now I can't see any routed option because:
>>
>> The gateway connected to the VLAN provider networks (bond1 on the
>> network nodes) is not able to route any traffic to my control nodes in
>> the spine-leaf layer3 backend network.
>>
>> And right now there is no br-ex at all nor any "streched" L2 domain
>> connecting all compute nodes.
>>
>>
>> So the only solution I can think of right now is to create an overlay
>> VxLAN in the spine-leaf backend network, connect all compute and
>> control nodes to this overlay L2 network, create a OVS bridge
>> connected to that network on the compute nodes and allow the Amphorae
>> to get an IPin this network as well.
>> Not to forget about DHCP... so the network nodes will need this bridge
>> as well.
>>
>> Am 10/24/18 um 10:01 PM schrieb Erik McCormick:
>>>
>>>
>>> On Wed, Oct 24, 2018, 3:33 PM Engelmann Florian
>>> <florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>> wrote:
>>>
>>> On the network nodes we've got a dedicated interface to deploy VLANs
>>> (like the provider network for internet access). What about creating
>>> another VLAN on the network nodes, give that bridge a IP which is
>>> part of the subnet of lb-mgmt-net and start the octavia worker,
>>> healthmanager and controller on the network nodes binding to that
>>> IP?
>>>
>>> The problem with that is you can't out an IP in the vlan interface
>>> and also use it as an OVS bridge, so the Octavia processes would have
>>> nothing to bind to.
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Erik McCormick <emccormick at cirrusseven.com
>>> <mailto:emccormick at cirrusseven.com>>
>>> *Sent:* Wednesday, October 24, 2018 6:18 PM
>>> *To:* Engelmann Florian
>>> *Cc:* openstack-operators
>>> *Subject:* Re: [Openstack-operators] [octavia][rocky] Octavia and
>>> VxLAN without DVR
>>>
>>>
>>> On Wed, Oct 24, 2018, 12:02 PM Florian Engelmann
>>> <florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>> wrote:
>>>
>>> Am 10/24/18 um 2:08 PM schrieb Erik McCormick:
>>> >
>>> >
>>> > On Wed, Oct 24, 2018, 3:14 AM Florian Engelmann
>>> > <florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>
>>> <mailto:florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>>>
>>> > wrote:
>>> >
>>> > Ohoh - thank you for your empathy :)
>>> > And those great details about how to setup this mgmt
>>> network.
>>> > I will try to do so this afternoon but solving that
>>> routing "puzzle"
>>> > (virtual network to control nodes) I will need our
>>> network guys to help
>>> > me out...
>>> >
>>> > But I will need to tell all Amphorae a static route to
>>> the gateway that
>>> > is routing to the control nodes?
>>> >
>>> >
>>> > Just set the default gateway when you create the neutron
>>> subnet. No need
>>> > for excess static routes. The route on the other connection
>>> won't
>>> > interfere with it as it lives in a namespace.
>>>
>>>
>>> My compute nodes have no br-ex and there is no L2 domain spread
>>> over all
>>> compute nodes. As far as I understood lb-mgmt-net is a provider
>>> network
>>> and has to be flat or VLAN and will need a "physical" gateway
>>> (as there
>>> is no virtual router).
>>> So the question - is it possible to get octavia up and running
>>> without a
>>> br-ex (L2 domain spread over all compute nodes) on the compute
>>> nodes?
>>>
>>>
>>> Sorry, I only meant it was *like* br-ex on your network nodes. You
>>> don't need that on your computes.
>>>
>>> The router here would be whatever does routing in your physical
>>> network. Setting the gateway in the neutron subnet simply adds that
>>> to the DHCP information sent to the amphorae.
>>>
>>> This does bring up another thingI forgot though. You'll probably
>>> want to add the management network / bridge to your network nodes or
>>> wherever you run the DHCP agents. When you create the subnet, be
>>> sure to leave some space in the address scope for the physical
>>> devices with static IPs.
>>>
>>> As for multiple L2 domains, I can't think of a way to go about that
>>> for the lb-mgmt network. It's a single network with a single subnet.
>>> Perhaps you could limit load balancers to an AZ in a single rack?
>>> Seems not very HA friendly.
>>>
>>>
>>>
>>> >
>>> >
>>> >
>>> > Am 10/23/18 um 6:57 PM schrieb Erik McCormick:
>>> > > So in your other email you said asked if there was a
>>> guide for
>>> > > deploying it with Kolla ansible...
>>> > >
>>> > > Oh boy. No there's not. I don't know if you've seen my
>>> recent
>>> > mails on
>>> > > Octavia, but I am going through this deployment
>>> process with
>>> > > kolla-ansible right now and it is lacking in a few
>>> areas.
>>> > >
>>> > > If you plan to use different CA certificates for
>>> client and server in
>>> > > Octavia, you'll need to add that into the playbook.
>>> Presently it only
>>> > > copies over ca_01.pem, cacert.key, and client.pem and
>>> uses them for
>>> > > everything. I was completely unable to make it work
>>> with only one CA
>>> > > as I got some SSL errors. It passes gate though, so I
>>> aasume it must
>>> > > work? I dunno.
>>> > >
>>> > > Networking comments and a really messy kolla-ansible /
>>> octavia
>>> > how-to below...
>>> > >
>>> > > On Tue, Oct 23, 2018 at 10:09 AM Florian Engelmann
>>> > > <florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>
>>> > <mailto:florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>>> wrote:
>>> > >>
>>> > >> Am 10/23/18 um 3:20 PM schrieb Erik McCormick:
>>> > >>> On Tue, Oct 23, 2018 at 7:53 AM Florian Engelmann
>>> > >>> <florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>
>>> > <mailto:florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>>> wrote:
>>> > >>>>
>>> > >>>> Hi,
>>> > >>>>
>>> > >>>> We did test Octavia with Pike (DVR deployment) and
>>> everything was
>>> > >>>> working right our of the box. We changed our
>>> underlay network to a
>>> > >>>> Layer3 spine-leaf network now and did not deploy
>>> DVR as we
>>> > don't wanted
>>> > >>>> to have that much cables in a rack.
>>> > >>>>
>>> > >>>> Octavia is not working right now as the lb-mgmt-net
>>> does not
>>> > exist on
>>> > >>>> the compute nodes nor does a br-ex.
>>> > >>>>
>>> > >>>> The control nodes running
>>> > >>>>
>>> > >>>> octavia_worker
>>> > >>>> octavia_housekeeping
>>> > >>>> octavia_health_manager
>>> > >>>> octavia_api
>>> > >>>>
>>> > Amphorae-VMs, z.b.
>>> >
>>> > lb-mgmt-net 172.16.0.0/16 <http://172.16.0.0/16>
>>> <http://172.16.0.0/16> default GW
>>> > >>>> and as far as I understood octavia_worker,
>>> > octavia_housekeeping and
>>> > >>>> octavia_health_manager have to talk to the amphora
>>> instances.
>>> > But the
>>> > >>>> control nodes are spread over three different
>>> leafs. So each
>>> > control
>>> > >>>> node in a different L2 domain.
>>> > >>>>
>>> > >>>> So the question is how to deploy a lb-mgmt-net
>>> network in our
>>> > setup?
>>> > >>>>
>>> > >>>> - Compute nodes have no "stretched" L2 domain
>>> > >>>> - Control nodes, compute nodes and network nodes
>>> are in L3
>>> > networks like
>>> > >>>> api, storage, ...
>>> > >>>> - Only network nodes are connected to a L2 domain
>>> (with a
>>> > separated NIC)
>>> > >>>> providing the "public" network
>>> > >>>>
>>> > >>> You'll need to add a new bridge to your compute
>>> nodes and create a
>>> > >>> provider network associated with that bridge. In my
>>> setup this is
>>> > >>> simply a flat network tied to a tagged interface. In
>>> your case it
>>> > >>> probably makes more sense to make a new VNI and
>>> create a vxlan
>>> > >>> provider network. The routing in your switches
>>> should handle
>>> > the rest.
>>> > >>
>>> > >> Ok that's what I try right now. But I don't get how
>>> to setup
>>> > something
>>> > >> like a VxLAN provider Network. I thought only vlan
>>> and flat is
>>> > supported
>>> > >> as provider network? I guess it is not possible to
>>> use the tunnel
>>> > >> interface that is used for tenant networks?
>>> > >> So I have to create a separated VxLAN on the control
>>> and compute
>>> > nodes like:
>>> > >>
>>> > >> # ip link add vxoctavia type vxlan id 42 dstport 4790
>>> group
>>> > 239.1.1.1
>>> > >> dev vlan3535 ttl 5
>>> > >> # ip addr add 172.16.1.11/20 <http://172.16.1.11/20>
>>> <http://172.16.1.11/20> dev vxoctavia
>>> > >> # ip link set vxoctavia up
>>> > >>
>>> > >> and use it like a flat provider network, true?
>>> > >>
>>> > > This is a fine way of doing things, but it's only half
>>> the battle.
>>> > > You'll need to add a bridge on the compute nodes and
>>> bind it to that
>>> > > new interface. Something like this if you're using
>>> openvswitch:
>>> > >
>>> > > docker exec openvswitch_db
>>> > > /usr/local/bin/kolla_ensure_openvswitch_configured
>>> br-mgmt vxoctavia
>>> > >
>>> > > Also you'll want to remove the IP address from that
>>> interface as it's
>>> > > going to be a bridge. Think of it like your public
>>> (br-ex) interface
>>> > > on your network nodes.
>>> > >
>>> > > From there you'll need to update the bridge mappings
>>> via kolla
>>> > > overrides. This would usually be in
>>> /etc/kolla/config/neutron. Create
>>> > > a subdirectory for your compute inventory group and
>>> create an
>>> > > ml2_conf.ini there. So you'd end up with something
>>> like:
>>> > >
>>> > > [root at kolla-deploy ~]# cat
>>> > /etc/kolla/config/neutron/compute/ml2_conf.ini
>>> > > [ml2_type_flat]
>>> > > flat_networks = mgmt-net
>>> > >
>>> > > [ovs]
>>> > > bridge_mappings = mgmt-net:br-mgmt
>>> > >
>>> > > run kolla-ansible --tags neutron reconfigure to push
>>> out the new
>>> > > configs. Note that there is a bug where the neutron
>>> containers
>>> > may not
>>> > > restart after the change, so you'll probably need to
>>> do a 'docker
>>> > > container restart neutron_openvswitch_agent' on each
>>> compute node.
>>> > >
>>> > > At this point, you'll need to create the provider
>>> network in the
>>> > admin
>>> > > project like:
>>> > >
>>> > > openstack network create --provider-network-type flat
>>> > > --provider-physical-network mgmt-net lb-mgmt-net
>>> > >
>>> > > And then create a normal subnet attached to this
>>> network with some
>>> > > largeish address scope. I wouldn't use 172.16.0.0/16
>>> <http://172.16.0.0/16>
>>> > <http://172.16.0.0/16> because docker
>>> > > uses that by default. I'm not sure if it matters since
>>> the network
>>> > > traffic will be isolated on a bridge, but it makes me
>>> paranoid so I
>>> > > avoided it.
>>> > >
>>> > > For your controllers, I think you can just let
>>> everything
>>> > function off
>>> > > your api interface since you're routing in your
>>> spines. Set up a
>>> > > gateway somewhere from that lb-mgmt network and save
>>> yourself the
>>> > > complication of adding an interface to your
>>> controllers. If you
>>> > choose
>>> > > to use a separate interface on your controllers,
>>> you'll need to make
>>> > > sure this patch is in your kolla-ansible install or
>>> cherry pick it.
>>> > >
>>> > >
>>> >
>>> https://github.com/openstack/kolla-ansible/commit/0b6e401c4fdb9aa4ff87d0bfd4b25c91b86e0d60#diff-6c871f6865aecf0057a5b5f677ae7d59
>>>
>>> > >
>>> > > I don't think that's been backported at all, so unless
>>> you're running
>>> > > off master you'll need to go get it.
>>> > >
>>> > > From here on out, the regular Octavia instruction
>>> should serve you.
>>> > > Create a flavor, Create a security group, and capture
>>> their UUIDs
>>> > > along with the UUID of the provider network you made.
>>> Override
>>> > them in
>>> > > globals.yml with:
>>> > >
>>> > > octavia_amp_boot_network_list: <uuid>
>>> > > octavia_amp_secgroup_list: <uuid>
>>> > > octavia_amp_flavor_id: <uuid>
>>> > >
>>> > > This is all from my scattered notes and bad memory.
>>> Hopefully it
>>> > makes
>>> > > sense. Corrections welcome.
>>> > >
>>> > > -Erik
>>> > >
>>> > >
>>> > >
>>> > >>
>>> > >>
>>> > >>>
>>> > >>> -Erik
>>> > >>>>
>>> > >>>> All the best,
>>> > >>>> Florian
>>> > >>>> _______________________________________________
>>> > >>>> OpenStack-operators mailing list
>>> > >>>> OpenStack-operators at lists.openstack.org
>>> <mailto:OpenStack-operators at lists.openstack.org>
>>> > <mailto:OpenStack-operators at lists.openstack.org
>>> <mailto:OpenStack-operators at lists.openstack.org>>
>>> > >>>>
>>> >
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> >
>>> > --
>>> >
>>> > EveryWare AG
>>> > Florian Engelmann
>>> > Systems Engineer
>>> > Zurlindenstrasse 52a
>>> > CH-8003 Zürich
>>> >
>>> > tel: +41 44 466 60 00
>>> > fax: +41 44 466 60 10
>>> > mail: mailto:florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>
>>> > <mailto:florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>>
>>> > web: http://www.everyware.ch
>>> >
>>>
>>> --
>>> EveryWare AG
>>> Florian Engelmann
>>> Systems Engineer
>>> Zurlindenstrasse 52a
>>> CH-8003 Zürich
>>>
>>> tel: +41 44 466 60 00
>>> fax: +41 44 466 60 10
>>> mail: mailto:florian.engelmann at everyware.ch
>>> <mailto:florian.engelmann at everyware.ch>
>>> web: http://www.everyware.ch
>>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
--
EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelmann at everyware.ch
web: http://www.everyware.ch
More information about the OpenStack-operators
mailing list