[Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR
Florian Engelmann
florian.engelmann at everyware.ch
Thu Oct 25 08:22:17 UTC 2018
Or could I create lb-mgmt-net as VxLAN and connect the control nodes to
this VxLAN? How to do something like that?
Am 10/25/18 um 10:03 AM schrieb Florian Engelmann:
> Hmm - so right now I can't see any routed option because:
>
> The gateway connected to the VLAN provider networks (bond1 on the
> network nodes) is not able to route any traffic to my control nodes in
> the spine-leaf layer3 backend network.
>
> And right now there is no br-ex at all nor any "streched" L2 domain
> connecting all compute nodes.
>
>
> So the only solution I can think of right now is to create an overlay
> VxLAN in the spine-leaf backend network, connect all compute and control
> nodes to this overlay L2 network, create a OVS bridge connected to that
> network on the compute nodes and allow the Amphorae to get an IPin this
> network as well.
> Not to forget about DHCP... so the network nodes will need this bridge
> as well.
>
> Am 10/24/18 um 10:01 PM schrieb Erik McCormick:
>>
>>
>> On Wed, Oct 24, 2018, 3:33 PM Engelmann Florian
>> <florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>> wrote:
>>
>> On the network nodes we've got a dedicated interface to deploy VLANs
>> (like the provider network for internet access). What about creating
>> another VLAN on the network nodes, give that bridge a IP which is
>> part of the subnet of lb-mgmt-net and start the octavia worker,
>> healthmanager and controller on the network nodes binding to that IP?
>>
>> The problem with that is you can't out an IP in the vlan interface and
>> also use it as an OVS bridge, so the Octavia processes would have
>> nothing to bind to.
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Erik McCormick <emccormick at cirrusseven.com
>> <mailto:emccormick at cirrusseven.com>>
>> *Sent:* Wednesday, October 24, 2018 6:18 PM
>> *To:* Engelmann Florian
>> *Cc:* openstack-operators
>> *Subject:* Re: [Openstack-operators] [octavia][rocky] Octavia and
>> VxLAN without DVR
>>
>>
>> On Wed, Oct 24, 2018, 12:02 PM Florian Engelmann
>> <florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>> wrote:
>>
>> Am 10/24/18 um 2:08 PM schrieb Erik McCormick:
>> >
>> >
>> > On Wed, Oct 24, 2018, 3:14 AM Florian Engelmann
>> > <florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>
>> <mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>>>
>> > wrote:
>> >
>> > Ohoh - thank you for your empathy :)
>> > And those great details about how to setup this mgmt
>> network.
>> > I will try to do so this afternoon but solving that
>> routing "puzzle"
>> > (virtual network to control nodes) I will need our
>> network guys to help
>> > me out...
>> >
>> > But I will need to tell all Amphorae a static route to
>> the gateway that
>> > is routing to the control nodes?
>> >
>> >
>> > Just set the default gateway when you create the neutron
>> subnet. No need
>> > for excess static routes. The route on the other connection
>> won't
>> > interfere with it as it lives in a namespace.
>>
>>
>> My compute nodes have no br-ex and there is no L2 domain spread
>> over all
>> compute nodes. As far as I understood lb-mgmt-net is a provider
>> network
>> and has to be flat or VLAN and will need a "physical" gateway
>> (as there
>> is no virtual router).
>> So the question - is it possible to get octavia up and running
>> without a
>> br-ex (L2 domain spread over all compute nodes) on the compute
>> nodes?
>>
>>
>> Sorry, I only meant it was *like* br-ex on your network nodes. You
>> don't need that on your computes.
>>
>> The router here would be whatever does routing in your physical
>> network. Setting the gateway in the neutron subnet simply adds that
>> to the DHCP information sent to the amphorae.
>>
>> This does bring up another thingI forgot though. You'll probably
>> want to add the management network / bridge to your network nodes or
>> wherever you run the DHCP agents. When you create the subnet, be
>> sure to leave some space in the address scope for the physical
>> devices with static IPs.
>>
>> As for multiple L2 domains, I can't think of a way to go about that
>> for the lb-mgmt network. It's a single network with a single subnet.
>> Perhaps you could limit load balancers to an AZ in a single rack?
>> Seems not very HA friendly.
>>
>>
>>
>> >
>> >
>> >
>> > Am 10/23/18 um 6:57 PM schrieb Erik McCormick:
>> > > So in your other email you said asked if there was a
>> guide for
>> > > deploying it with Kolla ansible...
>> > >
>> > > Oh boy. No there's not. I don't know if you've seen my
>> recent
>> > mails on
>> > > Octavia, but I am going through this deployment
>> process with
>> > > kolla-ansible right now and it is lacking in a few
>> areas.
>> > >
>> > > If you plan to use different CA certificates for
>> client and server in
>> > > Octavia, you'll need to add that into the playbook.
>> Presently it only
>> > > copies over ca_01.pem, cacert.key, and client.pem and
>> uses them for
>> > > everything. I was completely unable to make it work
>> with only one CA
>> > > as I got some SSL errors. It passes gate though, so I
>> aasume it must
>> > > work? I dunno.
>> > >
>> > > Networking comments and a really messy kolla-ansible /
>> octavia
>> > how-to below...
>> > >
>> > > On Tue, Oct 23, 2018 at 10:09 AM Florian Engelmann
>> > > <florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>
>> > <mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>>> wrote:
>> > >>
>> > >> Am 10/23/18 um 3:20 PM schrieb Erik McCormick:
>> > >>> On Tue, Oct 23, 2018 at 7:53 AM Florian Engelmann
>> > >>> <florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>
>> > <mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>>> wrote:
>> > >>>>
>> > >>>> Hi,
>> > >>>>
>> > >>>> We did test Octavia with Pike (DVR deployment) and
>> everything was
>> > >>>> working right our of the box. We changed our
>> underlay network to a
>> > >>>> Layer3 spine-leaf network now and did not deploy
>> DVR as we
>> > don't wanted
>> > >>>> to have that much cables in a rack.
>> > >>>>
>> > >>>> Octavia is not working right now as the lb-mgmt-net
>> does not
>> > exist on
>> > >>>> the compute nodes nor does a br-ex.
>> > >>>>
>> > >>>> The control nodes running
>> > >>>>
>> > >>>> octavia_worker
>> > >>>> octavia_housekeeping
>> > >>>> octavia_health_manager
>> > >>>> octavia_api
>> > >>>>
>> > Amphorae-VMs, z.b.
>> >
>> > lb-mgmt-net 172.16.0.0/16 <http://172.16.0.0/16>
>> <http://172.16.0.0/16> default GW
>> > >>>> and as far as I understood octavia_worker,
>> > octavia_housekeeping and
>> > >>>> octavia_health_manager have to talk to the amphora
>> instances.
>> > But the
>> > >>>> control nodes are spread over three different
>> leafs. So each
>> > control
>> > >>>> node in a different L2 domain.
>> > >>>>
>> > >>>> So the question is how to deploy a lb-mgmt-net
>> network in our
>> > setup?
>> > >>>>
>> > >>>> - Compute nodes have no "stretched" L2 domain
>> > >>>> - Control nodes, compute nodes and network nodes
>> are in L3
>> > networks like
>> > >>>> api, storage, ...
>> > >>>> - Only network nodes are connected to a L2 domain
>> (with a
>> > separated NIC)
>> > >>>> providing the "public" network
>> > >>>>
>> > >>> You'll need to add a new bridge to your compute
>> nodes and create a
>> > >>> provider network associated with that bridge. In my
>> setup this is
>> > >>> simply a flat network tied to a tagged interface. In
>> your case it
>> > >>> probably makes more sense to make a new VNI and
>> create a vxlan
>> > >>> provider network. The routing in your switches
>> should handle
>> > the rest.
>> > >>
>> > >> Ok that's what I try right now. But I don't get how
>> to setup
>> > something
>> > >> like a VxLAN provider Network. I thought only vlan
>> and flat is
>> > supported
>> > >> as provider network? I guess it is not possible to
>> use the tunnel
>> > >> interface that is used for tenant networks?
>> > >> So I have to create a separated VxLAN on the control
>> and compute
>> > nodes like:
>> > >>
>> > >> # ip link add vxoctavia type vxlan id 42 dstport 4790
>> group
>> > 239.1.1.1
>> > >> dev vlan3535 ttl 5
>> > >> # ip addr add 172.16.1.11/20 <http://172.16.1.11/20>
>> <http://172.16.1.11/20> dev vxoctavia
>> > >> # ip link set vxoctavia up
>> > >>
>> > >> and use it like a flat provider network, true?
>> > >>
>> > > This is a fine way of doing things, but it's only half
>> the battle.
>> > > You'll need to add a bridge on the compute nodes and
>> bind it to that
>> > > new interface. Something like this if you're using
>> openvswitch:
>> > >
>> > > docker exec openvswitch_db
>> > > /usr/local/bin/kolla_ensure_openvswitch_configured
>> br-mgmt vxoctavia
>> > >
>> > > Also you'll want to remove the IP address from that
>> interface as it's
>> > > going to be a bridge. Think of it like your public
>> (br-ex) interface
>> > > on your network nodes.
>> > >
>> > > From there you'll need to update the bridge mappings
>> via kolla
>> > > overrides. This would usually be in
>> /etc/kolla/config/neutron. Create
>> > > a subdirectory for your compute inventory group and
>> create an
>> > > ml2_conf.ini there. So you'd end up with something
>> like:
>> > >
>> > > [root at kolla-deploy ~]# cat
>> > /etc/kolla/config/neutron/compute/ml2_conf.ini
>> > > [ml2_type_flat]
>> > > flat_networks = mgmt-net
>> > >
>> > > [ovs]
>> > > bridge_mappings = mgmt-net:br-mgmt
>> > >
>> > > run kolla-ansible --tags neutron reconfigure to push
>> out the new
>> > > configs. Note that there is a bug where the neutron
>> containers
>> > may not
>> > > restart after the change, so you'll probably need to
>> do a 'docker
>> > > container restart neutron_openvswitch_agent' on each
>> compute node.
>> > >
>> > > At this point, you'll need to create the provider
>> network in the
>> > admin
>> > > project like:
>> > >
>> > > openstack network create --provider-network-type flat
>> > > --provider-physical-network mgmt-net lb-mgmt-net
>> > >
>> > > And then create a normal subnet attached to this
>> network with some
>> > > largeish address scope. I wouldn't use 172.16.0.0/16
>> <http://172.16.0.0/16>
>> > <http://172.16.0.0/16> because docker
>> > > uses that by default. I'm not sure if it matters since
>> the network
>> > > traffic will be isolated on a bridge, but it makes me
>> paranoid so I
>> > > avoided it.
>> > >
>> > > For your controllers, I think you can just let
>> everything
>> > function off
>> > > your api interface since you're routing in your
>> spines. Set up a
>> > > gateway somewhere from that lb-mgmt network and save
>> yourself the
>> > > complication of adding an interface to your
>> controllers. If you
>> > choose
>> > > to use a separate interface on your controllers,
>> you'll need to make
>> > > sure this patch is in your kolla-ansible install or
>> cherry pick it.
>> > >
>> > >
>> >
>>
>> https://github.com/openstack/kolla-ansible/commit/0b6e401c4fdb9aa4ff87d0bfd4b25c91b86e0d60#diff-6c871f6865aecf0057a5b5f677ae7d59
>>
>> > >
>> > > I don't think that's been backported at all, so unless
>> you're running
>> > > off master you'll need to go get it.
>> > >
>> > > From here on out, the regular Octavia instruction
>> should serve you.
>> > > Create a flavor, Create a security group, and capture
>> their UUIDs
>> > > along with the UUID of the provider network you made.
>> Override
>> > them in
>> > > globals.yml with:
>> > >
>> > > octavia_amp_boot_network_list: <uuid>
>> > > octavia_amp_secgroup_list: <uuid>
>> > > octavia_amp_flavor_id: <uuid>
>> > >
>> > > This is all from my scattered notes and bad memory.
>> Hopefully it
>> > makes
>> > > sense. Corrections welcome.
>> > >
>> > > -Erik
>> > >
>> > >
>> > >
>> > >>
>> > >>
>> > >>>
>> > >>> -Erik
>> > >>>>
>> > >>>> All the best,
>> > >>>> Florian
>> > >>>> _______________________________________________
>> > >>>> OpenStack-operators mailing list
>> > >>>> OpenStack-operators at lists.openstack.org
>> <mailto:OpenStack-operators at lists.openstack.org>
>> > <mailto:OpenStack-operators at lists.openstack.org
>> <mailto:OpenStack-operators at lists.openstack.org>>
>> > >>>>
>> >
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>> > --
>> >
>> > EveryWare AG
>> > Florian Engelmann
>> > Systems Engineer
>> > Zurlindenstrasse 52a
>> > CH-8003 Zürich
>> >
>> > tel: +41 44 466 60 00
>> > fax: +41 44 466 60 10
>> > mail: mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>
>> > <mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>>
>> > web: http://www.everyware.ch
>> >
>>
>> --
>> EveryWare AG
>> Florian Engelmann
>> Systems Engineer
>> Zurlindenstrasse 52a
>> CH-8003 Zürich
>>
>> tel: +41 44 466 60 00
>> fax: +41 44 466 60 10
>> mail: mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>
>> web: http://www.everyware.ch
>>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
--
EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelmann at everyware.ch
web: http://www.everyware.ch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5210 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20181025/b033f0ac/attachment.bin>
More information about the OpenStack-operators
mailing list