[Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR

Florian Engelmann florian.engelmann at everyware.ch
Thu Oct 25 15:37:21 UTC 2018


you mean deploy octavia into an openstack project? But I will than need 
to connect the octavia services with my galera DBs... so same problem.

Am 10/25/18 um 5:31 PM schrieb Fox, Kevin M:
> Would it make sense to move the control plane for this piece into the cluster? (vm in a mangement tenant?)
> 
> Thanks,
> Kevin
> ________________________________________
> From: Florian Engelmann [florian.engelmann at everyware.ch]
> Sent: Thursday, October 25, 2018 7:39 AM
> To: openstack-operators at lists.openstack.org
> Subject: Re: [Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR
> 
> It looks like devstack implemented some o-hm0 interface to connect the
> physical control host to a VxLAN.
> In our case there is no VxLAN at the control nodes nor is OVS.
> 
> Is it a option to deploy those Octavia services needing this conenction
> to the compute or network nodes and use o-hm0?
> 
> Am 10/25/18 um 10:22 AM schrieb Florian Engelmann:
>> Or could I create lb-mgmt-net as VxLAN and connect the control nodes to
>> this VxLAN? How to do something like that?
>>
>> Am 10/25/18 um 10:03 AM schrieb Florian Engelmann:
>>> Hmm - so right now I can't see any routed option because:
>>>
>>> The gateway connected to the VLAN provider networks (bond1 on the
>>> network nodes) is not able to route any traffic to my control nodes in
>>> the spine-leaf layer3 backend network.
>>>
>>> And right now there is no br-ex at all nor any "streched" L2 domain
>>> connecting all compute nodes.
>>>
>>>
>>> So the only solution I can think of right now is to create an overlay
>>> VxLAN in the spine-leaf backend network, connect all compute and
>>> control nodes to this overlay L2 network, create a OVS bridge
>>> connected to that network on the compute nodes and allow the Amphorae
>>> to get an IPin this network as well.
>>> Not to forget about DHCP... so the network nodes will need this bridge
>>> as well.
>>>
>>> Am 10/24/18 um 10:01 PM schrieb Erik McCormick:
>>>>
>>>>
>>>> On Wed, Oct 24, 2018, 3:33 PM Engelmann Florian
>>>> <florian.engelmann at everyware.ch
>>>> <mailto:florian.engelmann at everyware.ch>> wrote:
>>>>
>>>>      On the network nodes we've got a dedicated interface to deploy VLANs
>>>>      (like the provider network for internet access). What about creating
>>>>      another VLAN on the network nodes, give that bridge a IP which is
>>>>      part of the subnet of lb-mgmt-net and start the octavia worker,
>>>>      healthmanager and controller on the network nodes binding to that
>>>> IP?
>>>>
>>>> The problem with that is you can't out an IP in the vlan interface
>>>> and also use it as an OVS bridge, so the Octavia processes would have
>>>> nothing to bind to.
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>      *From:* Erik McCormick <emccormick at cirrusseven.com
>>>>      <mailto:emccormick at cirrusseven.com>>
>>>>      *Sent:* Wednesday, October 24, 2018 6:18 PM
>>>>      *To:* Engelmann Florian
>>>>      *Cc:* openstack-operators
>>>>      *Subject:* Re: [Openstack-operators] [octavia][rocky] Octavia and
>>>>      VxLAN without DVR
>>>>
>>>>
>>>>      On Wed, Oct 24, 2018, 12:02 PM Florian Engelmann
>>>>      <florian.engelmann at everyware.ch
>>>>      <mailto:florian.engelmann at everyware.ch>> wrote:
>>>>
>>>>          Am 10/24/18 um 2:08 PM schrieb Erik McCormick:
>>>>           >
>>>>           >
>>>>           > On Wed, Oct 24, 2018, 3:14 AM Florian Engelmann
>>>>           > <florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>
>>>>          <mailto:florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>>>
>>>>           > wrote:
>>>>           >
>>>>           >     Ohoh - thank you for your empathy :)
>>>>           >     And those great details about how to setup this mgmt
>>>> network.
>>>>           >     I will try to do so this afternoon but solving that
>>>>          routing "puzzle"
>>>>           >     (virtual network to control nodes) I will need our
>>>>          network guys to help
>>>>           >     me out...
>>>>           >
>>>>           >     But I will need to tell all Amphorae a static route to
>>>>          the gateway that
>>>>           >     is routing to the control nodes?
>>>>           >
>>>>           >
>>>>           > Just set the default gateway when you create the neutron
>>>>          subnet. No need
>>>>           > for excess static routes. The route on the other connection
>>>>          won't
>>>>           > interfere with it as it lives in a namespace.
>>>>
>>>>
>>>>          My compute nodes have no br-ex and there is no L2 domain spread
>>>>          over all
>>>>          compute nodes. As far as I understood lb-mgmt-net is a provider
>>>>          network
>>>>          and has to be flat or VLAN and will need a "physical" gateway
>>>>          (as there
>>>>          is no virtual router).
>>>>          So the question - is it possible to get octavia up and running
>>>>          without a
>>>>          br-ex (L2 domain spread over all compute nodes) on the compute
>>>>          nodes?
>>>>
>>>>
>>>>      Sorry, I only meant it was *like* br-ex on your network nodes. You
>>>>      don't need that on your computes.
>>>>
>>>>      The router here would be whatever does routing in your physical
>>>>      network. Setting the gateway in the neutron subnet simply adds that
>>>>      to the DHCP information sent to the amphorae.
>>>>
>>>>      This does bring up another thingI forgot  though. You'll probably
>>>>      want to add the management network / bridge to your network nodes or
>>>>      wherever you run the DHCP agents. When you create the subnet, be
>>>>      sure to leave some space in the address scope for the physical
>>>>      devices with static IPs.
>>>>
>>>>      As for multiple L2 domains, I can't think of a way to go about that
>>>>      for the lb-mgmt network. It's a single network with a single subnet.
>>>>      Perhaps you could limit load balancers to an AZ in a single rack?
>>>>      Seems not very HA friendly.
>>>>
>>>>
>>>>
>>>>           >
>>>>           >
>>>>           >
>>>>           >     Am 10/23/18 um 6:57 PM schrieb Erik McCormick:
>>>>           >      > So in your other email you said asked if there was a
>>>>          guide for
>>>>           >      > deploying it with Kolla ansible...
>>>>           >      >
>>>>           >      > Oh boy. No there's not. I don't know if you've seen my
>>>>          recent
>>>>           >     mails on
>>>>           >      > Octavia, but I am going through this deployment
>>>>          process with
>>>>           >      > kolla-ansible right now and it is lacking in a few
>>>> areas.
>>>>           >      >
>>>>           >      > If you plan to use different CA certificates for
>>>>          client and server in
>>>>           >      > Octavia, you'll need to add that into the playbook.
>>>>          Presently it only
>>>>           >      > copies over ca_01.pem, cacert.key, and client.pem and
>>>>          uses them for
>>>>           >      > everything. I was completely unable to make it work
>>>>          with only one CA
>>>>           >      > as I got some SSL errors. It passes gate though, so I
>>>>          aasume it must
>>>>           >      > work? I dunno.
>>>>           >      >
>>>>           >      > Networking comments and a really messy kolla-ansible /
>>>>          octavia
>>>>           >     how-to below...
>>>>           >      >
>>>>           >      > On Tue, Oct 23, 2018 at 10:09 AM Florian Engelmann
>>>>           >      > <florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>
>>>>           >     <mailto:florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>>> wrote:
>>>>           >      >>
>>>>           >      >> Am 10/23/18 um 3:20 PM schrieb Erik McCormick:
>>>>           >      >>> On Tue, Oct 23, 2018 at 7:53 AM Florian Engelmann
>>>>           >      >>> <florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>
>>>>           >     <mailto:florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>>> wrote:
>>>>           >      >>>>
>>>>           >      >>>> Hi,
>>>>           >      >>>>
>>>>           >      >>>> We did test Octavia with Pike (DVR deployment) and
>>>>          everything was
>>>>           >      >>>> working right our of the box. We changed our
>>>>          underlay network to a
>>>>           >      >>>> Layer3 spine-leaf network now and did not deploy
>>>>          DVR as we
>>>>           >     don't wanted
>>>>           >      >>>> to have that much cables in a rack.
>>>>           >      >>>>
>>>>           >      >>>> Octavia is not working right now as the lb-mgmt-net
>>>>          does not
>>>>           >     exist on
>>>>           >      >>>> the compute nodes nor does a br-ex.
>>>>           >      >>>>
>>>>           >      >>>> The control nodes running
>>>>           >      >>>>
>>>>           >      >>>> octavia_worker
>>>>           >      >>>> octavia_housekeeping
>>>>           >      >>>> octavia_health_manager
>>>>           >      >>>> octavia_api
>>>>           >      >>>>
>>>>           >     Amphorae-VMs, z.b.
>>>>           >
>>>>           >     lb-mgmt-net 172.16.0.0/16 <http://172.16.0.0/16>
>>>>          <http://172.16.0.0/16> default GW
>>>>           >      >>>> and as far as I understood octavia_worker,
>>>>           >     octavia_housekeeping and
>>>>           >      >>>> octavia_health_manager have to talk to the amphora
>>>>          instances.
>>>>           >     But the
>>>>           >      >>>> control nodes are spread over three different
>>>>          leafs. So each
>>>>           >     control
>>>>           >      >>>> node in a different L2 domain.
>>>>           >      >>>>
>>>>           >      >>>> So the question is how to deploy a lb-mgmt-net
>>>>          network in our
>>>>           >     setup?
>>>>           >      >>>>
>>>>           >      >>>> - Compute nodes have no "stretched" L2 domain
>>>>           >      >>>> - Control nodes, compute nodes and network nodes
>>>>          are in L3
>>>>           >     networks like
>>>>           >      >>>> api, storage, ...
>>>>           >      >>>> - Only network nodes are connected to a L2 domain
>>>>          (with a
>>>>           >     separated NIC)
>>>>           >      >>>> providing the "public" network
>>>>           >      >>>>
>>>>           >      >>> You'll need to add a new bridge to your compute
>>>>          nodes and create a
>>>>           >      >>> provider network associated with that bridge. In my
>>>>          setup this is
>>>>           >      >>> simply a flat network tied to a tagged interface. In
>>>>          your case it
>>>>           >      >>> probably makes more sense to make a new VNI and
>>>>          create a vxlan
>>>>           >      >>> provider network. The routing in your switches
>>>>          should handle
>>>>           >     the rest.
>>>>           >      >>
>>>>           >      >> Ok that's what I try right now. But I don't get how
>>>>          to setup
>>>>           >     something
>>>>           >      >> like a VxLAN provider Network. I thought only vlan
>>>>          and flat is
>>>>           >     supported
>>>>           >      >> as provider network? I guess it is not possible to
>>>>          use the tunnel
>>>>           >      >> interface that is used for tenant networks?
>>>>           >      >> So I have to create a separated VxLAN on the control
>>>>          and compute
>>>>           >     nodes like:
>>>>           >      >>
>>>>           >      >> # ip link add vxoctavia type vxlan id 42 dstport 4790
>>>>          group
>>>>           >     239.1.1.1
>>>>           >      >> dev vlan3535 ttl 5
>>>>           >      >> # ip addr add 172.16.1.11/20 <http://172.16.1.11/20>
>>>>          <http://172.16.1.11/20> dev vxoctavia
>>>>           >      >> # ip link set vxoctavia up
>>>>           >      >>
>>>>           >      >> and use it like a flat provider network, true?
>>>>           >      >>
>>>>           >      > This is a fine way of doing things, but it's only half
>>>>          the battle.
>>>>           >      > You'll need to add a bridge on the compute nodes and
>>>>          bind it to that
>>>>           >      > new interface. Something like this if you're using
>>>>          openvswitch:
>>>>           >      >
>>>>           >      > docker exec openvswitch_db
>>>>           >      > /usr/local/bin/kolla_ensure_openvswitch_configured
>>>>          br-mgmt vxoctavia
>>>>           >      >
>>>>           >      > Also you'll want to remove the IP address from that
>>>>          interface as it's
>>>>           >      > going to be a bridge. Think of it like your public
>>>>          (br-ex) interface
>>>>           >      > on your network nodes.
>>>>           >      >
>>>>           >      >  From there you'll need to update the bridge mappings
>>>>          via kolla
>>>>           >      > overrides. This would usually be in
>>>>          /etc/kolla/config/neutron. Create
>>>>           >      > a subdirectory for your compute inventory group and
>>>>          create an
>>>>           >      > ml2_conf.ini there. So you'd end up with something
>>>> like:
>>>>           >      >
>>>>           >      > [root at kolla-deploy ~]# cat
>>>>           >     /etc/kolla/config/neutron/compute/ml2_conf.ini
>>>>           >      > [ml2_type_flat]
>>>>           >      > flat_networks = mgmt-net
>>>>           >      >
>>>>           >      > [ovs]
>>>>           >      > bridge_mappings = mgmt-net:br-mgmt
>>>>           >      >
>>>>           >      > run kolla-ansible --tags neutron reconfigure to push
>>>>          out the new
>>>>           >      > configs. Note that there is a bug where the neutron
>>>>          containers
>>>>           >     may not
>>>>           >      > restart after the change, so you'll probably need to
>>>>          do a 'docker
>>>>           >      > container restart neutron_openvswitch_agent' on each
>>>>          compute node.
>>>>           >      >
>>>>           >      > At this point, you'll need to create the provider
>>>>          network in the
>>>>           >     admin
>>>>           >      > project like:
>>>>           >      >
>>>>           >      > openstack network create --provider-network-type flat
>>>>           >      > --provider-physical-network mgmt-net lb-mgmt-net
>>>>           >      >
>>>>           >      > And then create a normal subnet attached to this
>>>>          network with some
>>>>           >      > largeish address scope. I wouldn't use 172.16.0.0/16
>>>>          <http://172.16.0.0/16>
>>>>           >     <http://172.16.0.0/16> because docker
>>>>           >      > uses that by default. I'm not sure if it matters since
>>>>          the network
>>>>           >      > traffic will be isolated on a bridge, but it makes me
>>>>          paranoid so I
>>>>           >      > avoided it.
>>>>           >      >
>>>>           >      > For your controllers, I think you can just let
>>>> everything
>>>>           >     function off
>>>>           >      > your api interface since you're routing in your
>>>>          spines. Set up a
>>>>           >      > gateway somewhere from that lb-mgmt network and save
>>>>          yourself the
>>>>           >      > complication of adding an interface to your
>>>>          controllers. If you
>>>>           >     choose
>>>>           >      > to use a separate interface on your controllers,
>>>>          you'll need to make
>>>>           >      > sure this patch is in your kolla-ansible install or
>>>>          cherry pick it.
>>>>           >      >
>>>>           >      >
>>>>           >
>>>> https://github.com/openstack/kolla-ansible/commit/0b6e401c4fdb9aa4ff87d0bfd4b25c91b86e0d60#diff-6c871f6865aecf0057a5b5f677ae7d59
>>>>
>>>>           >      >
>>>>           >      > I don't think that's been backported at all, so unless
>>>>          you're running
>>>>           >      > off master you'll need to go get it.
>>>>           >      >
>>>>           >      >  From here on out, the regular Octavia instruction
>>>>          should serve you.
>>>>           >      > Create a flavor, Create a security group, and capture
>>>>          their UUIDs
>>>>           >      > along with the UUID of the provider network you made.
>>>>          Override
>>>>           >     them in
>>>>           >      > globals.yml with:
>>>>           >      >
>>>>           >      > octavia_amp_boot_network_list: <uuid>
>>>>           >      > octavia_amp_secgroup_list: <uuid>
>>>>           >      > octavia_amp_flavor_id: <uuid>
>>>>           >      >
>>>>           >      > This is all from my scattered notes and bad memory.
>>>>          Hopefully it
>>>>           >     makes
>>>>           >      > sense. Corrections welcome.
>>>>           >      >
>>>>           >      > -Erik
>>>>           >      >
>>>>           >      >
>>>>           >      >
>>>>           >      >>
>>>>           >      >>
>>>>           >      >>>
>>>>           >      >>> -Erik
>>>>           >      >>>>
>>>>           >      >>>> All the best,
>>>>           >      >>>> Florian
>>>>           >      >>>> _______________________________________________
>>>>           >      >>>> OpenStack-operators mailing list
>>>>           >      >>>> OpenStack-operators at lists.openstack.org
>>>>          <mailto:OpenStack-operators at lists.openstack.org>
>>>>           >     <mailto:OpenStack-operators at lists.openstack.org
>>>>          <mailto:OpenStack-operators at lists.openstack.org>>
>>>>           >      >>>>
>>>>           >
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>           >
>>>>           >     --
>>>>           >
>>>>           >     EveryWare AG
>>>>           >     Florian Engelmann
>>>>           >     Systems Engineer
>>>>           >     Zurlindenstrasse 52a
>>>>           >     CH-8003 Zürich
>>>>           >
>>>>           >     tel: +41 44 466 60 00
>>>>           >     fax: +41 44 466 60 10
>>>>           >     mail: mailto:florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>
>>>>           >     <mailto:florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>>
>>>>           >     web: http://www.everyware.ch
>>>>           >
>>>>
>>>>          --
>>>>          EveryWare AG
>>>>          Florian Engelmann
>>>>          Systems Engineer
>>>>          Zurlindenstrasse 52a
>>>>          CH-8003 Zürich
>>>>
>>>>          tel: +41 44 466 60 00
>>>>          fax: +41 44 466 60 10
>>>>          mail: mailto:florian.engelmann at everyware.ch
>>>>          <mailto:florian.engelmann at everyware.ch>
>>>>          web: http://www.everyware.ch
>>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
> 
> --
> 
> EveryWare AG
> Florian Engelmann
> Systems Engineer
> Zurlindenstrasse 52a
> CH-8003 Zürich
> 
> tel: +41 44 466 60 00
> fax: +41 44 466 60 10
> mail: mailto:florian.engelmann at everyware.ch
> web: http://www.everyware.ch
> 

-- 

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelmann at everyware.ch
web: http://www.everyware.ch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5210 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20181025/7cfb7e58/attachment.bin>


More information about the OpenStack-operators mailing list