<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Just a heads up to anybody; we had issues with certificates being
    broken using QEMU virtualization without full KVM.<br>
    We changed to using full virt supported hardware in our test
    environment and it worked.<br>
    <br>
    Best regards<br>
    <br>
    <div class="moz-cite-prefix">On 06/23/2019 06:55 AM, Erik McCormick
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAHUi5cPnHGF8HXxJa+t=im81-nMZg=ceog_-_zO=UPMzS6NZWA@mail.gmail.com">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <div dir="ltr">
        <div>Some goodies inline below.</div>
        <div><br>
        </div>
        <div dir="ltr">On Thu, Jun 20, 2019 at 12:34 PM Mark Goddard
          <<a href="mailto:mark@stackhpc.com" moz-do-not-send="true">mark@stackhpc.com</a>>
          wrote:<br>
        </div>
        <div class="gmail_quote">
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">
            On Thu, 20 Jun 2019 at 17:27, Michael Johnson <<a
              href="mailto:johnsomor@gmail.com" target="_blank"
              moz-do-not-send="true">johnsomor@gmail.com</a>> wrote:<br>
            ><br>
            > Hi Mark,<br>
            ><br>
            > I wanted to highlight that I wrote a detailed
            certificate<br>
            > configuration guide for Octavia here:<br>
            > <a
href="https://docs.openstack.org/octavia/latest/admin/guides/certificates.html"
              rel="noreferrer" target="_blank" moz-do-not-send="true">
https://docs.openstack.org/octavia/latest/admin/guides/certificates.html</a><br>
            ><br>
            > This could be used to automate the certificate
            generation for Kolla deployments.<br>
            <br>
            Great, that looks useful.<br>
          </blockquote>
          <div> </div>
          <div>Michael was super awesome creating this after Tobias and
            I (and a few other folks) ran into road blocks with this.
            Many many thanks for that.<br>
          </div>
          <div><br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">
            ><br>
            > Let me know if you have any questions about the guide
            or steps,<br>
            > Michael<br>
            ><br>
            > On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <<a
              href="mailto:mark@stackhpc.com" target="_blank"
              moz-do-not-send="true">mark@stackhpc.com</a>> wrote:<br>
            > ><br>
            > > On Thu, 20 Jun 2019 at 15:08, Alex Schultz <<a
              href="mailto:aschultz@redhat.com" target="_blank"
              moz-do-not-send="true">aschultz@redhat.com</a>> wrote:<br>
            > > ><br>
            > > ><br>
            > > ><br>
            > > > On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard
            <<a href="mailto:mark@stackhpc.com" target="_blank"
              moz-do-not-send="true">mark@stackhpc.com</a>> wrote:<br>
            > > >><br>
            > > >> Hi,<br>
            > > >><br>
            > > >> In the recent kolla meeting [1] we
            discussed the usability of octavia<br>
            > > >> in kolla ansible. We had feedback at the
            Denver summit [2] that this<br>
            > > >> service is difficult to deploy and
            requires a number of manual steps.<br>
            > > >> Certificates are one of the main
            headaches. It was stated that OSA [3]<br>
            > > >> may have some useful code we could look
            into.<br>
            > > ><br>
            > > ><br>
            > > > I second that Octavia is very painful to
            install. There is a requirement of a bunch of openstack
            cloud configurations (flavors/images/etc) that must be
            handled prior to actually configuring the service which
            means it's complex to deploy. IMHO it would have been
            beneficial for some of these items to actually have been
            rolled into the service itself (ie dynamically querying the
            services for flavor information rather than expecting an ID
            put into a configuration file).  That being said, we have
            managed to get it integrated into tripleo but it's rather
            complex. It does use ansible if you want to borrow some of
            the concepts for os_octavia.<br>
            > > ><br>
            > > > <a
href="https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/octavia"
              rel="noreferrer" target="_blank" moz-do-not-send="true">
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/octavia</a><br>
            > > > <a
href="https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/octavia-files.yaml"
              rel="noreferrer" target="_blank" moz-do-not-send="true">
https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/octavia-files.yaml</a><br>
            > > > <a
href="https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/roles"
              rel="noreferrer" target="_blank" moz-do-not-send="true">
https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/roles</a><br>
            > > ><br>
            > > > Additionally we're still leveraging some of
            the puppet-octavia code to manage configs/nova flavors.<br>
            > ><br>
            > > Thanks for sharing Alex & Carlos - useful
            source material.<br>
            > ><br>
          </blockquote>
          <div><br>
          </div>
          <div>Most of that bitching in Denver was me. Thanks to John
            for recording it for posterity. With Octavia being the
            defacto LBaaS standard in Openstack we need to do a much
            better job deploying it. This proved to be a bit of a
            headache with kolla-ansible, but I don't think it's terribly
            far off.</div>
          <div><br>
          </div>
          <div>The main issue, as mentioned in all the reference
            materials you provided, is certificate generation.
            Kolla-ansible presently only supports a single CA and does
            not help operators create that in any way. Also, you really
            need to have two CAs. Beyond that, Octavia is extremely
            fussy about its certificates and getting it right can be a
            royal pain.</div>
          <div><br>
          </div>
          <div>During my first attempt at doing this deployment, I went
            in search of some project that had the certificate
            generation as a component and found that OSA had the
            functionality and documented it well. I extracted the needed
            bits, ran it, and came out the other side with a working set
            of certificates. Then, like most operators, I saw a squirrel
            and forgot to do anything more with it. </div>
          <div><br>
          </div>
          <div>So with the introduction out of the way, here's what I
            used. It was entirely derived from OSA with much love. </div>
          <div><br>
          </div>
          <div><a
              href="https://github.com/emccormickva/octavia-cert-generate"
              moz-do-not-send="true">https://github.com/emccormickva/octavia-cert-generate</a> <br>
          </div>
          <div><br>
          </div>
          <div>I also had to make a few hacks to kolla-ansible to get
            everything going.</div>
          <div><br>
          </div>
          <div>1) In the octavia config template (octavia.conf.j2) I
            updated the config options to use the new certificates as
            follows:</div>
          <div><br>
          </div>
          <div>[certificates]<br>
            ca_private_key = /etc/octavia/certs/private/cakey.pem<br>
            ca_certificate = /etc/octavia/certs/ca_server_01.pem<br>
          </div>
          <div><br>
          </div>
          <div>[haproxy_amphora]<br>
            server_ca = /etc/octavia/certs/ca_server_01.pem<br>
            client_cert = /etc/octavia/certs/client.pem<br>
          </div>
          <div><br>
          </div>
          <div>[controller_worker]<br>
          </div>
          <div>client_ca = /etc/octavia/certs/ca_01.pem<br>
          </div>
          <div><br>
          </div>
          <div>2) Update kolla's config-yml to copy over all the certs
            for each container</div>
          <div><br>
          </div>
          <div>  with_items:<br>
                - cakey.pem<br>
                - ca_01.pem<br>
                - ca_server_01.pem<br>
                - client.pem<br>
          </div>
          <div><br>
          </div>
          <div>I think I had to make a few other hacks in Queens, but
            all of those seem to have been addressed already (just doing
            a diff of master vs. my current configs). If we can
            incorporate certificate generation, get 2 CAs, and copy /
            configure them properly, I think everything will be great.
            Maybe others have additional asks, but this would do it for
            me. If I can scrap together some time, I'll see if I can get
            some commits together to make it happen, but that's always a
            dodgy proposition. I'm also always willing to review
            whatever or answer questions from someone else who wants to
            take it on.</div>
          <div><br>
          </div>
          <div>Cheers,</div>
          <div>Erik</div>
          <div><br>
          </div>
          <div><br>
          </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">
            > > ><br>
            > > > Thanks,<br>
            > > > -Alex<br>
            > > ><br>
            > > >><br>
            > > >><br>
            > > >> As a starting point to improving this
            support, I'd like to gather<br>
            > > >> information from people who are using
            octavia in kolla ansible, and<br>
            > > >> what they have had to do to make it work.
            Please respond to this<br>
            > > >> email.<br>
            > > >><br>
            > > >> I've also tagged openstack-ansible and
            Tripleo - if there is any<br>
            > > >> useful information those teams have to
            share about this topic, it is<br>
            > > >> most welcome. Alternatively if your
            support for octavia also falls<br>
            > > >> short perhaps we could collaborate on
            improvements.<br>
            > > >><br>
            > > >> Thanks,<br>
            > > >> Mark<br>
            > > >><br>
            > > >> [1] <a
href="http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.log.html#l-86"
              rel="noreferrer" target="_blank" moz-do-not-send="true">
http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.log.html#l-86</a><br>
            > > >> [2] <a
              href="https://etherpad.openstack.org/p/DEN-train-kolla-feedback"
              rel="noreferrer" target="_blank" moz-do-not-send="true">
              https://etherpad.openstack.org/p/DEN-train-kolla-feedback</a><br>
            > > >> [3] <a
              href="https://opendev.org/openstack/openstack-ansible-os_octavia"
              rel="noreferrer" target="_blank" moz-do-not-send="true">
              https://opendev.org/openstack/openstack-ansible-os_octavia</a><br>
            > > >><br>
            > ><br>
            <br>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>