[kolla][octavia][openstack-ansible][tripleo] octavia in kolla ansible
Hi, In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into. As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email. I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements. Thanks, Mark [1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
On Thu, Jun 20, 2019 at 3:53 PM Mark Goddard <mark@stackhpc.com> wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
TripleO has a few Ansible roles that perform several post-deployment configurations like handling certificates (auto-generated or passed in by operator), amphora image upload to Glance, LB management network and interfaces provisioning, keypair create and setting in config file, etc. Have a look and feel free to ping the team if you need further info.
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
On Thu, Jun 20, 2019 at 4:01 PM Carlos Goncalves <cgoncalves@redhat.com> wrote:
On Thu, Jun 20, 2019 at 3:53 PM Mark Goddard <mark@stackhpc.com> wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
TripleO has a few Ansible roles that perform several post-deployment configurations like handling certificates (auto-generated or passed in by operator), amphora image upload to Glance, LB management network and interfaces provisioning, keypair create and setting in config file, etc.
Have a look and feel free to ping the team if you need further info.
Oops! Here: https://github.com/openstack/tripleo-common/tree/master/playbooks
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com> wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia. https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol... Additionally we're still leveraging some of the puppet-octavia code to manage configs/nova flavors. Thanks, -Alex
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com> wrote:
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com> wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia.
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol...
Additionally we're still leveraging some of the puppet-octavia code to manage configs/nova flavors.
Thanks for sharing Alex & Carlos - useful source material.
Thanks, -Alex
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
Hi Mark, I wanted to highlight that I wrote a detailed certificate configuration guide for Octavia here: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html This could be used to automate the certificate generation for Kolla deployments. Let me know if you have any questions about the guide or steps, Michael On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com> wrote:
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com> wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia.
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol...
Additionally we're still leveraging some of the puppet-octavia code to manage configs/nova flavors.
Thanks for sharing Alex & Carlos - useful source material.
Thanks, -Alex
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
On Thu, 20 Jun 2019 at 17:27, Michael Johnson <johnsomor@gmail.com> wrote:
Hi Mark,
I wanted to highlight that I wrote a detailed certificate configuration guide for Octavia here: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html
This could be used to automate the certificate generation for Kolla deployments.
Great, that looks useful.
Let me know if you have any questions about the guide or steps, Michael
On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com> wrote:
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com> wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia.
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol...
Additionally we're still leveraging some of the puppet-octavia code to manage configs/nova flavors.
Thanks for sharing Alex & Carlos - useful source material.
Thanks, -Alex
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
Some goodies inline below. On Thu, Jun 20, 2019 at 12:34 PM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 17:27, Michael Johnson <johnsomor@gmail.com> wrote:
Hi Mark,
I wanted to highlight that I wrote a detailed certificate configuration guide for Octavia here: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html
This could be used to automate the certificate generation for Kolla
deployments.
Great, that looks useful.
Michael was super awesome creating this after Tobias and I (and a few other folks) ran into road blocks with this. Many many thanks for that.
Let me know if you have any questions about the guide or steps, Michael
On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com>
wrote:
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com>
wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of
octavia
in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia.
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo...
https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct...
https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol...
Additionally we're still leveraging some of the puppet-octavia code
to manage configs/nova flavors.
Thanks for sharing Alex & Carlos - useful source material.
Most of that bitching in Denver was me. Thanks to John for recording it for posterity. With Octavia being the defacto LBaaS standard in Openstack we need to do a much better job deploying it. This proved to be a bit of a headache with kolla-ansible, but I don't think it's terribly far off. The main issue, as mentioned in all the reference materials you provided, is certificate generation. Kolla-ansible presently only supports a single CA and does not help operators create that in any way. Also, you really need to have two CAs. Beyond that, Octavia is extremely fussy about its certificates and getting it right can be a royal pain. During my first attempt at doing this deployment, I went in search of some project that had the certificate generation as a component and found that OSA had the functionality and documented it well. I extracted the needed bits, ran it, and came out the other side with a working set of certificates. Then, like most operators, I saw a squirrel and forgot to do anything more with it. So with the introduction out of the way, here's what I used. It was entirely derived from OSA with much love. https://github.com/emccormickva/octavia-cert-generate I also had to make a few hacks to kolla-ansible to get everything going. 1) In the octavia config template (octavia.conf.j2) I updated the config options to use the new certificates as follows: [certificates] ca_private_key = /etc/octavia/certs/private/cakey.pem ca_certificate = /etc/octavia/certs/ca_server_01.pem [haproxy_amphora] server_ca = /etc/octavia/certs/ca_server_01.pem client_cert = /etc/octavia/certs/client.pem [controller_worker] client_ca = /etc/octavia/certs/ca_01.pem 2) Update kolla's config-yml to copy over all the certs for each container with_items: - cakey.pem - ca_01.pem - ca_server_01.pem - client.pem I think I had to make a few other hacks in Queens, but all of those seem to have been addressed already (just doing a diff of master vs. my current configs). If we can incorporate certificate generation, get 2 CAs, and copy / configure them properly, I think everything will be great. Maybe others have additional asks, but this would do it for me. If I can scrap together some time, I'll see if I can get some commits together to make it happen, but that's always a dodgy proposition. I'm also always willing to review whatever or answer questions from someone else who wants to take it on. Cheers, Erik
Thanks, -Alex
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
Thanks, Mark
[1]
http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo...
[2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
Just a heads up to anybody; we had issues with certificates being broken using QEMU virtualization without full KVM. We changed to using full virt supported hardware in our test environment and it worked. Best regards On 06/23/2019 06:55 AM, Erik McCormick wrote:
Some goodies inline below.
On Thu, Jun 20, 2019 at 12:34 PM Mark Goddard <mark@stackhpc.com <mailto:mark@stackhpc.com>> wrote:
On Thu, 20 Jun 2019 at 17:27, Michael Johnson <johnsomor@gmail.com <mailto:johnsomor@gmail.com>> wrote: > > Hi Mark, > > I wanted to highlight that I wrote a detailed certificate > configuration guide for Octavia here: > https://docs.openstack.org/octavia/latest/admin/guides/certificates.html > > This could be used to automate the certificate generation for Kolla deployments.
Great, that looks useful.
Michael was super awesome creating this after Tobias and I (and a few other folks) ran into road blocks with this. Many many thanks for that.
> > Let me know if you have any questions about the guide or steps, > Michael > > On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <mark@stackhpc.com <mailto:mark@stackhpc.com>> wrote: > > > > On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com <mailto:aschultz@redhat.com>> wrote: > > > > > > > > > > > > On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com <mailto:mark@stackhpc.com>> wrote: > > >> > > >> Hi, > > >> > > >> In the recent kolla meeting [1] we discussed the usability of octavia > > >> in kolla ansible. We had feedback at the Denver summit [2] that this > > >> service is difficult to deploy and requires a number of manual steps. > > >> Certificates are one of the main headaches. It was stated that OSA [3] > > >> may have some useful code we could look into. > > > > > > > > > I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia. > > > > > > https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo... > > > https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct... > > > https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol... > > > > > > Additionally we're still leveraging some of the puppet-octavia code to manage configs/nova flavors. > > > > Thanks for sharing Alex & Carlos - useful source material. > >
Most of that bitching in Denver was me. Thanks to John for recording it for posterity. With Octavia being the defacto LBaaS standard in Openstack we need to do a much better job deploying it. This proved to be a bit of a headache with kolla-ansible, but I don't think it's terribly far off.
The main issue, as mentioned in all the reference materials you provided, is certificate generation. Kolla-ansible presently only supports a single CA and does not help operators create that in any way. Also, you really need to have two CAs. Beyond that, Octavia is extremely fussy about its certificates and getting it right can be a royal pain.
During my first attempt at doing this deployment, I went in search of some project that had the certificate generation as a component and found that OSA had the functionality and documented it well. I extracted the needed bits, ran it, and came out the other side with a working set of certificates. Then, like most operators, I saw a squirrel and forgot to do anything more with it.
So with the introduction out of the way, here's what I used. It was entirely derived from OSA with much love.
https://github.com/emccormickva/octavia-cert-generate
I also had to make a few hacks to kolla-ansible to get everything going.
1) In the octavia config template (octavia.conf.j2) I updated the config options to use the new certificates as follows:
[certificates] ca_private_key = /etc/octavia/certs/private/cakey.pem ca_certificate = /etc/octavia/certs/ca_server_01.pem
[haproxy_amphora] server_ca = /etc/octavia/certs/ca_server_01.pem client_cert = /etc/octavia/certs/client.pem
[controller_worker] client_ca = /etc/octavia/certs/ca_01.pem
2) Update kolla's config-yml to copy over all the certs for each container
with_items: - cakey.pem - ca_01.pem - ca_server_01.pem - client.pem
I think I had to make a few other hacks in Queens, but all of those seem to have been addressed already (just doing a diff of master vs. my current configs). If we can incorporate certificate generation, get 2 CAs, and copy / configure them properly, I think everything will be great. Maybe others have additional asks, but this would do it for me. If I can scrap together some time, I'll see if I can get some commits together to make it happen, but that's always a dodgy proposition. I'm also always willing to review whatever or answer questions from someone else who wants to take it on.
Cheers, Erik
> > > > > > Thanks, > > > -Alex > > > > > >> > > >> > > >> As a starting point to improving this support, I'd like to gather > > >> information from people who are using octavia in kolla ansible, and > > >> what they have had to do to make it work. Please respond to this > > >> email. > > >> > > >> I've also tagged openstack-ansible and Tripleo - if there is any > > >> useful information those teams have to share about this topic, it is > > >> most welcome. Alternatively if your support for octavia also falls > > >> short perhaps we could collaborate on improvements. > > >> > > >> Thanks, > > >> Mark > > >> > > >> [1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... > > >> [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback <https://etherpad.openstack.org/p/DEN-train-kolla-feedback> > > >> [3] https://opendev.org/openstack/openstack-ansible-os_octavia <https://opendev.org/openstack/openstack-ansible-os_octavia> > > >> > >
That's really interesting Tobias -- in not sure what that would have to do with the certificates, except maybe if it is possible you were experiencing severe clock drift with QEMU? That could cause the requests to be rejected, but it has to be pretty extreme (in the scheme of things). That said, we are definitely aware that certificates (and especially somewhat ambiguously or confusingly worded configuration options for them) are difficult right now in Octavia. We do have the guide Michael mentioned, but I personally agree that it shouldn't be nearly as difficult as it is presently. I do have some thoughts about fixing it, but I'm afraid that changing up most of the configuration options around this would be similarly frustrating to the group of people who actually do have it figured out. Possibly it'd be ok with a long deprecation period. We can definitely also help with getting kolla-ansible doing the correct thing from the outset, it's just a matter of available resources. As much as I'd love to drive that effort, I'm already actively overcommitted right now, so unfortunately the only help I can give in the VERY near future would be guidance in our IRC channel to anyone else willing to spend some time on this. --Adam On Mon, Jun 24, 2019, 16:54 Tobias Urdin <tobias.urdin@binero.se> wrote:
Just a heads up to anybody; we had issues with certificates being broken using QEMU virtualization without full KVM. We changed to using full virt supported hardware in our test environment and it worked.
Best regards
On 06/23/2019 06:55 AM, Erik McCormick wrote:
Some goodies inline below.
On Thu, Jun 20, 2019 at 12:34 PM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 17:27, Michael Johnson <johnsomor@gmail.com> wrote:
Hi Mark,
I wanted to highlight that I wrote a detailed certificate configuration guide for Octavia here:
https://docs.openstack.org/octavia/latest/admin/guides/certificates.html
This could be used to automate the certificate generation for Kolla
deployments.
Great, that looks useful.
Michael was super awesome creating this after Tobias and I (and a few other folks) ran into road blocks with this. Many many thanks for that.
Let me know if you have any questions about the guide or steps, Michael
On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com>
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com>
wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of
octavia
in kolla ansible. We had feedback at the Denver summit [2] that
wrote: this
service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia.
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo...
https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct...
https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol...
Additionally we're still leveraging some of the puppet-octavia code
to manage configs/nova flavors.
Thanks for sharing Alex & Carlos - useful source material.
Most of that bitching in Denver was me. Thanks to John for recording it for posterity. With Octavia being the defacto LBaaS standard in Openstack we need to do a much better job deploying it. This proved to be a bit of a headache with kolla-ansible, but I don't think it's terribly far off.
The main issue, as mentioned in all the reference materials you provided, is certificate generation. Kolla-ansible presently only supports a single CA and does not help operators create that in any way. Also, you really need to have two CAs. Beyond that, Octavia is extremely fussy about its certificates and getting it right can be a royal pain.
During my first attempt at doing this deployment, I went in search of some project that had the certificate generation as a component and found that OSA had the functionality and documented it well. I extracted the needed bits, ran it, and came out the other side with a working set of certificates. Then, like most operators, I saw a squirrel and forgot to do anything more with it.
So with the introduction out of the way, here's what I used. It was entirely derived from OSA with much love.
https://github.com/emccormickva/octavia-cert-generate
I also had to make a few hacks to kolla-ansible to get everything going.
1) In the octavia config template (octavia.conf.j2) I updated the config options to use the new certificates as follows:
[certificates] ca_private_key = /etc/octavia/certs/private/cakey.pem ca_certificate = /etc/octavia/certs/ca_server_01.pem
[haproxy_amphora] server_ca = /etc/octavia/certs/ca_server_01.pem client_cert = /etc/octavia/certs/client.pem
[controller_worker] client_ca = /etc/octavia/certs/ca_01.pem
2) Update kolla's config-yml to copy over all the certs for each container
with_items: - cakey.pem - ca_01.pem - ca_server_01.pem - client.pem
I think I had to make a few other hacks in Queens, but all of those seem to have been addressed already (just doing a diff of master vs. my current configs). If we can incorporate certificate generation, get 2 CAs, and copy / configure them properly, I think everything will be great. Maybe others have additional asks, but this would do it for me. If I can scrap together some time, I'll see if I can get some commits together to make it happen, but that's always a dodgy proposition. I'm also always willing to review whatever or answer questions from someone else who wants to take it on.
Cheers, Erik
Thanks, -Alex
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it
is
most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
On Sun, 23 Jun 2019 at 05:52, Erik McCormick <emccormick@cirrusseven.com> wrote:
Some goodies inline below.
On Thu, Jun 20, 2019 at 12:34 PM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 17:27, Michael Johnson <johnsomor@gmail.com> wrote:
Hi Mark,
I wanted to highlight that I wrote a detailed certificate configuration guide for Octavia here: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html
This could be used to automate the certificate generation for Kolla deployments.
Great, that looks useful.
Michael was super awesome creating this after Tobias and I (and a few other folks) ran into road blocks with this. Many many thanks for that.
Let me know if you have any questions about the guide or steps, Michael
On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com> wrote:
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com> wrote:
Hi,
In the recent kolla meeting [1] we discussed the usability of octavia in kolla ansible. We had feedback at the Denver summit [2] that this service is difficult to deploy and requires a number of manual steps. Certificates are one of the main headaches. It was stated that OSA [3] may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia.
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol...
Additionally we're still leveraging some of the puppet-octavia code to manage configs/nova flavors.
Thanks for sharing Alex & Carlos - useful source material.
Most of that bitching in Denver was me. Thanks to John for recording it for posterity. With Octavia being the defacto LBaaS standard in Openstack we need to do a much better job deploying it. This proved to be a bit of a headache with kolla-ansible, but I don't think it's terribly far off.
The main issue, as mentioned in all the reference materials you provided, is certificate generation. Kolla-ansible presently only supports a single CA and does not help operators create that in any way. Also, you really need to have two CAs. Beyond that, Octavia is extremely fussy about its certificates and getting it right can be a royal pain.
Kolla Ansible (IMO) quite nicely separates certificate generation for the API from configuration of which cert to use. So you can generate your own cert and drop it in or use the kolla-ansible certificates command. I'd like to stick to this pattern
During my first attempt at doing this deployment, I went in search of some project that had the certificate generation as a component and found that OSA had the functionality and documented it well. I extracted the needed bits, ran it, and came out the other side with a working set of certificates. Then, like most operators, I saw a squirrel and forgot to do anything more with it.
So with the introduction out of the way, here's what I used. It was entirely derived from OSA with much love.
https://github.com/emccormickva/octavia-cert-generate
I also had to make a few hacks to kolla-ansible to get everything going.
1) In the octavia config template (octavia.conf.j2) I updated the config options to use the new certificates as follows:
[certificates] ca_private_key = /etc/octavia/certs/private/cakey.pem ca_certificate = /etc/octavia/certs/ca_server_01.pem
[haproxy_amphora] server_ca = /etc/octavia/certs/ca_server_01.pem client_cert = /etc/octavia/certs/client.pem
[controller_worker] client_ca = /etc/octavia/certs/ca_01.pem
2) Update kolla's config-yml to copy over all the certs for each container
with_items: - cakey.pem - ca_01.pem - ca_server_01.pem - client.pem
I think I had to make a few other hacks in Queens, but all of those seem to have been addressed already (just doing a diff of master vs. my current configs). If we can incorporate certificate generation, get 2 CAs, and copy / configure them properly, I think everything will be great. Maybe others have additional asks, but this would do it for me. If I can scrap together some time, I'll see if I can get some commits together to make it happen, but that's always a dodgy proposition. I'm also always willing to review whatever or answer questions from someone else who wants to take it on.
Cheers, Erik
Thanks, -Alex
As a starting point to improving this support, I'd like to gather information from people who are using octavia in kolla ansible, and what they have had to do to make it work. Please respond to this email.
I've also tagged openstack-ansible and Tripleo - if there is any useful information those teams have to share about this topic, it is most welcome. Alternatively if your support for octavia also falls short perhaps we could collaborate on improvements.
Thanks, Mark
[1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback [3] https://opendev.org/openstack/openstack-ansible-os_octavia
On Tue, 25 Jun 2019 at 09:44, Mark Goddard <mark@stackhpc.com> wrote:
On Sun, 23 Jun 2019 at 05:52, Erik McCormick <emccormick@cirrusseven.com> wrote:
Some goodies inline below.
On Thu, Jun 20, 2019 at 12:34 PM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 17:27, Michael Johnson <johnsomor@gmail.com> wrote:
Hi Mark,
I wanted to highlight that I wrote a detailed certificate configuration guide for Octavia here: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html
This could be used to automate the certificate generation for Kolla deployments.
Great, that looks useful.
Michael was super awesome creating this after Tobias and I (and a few other folks) ran into road blocks with this. Many many thanks for that.
Let me know if you have any questions about the guide or steps, Michael
On Thu, Jun 20, 2019 at 7:31 AM Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 20 Jun 2019 at 15:08, Alex Schultz <aschultz@redhat.com> wrote:
On Thu, Jun 20, 2019 at 8:00 AM Mark Goddard <mark@stackhpc.com> wrote: > > Hi, > > In the recent kolla meeting [1] we discussed the usability of octavia > in kolla ansible. We had feedback at the Denver summit [2] that this > service is difficult to deploy and requires a number of manual steps. > Certificates are one of the main headaches. It was stated that OSA [3] > may have some useful code we could look into.
I second that Octavia is very painful to install. There is a requirement of a bunch of openstack cloud configurations (flavors/images/etc) that must be handled prior to actually configuring the service which means it's complex to deploy. IMHO it would have been beneficial for some of these items to actually have been rolled into the service itself (ie dynamically querying the services for flavor information rather than expecting an ID put into a configuration file). That being said, we have managed to get it integrated into tripleo but it's rather complex. It does use ansible if you want to borrow some of the concepts for os_octavia.
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deplo... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/oct... https://opendev.org/openstack/tripleo-common/src/branch/master/playbooks/rol...
Additionally we're still leveraging some of the puppet-octavia code to manage configs/nova flavors.
Thanks for sharing Alex & Carlos - useful source material.
Most of that bitching in Denver was me. Thanks to John for recording it for posterity. With Octavia being the defacto LBaaS standard in Openstack we need to do a much better job deploying it. This proved to be a bit of a headache with kolla-ansible, but I don't think it's terribly far off.
The main issue, as mentioned in all the reference materials you provided, is certificate generation. Kolla-ansible presently only supports a single CA and does not help operators create that in any way. Also, you really need to have two CAs. Beyond that, Octavia is extremely fussy about its certificates and getting it right can be a royal pain.
Kolla Ansible (IMO) quite nicely separates certificate generation for the API from configuration of which cert to use. So you can generate your own cert and drop it in or use the kolla-ansible certificates command. I'd like to stick to this pattern
Pressed send too early :( I'd like to first make it possible to drop in certs for octavia, which should just be the code you've included below. If you have time to tidy that up and push it that would be great. A second step would be to incorporate certificate generation, and your role and OSA might be a good starting point for that.
During my first attempt at doing this deployment, I went in search of some project that had the certificate generation as a component and found that OSA had the functionality and documented it well. I extracted the needed bits, ran it, and came out the other side with a working set of certificates. Then, like most operators, I saw a squirrel and forgot to do anything more with it.
So with the introduction out of the way, here's what I used. It was entirely derived from OSA with much love.
https://github.com/emccormickva/octavia-cert-generate
I also had to make a few hacks to kolla-ansible to get everything going.
1) In the octavia config template (octavia.conf.j2) I updated the config options to use the new certificates as follows:
[certificates] ca_private_key = /etc/octavia/certs/private/cakey.pem ca_certificate = /etc/octavia/certs/ca_server_01.pem
[haproxy_amphora] server_ca = /etc/octavia/certs/ca_server_01.pem client_cert = /etc/octavia/certs/client.pem
[controller_worker] client_ca = /etc/octavia/certs/ca_01.pem
2) Update kolla's config-yml to copy over all the certs for each container
with_items: - cakey.pem - ca_01.pem - ca_server_01.pem - client.pem
I think I had to make a few other hacks in Queens, but all of those seem to have been addressed already (just doing a diff of master vs. my current configs). If we can incorporate certificate generation, get 2 CAs, and copy / configure them properly, I think everything will be great. Maybe others have additional asks, but this would do it for me. If I can scrap together some time, I'll see if I can get some commits together to make it happen, but that's always a dodgy proposition. I'm also always willing to review whatever or answer questions from someone else who wants to take it on.
Cheers, Erik
Thanks, -Alex
> > > As a starting point to improving this support, I'd like to gather > information from people who are using octavia in kolla ansible, and > what they have had to do to make it work. Please respond to this > email. > > I've also tagged openstack-ansible and Tripleo - if there is any > useful information those teams have to share about this topic, it is > most welcome. Alternatively if your support for octavia also falls > short perhaps we could collaborate on improvements. > > Thanks, > Mark > > [1] http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.lo... > [2] https://etherpad.openstack.org/p/DEN-train-kolla-feedback > [3] https://opendev.org/openstack/openstack-ansible-os_octavia >
participants (7)
-
Adam Harwell
-
Alex Schultz
-
Carlos Goncalves
-
Erik McCormick
-
Mark Goddard
-
Michael Johnson
-
Tobias Urdin