<div><div dir="auto">Doing the same documentation process here as well (except that I’m using kolla). The only annoying thing is the doc submission process :-/.</div></div><div><br><div class="gmail_quote"><div dir="ltr">Le lun. 22 oct. 2018 à 16:50, Erik McCormick <<a href="mailto:emccormick@cirrusseven.com">emccormick@cirrusseven.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Oops, dropped Operators. Can't wait until it's all one list...<br>
On Mon, Oct 22, 2018 at 10:44 AM Erik McCormick<br>
<<a href="mailto:emccormick@cirrusseven.com" target="_blank">emccormick@cirrusseven.com</a>> wrote:<br>
><br>
> On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin <<a href="mailto:tobias.urdin@binero.se" target="_blank">tobias.urdin@binero.se</a>> wrote:<br>
> ><br>
> > Hello,<br>
> ><br>
> > I've been having a lot of issues with SSL certificates myself, on my<br>
> > second trip now trying to get it working.<br>
> ><br>
> > Before I spent a lot of time walking through every line in the DevStack<br>
> > plugin and fixing my config options, used the generate<br>
> > script [1] and still it didn't work.<br>
> ><br>
> > When I got the "invalid padding" issue it was because of the DN I used<br>
> > for the CA and the certificate IIRC.<br>
> ><br>
> > > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING<br>
> > octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect<br>
> > to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa<br>
> > routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),<br>
> > ('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),<br>
> > ('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)<br>
> > > 19:47 < tobias-urdin> after a quick google "The problem was that my<br>
> > CA DN was the same as the certificate DN."<br>
> ><br>
> > IIRC I think that solved it, but then again I wouldn't remember fully<br>
> > since I've been at so many different angles by now.<br>
> ><br>
> > Here is my IRC logs history from the #openstack-lbaas channel, perhaps<br>
> > it can help you out<br>
> > <a href="http://paste.openstack.org/show/732575/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/732575/</a><br>
> ><br>
><br>
> Tobias, I owe you a beer. This was precisely the issue. I'm deploying<br>
> Octavia with kolla-ansible. It only deploys a single CA. After hacking<br>
> the templates and playbook to incorporate a separate server CA, the<br>
> amphorae now load and provision the required namespace. I'm adding a<br>
> kolla tag to the subject of this in hopes that someone might want to<br>
> take on changing this behavior in the project. Hopefully after I get<br>
> through Upstream Institute in Berlin I'll be able to do it myself if<br>
> nobody else wants to do it.<br>
><br>
> For certificate generation, I extracted the contents of<br>
> octavia_certs_install.yml (which sets up the directory structure,<br>
> openssl.cnf, and the client CA), and octavia_certs.yml (which creates<br>
> the server CA and the client certificate) and mashed them into a<br>
> separate playbook just for this purpose. At the end I get:<br>
><br>
> ca_01.pem - Client CA Certificate<br>
> ca_01.key - Client CA Key<br>
> ca_server_01.pem - Server CA Certificate<br>
> cakey.pem - Server CA Key<br>
> client.pem - Concatenated Client Key and Certificate<br>
><br>
> If it would help to have the playbook, I can stick it up on github<br>
> with a huge "This is a hack" disclaimer on it.<br>
><br>
> > -----<br>
> ><br>
> > Sorry for hijacking the thread but I'm stuck as well.<br>
> ><br>
> > I've in the past tried to generate the certificates with [1] but now<br>
> > moved on to using the openstack-ansible way of generating them [2]<br>
> > with some modifications.<br>
> ><br>
> > Right now I'm just getting: Could not connect to instance. Retrying.:<br>
> > SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)<br>
> > from the amphoras, haven't got any further but I've eliminated a lot of<br>
> > stuck in the middle.<br>
> ><br>
> > Tried deploying Ocatavia on Ubuntu with python3 to just make sure there<br>
> > wasn't an issue with CentOS and OpenSSL versions since it tends to lag<br>
> > behind.<br>
> > Checking the amphora with openssl s_client [3] it gives the same one,<br>
> > but the verification is successful just that I don't understand what the<br>
> > bad signature<br>
> > part is about, from browsing some OpenSSL code it seems to be related to<br>
> > RSA signatures somehow.<br>
> ><br>
> > 140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad<br>
> > signature:s3_clnt.c:2032:<br>
> ><br>
> > So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS<br>
> > (openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm<br>
> > back to something related<br>
> > to the certificates or the communication between the endpoints, or what<br>
> > actually responds inside the amphora (gunicorn IIUC?). Based on the<br>
> > "verify" functions actually causing that bad signature error I would<br>
> > assume it's the generated certificate that the amphora presents that is<br>
> > causing it.<br>
> ><br>
> > I'll have to continue the troubleshooting to the inside of the amphora,<br>
> > I've used the test-only amphora image before but have now built my own<br>
> > one that is<br>
> > using the amphora-agent from the actual stable branch, but same issue<br>
> > (bad signature).<br>
> ><br>
> > For verbosity this is the config options set for the certificates in<br>
> > octavia.conf and which file it was copied from [4], same here, a<br>
> > replication of what openstack-ansible does.<br>
> ><br>
> > Appreciate any feedback or help :)<br>
> ><br>
> > Best regards<br>
> > Tobias<br>
> ><br>
> > [1]<br>
> > <a href="https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh" rel="noreferrer" target="_blank">https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh</a><br>
> > [2] <a href="http://paste.openstack.org/show/732483/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/732483/</a><br>
> > [3] <a href="http://paste.openstack.org/show/732486/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/732486/</a><br>
> > [4] <a href="http://paste.openstack.org/show/732487/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/732487/</a><br>
> ><br>
> > On 10/20/2018 01:53 AM, Michael Johnson wrote:<br>
> > > Hi Erik,<br>
> > ><br>
> > > Sorry to hear you are still having certificate issues.<br>
> > ><br>
> > > Issue #2 is probably caused by issue #1. Since we hot-plug the tenant<br>
> > > network for the VIP, one of the first steps after the worker connects<br>
> > > to the amphora agent is finishing the required configuration of the<br>
> > > VIP interface inside the network namespace on the amphroa.<br>
> > ><br>
> Thanks for the hint on the workflow of this. I hadn't gotten deep<br>
> enough into the code to find that yet, but I suspected it was blocking<br>
> since the namespace never got created either. Thanks<br>
><br>
> > > If I remember correctly, you are attempting to configure Octavia with<br>
> > > the dual CA option (which is good for non-development use).<br>
> > ><br>
> > > This is what I have for notes:<br>
> > ><br>
> > > [certificates] gets the following:<br>
> > > cert_generator = local_cert_generator<br>
> > > ca_certificate = server CA's "server.pem" file<br>
> > > ca_private_key = server CA's "server.key" file<br>
> > > ca_private_key_passphrase = pass phrase for ca_private_key<br>
> > > [controller_worker]<br>
> > > client_ca = Client CA's ca_cert file<br>
> > > [haproxy_amphora]<br>
> > > client_cert = Client CA's client.pem file (I think with it's key<br>
> > > concatenated is what rm_work said the other day)<br>
> > > server_ca = Server CA's ca_cert file<br>
> > ><br>
><br>
> This is all very helpful. It's a bit difficult to know what goes where<br>
> the way the documentation is written presently. For something that's<br>
> going to be the defacto standard for loadbalancing, we as a community<br>
> need to do a better job of documenting how to set up, configure, and<br>
> manage this in production. I'm trying to capture my lessons learned<br>
> and processes as I go to help with that if I can.<br>
><br>
> -Erik<br>
><br>
> > > That said, I can probably run through this and write something up next<br>
> > > week that is more step-by-step/detailed.<br>
> > ><br>
> > > Michael<br>
> > ><br>
> > > On Fri, Oct 19, 2018 at 2:31 PM Erik McCormick<br>
> > > <<a href="mailto:emccormick@cirrusseven.com" target="_blank">emccormick@cirrusseven.com</a>> wrote:<br>
> > >> Apologies for cross-posting, but in the event that these might be<br>
> > >> worth filing as bugs, I wanted the Octavia devs to see it as well...<br>
> > >><br>
> > >> I've been wrestling with getting Octavia up and running and have<br>
> > >> become stuck on two issues. I'm hoping someone has run into these<br>
> > >> before. My google foo has come up empty.<br>
> > >><br>
> > >> Issue 1:<br>
> > >> When the Octavia controller tries to poll the amphora instance, it<br>
> > >> tries repeatedly and eventually fails. The error on the controller<br>
> > >> side is:<br>
> > >><br>
> > >> 2018-10-19 14:17:39.181 26 ERROR<br>
> > >> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection<br>
> > >> retries (currently set to 300) exhausted. The amphora is unavailable.<br>
> > >> Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries<br>
> > >> exceeded with url: /0.5/plug/vip/<a href="http://10.250.20.15" rel="noreferrer" target="_blank">10.250.20.15</a> (Caused by<br>
> > >> SSLError(SSLError("bad handshake: Error([('rsa routines',<br>
> > >> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',<br>
> > >> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding<br>
> > >> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',<br>
> > >> 'tls_process_server_certificate', 'certificate verify<br>
> > >> failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112',<br>
> > >> port=9443): Max retries exceeded with url: /0.5/plug/vip/<a href="http://10.250.20.15" rel="noreferrer" target="_blank">10.250.20.15</a><br>
> > >> (Caused by SSLError(SSLError("bad handshake: Error([('rsa routines',<br>
> > >> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines',<br>
> > >> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding<br>
> > >> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines',<br>
> > >> 'tls_process_server_certificate', 'certificate verify<br>
> > >> failed')],)",),))<br>
> > >><br>
> > >> On the amphora side I see:<br>
> > >> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Error processing SSL request.<br>
> > >> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Invalid request from<br>
> > >> ip=::ffff:<a href="http://10.7.0.40" rel="noreferrer" target="_blank">10.7.0.40</a>: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake<br>
> > >> failure (_ssl.c:1754)<br>
> > >><br>
> > >> I've generated certificates both with the script in the Octavia git<br>
> > >> repo, and with the Openstack Ansible playbook. I can see that they are<br>
> > >> present in /etc/octavia/certs.<br>
> > >><br>
> > >> I'm using the Kolla (Queens) containers for the control plane so I'm<br>
> > >> sure I've satisfied all the python library constraints.<br>
> > >><br>
> > >> Issue 2:<br>
> > >> I"m not sure how it gets configured, but the tenant network interface<br>
> > >> (ens6) never comes up. I can spawn other instances on that network<br>
> > >> with no issue, and I can see that Neutron has the port attached to the<br>
> > >> instance. However, in the instance this is all I get:<br>
> > >><br>
> > >> ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a<br>
> > >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN<br>
> > >> group default qlen 1<br>
> > >> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>
> > >> inet <a href="http://127.0.0.1/8" rel="noreferrer" target="_blank">127.0.0.1/8</a> scope host lo<br>
> > >> valid_lft forever preferred_lft forever<br>
> > >> inet6 ::1/128 scope host<br>
> > >> valid_lft forever preferred_lft forever<br>
> > >> 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast<br>
> > >> state UP group default qlen 1000<br>
> > >> link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff<br>
> > >> inet <a href="http://10.7.0.112/16" rel="noreferrer" target="_blank">10.7.0.112/16</a> brd 10.7.255.255 scope global ens3<br>
> > >> valid_lft forever preferred_lft forever<br>
> > >> inet6 fe80::f816:3eff:fe30:c460/64 scope link<br>
> > >> valid_lft forever preferred_lft forever<br>
> > >> 3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group<br>
> > >> default qlen 1000<br>
> > >> link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff<br>
> > >><br>
> > >> There's no evidence of the interface anywhere else including udev rules.<br>
> > >><br>
> > >> Any help with either or both issues would be greatly appreciated.<br>
> > >><br>
> > >> Cheers,<br>
> > >> Erik<br>
> > >><br>
> > >> __________________________________________________________________________<br>
> > >> OpenStack Development Mailing List (not for usage questions)<br>
> > >> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > __________________________________________________________________________<br>
> > > OpenStack Development Mailing List (not for usage questions)<br>
> > > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > ><br>
> ><br>
> ><br>
> > __________________________________________________________________________<br>
> > OpenStack Development Mailing List (not for usage questions)<br>
> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</blockquote></div></div>