<div dir="ltr">Gilles,<div><br></div><div>5xx errors like 503 and 502/504 could always be intermittent operational issues. E.g. when you access your keystone backends through some proxy and there is a connectivity issue between the proxy and backends which disappears in 10 seconds, you do not need to rerun the puppet completely - just retry the request.</div><div><br></div><div>Regarding "<span style="font-size:12.8000001907349px">REST interfaces for all Openstack API" - this is very close to another topic that I raised ([0]) - using native Ruby application and handle the exceptions. Otherwise whenever we have an OpenStack client (generic or neutron/glance/etc. one) sending us a message like '[111] Connection refused' this message is very much determined by the framework that OpenStack is using within this release for clients. It could be `requests` or any other type of framework which sends different text message depending on its version. So it is very bothersome to write a bunch of 'if' clauses or gigantic regexps instead of handling simple Ruby exception. So I agree with you here - we need to work with the API directly. And, by the way, if you also support switching to native Ruby OpenStack API client, please feel free to support movement towards it in the thread [0]</span><br></div><div><br></div><div>Matt and Gilles,</div><div><br></div><div>Regarding puppet-healthcheck - I do not think that puppet-healtcheck handles exactly what I am mentioning here - it is not running exactly at the same time as we run the request.</div><div><br></div><div>E.g. 10 seconds ago everything was OK, then we had a temporary connectivity issue, then everything is ok again in 10 seconds. Could you please describe how puppet-healthcheck can help us solve this problem? </div><div><br></div><div>Or another example - there was an issue with keystone accessing token database when you have several keystone instances running, or there was some desync between these instances, e.g. you fetched the token at keystone #1 and then you verify it again keystone #2. Keystone #2 had some issues verifying it not due to the fact that token was bad, but due to the fact that that keystone #2 had some issues. We would get 401 error and instead of trying to rerun the puppet we would need just to handle this issue locally by retrying the request.</div><div><br></div><div>[0] <a href="http://permalink.gmane.org/gmane.comp.cloud.openstack.devel/66423">http://permalink.gmane.org/gmane.comp.cloud.openstack.devel/66423</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 15, 2015 at 12:23 PM, Gilles Dubreuil <span dir="ltr"><<a href="mailto:gilles@redhat.com" target="_blank">gilles@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
On 15/10/15 12:42, Matt Fischer wrote:<br>
><br>
><br>
> On Thu, Oct 8, 2015 at 5:38 AM, Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a><br>
</span><span class="">> <mailto:<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a>>> wrote:<br>
><br>
> Hi, folks<br>
><br>
> * Intro<br>
><br>
> Per our discussion at Meeting #54 [0] I would like to propose the<br>
> uniform approach of exception handling for all puppet-openstack<br>
> providers accessing any types of OpenStack APIs.<br>
><br>
> * Problem Description<br>
><br>
> While working on Fuel during deployment of multi-node HA-aware<br>
> environments we faced many intermittent operational issues, e.g.:<br>
><br>
> 401/403 authentication failures when we were doing scaling of<br>
> OpenStack controllers due to difference in hashing view between<br>
> keystone instances<br>
> 503/502/504 errors due to temporary connectivity issues<br>
<br>
</span>The 5xx errors are not connectivity issues:<br>
<br>
500 Internal Server Error<br>
501 Not Implemented<br>
502 Bad Gateway<br>
503 Service Unavailable<br>
504 Gateway Timeout<br>
505 HTTP Version Not Supported<br>
<br>
I believe nothing should be done to trap them.<br>
<br>
The connectivity issues are different matter (to be addressed as<br>
mentioned by Matt)<br>
<span class=""><br>
> non-idempotent operations like deletion or creation - e.g. if you<br>
> are deleting an endpoint and someone is deleting on the other node<br>
> and you get 404 - you should continue with success instead of<br>
> failing. 409 Conflict error should also signal us to re-fetch<br>
> resource parameters and then decide what to do with them.<br>
><br>
> Obviously, it is not optimal to rerun puppet to correct such errors<br>
> when we can just handle an exception properly.<br>
><br>
> * Current State of Art<br>
><br>
> There is some exception handling, but it does not cover all the<br>
> aforementioned use cases.<br>
><br>
> * Proposed solution<br>
><br>
> Introduce a library of exception handling methods which should be<br>
> the same for all puppet openstack providers as these exceptions seem<br>
> to be generic. Then, for each of the providers we can introduce<br>
> provider-specific libraries that will inherit from this one.<br>
><br>
> Our mos-puppet team could add this into their backlog and could work<br>
> on that in upstream or downstream and propose it upstream.<br>
><br>
> What do you think on that, puppet folks?<br>
><br>
<br>
</span>The real issue is because we're dealing with openstackclient, a CLI tool<br>
and not an API. Therefore no error propagation is expected.<br>
<br>
Using REST interfaces for all Openstack API would provide all HTTP errors:<br>
<br>
Check for "HTTP Response Classes" in<br>
<a href="http://ruby-doc.org/stdlib-2.2.3/libdoc/net/http/rdoc/Net/HTTP.html" rel="noreferrer" target="_blank">http://ruby-doc.org/stdlib-2.2.3/libdoc/net/http/rdoc/Net/HTTP.html</a><br>
<span class=""><br>
<br>
> [0] <a href="http://eavesdrop.openstack.org/meetings/puppet_openstack/2015/puppet_openstack.2015-10-06-15.00.html" rel="noreferrer" target="_blank">http://eavesdrop.openstack.org/meetings/puppet_openstack/2015/puppet_openstack.2015-10-06-15.00.html</a><br>
><br>
><br>
> I think that we should look into some solutions here as I'm generally<br>
> for something we can solve once and re-use. Currently we solve some of<br>
> this at TWC by serializing our deploys and disabling puppet site wide<br>
> while we do so. This avoids the issue of Keystone on one node removing<br>
> and endpoint while the other nodes (who still have old code) keep trying<br>
> to add it back.<br>
><br>
> For connectivity issues especially after service restarts, we're using<br>
> puppet-healthcheck [0] and I'd like to discuss that more in Tokyo as an<br>
> alternative to explicit retries and delays. It's in the etherpad so<br>
> hopefully you can attend.<br>
<br>
</span>+1<br>
<br>
><br>
> [0] - <a href="https://github.com/puppet-community/puppet-healthcheck" rel="noreferrer" target="_blank">https://github.com/puppet-community/puppet-healthcheck</a><br>
<div class="HOEnZb"><div class="h5">><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Yours Faithfully,<br>Vladimir Kuklin,<br>Fuel Library Tech Lead,<br>Mirantis, Inc.<br>+7 (495) 640-49-04<br>+7 (926) 702-39-68<br>Skype kuklinvv<br>35bk3, Vorontsovskaya Str.<br>Moscow, Russia,<br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.com</a><br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.ru</a><br><a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a></div></div></div></div>
</div>