<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello,</p>
<p>Same here, I will continue looking at this aswell.</p>
<p>Would be great if we could get some input from a neutron dev with good insight into the project.</p>
<p><br>
</p>
<p>Can we backtrace the timed out message from where it's thrown/returned.<br>
</p>
<p>Error: Request to <a href="https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c" rel="noreferrer" target="_blank">
https://127.0.0.1:9696/v2.0/<wbr>floatingips/2e3fa334-d6ac-<wbr>443c-b5ba-eeb521d6324c</a> timed out', u'code': 400}</p>
<p>I'm interested why we would get 400 code back, the floating ip operations should be async right so this would be the response from the API layer with information from the database that should</p>
<p>return more information.</p>
<p><br>
</p>
<p>Best regards</p>
<br>
<div class="moz-cite-prefix">On 11/14/2017 07:41 AM, Arnaud MORIN wrote:<br>
</div>
<blockquote type="cite" cite="mid:CALn_SgaSumuuTyk+YUe=7r08TYbugDu-9ibFwizzcUV0XZn=fg@mail.gmail.com">
<div dir="auto">Hey,
<div dir="auto">Do you know if the bug appears on a specific Ubuntu / openstack version?</div>
<div dir="auto">As far as I remember it was not related to the puppet branch? I mean the bug is existing on master but also on newton puppet branches, right?</div>
<div dir="auto"><br>
</div>
<div dir="auto">We are using Ubuntu in my company so we would love to see that continue ;)</div>
<div dir="auto">I'll try to take a look again.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Cheers.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">Le 14 nov. 2017 00:00, "Mohammed Naser" <<a href="mailto:mnaser@vexxhost.com" moz-do-not-send="true">mnaser@vexxhost.com</a>> a écrit :<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
Hope that everyone had safe travels and enjoyed their time at Sydney<br>
(and those who weren't there enjoyed a bit of quiet time!). I'm just<br>
sending this email if anyone had a chance to look more into this (or<br>
perhaps we can get some help if there are any Canonical folks on the<br>
list?)<br>
<br>
I would be really sad if we had to drop Ubuntu/Debian support because<br>
we cannot test it. I think there are a lot of users out there using<br>
it! I'd be more than happy to provide any assistance/information in<br>
troubleshooting this.<br>
<br>
Thank you,<br>
Mohammed<br>
<br>
On Thu, Nov 2, 2017 at 1:10 PM, Mohammed Naser <<a href="mailto:mnaser@vexxhost.com" moz-do-not-send="true">mnaser@vexxhost.com</a>> wrote:<br>
> On Thu, Nov 2, 2017 at 1:02 PM, Tobias Urdin <<a href="mailto:tobias.urdin@crystone.com" moz-do-not-send="true">tobias.urdin@crystone.com</a>> wrote:<br>
>> I've been staring at this for almost an hour now going through all the logs<br>
>> and I can't really pin a point from<br>
>><br>
>> where that error message is generated. Cannot find any references for the<br>
>> timed out message that the API returns or the unable to associate part.<br>
>><br>
>><br>
>> What I'm currently staring at is why would the instance fixed ip 172.24.5.17<br>
>> be references as a network:router_gateway port in the OVS agent logs.<br>
>><br>
>> 2017-10-29 23:19:27.591 11856 INFO<br>
>> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br>
>> [req-7274c6f7-18ef-420d-ad5a-<wbr>9d0fe4eb35c6 - - - - -] Port<br>
>> 053a625c-4227-41fb-9a26-<wbr>45eda7bd2055 updated. Details: {'profile': {},<br>
>> 'network_qos_policy_id': None, 'qos_policy_id': None,<br>
>> 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id':<br>
>> 'f9647756-41ad-4ec5-af49-<wbr>daefe410815e', 'segmentation_id': None,<br>
>> 'fixed_ips': [{'subnet_id': 'a31c7115-1f3e-4220-8bdb-<wbr>981b6df2e18c',<br>
>> 'ip_address': '172.24.5.17'}], 'device_owner': u'network:router_gateway',<br>
>> 'physical_network': u'external', 'mac_address': 'fa:16:3e:3b:ec:c3',<br>
>> 'device': u'053a625c-4227-41fb-9a26-<wbr>45eda7bd2055', 'port_security_enabled':<br>
>> False, 'port_id': '053a625c-4227-41fb-9a26-<wbr>45eda7bd2055', 'network_type':<br>
>> u'flat', 'security_groups': []}<br>
>><br>
>><br>
>> Anybody else seen anything interesting?<br>
><br>
> Hi Tobias,<br>
><br>
> Thanks for looking out. I've spent a lot of time and I haven't<br>
> successfully identified much, I can add the following<br>
><br>
> - This issue is intermittent in CI<br>
> - It does *not* happen on any specific providers, I've seen it fail on<br>
> both OVH and Rackspace.<br>
> - Not *all* floating iP assignments fail, if you look at the logs, you<br>
> can see it attach a few successfully before failing<br>
><br>
> But yeah, I'm still quite at a loss and not having this coverage isn't fun.<br>
><br>
>><br>
>><br>
>> On 10/30/2017 11:08 PM, Brian Haley wrote:<br>
>><br>
>> On 10/30/2017 05:46 PM, Matthew Treinish wrote:<br>
>><br>
>> From a quick glance at the logs my guess is that the issue is related<br>
>> to this stack trace in the l3 agent logs:<br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-l3-agent.txt.gz?level=TRACE#_2017-10-29_23_11_15_146" rel="noreferrer" target="_blank" moz-do-not-send="true">
http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/logs/neutron/<wbr>neutron-l3-agent.txt.gz?level=<wbr>TRACE#_2017-10-29_23_11_15_146</a><br>
>><br>
>> I'm not sure what's causing it to complain there. But, I'm on a plane<br>
>> right now (which is why this is a top post, sorry) so I can't really dig<br>
>> much more than that. I'll try to take a deeper look at things later when<br>
>> I'm on solid ground. (hopefully someone will beat me to it by then though)<br>
>><br>
>> I don't think that l3-agent trace is it, as the failure is coming from<br>
>> the API. It's actually a trace that's happening due to the async nature<br>
>> of how the agent runs arping, fix is<br>
>> <a href="https://review.openstack.org/#/c/507914/" rel="noreferrer" target="_blank" moz-do-not-send="true">
https://review.openstack.org/#<wbr>/c/507914/</a> but it only removes the log noise.<br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-server.txt.gz" rel="noreferrer" target="_blank" moz-do-not-send="true">
http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/logs/neutron/<wbr>neutron-server.txt.gz</a><br>
>> has some tracebacks that look config related, possible missing DB table?<br>
>> But I haven't looked very closely.<br>
>><br>
>> -Brian<br>
>><br>
>><br>
>> On October 31, 2017 1:25:55 AM GMT+04:00, Mohammed Naser<br>
>> <<a href="mailto:mnaser@vexxhost.com" moz-do-not-send="true">mnaser@vexxhost.com</a>> wrote:<br>
>><br>
>> Hi everyone,<br>
>><br>
>> I'm looking for some help regarding an issue that we're having with<br>
>> the Puppet OpenStack modules, we've had very inconsistent failures in<br>
>> the Xenial with the following error:<br>
>><br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/" rel="noreferrer" target="_blank" moz-do-not-send="true">
http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/</a><br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/testr_results.html.gz" rel="noreferrer" target="_blank" moz-do-not-send="true">
http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/logs/testr_<wbr>results.html.gz</a><br>
>> Details: {u'message': u'Unable to associate floating IP<br>
>> 172.24.5.17 <<a href="http://172.24.5.17" rel="noreferrer" target="_blank" moz-do-not-send="true">http://172.24.5.17</a>> to fixed IP10.100.0.8<br>
>> <<a href="http://10.100.0.8" rel="noreferrer" target="_blank" moz-do-not-send="true">http://10.100.0.8</a>> for instance<br>
>> d265626a-77c1-4d2f-8260-<wbr>46abe548293e. Error: Request to<br>
>><br>
>> <a href="https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c" rel="noreferrer" target="_blank" moz-do-not-send="true">
https://127.0.0.1:9696/v2.0/<wbr>floatingips/2e3fa334-d6ac-<wbr>443c-b5ba-eeb521d6324c</a><br>
>> timed out', u'code': 400}<br>
>><br>
>> At this point, we're at a bit of a loss. I've tried my best in order<br>
>> to find the root cause however we have not been able to do this. It<br>
>> was persistent enough that we elected to go non-voting for our Xenial<br>
>> gates, however, with no fix ahead of us, I feel like this is a waste<br>
>> of resources and we need to either fix this or drop CI for Ubuntu. We<br>
>> don't deploy on Ubuntu and most of the developers working on the<br>
>> project don't either at this point, so we need a bit of resources.<br>
>><br>
>> If you're a user of Puppet on Xenial, we need your help! Without any<br>
>> resources going to fix this, we'd unfortunately have to drop support<br>
>> for Ubuntu because of the lack of resources to maintain it (or<br>
>> assistance). We (Puppet OpenStack team) would be more than happy to<br>
>> work together to fix this so pop-in at #puppet-openstack or reply to<br>
>> this email and let's get this issue fixed.<br>
>><br>
>> Thanks,<br>
>> Mohammed<br>
>><br>
>> ------------------------------<wbr>------------------------------<wbr>------------<br>
>><br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe:<br>
>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank" moz-do-not-send="true">
OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>______________________________<wbr>______________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank" moz-do-not-send="true">
OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" moz-do-not-send="true">
http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
>> ______________________________<wbr>______________________________<wbr>______________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank" moz-do-not-send="true">
OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" moz-do-not-send="true">
http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>______________________________<wbr>______________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank" moz-do-not-send="true">
OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" moz-do-not-send="true">
http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank" moz-do-not-send="true">
OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
<br>
</body>
</html>