<div dir="auto">Hey,<div dir="auto">Do you know if the bug appears on a specific Ubuntu / openstack version?</div><div dir="auto">As far as I remember it was not related to the puppet branch? I mean the bug is existing on master but also on newton puppet branches, right?</div><div dir="auto"><br></div><div dir="auto">We are using Ubuntu in my company so we would love to see that continue ;)</div><div dir="auto">I'll try to take a look again.</div><div dir="auto"><br></div><div dir="auto">Cheers.</div></div><div class="gmail_extra"><br><div class="gmail_quote">Le 14 nov. 2017 00:00, "Mohammed Naser" <<a href="mailto:mnaser@vexxhost.com">mnaser@vexxhost.com</a>> a écrit :<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Hope that everyone had safe travels and enjoyed their time at Sydney<br>
(and those who weren't there enjoyed a bit of quiet time!). I'm just<br>
sending this email if anyone had a chance to look more into this (or<br>
perhaps we can get some help if there are any Canonical folks on the<br>
list?)<br>
<br>
I would be really sad if we had to drop Ubuntu/Debian support because<br>
we cannot test it. I think there are a lot of users out there using<br>
it! I'd be more than happy to provide any assistance/information in<br>
troubleshooting this.<br>
<br>
Thank you,<br>
Mohammed<br>
<br>
On Thu, Nov 2, 2017 at 1:10 PM, Mohammed Naser <<a href="mailto:mnaser@vexxhost.com">mnaser@vexxhost.com</a>> wrote:<br>
> On Thu, Nov 2, 2017 at 1:02 PM, Tobias Urdin <<a href="mailto:tobias.urdin@crystone.com">tobias.urdin@crystone.com</a>> wrote:<br>
>> I've been staring at this for almost an hour now going through all the logs<br>
>> and I can't really pin a point from<br>
>><br>
>> where that error message is generated. Cannot find any references for the<br>
>> timed out message that the API returns or the unable to associate part.<br>
>><br>
>><br>
>> What I'm currently staring at is why would the instance fixed ip 172.24.5.17<br>
>> be references as a network:router_gateway port in the OVS agent logs.<br>
>><br>
>> 2017-10-29 23:19:27.591 11856 INFO<br>
>> neutron.plugins.ml2.drivers.<wbr>openvswitch.agent.ovs_neutron_<wbr>agent<br>
>> [req-7274c6f7-18ef-420d-ad5a-<wbr>9d0fe4eb35c6 - - - - -] Port<br>
>> 053a625c-4227-41fb-9a26-<wbr>45eda7bd2055 updated. Details: {'profile': {},<br>
>> 'network_qos_policy_id': None, 'qos_policy_id': None,<br>
>> 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id':<br>
>> 'f9647756-41ad-4ec5-af49-<wbr>daefe410815e', 'segmentation_id': None,<br>
>> 'fixed_ips': [{'subnet_id': 'a31c7115-1f3e-4220-8bdb-<wbr>981b6df2e18c',<br>
>> 'ip_address': '172.24.5.17'}], 'device_owner': u'network:router_gateway',<br>
>> 'physical_network': u'external', 'mac_address': 'fa:16:3e:3b:ec:c3',<br>
>> 'device': u'053a625c-4227-41fb-9a26-<wbr>45eda7bd2055', 'port_security_enabled':<br>
>> False, 'port_id': '053a625c-4227-41fb-9a26-<wbr>45eda7bd2055', 'network_type':<br>
>> u'flat', 'security_groups': []}<br>
>><br>
>><br>
>> Anybody else seen anything interesting?<br>
><br>
> Hi Tobias,<br>
><br>
> Thanks for looking out. I've spent a lot of time and I haven't<br>
> successfully identified much, I can add the following<br>
><br>
> - This issue is intermittent in CI<br>
> - It does *not* happen on any specific providers, I've seen it fail on<br>
> both OVH and Rackspace.<br>
> - Not *all* floating iP assignments fail, if you look at the logs, you<br>
> can see it attach a few successfully before failing<br>
><br>
> But yeah, I'm still quite at a loss and not having this coverage isn't fun.<br>
><br>
>><br>
>><br>
>> On 10/30/2017 11:08 PM, Brian Haley wrote:<br>
>><br>
>> On 10/30/2017 05:46 PM, Matthew Treinish wrote:<br>
>><br>
>> From a quick glance at the logs my guess is that the issue is related<br>
>> to this stack trace in the l3 agent logs:<br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-l3-agent.txt.gz?level=TRACE#_2017-10-29_23_11_15_146" rel="noreferrer" target="_blank">http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/logs/neutron/<wbr>neutron-l3-agent.txt.gz?level=<wbr>TRACE#_2017-10-29_23_11_15_146</a><br>
>><br>
>> I'm not sure what's causing it to complain there. But, I'm on a plane<br>
>> right now (which is why this is a top post, sorry) so I can't really dig<br>
>> much more than that. I'll try to take a deeper look at things later when<br>
>> I'm on solid ground. (hopefully someone will beat me to it by then though)<br>
>><br>
>> I don't think that l3-agent trace is it, as the failure is coming from<br>
>> the API. It's actually a trace that's happening due to the async nature<br>
>> of how the agent runs arping, fix is<br>
>> <a href="https://review.openstack.org/#/c/507914/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/507914/</a> but it only removes the log noise.<br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-server.txt.gz" rel="noreferrer" target="_blank">http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/logs/neutron/<wbr>neutron-server.txt.gz</a><br>
>> has some tracebacks that look config related, possible missing DB table?<br>
>> But I haven't looked very closely.<br>
>><br>
>> -Brian<br>
>><br>
>><br>
>> On October 31, 2017 1:25:55 AM GMT+04:00, Mohammed Naser<br>
>> <<a href="mailto:mnaser@vexxhost.com">mnaser@vexxhost.com</a>> wrote:<br>
>><br>
>> Hi everyone,<br>
>><br>
>> I'm looking for some help regarding an issue that we're having with<br>
>> the Puppet OpenStack modules, we've had very inconsistent failures in<br>
>> the Xenial with the following error:<br>
>><br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/" rel="noreferrer" target="_blank">http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/</a><br>
>><br>
>> <a href="http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/testr_results.html.gz" rel="noreferrer" target="_blank">http://logs.openstack.org/47/<wbr>514347/1/check/puppet-<wbr>openstack-integration-4-<wbr>scenario001-tempest-ubuntu-<wbr>xenial/ed5a657/logs/testr_<wbr>results.html.gz</a><br>
>> Details: {u'message': u'Unable to associate floating IP<br>
>> 172.24.5.17 <<a href="http://172.24.5.17" rel="noreferrer" target="_blank">http://172.24.5.17</a>> to fixed IP10.100.0.8<br>
>> <<a href="http://10.100.0.8" rel="noreferrer" target="_blank">http://10.100.0.8</a>> for instance<br>
>> d265626a-77c1-4d2f-8260-<wbr>46abe548293e. Error: Request to<br>
>><br>
>> <a href="https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c" rel="noreferrer" target="_blank">https://127.0.0.1:9696/v2.0/<wbr>floatingips/2e3fa334-d6ac-<wbr>443c-b5ba-eeb521d6324c</a><br>
>> timed out', u'code': 400}<br>
>><br>
>> At this point, we're at a bit of a loss. I've tried my best in order<br>
>> to find the root cause however we have not been able to do this. It<br>
>> was persistent enough that we elected to go non-voting for our Xenial<br>
>> gates, however, with no fix ahead of us, I feel like this is a waste<br>
>> of resources and we need to either fix this or drop CI for Ubuntu. We<br>
>> don't deploy on Ubuntu and most of the developers working on the<br>
>> project don't either at this point, so we need a bit of resources.<br>
>><br>
>> If you're a user of Puppet on Xenial, we need your help! Without any<br>
>> resources going to fix this, we'd unfortunately have to drop support<br>
>> for Ubuntu because of the lack of resources to maintain it (or<br>
>> assistance). We (Puppet OpenStack team) would be more than happy to<br>
>> work together to fix this so pop-in at #puppet-openstack or reply to<br>
>> this email and let's get this issue fixed.<br>
>><br>
>> Thanks,<br>
>> Mohammed<br>
>><br>
>> ------------------------------<wbr>------------------------------<wbr>------------<br>
>><br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe:<br>
>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>______________________________<wbr>______________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
>> ______________________________<wbr>______________________________<wbr>______________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>______________________________<wbr>______________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>><br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
</blockquote></div></div>