Open Stack

Thanks Mikhail for the hints,
As a first step we decided to add "unknown project from 347574,1" into dependency chain ....

Michal

-----Original Message-----
From: Mikhail Medvedev [mailto:mihailmed at gmail.com] 
Sent: Wednesday, July 27, 2016 5:13 PM
To: Ptacek, MichalX <michalx.ptacek at intel.com>
Cc: openstack-infra at lists.openstack.org; Znoinski, Waldemar <waldemar.znoinski at intel.com>
Subject: Re: [OpenStack-Infra] zuul stopped working and keeps logging AtrributeError

On Wed, Jul 27, 2016 at 8:23 AM, Ptacek, MichalX <michalx.ptacek at intel.com> wrote:
> Hi all.
>
>
>
> we are managing some 3rd party CI’s and have repeatedly encountered 
> some strange issue with our zuul service,
>
> from debug.log I can see that zuul becomes not happy with one patchset
>
>
>
> 2016-07-27 00:05:23,545 INFO zuul.IndependentPipelineManager: Change 
> <Change
> 0x7fa052eea250 347574,1> depends on changes []
>
> 2016-07-27 00:05:23,546 ERROR zuul.Scheduler: Exception in run handler:
>
> Traceback (most recent call last):
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 971, in run
>
>     while pipeline.manager.processQueue():
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 1607, in processQueue
>
>     item, nnfi)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 1565, in _processOneItem
>
>     ready = self.prepareRef(item)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 1465, in prepareRef
>
>     merger_items = map(self._makeMergerItem, all_items)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 1437, in _makeMergerItem
>
>     return dict(project=item.change.project.name,
>
> AttributeError: 'NoneType' object has no attribute 'name'
>
> 2016-07-27 00:05:23,546 DEBUG zuul.Scheduler: Run handler sleeping
>
> 2016-07-27 00:05:23,546 DEBUG zuul.Scheduler: Run handler awake
>
>
>
> and after some time it stopped logging everything except those 
> tracebacks
>
>
>
> Traceback (most recent call last):
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 971, in run
>
>     while pipeline.manager.processQueue():
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 1607, in processQueue
>
>     item, nnfi)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 1565, in _processOneItem
>
>     ready = self.prepareRef(item)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", 
> line 1471, in prepareRef
>
>     url = self.pipeline.source.getGitUrl(item.change.project)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/source/gerrit.py", 
> line 354, in getGitUrl
>
>     return self.connection.getGitUrl(project)
>
>   File 
> "/usr/local/lib/python2.7/dist-packages/zuul/connection/gerrit.py",
> line 432, in getGitUrl
>
>     project.name)
>
> AttributeError: 'NoneType' object has no attribute 'name'
>
>
>
> and it has to be restarted as it’s not processing any patchsets at 
> all, Is it a known issue  ?
>
>

It was happening for a long time (a year?) in our third-party CI.
There were attempts to fix what seemed to be a problem with the same signature, but I am not sure it worked, did not have time to upgrade yet. We are currently on 2.1.1.dev122. I noticed that it more often happens when you change configuration and zuul reloads it, and often before that there are patches in zuul queue that have either 'unknown project' in dependency chain, or failed to get passed to jenkins.
Another problem I see is that zuul @dev122 leaks memory (or it might not be a leak, but it does consume all 8G on our VM in about 10 days).
In any case, I restart zuul weekly to deal with memory issue and purge hanging patches, and then watch it when we change any configuration to deal with the problem you described.

>
> zuul-merger keeps logging “normal debugging data” into 
> merger-debug.log but after some time it stopped logging with following
>
> and became active just when restarted as well …
>
>
>
> 2016-07-27 04:58:34,565 DEBUG zuul.Repo: Resetting repository 
> /var/lib/zuul/git/openstack/astara
>
> 2016-07-27 04:58:34,566 DEBUG zuul.Repo: Updating repository 
> /var/lib/zuul/git/openstack/astara
>
> 2016-07-27 04:58:36,942 DEBUG zuul.Repo: Checking out
> 6cb543ac99cb1a0a0e5a3e0e5974a997a11ae508
>
> 2016-07-27 04:58:39,066 DEBUG zuul.Repo: Merging 
> refs/changes/56/345756/1 with args ['-s', 'resolve', 'FETCH_HEAD']
>
> 2016-07-27 04:58:39,134 DEBUG zuul.Repo: CreateZuulRef
> master/Zc3c321436f5c42bf9c073d1978c5cd03 at 
> c59a924b3251297d3fc5f9404731160c71f6f3da on <git.Repo 
> "/var/lib/zuul/git/openstack/astara/.git">
>
>
>
>
>
>
>
> we are using Zuul version: 2.1.1.dev149
>
>
>
>
>
> Thanks for any hint/suggestion
>
> Michal
>
>
>
>
>
>
>
>
>
>
>
> --------------------------------------------------------------
> Intel Research and Development Ireland Limited Registered in Ireland 
> Registered Office: Collinstown Industrial Park, Leixlip, County 
> Kildare Registered Number: 308263
>
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>

---
Mikhail Medvedev
IBM
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.