Open Stack

On Wed, Jul 27, 2016 at 8:23 AM, Ptacek, MichalX
<michalx.ptacek at intel.com> wrote:
> Hi all.
>
>
>
> we are managing some 3rd party CI’s and have repeatedly encountered some
> strange issue with our zuul service,
>
> from debug.log I can see that zuul becomes not happy with one patchset
>
>
>
> 2016-07-27 00:05:23,545 INFO zuul.IndependentPipelineManager: Change <Change
> 0x7fa052eea250 347574,1> depends on changes []
>
> 2016-07-27 00:05:23,546 ERROR zuul.Scheduler: Exception in run handler:
>
> Traceback (most recent call last):
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 971,
> in run
>
>     while pipeline.manager.processQueue():
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line
> 1607, in processQueue
>
>     item, nnfi)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line
> 1565, in _processOneItem
>
>     ready = self.prepareRef(item)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line
> 1465, in prepareRef
>
>     merger_items = map(self._makeMergerItem, all_items)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line
> 1437, in _makeMergerItem
>
>     return dict(project=item.change.project.name,
>
> AttributeError: 'NoneType' object has no attribute 'name'
>
> 2016-07-27 00:05:23,546 DEBUG zuul.Scheduler: Run handler sleeping
>
> 2016-07-27 00:05:23,546 DEBUG zuul.Scheduler: Run handler awake
>
>
>
> and after some time it stopped logging everything except those tracebacks
>
>
>
> Traceback (most recent call last):
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 971,
> in run
>
>     while pipeline.manager.processQueue():
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line
> 1607, in processQueue
>
>     item, nnfi)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line
> 1565, in _processOneItem
>
>     ready = self.prepareRef(item)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line
> 1471, in prepareRef
>
>     url = self.pipeline.source.getGitUrl(item.change.project)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/source/gerrit.py", line
> 354, in getGitUrl
>
>     return self.connection.getGitUrl(project)
>
>   File "/usr/local/lib/python2.7/dist-packages/zuul/connection/gerrit.py",
> line 432, in getGitUrl
>
>     project.name)
>
> AttributeError: 'NoneType' object has no attribute 'name'
>
>
>
> and it has to be restarted as it’s not processing any patchsets at all, Is
> it a known issue  ?
>
>

It was happening for a long time (a year?) in our third-party CI.
There were attempts to fix what seemed to be a problem with the same
signature, but I am not sure it worked, did not have time to upgrade
yet. We are currently on 2.1.1.dev122. I noticed that it more often
happens when you change configuration and zuul reloads it, and often
before that there are patches in zuul queue that have either 'unknown
project' in dependency chain, or failed to get passed to jenkins.
Another problem I see is that zuul @dev122 leaks memory (or it might
not be a leak, but it does consume all 8G on our VM in about 10 days).
In any case, I restart zuul weekly to deal with memory issue and purge
hanging patches, and then watch it when we change any configuration to
deal with the problem you described.

>
> zuul-merger keeps logging “normal debugging data” into merger-debug.log but
> after some time it stopped logging with following
>
> and became active just when restarted as well …
>
>
>
> 2016-07-27 04:58:34,565 DEBUG zuul.Repo: Resetting repository
> /var/lib/zuul/git/openstack/astara
>
> 2016-07-27 04:58:34,566 DEBUG zuul.Repo: Updating repository
> /var/lib/zuul/git/openstack/astara
>
> 2016-07-27 04:58:36,942 DEBUG zuul.Repo: Checking out
> 6cb543ac99cb1a0a0e5a3e0e5974a997a11ae508
>
> 2016-07-27 04:58:39,066 DEBUG zuul.Repo: Merging refs/changes/56/345756/1
> with args ['-s', 'resolve', 'FETCH_HEAD']
>
> 2016-07-27 04:58:39,134 DEBUG zuul.Repo: CreateZuulRef
> master/Zc3c321436f5c42bf9c073d1978c5cd03 at
> c59a924b3251297d3fc5f9404731160c71f6f3da on <git.Repo
> "/var/lib/zuul/git/openstack/astara/.git">
>
>
>
>
>
>
>
> we are using Zuul version: 2.1.1.dev149
>
>
>
>
>
> Thanks for any hint/suggestion
>
> Michal
>
>
>
>
>
>
>
>
>
>
>
> --------------------------------------------------------------
> Intel Research and Development Ireland Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
>
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by others
> is strictly prohibited. If you are not the intended recipient, please
> contact the sender and delete all copies.
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>

---
Mikhail Medvedev
IBM