Open Stack

I experimentally changed zuul's configs and latest findings are below.

(Seems comments used in configs just make situation worse time to time, but are not the cause.)

I created basic layout.yaml file

[1] 1 gerrit account - Using basic layout.yaml & zuul.conf with just 1 gerrit account is fine. 
I start zuul & zuul-merger with this configuration and trigger patchset via comment. All works.

yamllint layout.yaml
layout.yaml
  7:81      error    line too long (116 > 80 characters)  (line-length)
  8:81      error    line too long (117 > 80 characters)  (line-length)
  14:81     error    line too long (84 > 80 characters)  (line-length)

[1] 1-gerrit account
/etc/zuul/layout/layout.yaml
>>
---
includes:
  - python-file: openstack_functions.py

pipelines:
  - name: sandbox
    description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
    failure-message: "Build failed. For 3rd party CI contact info: https://wiki.openstack.org/wiki/ThirdPartySystems"
    manager: IndependentPipelineManager
    source: gerrit
    trigger:
      gerrit:
        - event: comment-added
          comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck intel\s*$
    success:
      gerrit:
        verified: 1
    failure:
      gerrit:
        verified: -1

projects:
  - name: openstack-dev/ci-sandbox
    sandbox:
      - noop-check-communication
>>

/etc/zuul/zuul.conf
>>
[connection gerrit]
driver=gerrit
server=review.openstack.org
user=intel-nfv-ci
sshkey=/var/lib/zuul/ssh/id_rsa
>>


When I stop zuul, change zuul.conf config to have also 2nd gerrit account and start zuul again with the same layout (this layout do not need to use 2nd gerrit account at all),
after triggering the same patchset with the same comment,
[2] 2 gerrit accounts - Using basic layout.yaml & zuul.conf with 2 gerrit account breaks.

[2] 2 gerrit accounts
/etc/zuul/layout/layout.yaml is the same
I only added 2nd gerrit account into /etc/zuul.conf so it look like

/etc/zuul/zuul.conf
>>
[connection gerrit]
driver=gerrit
server=review.openstack.org
user=intel-nfv-ci
sshkey=/var/lib/zuul/ssh/id_rsa

[connection gerrit_networking]
driver=gerrit
server=review.openstack.org
user=intel-networking-ci
sshkey=/var/lib/zuul/ssh/id_rsa_intel-networking-ci at review.openstack.org
>>

Errors:
>>
2016-04-07 11:59:56,888 INFO zuul.Server: Starting scheduler
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager: Configured Pipeline Manager sandbox
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:   Source: <zuul.source.gerrit.GerritSource object at 0x7f04004f7490>
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:   Requirements:
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:   Events:
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:     <EventFilter types: comment-added ignore_deletes: True comments: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck intel\s*$>
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:   Projects:
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:     openstack-dev/ci-sandbox
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:       <Job noop-check-communication>
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:   On start:
2016-04-07 11:59:57,178 INFO zuul.IndependentPipelineManager:     []
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:   On success:
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f04004f7590>]
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:   On failure:
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f04004f75d0>]
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:   On merge-failure:
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f04004f75d0>]
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:   When disabled:
2016-04-07 11:59:57,179 INFO zuul.IndependentPipelineManager:     []
2016-04-07 11:59:57,180 INFO zuul.Server: Starting Webapp
2016-04-07 11:59:57,181 INFO zuul.Server: Starting RPC
2016-04-07 12:00:08,627 INFO zuul.Scheduler: Adding openstack-dev/ci-sandbox, <Change 0x7f04004e20d0 282323,1> to <Pipeline sandbox>
2016-04-07 12:00:08,628 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 959, in run
    self.process_event_queue()
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1015, in process_event_queue
    pipeline.manager.addChange(change)
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1363, in addChange
    with self.getChangeQueue(change, change_queue) as change_queue:
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1889, in getChangeQueue
    change_queue.addProject(change.project)
  File "/usr/local/lib/python2.7/dist-packages/zuul/model.py", line 332, in addProject
    names = [x.name for x in self.projects]
AttributeError: 'NoneType' object has no attribute 'name'
2016-04-07 12:00:11,708 INFO zuul.Scheduler: Adding openstack-dev/ci-sandbox, <Change 0x7f04004e20d0 282323,1> to <Pipeline sandbox>
2016-04-07 12:00:11,709 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 959, in run
    self.process_event_queue()
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1015, in process_event_queue
    pipeline.manager.addChange(change)
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1363, in addChange
    with self.getChangeQueue(change, change_queue) as change_queue:
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1886, in getChangeQueue
    if change.project not in self.pipeline.getProjects():
  File "/usr/local/lib/python2.7/dist-packages/zuul/model.py", line 111, in getProjects
    return sorted(self.job_trees.keys(), lambda a, b: cmp(a.name, b.name))
  File "/usr/local/lib/python2.7/dist-packages/zuul/model.py", line 111, in <lambda>
    return sorted(self.job_trees.keys(), lambda a, b: cmp(a.name, b.name))
AttributeError: 'NoneType' object has no attribute 'name'
>>

So, changing layout itself, adding / removing comments those are just misleading paths why I thought it doesn't work.
The cure for this is to REMOVE 2nd account from zuul.conf again and just use one
Stop and start zuul, trigger patchset and it works well as before.
>>
2016-04-07 12:18:53,967 INFO zuul.Server: Starting scheduler
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager: Configured Pipeline Manager sandbox
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:   Source: <zuul.source.gerrit.GerritSource object at 0x7f4b0027f310>
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:   Requirements:
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:   Events:
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:     <EventFilter types: comment-added ignore_deletes: True comments: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck intel\s*$>
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:   Projects:
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:     openstack-dev/ci-sandbox
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:       <Job noop-check-communication>
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:   On start:
2016-04-07 12:18:54,244 INFO zuul.IndependentPipelineManager:     []
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:   On success:
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f4b0027f410>]
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:   On failure:
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f4b0027f450>]
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:   On merge-failure:
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f4b0027f450>]
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:   When disabled:
2016-04-07 12:18:54,245 INFO zuul.IndependentPipelineManager:     []
2016-04-07 12:18:54,246 INFO zuul.Server: Starting Webapp
2016-04-07 12:18:54,247 INFO zuul.Server: Starting RPC
2016-04-07 12:19:44,084 INFO zuul.Scheduler: Adding openstack-dev/ci-sandbox, <Change 0x7f4b00090d50 282323,1> to <Pipeline sandbox>
2016-04-07 12:19:44,085 INFO zuul.IndependentPipelineManager: Change <Change 0x7f4b00090d50 282323,1> depends on changes []
2016-04-07 12:19:49,897 INFO zuul.IndependentPipelineManager: Change <Change 0x7f4b00090d50 282323,1> depends on changes []
2016-04-07 12:19:49,897 INFO zuul.Gearman: Launch job noop-check-communication (uuid: d34e7e0ba19b4cf3b8f12b8c6553e437) for change <Change 0x7f4b00090d50 282323,1> with dependent changes []
2016-04-07 12:19:49,955 INFO zuul.Gearman: Build <gear.Job 0x7f4adc728e10 handle: H:127.0.0.1:2 name: build:noop-check-communication unique: d34e7e0ba19b4cf3b8f12b8c6553e437> started
2016-04-07 12:19:51,981 INFO zuul.Gearman: Build <gear.Job 0x7f4adc728e10 handle: H:127.0.0.1:2 name: build:noop-check-communication unique: d34e7e0ba19b4cf3b8f12b8c6553e437> complete, result SUCCESS
2016-04-07 12:19:51,984 INFO zuul.IndependentPipelineManager: Reporting item <QueueItem 0x7f4adc728d50 for <Change 0x7f4b00090d50 282323,1> in sandbox>, actions: [<zuul.reporter.gerrit.GerritReporter object at 0x7f4b0027f410>]
>>

When I change zuul.conf and layout.yaml so that both use 2nd gerrit account, triggering works for 1st patchset or randomly.
[3] 2 gerrit accounts - changed both zuul.conf and layout.yaml

/etc/zuul/zuul.conf
>>
[connection gerrit]
driver=gerrit
server=review.openstack.org
user=intel-nfv-ci
sshkey=/var/lib/zuul/ssh/id_rsa

[connection gerrit_networking]
driver=gerrit
server=review.openstack.org
user=intel-networking-ci
sshkey=/var/lib/zuul/ssh/id_rsa_intel-networking-ci at review.openstack.org
>>

>>
---
includes:
  - python-file: openstack_functions.py

pipelines:
  - name: sandbox
    description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
    failure-message: "Build failed. For 3rd party CI contact info: https://wiki.openstack.org/wiki/ThirdPartySystems"
    manager: IndependentPipelineManager
    source: gerrit
    trigger:
      gerrit:
        - event: comment-added
          comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck intel\s*$
    success:
      gerrit_networking:
        verified: 1
    failure:
      gerrit_networking:
        verified: -1

projects:
  - name: openstack-dev/ci-sandbox
    sandbox:
      - noop-check-communication
>>

Errors
>>
2016-04-07 12:35:48,953 INFO zuul.Server: Starting scheduler
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager: Configured Pipeline Manager sandbox
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager:   Source: <zuul.source.gerrit.GerritSource object at 0x7f07700b6c50>
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager:   Requirements:
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager:   Events:
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager:     <EventFilter types: comment-added ignore_deletes: True comments: (?i)^(Patch Set [0-9]+:)?(
[\w\\+-]*)*(\n\n)?\s*recheck intel\s*$>
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager:   Projects:
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager:     openstack-dev/ci-sandbox
2016-04-07 12:35:49,235 INFO zuul.IndependentPipelineManager:       <Job noop-check-communication>
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:   On start:
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:     []
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:   On success:
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f07700b6d50>]
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:   On failure:
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f07700b6d90>]
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:   On merge-failure:
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:     [<zuul.reporter.gerrit.GerritReporter object at 0x7f07700b6d90>]
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:   When disabled:
2016-04-07 12:35:49,236 INFO zuul.IndependentPipelineManager:     []
2016-04-07 12:35:49,237 INFO zuul.Server: Starting Webapp
2016-04-07 12:35:49,238 INFO zuul.Server: Starting RPC
2016-04-07 12:36:08,647 INFO zuul.Scheduler: Adding openstack-dev/ci-sandbox, <Change 0x7f07605b6fd0 302263,21> to <Pipeline sandbox>
2016-04-07 12:36:11,112 INFO zuul.Scheduler: Adding openstack-dev/ci-sandbox, <Change 0x7f07605b6fd0 302263,21> to <Pipeline sandbox>
2016-04-07 12:36:11,124 INFO zuul.IndependentPipelineManager: Change <Change 0x7f07605b6fd0 302263,21> depends on changes []
2016-04-07 12:36:16,552 INFO zuul.IndependentPipelineManager: Change <Change 0x7f07605b6fd0 302263,21> depends on changes []
2016-04-07 12:36:16,553 INFO zuul.Gearman: Launch job noop-check-communication (uuid: 2b70368b6c3247c2902a469d80219e41) for change <Change 0x7f07605b6fd0 302263,21> with dependent changes []
2016-04-07 12:36:16,607 INFO zuul.Gearman: Build <gear.Job 0x7f0770094650 handle: H:127.0.0.1:2 name: build:noop-check-communication unique: 2b70368b6c3247c2902a469d80219e41> started
2016-04-07 12:36:18,605 INFO zuul.Gearman: Build <gear.Job 0x7f0770094650 handle: H:127.0.0.1:2 name: build:noop-check-communication unique: 2b70368b6c3247c2902a469d80219e41> complete, result SUCCESS
2016-04-07 12:36:18,608 INFO zuul.IndependentPipelineManager: Reporting item <QueueItem 0x7f07601f5790 for <Change 0x7f07605b6fd0 302263,21> in sandbox>, actions: [<zuul.reporter.gerrit.GerritReporter object at 0x7f07700b6d50>]
2016-04-07 12:36:23,797 INFO zuul.Scheduler: Adding openstack-dev/ci-sandbox, <Change 0x7f07605b6b90 282323,1> to <Pipeline sandbox>
2016-04-07 12:36:23,797 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 959, in run
    self.process_event_queue()
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1015, in process_event_queue
    pipeline.manager.addChange(change)
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1363, in addChange
    with self.getChangeQueue(change, change_queue) as change_queue:
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1889, in getChangeQueue
    change_queue.addProject(change.project)
  File "/usr/local/lib/python2.7/dist-packages/zuul/model.py", line 332, in addProject
    names = [x.name for x in self.projects]
AttributeError: 'NoneType' object has no attribute 'name'
2016-04-07 12:36:25,042 INFO zuul.Scheduler: Adding openstack-dev/ci-sandbox, <Change 0x7f07605b6b90 282323,1> to <Pipeline sandbox>
2016-04-07 12:36:25,042 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 959, in run
    self.process_event_queue()
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1015, in process_event_queue
    pipeline.manager.addChange(change)
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1363, in addChange
    with self.getChangeQueue(change, change_queue) as change_queue:
  File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1886, in getChangeQueue
    if change.project not in self.pipeline.getProjects():
  File "/usr/local/lib/python2.7/dist-packages/zuul/model.py", line 111, in getProjects
    return sorted(self.job_trees.keys(), lambda a, b: cmp(a.name, b.name))
  File "/usr/local/lib/python2.7/dist-packages/zuul/model.py", line 111, in <lambda>
    return sorted(self.job_trees.keys(), lambda a, b: cmp(a.name, b.name))
AttributeError: 'NoneType' object has no attribute 'name'
>>

Setup with 2 gerrit accounts worked for me few days ago, but it was time to time raising the same error. This was case [3]

Vaclav
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.