Open Stack

On Wed, May 31, 2017, at 07:14 AM, Wang Shilong wrote:
> Ping.
> 
> Anyone has some good ideas that i could debug further?
> 
> Thanks,
> Shilong
> 
> 
> _______________________________________
> From: Wang Shilong
> Sent: Wednesday, May 24, 2017 19:37
> To: openstack-infra ‎[openstack-infra at lists.openstack.org]‎
> Cc: fungi at yuggoth.org
> Subject: ERROR gear.Server: Exception in poll loop
> 
> Hi Guys,
> 
>       I hit an error in My CI setup, somehow Gear could not call
> Jenkins jobs. zuul could watch gerrit events fine.
> 
> And I tried this 'root at r52:~/gearman-plugin-client# python gear_client.py
> -s localhost --function=build:noop-check-communication'  it could work.
> 
> But i hit gear errors
> $ cat /var/log/zuul/gearman-server.log
> 2017-05-24 20:22:13,043 ERROR gear.Server: Exception in poll loop:
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/gear/__init__.py", line
>   2903, in _doPollLoop
>     self.log.exception("Exception in poll loop:")
>   File "/usr/local/lib/python2.7/dist-packages/gear/__init__.py", line
>   2916, in _pollLoop
>     for fd, event in ret:
> IOError: [Errno 4] Interrupted system call
> 2017-05-24 20:22:14,081 ERROR gear.Server: Exception in poll loop:
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/gear/__init__.py", line
>   2903, in _doPollLoop
>     self.log.exception("Exception in poll loop:")
>   File "/usr/local/lib/python2.7/dist-packages/gear/__init__.py", line
>   2916, in _pollLoop
>     for fd, event in ret:
> IOError: [Errno 4] Interrupted system call
> 
> Zuul logs like following:
> 2017-05-24 11:42:41,857 ERROR gear.Client.unknown: Exception in poll
> loop:
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/gear/__init__.py", line
>   830, in _doPollLoop
>     try:
>   File "/usr/local/lib/python2.7/dist-packages/gear/__init__.py", line
>   856, in _pollLoop
>     len(self.active_connections))
> error: (4, 'Interrupted system call')
> 2017-05-24 11:45:24,565 INFO zuul.Server: Starting scheduler
> 2017-05-24 11:45:53,960 INFO zuul.Scheduler: Registering foreign project:
> openstack/diskimage-builder
> 2017-05-24 11:45:54,417 INFO zuul.Scheduler: Registering foreign project:
> openstack/nova
> 2017-05-24 11:46:03,840 INFO zuul.Scheduler: Registering foreign project:
> openstack/openstack-helm
> 2017-05-24 11:46:10,344 INFO zuul.Scheduler: Registering foreign project:
> openstack/daisycloud-core
> 2017-05-24 11:46:45,066 INFO zuul.Scheduler: Registering foreign project:
> openstack/neutron-lib
> 2017-05-24 11:46:57,904 WARNING zuul.GerritEventConnector: Received
> unrecognized event type 'ref-replicated' from Gerrit.                   
> Can not get account information.
> 2017-05-24 11:46:58,045 WARNING zuul.GerritEventConnector: Received
> unrecognized event type 'ref-replicated' from Gerrit.                   
> Can not get account information.
> 2017-05-24 11:46:58,280 WARNING zuul.GerritEventConnector: Received
> unrecognized event type 'ref-replicated' from Gerrit.                   
> Can not get account information.
> 2017-05-24 11:46:58,422 WARNING zuul.GerritEventConnector: Received
> unrecognized event type 'ref-replicated' from Gerrit.                   
> Can not get account information.
> 2017-05-24 11:46:58,423 WARNING zuul.GerritEventConnector: Received
> unrecognized event type 'ref-replicated' from Gerrit.                   
> Can not get account information.
> ....
> 
> Any help is much appreciated!
> 
> Thanks,
> Shilong

The interrupted system calls are "normal" in that it is something that
happens when signals are sent to the application and the syscall is
sitting around, it gets interrupted. The gear code should handle this
(and a quick review of said code shows that the poll loop is wrapped in
an exception handler that should do this properly.

What I would check is that your Jenkins server has the gearman plugin
installed, that it can talk to the gear server running under zuul, and
that you have Jenkins slaves with labels that match up to jobs so that
they will get registered in Gearman.

To do this check your global jenkins config for the gearman plugin
settings, there should be a button to test the connection. Then make
sure the labels on slaves in jenkins match the labels on your jobs.
Finally you can run `telnet $zuul_server 4730` and enter 'status' to get
the job registrations. The fields there are job name, queued jobs,
running jobs, workers able to run the job.

If all that looks good then we'll need to dig deeper but that should
give you somewhere to start.

Hope this helps,
Clark