[openstack-dev] [CI] How to set a proxy for zuul.

Abhishek Shrivastava abhishek at cloudbyte.com
Mon Jul 20 06:09:06 UTC 2015 

Hi Tang,



On Mon, Jul 20, 2015 at 11:18 AM, Tang Chen <tangchen at cn.fujitsu.com> wrote:

>  Hi Asselin, Abhishek,
>
> Thanks for the reply. :)
>
>
> On 07/19/2015 02:41 AM, Asselin, Ramy wrote:
>
>  HI Abhi & Tang,
>
>
>
> Sorry I missed this thread. Let me know if you’ve resolved your issues.
>
>
>
> My repo is undergoing migrations to reuse components in
> openstack-infra/puppet-openstackci.
>
>
>
> For single-use-nodes, the file you need has been removed here [1]: But I
> see now that it is still needed, or a different function is needed based on
> this version used by infra: [2]. I will explore a solution.
>
>
>
> A couple other notes, please use ci-sandbox  [3] instead of sandbox.
>
>
> OK.
>
>
>
> Zuul use behind a proxy: seems you got past this? Could you share your
> solution?
>
>
> The root cause is that zuul uses a library named paramiko to create
> connection
> with low level socket APIs. And it doesn't provide proxy functionary.
>
> I tried to use tools like proxychain, to redirect the zuul connection to
> go through
> my proxy. But it doesn't work. If I use proxychain to run zuul service, it
> doesn't
> output anything to the log file, and the service will die soon.
>
> I think, there are two solutions:
> 1. Add proxy functionality to paramiko.
> 2. Add proxy functionality to zuul. That means, maybe,
>     zuul does not use paramiko to create connection, but by itself.
>
> Solution 1 is much simpler, so for now, I just modified the source code of
> paramiko.
>
> I'm using python-socksipy package, and modified
> /usr/local/lib/python2.7/dist-packages/paramiko/client.py like this:
>
> diff --git a/client.py b/client.py
> index 15ff696..d7225ed 100644
> --- a/client.py
> +++ b/client.py
> @@ -24,6 +24,7 @@ from binascii import hexlify
>  import getpass
>  import os
>  import socket
> +import socks
>  import warnings
>  import pdb
>
> @@ -235,6 +236,7 @@ class SSHClient (ClosingContextManager):
>              ``gss_deleg_creds`` and ``gss_host`` arguments.
>          """
>
> +       """
>          if not sock:
>              for (family, socktype, proto, canonname, sockaddr) in
> socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM):
>                  if socktype == socket.SOCK_STREAM:
> @@ -251,6 +253,13 @@ class SSHClient (ClosingContextManager):
>                  except:
>                      pass
>              retry_on_signal(lambda: sock.connect(addr))
> +       """
> +
> +       if not sock:
> +               sock = socks.socksocket()
> +               sock.setproxy(socks.PROXY_TYPE_SOCKS5, "MY_PROXY_IP",
> MY_PROXY_PORT, username="XXX", password="XXX")
>
>                   # This is review.openstack.org
> +               addr = ('104.130.159.134', 29418)
> +               retry_on_signal(lambda: sock.connect(addr))
>
>          t = self._transport = Transport(sock, gss_kex=gss_kex,
> gss_deleg_creds=gss_deleg_creds)
>          t.use_compression(compress=compress)
>
>
> ​Quite a nice explanation +1 for all the things you have done. Just one
more thing to ask, have you made your slave node static?​



> Of course, this is just a draft. It is only for my case, not for all.
>
>
> BTW, I'm now working on Fujitsu CI System, and really want to join into
> the development of openstack-infra.
> I think the proxy functionality is necessary for many companies. So if you
> are planing to add the proxy support,
> I think I can help.
>
> Thanks. :)
>
>
>
> Also, feel free to join 3rd party ci IRC  meetings on freenode [4]. It’s
> a great place to ask questions and meet others setting up or maintaining
> these systems.
>
>
>
> Thanks,
>
> Ramy
>
> IRC: asselin
>
>
>
> [1]
> https://github.com/rasselin/os-ext-testing/commit/dafe822be7813522a6c7361993169da20b37ffb7
>
> [2]
> https://github.com/openstack-infra/project-config/blob/master/zuul/openstack_functions.py
>
> [3] http://git.openstack.org/cgit/openstack-dev/ci-sandbox/
>
> [4] http://eavesdrop.openstack.org/#Third_Party_Meeting
>
>
>
>
>
>
>
> *From:* Abhishek Shrivastava [mailto:abhishek at cloudbyte.com
> <abhishek at cloudbyte.com>]
> *Sent:* Monday, July 13, 2015 11:51 PM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [CI] How to set a proxy for zuul.
>
>
>
> Also if you want to change it you will need to talk with Asselin Ramy who
> is the owner of the repo you followed.
>
>
>
> On Tue, Jul 14, 2015 at 12:18 PM, Abhishek Shrivastava <
> abhishek at cloudbyte.com> wrote:
>
>  ​Basically it is not required, and if you see /etc/jenkins_jobs/config
> folder you will find one dsvm-cinder-tempest.yaml which is to be used
> basically not examples.yaml. So its not an issue.​
>
>
>
> On Tue, Jul 14, 2015 at 12:07 PM, Tang Chen <tangchen at cn.fujitsu.com>
> wrote:
>
>
>
> On 07/14/2015 01:46 PM, Abhishek Shrivastava wrote:
>
>  Instead of it use reusable_node option.
>
>
> Thanks. Problem resolved. :)
>
> BTW, single_use_node is written in layout.yaml by default.
> If it doesn't exist anymore, do we need a patch to fix it ?
>
> For someone who uses CI for the first time, it is really a problem......
>
> And also, if I want to post patch for zuul, where should I post the patch ?
>
> Thanks.
>
>
>
>
>
>
> On Tue, Jul 14, 2015 at 9:12 AM, Tang Chen <tangchen at cn.fujitsu.com>
> wrote:
>
>  Hi Abhishek, All,
>
> I found the problem.
>
> My /etc/zuul/layout/layout.yaml has the following config:
>
> jobs:
>   - name: ^dsvm-tempest.*$
>     parameter-function: single_use_node
>
> But in _parseConfig() in zuul/scheduler.py, it failed to find
> single_use_node().
>
>             fname = config_job.get('parameter-function', None)
>             if fname:
>                 func = config_env.get(fname, None)
>                 if not func:
>                     raise Exception("Unable to find function %s" % fname)
>
> So projects section was not parsed.
>
> Does anyone know why ?
>
> Thanks.
>
>
>
>
>  On 07/14/2015 10:54 AM, Tang Chen wrote:
>
> Hi Abhishek,
>
> I printed the self.layout.projects in zuul/scheduler.py, it is empty.
> So the project was not found.
>
> But I did do the *jenkins-jobs --flush-cache update*
> */etc/jenkins_jobs/config/*
> And I did configure openstack-dev/sandbox in layout.yaml.
>
> Do you have any idea what's wrong here ?
>
> Thanks.
>
>  On 07/13/2015 05:58 PM, Tang Chen wrote:
>
>
>
> On 07/13/2015 04:35 PM, Abhishek Shrivastava wrote:
>
>  Updating jobs using "*sudo jenkins-jobs --flush-cache update
> /etc/jenkins_jobs/config/". *Also update the myvendor in examples.yaml
>
>
>
>
> Sorry, I updated the jobs, restart the whole machine. But it still doesn't
> work.
>
> By the way, there is no vendor in examples.yaml.
>
> It is still this error: Project openstack-dev/sandbox not found
>
> Anything else should I pay attention to?
>
> Thanks.
>
>
>
>
> On Mon, Jul 13, 2015 at 1:45 PM, Tang Chen <tangchen at cn.fujitsu.com>
> wrote:
>
>
>
> On 07/13/2015 03:50 PM, Abhishek Shrivastava wrote:
>
>  ​ Use tester or something, also are you updating the jobs or not?​
>
>
> I used tester as my vendor. It doesn't work.
>
> And what do you mean by updating the jobs ? I built the
> noop-check-cimmunitication
> job once in Jenkins UI. Does it matter ? All the others are not touched.
>
> And referring to the error, "Project openstack-dev/sandbox not found", it
> seems like
> somewhere the project name was wrong.
>
> right ?
>
>
>
> Thanks.
>
>
>
>
> On Mon, Jul 13, 2015 at 1:16 PM, Tang Chen <tangchen at cn.fujitsu.com>
> wrote:
>
>  Hi Abhishek,
>
> Thanks for the quick reply.
>
> On 07/13/2015 03:16 PM, Abhishek Shrivastava wrote:
>
>  Also check that Gearman is connecting or not through Jenkins UI.
>
>
>
> On Mon, Jul 13, 2015 at 12:45 PM, Abhishek Shrivastava <
> abhishek at cloudbyte.com> wrote:
>
>  First of all, change the "vendor" to your vendor name in /etc/jenkins_jobs/config/projects.yaml
> file. Also, restart the zuul and zuul merger.
>
>
> I did the check. Gearman plugin works find. In Jenkins UI, I tested the
> connection, and it succeeded.
>
> And also, I restarted zuul and zuul merger every time I modified the yaml
> files.
>
> But it doesn't work.
>
> And the vendor, does that matter ?  And what vendor name should I provide ?
> I cannot find any vendor info in my Gerrit service account profile.
> For example, is XXX OK ?
>
> Thanks.
>
>
>
>
>
>
>
>
> On Mon, Jul 13, 2015 at 12:29 PM, Tang Chen <tangchen at cn.fujitsu.com>
> wrote:
>
> Hi all,
>
> I have constructed my CI system.
> When I tested it with sandbox project, I posted a patch to Gerrit.
>
> https://review.openstack.org/201002
>
> But I got this error in zuul scheduler:
>
> 2015-07-14 14:07:24,921 DEBUG zuul.Scheduler: Run handler awake
> 2015-07-14 14:07:24,921 DEBUG zuul.Scheduler: Fetching trigger event
> 2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Processing trigger event
> <TriggerEvent comment-added openstack-dev/sandbox master 201002,1
> Verified:1>
> 2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Project
> openstack-dev/sandbox not found
> 2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Run handler sleeping
>
>
> My /etc/zuul/layout/layout.yaml looks like this:
>
> projects:
>   - name: openstack-dev/sandbox
>     check:
>       - noop-check-communication
>
>
> My /etc/jenkins_jobs/config/projects.yaml looks like this:
>
> - project:
>     name: sandbox
>     github-org: openstack-dev
>     node: master
>     vendor: myvendor
>
>     jobs:
>         - noop-check-communication
>         - dsvm-tempest-full:
>             node: 'devstack_slave || devstack-precise-check || d-p-c'
>
>
> And Jenkins master works fine.
>
>
> Does anyone know what is going on here ?
>
>
>
> Thanks.
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
> --
>
>   *Thanks & Regards,*
>
> *Abhishek*
>
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
>
>
>
>
> --
>
>   *Thanks & Regards,*
>
> *Abhishek*
>
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
> --
>
>   *Thanks & Regards,*
>
> *Abhishek*
>
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
> --
>
>   *Thanks & Regards,*
>
> *Abhishek*
>
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>  __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>  __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
> --
>
>   *Thanks & Regards,*
>
> *Abhishek*
>
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
> --
>
>   *Thanks & Regards,*
>
> *Abhishek*
>
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
>
>
>
>
> --
>
>   *Thanks & Regards,*
>
> *Abhishek*
>
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 


*Thanks & Regards,*
*Abhishek*
*Cloudbyte Inc. <http://www.cloudbyte.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150720/205651f4/attachment.html>

More information about the OpenStack-dev mailing list