[openstack-dev] [CI] How to set a proxy for zuul.
Tang Chen
tangchen at cn.fujitsu.com
Mon Jul 20 05:48:28 UTC 2015
Hi Asselin, Abhishek,
Thanks for the reply. :)
On 07/19/2015 02:41 AM, Asselin, Ramy wrote:
>
> HI Abhi & Tang,
>
> Sorry I missed this thread. Let me know if you've resolved your issues.
>
> My repo is undergoing migrations to reuse components in
> openstack-infra/puppet-openstackci.
>
> For single-use-nodes, the file you need has been removed here [1]: But
> I see now that it is still needed, or a different function is needed
> based on this version used by infra: [2]. I will explore a solution.
>
> A couple other notes, please use ci-sandbox [3] instead of sandbox.
>
OK.
> Zuul use behind a proxy: seems you got past this? Could you share your
> solution?
>
The root cause is that zuul uses a library named paramiko to create
connection
with low level socket APIs. And it doesn't provide proxy functionary.
I tried to use tools like proxychain, to redirect the zuul connection to
go through
my proxy. But it doesn't work. If I use proxychain to run zuul service,
it doesn't
output anything to the log file, and the service will die soon.
I think, there are two solutions:
1. Add proxy functionality to paramiko.
2. Add proxy functionality to zuul. That means, maybe,
zuul does not use paramiko to create connection, but by itself.
Solution 1 is much simpler, so for now, I just modified the source code
of paramiko.
I'm using python-socksipy package, and modified
/usr/local/lib/python2.7/dist-packages/paramiko/client.py like this:
diff --git a/client.py b/client.py
index 15ff696..d7225ed 100644
--- a/client.py
+++ b/client.py
@@ -24,6 +24,7 @@ from binascii import hexlify
import getpass
import os
import socket
+import socks
import warnings
import pdb
@@ -235,6 +236,7 @@ class SSHClient (ClosingContextManager):
``gss_deleg_creds`` and ``gss_host`` arguments.
"""
+ """
if not sock:
for (family, socktype, proto, canonname, sockaddr) in
socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM):
if socktype == socket.SOCK_STREAM:
@@ -251,6 +253,13 @@ class SSHClient (ClosingContextManager):
except:
pass
retry_on_signal(lambda: sock.connect(addr))
+ """
+
+ if not sock:
+ sock = socks.socksocket()
+ sock.setproxy(socks.PROXY_TYPE_SOCKS5, "MY_PROXY_IP",
MY_PROXY_PORT, username="XXX", password="XXX")
# This is review.openstack.org
+ addr = ('104.130.159.134', 29418)
+ retry_on_signal(lambda: sock.connect(addr))
t = self._transport = Transport(sock, gss_kex=gss_kex,
gss_deleg_creds=gss_deleg_creds)
t.use_compression(compress=compress)
Of course, this is just a draft. It is only for my case, not for all.
BTW, I'm now working on Fujitsu CI System, and really want to join into
the development of openstack-infra.
I think the proxy functionality is necessary for many companies. So if
you are planing to add the proxy support,
I think I can help.
Thanks. :)
> Also, feel free to join 3^rd party ci IRC meetings on freenode [4].
> It's a great place to ask questions and meet others setting up or
> maintaining these systems.
>
> Thanks,
>
> Ramy
>
> IRC: asselin
>
> [1]
> https://github.com/rasselin/os-ext-testing/commit/dafe822be7813522a6c7361993169da20b37ffb7
>
> [2]
> https://github.com/openstack-infra/project-config/blob/master/zuul/openstack_functions.py
>
> [3] http://git.openstack.org/cgit/openstack-dev/ci-sandbox/
>
> [4] http://eavesdrop.openstack.org/#Third_Party_Meeting
>
> *From:*Abhishek Shrivastava [mailto:abhishek at cloudbyte.com]
> *Sent:* Monday, July 13, 2015 11:51 PM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [CI] How to set a proxy for zuul.
>
> Also if you want to change it you will need to talk with Asselin Ramy
> who is the owner of the repo you followed.
>
> On Tue, Jul 14, 2015 at 12:18 PM, Abhishek Shrivastava
> <abhishek at cloudbyte.com <mailto:abhishek at cloudbyte.com>> wrote:
>
> Basically it is not required, and if you see
> /etc/jenkins_jobs/config folder you will find one
> dsvm-cinder-tempest.yaml which is to be used basically not
> examples.yaml. So its not an issue.
>
> On Tue, Jul 14, 2015 at 12:07 PM, Tang Chen
> <tangchen at cn.fujitsu.com <mailto:tangchen at cn.fujitsu.com>> wrote:
>
> On 07/14/2015 01:46 PM, Abhishek Shrivastava wrote:
>
> Instead of it use reusable_node option.
>
>
> Thanks. Problem resolved. :)
>
> BTW, single_use_node is written in layout.yaml by default.
> If it doesn't exist anymore, do we need a patch to fix it ?
>
> For someone who uses CI for the first time, it is really a
> problem......
>
> And also, if I want to post patch for zuul, where should I
> post the patch ?
>
> Thanks.
>
>
>
>
> On Tue, Jul 14, 2015 at 9:12 AM, Tang Chen
> <tangchen at cn.fujitsu.com <mailto:tangchen at cn.fujitsu.com>>
> wrote:
>
> Hi Abhishek, All,
>
> I found the problem.
>
> My /etc/zuul/layout/layout.yaml has the following config:
>
> jobs:
> - name: ^dsvm-tempest.*$
> parameter-function: single_use_node
>
> But in _parseConfig() in zuul/scheduler.py, it failed
> to find single_use_node().
>
> fname =
> config_job.get('parameter-function', None)
> if fname:
> func = config_env.get(fname, None)
> if not func:
> raise Exception("Unable to find
> function %s" % fname)
>
> So projects section was not parsed.
>
> Does anyone know why ?
>
> Thanks.
>
>
>
>
> On 07/14/2015 10:54 AM, Tang Chen wrote:
>
> Hi Abhishek,
>
> I printed the self.layout.projects in
> zuul/scheduler.py, it is empty.
> So the project was not found.
>
> But I did do the *jenkins-jobs --flush-cache
> update* */etc/jenkins_jobs/config/*
> And I did configure openstack-dev/sandbox in
> layout.yaml.
>
> Do you have any idea what's wrong here ?
>
> Thanks.
>
> On 07/13/2015 05:58 PM, Tang Chen wrote:
>
> On 07/13/2015 04:35 PM, Abhishek Shrivastava
> wrote:
>
> Updating jobs using "*sudo jenkins-jobs
> --flush-cache update
> /etc/jenkins_jobs/config/". *Also update
> the myvendor in examples.yaml
>
>
> Sorry, I updated the jobs, restart the whole
> machine. But it still doesn't work.
>
> By the way, there is no vendor in examples.yaml.
>
> It is still this error: Project
> openstack-dev/sandbox not found
>
> Anything else should I pay attention to?
>
> Thanks.
>
>
> On Mon, Jul 13, 2015 at 1:45 PM, Tang Chen
> <tangchen at cn.fujitsu.com
> <mailto:tangchen at cn.fujitsu.com>> wrote:
>
> On 07/13/2015 03:50 PM, Abhishek
> Shrivastava wrote:
>
> Use tester or something, also are
> you updating the jobs or not?
>
>
> I used tester as my vendor. It doesn't
> work.
>
> And what do you mean by updating the
> jobs ? I built the
> noop-check-cimmunitication
> job once in Jenkins UI. Does it matter
> ? All the others are not touched.
>
> And referring to the error, "Project
> openstack-dev/sandbox not found", it
> seems like
> somewhere the project name was wrong.
>
> right ?
>
>
>
> Thanks.
>
>
> On Mon, Jul 13, 2015 at 1:16 PM,
> Tang Chen <tangchen at cn.fujitsu.com
> <mailto:tangchen at cn.fujitsu.com>>
> wrote:
>
> Hi Abhishek,
>
> Thanks for the quick reply.
>
> On 07/13/2015 03:16 PM,
> Abhishek Shrivastava wrote:
>
> Also check that Gearman is
> connecting or not through
> Jenkins UI.
>
> On Mon, Jul 13, 2015 at
> 12:45 PM, Abhishek
> Shrivastava
> <abhishek at cloudbyte.com
> <mailto:abhishek at cloudbyte.com>>
> wrote:
>
> First of all, change
> the "vendor" to your
> vendor name in
> /etc/jenkins_jobs/config/projects.yaml
> file. Also, restart
> the zuul and zuul merger.
>
>
> I did the check. Gearman
> plugin works find. In Jenkins
> UI, I tested the connection,
> and it succeeded.
>
> And also, I restarted zuul and
> zuul merger every time I
> modified the yaml files.
>
> But it doesn't work.
>
> And the vendor, does that
> matter ? And what vendor name
> should I provide ?
> I cannot find any vendor info
> in my Gerrit service account
> profile.
> For example, is XXX OK ?
>
> Thanks.
>
>
>
>
> On Mon, Jul 13, 2015
> at 12:29 PM, Tang Chen
> <tangchen at cn.fujitsu.com
> <mailto:tangchen at cn.fujitsu.com>>
> wrote:
>
> Hi all,
>
> I have constructed
> my CI system.
> When I tested it
> with sandbox
> project, I posted
> a patch to Gerrit.
>
> https://review.openstack.org/201002
>
> But I got this
> error in zuul
> scheduler:
>
> 2015-07-14
> 14:07:24,921 DEBUG
> zuul.Scheduler:
> Run handler awake
> 2015-07-14
> 14:07:24,921 DEBUG
> zuul.Scheduler:
> Fetching trigger event
> 2015-07-14
> 14:07:24,922 DEBUG
> zuul.Scheduler:
> Processing trigger
> event
> <TriggerEvent
> comment-added
> openstack-dev/sandbox
> master 201002,1
> Verified:1>
> 2015-07-14
> 14:07:24,922 DEBUG
> zuul.Scheduler:
> Project
> openstack-dev/sandbox
> not found
> 2015-07-14
> 14:07:24,922 DEBUG
> zuul.Scheduler:
> Run handler sleeping
>
>
> My
> /etc/zuul/layout/layout.yaml
> looks like this:
>
> projects:
> - name:
> openstack-dev/sandbox
> check:
> -
> noop-check-communication
>
>
> My
> /etc/jenkins_jobs/config/projects.yaml
> looks like this:
>
> - project:
> name: sandbox
> github-org:
> openstack-dev
> node: master
> vendor: myvendor
>
> jobs:
> -
> noop-check-communication
> -
> dsvm-tempest-full:
> node:
> 'devstack_slave ||
> devstack-precise-check
> || d-p-c'
>
>
> And Jenkins master
> works fine.
>
>
> Does anyone know
> what is going on
> here ?
>
>
>
> Thanks.
>
>
>
>
>
> __________________________________________________________________________
> OpenStack
> Development
> Mailing List (not
> for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
>
> *Thanks & Regards,*
>
> *Abhishek*
>
> /_Cloudbyte Inc.
> <http://www.cloudbyte.com>_/
>
>
>
> --
>
> *Thanks & Regards,*
>
> *Abhishek*
>
> /_Cloudbyte Inc.
> <http://www.cloudbyte.com>_/
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing
> List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
>
> *Thanks & Regards,*
>
> *Abhishek*
>
> /_Cloudbyte Inc.
> <http://www.cloudbyte.com>_/
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List
> (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
>
> **
>
> *Thanks & Regards,*
>
> *Abhishek*
>
> /_Cloudbyte Inc. <http://www.cloudbyte.com>_/
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage
> questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
>
> **
>
> *Thanks & Regards,*
>
> *Abhishek*
>
> /_Cloudbyte Inc. <http://www.cloudbyte.com>_/
>
> __________________________________________________________________________
>
> OpenStack Development Mailing List (not for usage questions)
>
> Unsubscribe:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
>
> **
>
> *Thanks & Regards,*
>
> *Abhishek*
>
> /_Cloudbyte Inc. <http://www.cloudbyte.com>_/
>
>
>
> --
>
> **
>
> *Thanks & Regards,*
>
> *Abhishek*
>
> /_Cloudbyte Inc. <http://www.cloudbyte.com>_/
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150720/3fbc6ed8/attachment.html>
More information about the OpenStack-dev
mailing list