[openstack-dev] [CI] How to set a proxy for zuul.
Asselin, Ramy
ramy.asselin at hp.com
Mon Jul 20 13:58:32 UTC 2015
I found that paramiko does support proxies [1].
I remember looking at this a long time ago and you should be able to update zuul code to use it [2].
Ramy
[1] http://blog.adimian.com/2014/10/paramiko-and-corporate-proxies/
[2] http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/lib/gerrit.py#n61
From: Tang Chen [mailto:tangchen at cn.fujitsu.com]
Sent: Sunday, July 19, 2015 10:48 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [CI] How to set a proxy for zuul.
Hi Asselin, Abhishek,
Thanks for the reply. :)
On 07/19/2015 02:41 AM, Asselin, Ramy wrote:
HI Abhi & Tang,
Sorry I missed this thread. Let me know if you’ve resolved your issues.
My repo is undergoing migrations to reuse components in openstack-infra/puppet-openstackci.
For single-use-nodes, the file you need has been removed here [1]: But I see now that it is still needed, or a different function is needed based on this version used by infra: [2]. I will explore a solution.
A couple other notes, please use ci-sandbox [3] instead of sandbox.
OK.
Zuul use behind a proxy: seems you got past this? Could you share your solution?
The root cause is that zuul uses a library named paramiko to create connection
with low level socket APIs. And it doesn't provide proxy functionary.
I tried to use tools like proxychain, to redirect the zuul connection to go through
my proxy. But it doesn't work. If I use proxychain to run zuul service, it doesn't
output anything to the log file, and the service will die soon.
I think, there are two solutions:
1. Add proxy functionality to paramiko.
2. Add proxy functionality to zuul. That means, maybe,
zuul does not use paramiko to create connection, but by itself.
Solution 1 is much simpler, so for now, I just modified the source code of paramiko.
I'm using python-socksipy package, and modified
/usr/local/lib/python2.7/dist-packages/paramiko/client.py like this:
diff --git a/client.py b/client.py
index 15ff696..d7225ed 100644
--- a/client.py
+++ b/client.py
@@ -24,6 +24,7 @@ from binascii import hexlify
import getpass
import os
import socket
+import socks
import warnings
import pdb
@@ -235,6 +236,7 @@ class SSHClient (ClosingContextManager):
``gss_deleg_creds`` and ``gss_host`` arguments.
"""
+ """
if not sock:
for (family, socktype, proto, canonname, sockaddr) in socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM):
if socktype == socket.SOCK_STREAM:
@@ -251,6 +253,13 @@ class SSHClient (ClosingContextManager):
except:
pass
retry_on_signal(lambda: sock.connect(addr))
+ """
+
+ if not sock:
+ sock = socks.socksocket()
+ sock.setproxy(socks.PROXY_TYPE_SOCKS5, "MY_PROXY_IP", MY_PROXY_PORT, username="XXX", password="XXX")
# This is review.openstack.org
+ addr = ('104.130.159.134', 29418)
+ retry_on_signal(lambda: sock.connect(addr))
t = self._transport = Transport(sock, gss_kex=gss_kex, gss_deleg_creds=gss_deleg_creds)
t.use_compression(compress=compress)
Of course, this is just a draft. It is only for my case, not for all.
BTW, I'm now working on Fujitsu CI System, and really want to join into the development of openstack-infra.
I think the proxy functionality is necessary for many companies. So if you are planing to add the proxy support,
I think I can help.
Thanks. :)
Also, feel free to join 3rd party ci IRC meetings on freenode [4]. It’s a great place to ask questions and meet others setting up or maintaining these systems.
Thanks,
Ramy
IRC: asselin
[1] https://github.com/rasselin/os-ext-testing/commit/dafe822be7813522a6c7361993169da20b37ffb7
[2] https://github.com/openstack-infra/project-config/blob/master/zuul/openstack_functions.py
[3] http://git.openstack.org/cgit/openstack-dev/ci-sandbox/
[4] http://eavesdrop.openstack.org/#Third_Party_Meeting
From: Abhishek Shrivastava [mailto:abhishek at cloudbyte.com]
Sent: Monday, July 13, 2015 11:51 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [CI] How to set a proxy for zuul.
Also if you want to change it you will need to talk with Asselin Ramy who is the owner of the repo you followed.
On Tue, Jul 14, 2015 at 12:18 PM, Abhishek Shrivastava <abhishek at cloudbyte.com<mailto:abhishek at cloudbyte.com>> wrote:
Basically it is not required, and if you see /etc/jenkins_jobs/config folder you will find one dsvm-cinder-tempest.yaml which is to be used basically not examples.yaml. So its not an issue.
On Tue, Jul 14, 2015 at 12:07 PM, Tang Chen <tangchen at cn.fujitsu.com<mailto:tangchen at cn.fujitsu.com>> wrote:
On 07/14/2015 01:46 PM, Abhishek Shrivastava wrote:
Instead of it use reusable_node option.
Thanks. Problem resolved. :)
BTW, single_use_node is written in layout.yaml by default.
If it doesn't exist anymore, do we need a patch to fix it ?
For someone who uses CI for the first time, it is really a problem......
And also, if I want to post patch for zuul, where should I post the patch ?
Thanks.
On Tue, Jul 14, 2015 at 9:12 AM, Tang Chen <tangchen at cn.fujitsu.com<mailto:tangchen at cn.fujitsu.com>> wrote:
Hi Abhishek, All,
I found the problem.
My /etc/zuul/layout/layout.yaml has the following config:
jobs:
- name: ^dsvm-tempest.*$
parameter-function: single_use_node
But in _parseConfig() in zuul/scheduler.py, it failed to find single_use_node().
fname = config_job.get('parameter-function', None)
if fname:
func = config_env.get(fname, None)
if not func:
raise Exception("Unable to find function %s" % fname)
So projects section was not parsed.
Does anyone know why ?
Thanks.
On 07/14/2015 10:54 AM, Tang Chen wrote:
Hi Abhishek,
I printed the self.layout.projects in zuul/scheduler.py, it is empty.
So the project was not found.
But I did do the jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/
And I did configure openstack-dev/sandbox in layout.yaml.
Do you have any idea what's wrong here ?
Thanks.
On 07/13/2015 05:58 PM, Tang Chen wrote:
On 07/13/2015 04:35 PM, Abhishek Shrivastava wrote:
Updating jobs using "sudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/". Also update the myvendor in examples.yaml
Sorry, I updated the jobs, restart the whole machine. But it still doesn't work.
By the way, there is no vendor in examples.yaml.
It is still this error: Project openstack-dev/sandbox not found
Anything else should I pay attention to?
Thanks.
On Mon, Jul 13, 2015 at 1:45 PM, Tang Chen <tangchen at cn.fujitsu.com<mailto:tangchen at cn.fujitsu.com>> wrote:
On 07/13/2015 03:50 PM, Abhishek Shrivastava wrote:
Use tester or something, also are you updating the jobs or not?
I used tester as my vendor. It doesn't work.
And what do you mean by updating the jobs ? I built the noop-check-cimmunitication
job once in Jenkins UI. Does it matter ? All the others are not touched.
And referring to the error, "Project openstack-dev/sandbox not found", it seems like
somewhere the project name was wrong.
right ?
Thanks.
On Mon, Jul 13, 2015 at 1:16 PM, Tang Chen <tangchen at cn.fujitsu.com<mailto:tangchen at cn.fujitsu.com>> wrote:
Hi Abhishek,
Thanks for the quick reply.
On 07/13/2015 03:16 PM, Abhishek Shrivastava wrote:
Also check that Gearman is connecting or not through Jenkins UI.
On Mon, Jul 13, 2015 at 12:45 PM, Abhishek Shrivastava <abhishek at cloudbyte.com<mailto:abhishek at cloudbyte.com>> wrote:
First of all, change the "vendor" to your vendor name in /etc/jenkins_jobs/config/projects.yaml file. Also, restart the zuul and zuul merger.
I did the check. Gearman plugin works find. In Jenkins UI, I tested the connection, and it succeeded.
And also, I restarted zuul and zuul merger every time I modified the yaml files.
But it doesn't work.
And the vendor, does that matter ? And what vendor name should I provide ?
I cannot find any vendor info in my Gerrit service account profile.
For example, is XXX OK ?
Thanks.
On Mon, Jul 13, 2015 at 12:29 PM, Tang Chen <tangchen at cn.fujitsu.com<mailto:tangchen at cn.fujitsu.com>> wrote:
Hi all,
I have constructed my CI system.
When I tested it with sandbox project, I posted a patch to Gerrit.
https://review.openstack.org/201002
But I got this error in zuul scheduler:
2015-07-14 14:07:24,921 DEBUG zuul.Scheduler: Run handler awake
2015-07-14 14:07:24,921 DEBUG zuul.Scheduler: Fetching trigger event
2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Processing trigger event <TriggerEvent comment-added openstack-dev/sandbox master 201002,1 Verified:1>
2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Project openstack-dev/sandbox not found
2015-07-14 14:07:24,922 DEBUG zuul.Scheduler: Run handler sleeping
My /etc/zuul/layout/layout.yaml looks like this:
projects:
- name: openstack-dev/sandbox
check:
- noop-check-communication
My /etc/jenkins_jobs/config/projects.yaml looks like this:
- project:
name: sandbox
github-org: openstack-dev
node: master
vendor: myvendor
jobs:
- noop-check-communication
- dsvm-tempest-full:
node: 'devstack_slave || devstack-precise-check || d-p-c'
And Jenkins master works fine.
Does anyone know what is going on here ?
Thanks.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Thanks & Regards,
Abhishek
Cloudbyte Inc.<http://www.cloudbyte.com>
--
Thanks & Regards,
Abhishek
Cloudbyte Inc.<http://www.cloudbyte.com>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Thanks & Regards,
Abhishek
Cloudbyte Inc.<http://www.cloudbyte.com>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
[https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ]
Thanks & Regards,
Abhishek
Cloudbyte Inc.<http://www.cloudbyte.com>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
[https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ]
Thanks & Regards,
Abhishek
Cloudbyte Inc.<http://www.cloudbyte.com>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
[https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ]
Thanks & Regards,
Abhishek
Cloudbyte Inc.<http://www.cloudbyte.com>
--
[https://docs.google.com/uc?export=download&id=0Byq0j7ZjFlFKV3ZCWnlMRXBCcU0&revid=0Byq0j7ZjFlFKa2V5VjdBSjIwUGx6bUROS2IrenNwc0kzd2IwPQ]
Thanks & Regards,
Abhishek
Cloudbyte Inc.<http://www.cloudbyte.com>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<mailto:OpenStack-dev-request at lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150720/45a40a62/attachment-0001.html>
More information about the OpenStack-dev
mailing list