[Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

Jesse Pretorius Jesse.Pretorius at rackspace.co.uk
Thu Nov 17 11:06:52 UTC 2016


Please continue to make use of the list instead of going off list. The list archives are a reference point for others who may find themselves in the same situation.

Right, so the service is up. The load balancer is up, so we’re down to networking issues or perhaps something a little more obscure.

Can you confirm whether you’re able to reach the repo contents from each of the following points: repo container, repo container host, deployment host

ie

ansible repo_all -m shell -a "curl http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt"
ansible hosts -m shell -a "curl http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt"
curl http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt

J

From: Achi Hamza <h16mara at gmail.com>
Date: Thursday, November 17, 2016 at 10:59 AM
To: Jesse Pretorius <Jesse.Pretorius at rackspace.co.uk>
Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

Thank you Jesse.

here are the output of the 2 commands:

nginx server is UP:
root at maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a "service nginx status"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
node01_repo_container-82b4e1f6 | SUCCESS | rc=0 >>
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2016-11-16 13:12:57 UTC; 21h ago
 Main PID: 2729 (nginx)
    Tasks: 3
   Memory: 956.0K
      CPU: 5.832s
   CGroup: /system.slice/nginx.service
           ├─2729 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
           ├─2730 nginx: worker process
           └─2731 nginx: worker process

Nov 16 13:12:57 node01-repo-container-82b4e1f6 systemd[1]: Starting A high performance web server and a reverse proxy server...
Nov 16 13:12:57 node01-repo-container-82b4e1f6 systemd[1]: Started A high performance web server and a reverse proxy server.

the repo-all* is also UP:
root at node01:~# hatop -s /var/run/haproxy.stat
>>> rabbitmq_mgmt-back
..ner-2148a5b8        1 DOWN       L4CON             1      0        0        0         0         0         0         0
BACKEND               0 DOWN                         0      0        0        0         0         0       410         0

>>> repo_all-front-1
FRONTEND              0 OPEN                         0      0        0        0         0         1      4096         0

>>> repo_all-back
..ner-82b4e1f6        1 UP         L7OK              1      0        0        0         0         0         0         0
BACKEND               1 UP                           1      0        0        0         0         0       410         0

>>> repo_cache-front-1
FRONTEND              0 OPEN                         0      0        0        0         0         1      4096         0

>>> repo_cache-back
..ner-82b4e1f6        1 UP         L7OK              1      0        0        0         0         0         0         0
BACKEND               1 UP                           1      0        0        0         0         0       410         0

>>> repo_git-front-1
FRONTEND              0 OPEN                         0      0        0        0         0         0      4096         0

>>> repo_git-back
..ner-82b4e1f6        1 UP         L4OK              1      0        0        0         0         0         0         0
BACKEND               1 UP                           1      0        0        0         0         0       410         0

 1-STATUS  2-TRAFFIC  3-HTTP  4-ERRORS  5-CLI                                                                 [#30/#0]

What else please ?

On 17 November 2016 at 11:47, Jesse Pretorius <Jesse.Pretorius at rackspace.co.uk<mailto:Jesse.Pretorius at rackspace.co.uk>> wrote:
OK, so it’s clearly there in terms of data.

Can you now verify whether the web server is up on the repo server?

ansible repo_all -m shell -a "service nginx status"

Also check haproxy by accessing the haproxy host and running:

hatop –s /var/run/haproxy.stat
Look specifically for the repo_all-* entries and check whether they’re up/down.
There should be at least one front end and one back end.

Thanks,

Jesse

From: Achi Hamza <h16mara at gmail.com<mailto:h16mara at gmail.com>>
Date: Thursday, November 17, 2016 at 10:39 AM
To: Jesse Pretorius <Jesse.Pretorius at rackspace.co.uk<mailto:Jesse.Pretorius at rackspace.co.uk>>
Cc: "OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>" <OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>>

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

Hi Jesse,

Yes, i can:

root at maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a "find /var/www/repo/os-releases -type f"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
node01_repo_container-82b4e1f6 | SUCCESS | rc=0 >>
/var/www/repo/os-releases/14.0.1/requirements.txt
/var/www/repo/os-releases/14.0.1/requirements_absolute_requirements.txt
/var/www/repo/os-releases/14.0.1/MANIFEST.in
/var/www/repo/os-releases/14.0.1/index.html
/var/www/repo/os-releases/14.0.1/requirements_constraints.txt


On 17 November 2016 at 11:35, Jesse Pretorius <Jesse.Pretorius at rackspace.co.uk<mailto:Jesse.Pretorius at rackspace.co.uk>> wrote:
Hi Achi,

Can you list the files in /var/www/repo/os-release?

ansible repo_all -m shell -a "find /var/www/repo/os-releases -type f"

Thanks,

Jesse
IRC: odyssey4me

From: Achi Hamza <h16mara at gmail.com<mailto:h16mara at gmail.com>>
Date: Thursday, November 17, 2016 at 10:01 AM
To: "OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>" <OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>>, "openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

Hi Jesse and Jean-Philippe,

Thank you both for your responses.

@Jesse
I am just having one node for the infra and one repo, for now:

 root at maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a "du -sh /var/www/repo/*"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml "
node01_repo_container-82b4e1f6 | SUCCESS | rc=0 >>
900K
/var/www/repo/links
656M
/var/www/repo/openstackgit
964K
/var/www/repo/os-releases
4.0K
/var/www/repo/pkg-cache
105M
/var/www/repo/pools
273M
/var/www/repo/venvs

@Jean-Philippe
Yes, i am using haproxy. I cannot wget the http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt file neither form the bare metal node nor from within the container:

root at node01-galera-container-368f269a:~# wget -c http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
--2016-11-17 09:53:19--  http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
--2016-11-17 09:53:20--  (try: 2)  http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
--2016-11-17 09:53:22--  (try: 3)  http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

I attached my haproxy.conf file, can you please check it ? If the problem persists i will pop into the irc channel.

Thank you,
Hamza






On 17 November 2016 at 10:10, Jean-Philippe Evrard <Jean-Philippe.Evrard at rackspace.co.uk<mailto:Jean-Philippe.Evrard at rackspace.co.uk>> wrote:
Hello,

Is this a greenfield newton?
Could you wget your http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt  file reliably?
Are you using haproxy?  Are all the backends OK there?

Don’t hesitate to come by our irc channel, it’s probably easier to have a conversation there.

Best regards,
Jean-Philippe

From: Achi Hamza <h16mara at gmail.com<mailto:h16mara at gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Thursday, 17 November 2016 at 08:56
To: "openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: [openstack-dev] [openstack-ansible] pip issues


Hi all,

Context:
openstact-ansible newton, tag: 14.0.1
OS: ubuntu 16.04 LTS

the pip command is unable to download packages form the repo container. I can reach the repo container from within galera container through the port 8181:

root at node01:~# lxc-attach -n node01_galera_container-368f269a
root at node01-galera-container-368f269a:~# nc -nzv 172.16.1.2 8181
Connection to 172.16.1.2 8181 port [tcp/*] succeeded!

However, when i run galera-install.yml playbook it throws the following error (but the root cause is pip_install role)  :
root at maas:/opt/openstack-ansible/playbooks# openstack-ansible galera-install.yml
………..............................................................................
TASK [pip_install : Install pip packages (fall back mode)] *********************
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (5 retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (4 retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (3 retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (2 retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (1 retries left).fatal: [node01_galera_container-368f269a]: FAILED! => {"changed": false, "cmd": "/usr/local/bin/pip install -U --isolated --constraint http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt  ndg-httpsclient requests", "failed": true, "msg": "\n:stderr: Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': /os-releases/14.0.1/requirements_constraints.txt\nRetrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': /os-releases/14.0.1/requirements_constraints.txt\nRetrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': /os-releases/14.0.1/requirements_constraints.txt\nRetrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': /os-releases/14.0.1/requirements_constraints.txt\nRetrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': /os-releases/14.0.1/requirements_constraints.txt\nException:\nTraceback (most recent call last):\n  File \"/usr/local/lib/python2.7/dist-packages/pip/basecommand.py\", line 215, in main\n    status = self.run(options, args)\n  File \"/usr/local/lib/python2.7/dist-packages/pip/commands/install.py\", line 287, in run\n    wheel_cache\n  File \"/usr/local/lib/python2.7/dist-packages/pip/basecommand.py\", line 269, in populate_requirement_set\n    session=session, wheel_cache=wheel_cache):\n  File \"/usr/local/lib/python2.7/dist-packages/pip/req/req_file.py\", line 84, in parse_requirements\n    filename, comes_from=comes_from, session=session\n  File \"/usr/local/lib/python2.7/dist-packages/pip/download.py\", line 409, in get_file_content\n    resp = session.get(url)\n  File \"/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py\", line 487, in get\n    return self.request('GET', url, **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/pip/download.py\", line 378, in request\n    return super(PipSession, self).request(method, url, *args, **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py\", line 475, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/sessions.py\", line 585, in send\n    r = adapter.send(request, **kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/adapters.py\", line 467, in send\n    raise ConnectionError(e, request=request)\nConnectionError: HTTPConnectionPool(host='172.16.1.2', port=8181): Max retries exceeded with url: /os-releases/14.0.1/requirements_constraints.txt (Caused by ProtocolError('Connection aborted.', BadStatusLine(\"''\",)))\nYou are using pip version 8.1.2, however version 9.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n"}


Please advice on what i am doing wrong. I am stuck :)

Thank you,
Hamza

________________________________
Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy<http://www.rackspace.co.uk/legal/privacy-policy> - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse at rackspace.com<mailto:abuse at rackspace.com> and delete the original message. Your cooperation is appreciated.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161117/8b2ec7bc/attachment.html>


More information about the OpenStack-operators mailing list