[openstack-ansible]Playbooks failing after upgrading lxc containers from ubuntu18 to ubuntu20 due to python version mismatch.
Dear All, We are upgrading one of our openstack cluster from ussuri to wallaby and for this we are first upgrading OS of lxc container from ubuntu to 18 to 20(hosts OS is also upgraded to 20). We are destroying LXC containers(which don't form quorum like nova,neutron,heat,cinder,horizon) of one infra node and then creating it again. The new containers are getting created with Ubuntu 20 as the host OS is 20. Playbook for cinder, nova neutron were completed successfully for new containers of ubuntu 20. But Playbook is failing for heat, placement and memcached due to python version mismatch: fatal: [host5_heat_api_container-25f90e52]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/heat-21.2.5/bin/pip3", "install", "-U", "--constraint", "/openstack/venvs/heat-21.2.5/global-constraints.txt", "--constraint", "/openstack/venvs/heat-21.2.5/constraints.txt", "--pre", "--log", "/var/log/python_venv_build.log", "cryptography", "keystonemiddleware", "openstack-heat", "osprofiler", "pymemcache", "PyMySQL", "python-ceilometerclient", "python-glanceclient", "python-heatclient", "python-keystoneclient", "python-memcached", "python-neutronclient", "python-novaclient", "python-openstackclient", "python-swiftclient", "python-troveclient", "systemd-python"], "msg": "stdout: Ignoring python-saharaclient: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring python-saharaclient: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring python-saharaclient: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring os-resource-classes: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring os-resource-classes: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring os-resource-classes: markers I understand that 2.7 version is depricated but where should make changes to fix it? https://docs.openstack.org/releasenotes/python-heatclient/ussuri.html I tried updating few variable to fix python version like: /etc/ansible/roles/os_heat/defaults/main.yml #heat_venv_python_executable: "{{ openstack_venv_python_executable | default('python3') }}" updated it from python2 to python 3 Tried few more variables but that didn't work. Any idea how we can fix this version mismatch issue ? Regards, Danish
Hey, So, I assume you're still on Ussuri right now? As from what I see in compatibility matrix, Ussuri was the first release having 20.04 support: https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/compatibi... But also we should be using python 3 as default since Train except for CentOS according to this: https://opendev.org/openstack/openstack-ansible/src/tag/ussuri-eol/inventory... It would be actually interesting to look through the log file /var/log/python_venv_build.log from host5_heat_api_container-25f90e52 as it might contain some more relevant reasons for task failure. Also I'd suggest not to mess up with role defaults manually. If you want to change them - better use user_variables file, just because role defaults have very low precedence in Ansible, and can be already overridden somewhere in role or in inventory group_vars, while user_*.yml files are loaded as extra-vars and have highest precedence, thus should be always respected. Another thing I'd suggest to try, is to run with `-e venv_rebuild=true` to flush pre-build requirements and start the venv and wheels build process from scratch. Eventually, I think this issue can boil down to the OS version of repo containers and how wheels sync worked back in the releases. Just in case - it used to be problematic due to the logic of LSYNCD which was deleting freshly built wheels. And OS upgrades were specifically a trouble for it - now we've replaced lsyncd with a shared FS which by default is implemented through glusterfs. If this is the case - you might try stopping all except one repo hosts (you need to leave the one having Ubuntu 20.04) and explicitly define `venv_build_host` to be this host. Alternatively - diable wheels build by `venv_wheel_build_enable: False`, but that is not recommended as can result in issuing tons of requests towards OpenDev infra servers potentially causing a DoS. пн, 17 мар. 2025 г. в 08:46, Danish Khan <danish52.jmi@gmail.com>:
Dear All,
We are upgrading one of our openstack cluster from ussuri to wallaby and for this we are first upgrading OS of lxc container from ubuntu to 18 to 20(hosts OS is also upgraded to 20).
We are destroying LXC containers(which don't form quorum like nova,neutron,heat,cinder,horizon) of one infra node and then creating it again. The new containers are getting created with Ubuntu 20 as the host OS is 20.
Playbook for cinder, nova neutron were completed successfully for new containers of ubuntu 20.
But Playbook is failing for heat, placement and memcached due to python version mismatch: fatal: [host5_heat_api_container-25f90e52]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/heat-21.2.5/bin/pip3", "install", "-U", "--constraint", "/openstack/venvs/heat-21.2.5/global-constraints.txt", "--constraint", "/openstack/venvs/heat-21.2.5/constraints.txt", "--pre", "--log", "/var/log/python_venv_build.log", "cryptography", "keystonemiddleware", "openstack-heat", "osprofiler", "pymemcache", "PyMySQL", "python-ceilometerclient", "python-glanceclient", "python-heatclient", "python-keystoneclient", "python-memcached", "python-neutronclient", "python-novaclient", "python-openstackclient", "python-swiftclient", "python-troveclient", "systemd-python"], "msg": "stdout: Ignoring python-saharaclient: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring python-saharaclient: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring python-saharaclient: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring os-resource-classes: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring os-resource-classes: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring os-resource-classes: markers
I understand that 2.7 version is depricated but where should make changes to fix it? https://docs.openstack.org/releasenotes/python-heatclient/ussuri.html
I tried updating few variable to fix python version like: /etc/ansible/roles/os_heat/defaults/main.yml #heat_venv_python_executable: "{{ openstack_venv_python_executable | default('python3') }}"
updated it from python2 to python 3
Tried few more variables but that didn't work.
Any idea how we can fix this version mismatch issue ?
Regards, Danish
participants (2)
-
Danish Khan
-
Dmitriy Rabotyagov