We are stoked to announce the release of: nova 19.0.3: Cloud computing fabric controller This release is part of the stein stable release series. The source is available from: https://opendev.org/openstack/nova Download the package from: https://tarballs.openstack.org/nova/ Please report issues through: https://bugs.launchpad.net/nova/+bugs For more details, please see below. 19.0.3 ^^^^^^ Known Issues * Operators should be aware that nova-api has a dependency on eventlet for executing parallel queries across multiple cells and is monkey- patched accordingly. When nova-api is running under uWSGI or mod_wsgi, the wsgi app will pause after idle time. While the wsgi app is paused, rabbitmq heartbeats will not be sent and log messages related to this can be seen in the nova-api logs when the wsgi app resumes when new requests arrive to the nova-api. These messages are not harmful. When the wsgi app resumes, oslo.messaging will reconnect to rabbitmq and requests will be served successfully. There is one caveat, which is that the wsgi app configuration must be left as the default "threads=1" or set explicitly to "threads=1" to ensure that reconnection will work properly. When threads > 1, it is not guaranteed that oslo.messaging will reconnect to rabbitmq when the wsgi app resumes after pausing during idle time. Threads are used internally by oslo.messaging for heartbeats and more, and it may fail in a variety of ways if run under eventlet with an app that violates eventlet's threading guarantees. When oslo.messaging does not reconnect to rabbitmq after a wsgi app pause, RPC requests will fail with a "MessagingTimeout" error. So, it is necessary to have the wsgi app configured with "threads=1" for reconnection to work properly. If running with "threads=1" is not an option in a particular environment, there are two other workarounds: * Use the eventlet wsgi server instead of uWSGI or mod_wsgi, or * Disable eventlet monkey-patching using the environment variable "OS_NOVA_DISABLE_EVENTLET_PATCHING=yes". Note that disabling eventlet monkey-patching will cause queries across multiple cells to be serialized instead of running in parallel and this may be undesirable in a large deployment with multiple cells, for performance reasons. Please see the following related bugs for more details: * https://bugs.launchpad.net/nova/+bug/1825584 * https://bugs.launchpad.net/nova/+bug/1829062 * In Stein the Placement service is available either as part of Nova, or independently packaged from its own project. This is to allow easier migration from one to another. See the upgrade notes for more information. When using the Placement packaged from Nova, some deployment strategies can lead to the service stalling with error messages similar to: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 460, in fire_timers timer() File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 59, in __call__ cb(*args, **kw) File "/usr/lib/python2.7/site-packages/eventlet/semaphore.py", line 147, in _do_acquire waiter.switch() error: cannot switch to a different thread The reasons this is happening are discussed in bug 1829062. There are three workarounds available: * In the environment of the web server running the placement service, set "OS_NOVA_DISABLE_EVENTLET_PATCHING=yes" so that eventlet does not conflict with thread handling in the web server. * Turn off threading in the web server. For example, if using "mod_wsgi" or "uwsgi", set "threads=1" in their respective configurations. * Switch to using the extracted placement. It does not suffer from eventlet. (https://docs.openstack.org/placement/latest/upgrade/to-stein.html) (https://bugs.launchpad.net/nova/+bug/1829062) Changes in nova 19.0.2..19.0.3 ------------------------------ ff29f7018f Reduce scope of 'path' query parameter to noVNC consoles 66bf82864d Handle legacy request spec dict in ComputeTaskManager._cold_migrate fd491c7415 Func test for migrate reschedule with pinned compute rpc 7d54f91e9f Make nova.compute.rpcapi.ComputeAPI.router a singleton a694952eac Add reno about nova-api eventlet monkey-patching and rabbitmq d6b5e81df5 libvirt: stub logging of host capabilities 4adf563da0 Find instance in another cell during floating IP re-association 38fc7f6f16 Trap and log errors from _update_inst_info_cache_for_disassociated_fip faeb890f36 neutron: refactor nw info cache refresh out of associate_floating_ip bd669a5352 Fix a misuse of assertGreaterEqual ec4993ce0f Fix race in _test_live_migration_force_complete 8a9efe1cbd Log notifications if assertion in _test_live_migration_force_complete fails c2b88839c4 Fixing broken links c2b8496bd2 doc: Fix a broken reference link c21cbf2964 Fix rebuild of baremetal instance when vm_state is ERROR b6c3ff6c77 Remove descriptions of nonexistent hacking rules e5b304af45 doc: remove confusing docs about aggregate allocation ratios 88b2206f2d Fix wrong assertions in unit tests fa59033c40 Fix 'has_calls' method calls in unit tests d7996ff5e7 Add an issue releasenote for placement eventlet stall d90186068b Fix non-existent method of Mock 1b02166528 Restore soft-deleted compute node with same uuid e7109d43d6 Add functional regression recreate test for bug 1839560 e135afec85 lxc: make use of filter python3 compatible a8e19afb3d Fix misuse of nova.objects.base.obj_equal_prims 648770bd68 rt: only map compute node if we created it 677c95280b Cache host to cell mapping in HostManager 3055229841 Convert HostMapping.cells to a dict 77a077573b Delete unused get_all_host_states method d7e4e47d57 Retrun 400 if invalid query parameters are specified bd03723a0c Add useful error log when _determine_version_cap raises DBNotAllowed 5eb4e01a83 Fix the server group "policy" field type in api-ref Diffstat (except docs and test files) ------------------------------------- HACKING.rst | 6 +- api-ref/source/parameters.yaml | 20 +- .../get-rdp-console-post-resp.json | 4 +- .../get-serial-console-post-resp.json | 4 +- .../get-spice-console-post-resp.json | 2 +- .../admin/configuration/hypervisor-xen-api.rst | 5 +- .../admin/configuration/hypervisor-xen-libvirt.rst | 2 +- .../contributor/testing/zero-downtime-upgrade.rst | 2 +- nova/api/openstack/compute/schemas/flavors.py | 16 +- nova/cmd/manage.py | 4 + nova/compute/manager.py | 2 +- nova/compute/resource_tracker.py | 6 +- nova/compute/rpcapi.py | 84 ++++++-- nova/conductor/manager.py | 18 +- nova/conductor/rpcapi.py | 2 + nova/conf/database.py | 5 +- nova/db/sqlalchemy/api.py | 46 ++++- nova/network/neutronv2/api.py | 103 ++++++++-- nova/objects/console_auth_token.py | 14 +- nova/scheduler/host_manager.py | 72 ++++--- nova/scheduler/manager.py | 3 + nova/test.py | 4 + .../get-rdp-console-post-resp.json.tpl | 2 +- .../get-serial-console-post-resp.json.tpl | 2 +- .../get-spice-console-post-resp.json.tpl | 2 +- .../functional/regressions/test_bug_1839560.py | 114 +++++++++++ .../functional/regressions/test_bug_1843090.py | 133 +++++++++++++ nova/virt/disk/mount/nbd.py | 5 +- nova/virt/fake.py | 14 ++ nova/virt/ironic/driver.py | 6 +- nova/virt/libvirt/host.py | 9 +- .../eventlet-monkey-patch-5f734ef581aa550e.yaml | 42 ++++ .../placement-eventlet-stall-ffcca23a6a364c78.yaml | 34 ++++ 63 files changed, 1457 insertions(+), 355 deletions(-)