Hi everyone, Following our office hours session at the Hibiscus PTG, I'd like to share a brief summary of our Gazpacho retrospective, the key achievements we've celebrated, and the priority actions we've identified for the Hibiscus cycle. ## Gazpacho Retrospective The Gazpacho cycle marked a significant turning point in our eventlet removal journey. Three projects reached full migration completion: Designate, Neutron, and Cyborg joined the ranks of fully migrated projects alongside Octavia, Mistral, Ironic, Heat, and Barbican. This brings us to eight major projects that have completely removed their eventlet dependencies. Nova made remarkable progress with five out of seven services now defaulting to threading mode. The API service, metadata API, and scheduler switched to threading by default in Gazpacho, while compute and conductor followed suit in early Hibiscus. Watcher achieved a similar milestone by switching all components to threading mode by default. Manila developed a reusable Rally-based performance testing framework that demonstrated no significant performance differences between eventlet and threading modes, providing crucial validation for the entire initiative. On the infrastructure side, the eventlet library itself now fully supports Python 3.14 stable after resolving compatibility issues with asyncio. Oslo.service introduced multiprocessing spawn support to reduce fork instability in multi-threaded environments, particularly benefiting Manila deployments. ## Hibiscus Call to Actions The Hibiscus cycle represents a critical phase where our focus shifts from initial migration to defaulting projects to threading mode and addressing remaining technical blockers. Nova will prioritize completing the console proxy migration for noVNC, spice, and serial consoles, which remain the last services pending threading support. Projects that have completed their core migration work should now turn their attention to converting functional tests to threading mode and expanding test coverage. With Ubuntu 26.04 releasing on April 29, 2026, we need to enable Python 3.14 testing across all projects to validate our compatibility work. This testing will be essential for ensuring smooth deployments on the new LTS release. Projects should also begin defaulting their services to threading mode following the recommended three-release pattern: one cycle supporting both modes with eventlet as default, the next cycle switching to threading as default, and the final cycle removing eventlet support entirely. Library maintainers should focus on cleaning up unconditional eventlet imports across Oslo libraries and making eventlet dependencies truly optional. This cleanup work will unblock projects from declaring themselves fully eventlet-free even when they depend on Oslo components. ## Critical Concerns Requiring Attention Three critical issues emerged during our discussions that need immediate community focus. The picklable problem affects multiple projects including Nova and Neutron. Objects passed to multiprocessing spawn must be picklable, but config objects from Oslo.config currently are not. Python 3.14 changes the default multiprocessing context to fork_server, and the fork context is being deprecated, making this issue increasingly urgent. Root wrap compatibility in threading mode remains unclear and potentially blocks projects like Nova and Cinder through their OS-brick dependency. We need coordinated testing and potential workarounds or alternatives to ensure this critical privileged operation mechanism works correctly in the new threading model. The heartbeat performance regression issue has a clear solution path: reverting the problematic patch that introduced a half-second sleep in threading mode. The community agreed to proceed with this revert and explore TCP keepalive as an alternative, which already works successfully for Tooz and Oslo.messaging. ## Performance Validation and Testing Insights One of the most encouraging outcomes from Gazpacho was Manila's performance testing work. Their Rally comparison framework runs identical workloads on both eventlet and threading deployments, generating comparative reports that show no significant performance differences. This framework is adaptable for other projects and provides the concrete validation needed to build operator confidence in the migration. Swift faces unique challenges with erasure coding that requires predictable request ordering for fragment reconstruction. The team is carefully distinguishing between actual requirements for deterministic behavior and test artifacts that assumed eventlet's specific co-routine switching patterns. This work will provide valuable insights for any project dealing with complex ordering requirements. ## Timeline and Long-term Vision The community reaffirmed the timeline for complete eventlet removal. By 2027.1, all projects should support threading mode with threading as the default configuration, though eventlet may still be available for operators who need it. The 2027.2 release will focus on removing eventlet from Oslo libraries and cleaning up all transitive dependencies, marking the completion of the initiative. After eventlet is fully removed, the community will evaluate whether async.io adoption makes sense for specific use cases. The consensus is clear: we don't want to run three concurrent runtimes simultaneously, so async.io consideration must wait until eventlet is completely eliminated. For a comprehensive analysis including detailed project status, technical discussion transcripts, and the complete list of concerns and solutions, please visit the full report at https://removal.eventlet.org/guide/openstack/hibiscus-ptg/. The report also includes the complete PTG session recording for those who want to review the discussions in detail. Thank you to everyone who participated in the PTG session and continues to drive this critical initiative forward. The progress we've made demonstrates that full migration is not only possible but achievable on a reasonable timeline. Let's maintain this momentum through Hibiscus and continue moving toward a fully threading-based OpenStack. As always, you're welcome to join us on IRC at #openstack-eventlet-removal, and you can find guidance, resources, and updates at https://removal.eventlet.org/ Best regards, Hervé -- Hervé Beraud Principal Software Engineer at Red Hat irc: hberaud