Hello,
I first would like to apologize for not keeping everybody in the loop but I’ve been very busy with a lot of work.
I just wanted to give a quick update on some potential progress.
First off, the Oslo meeting on the 19th of September ([1] log here) kicked us into a direction of how we could use the official nats.py python client [2] to talk to NATS as we quickly ruled out the nats-python client [3] as it’s pretty unmaintained and incomplete.
We did however get a working POC driver [4] (see patchset 14) with [3] that confirmed we could get a Devstack job working even though the driver missed a lot of critical functionality such as error handling, retries, TLS etc etc.
Since the nats.py client [2] is asyncio based we investigated a path forward that meant where we could utilize asyncio in the oslo.messaging library and expose an API up to projects consuming oslo.messaging.
We decided to do some POC patches into oslo.messaging and using the Blazar project as a testing ground [4]. We very quickly found ourselves digging into oslo.service to implement asyncio, checking SQLAlchemy for support, API frameworks would need asyncio support, moving from WSGI to an ASGI compatible server, duplicating code for supporting sync and async code paths and more.
We came to the conclussion that introducing a NATS oslo.messaging driver meant requiring us to implement asyncio across our ecosystem which would take an massive effort. Some good came out of it in form of a patch [5] further improving the consistency of the oslo.messaging API, we also gained a lot of insight in the future if we would like to start introducing asyncio functionality into oslo projects to support it for the future.
Now moving forward to today.
I’ve talked with the maintainer of nats.py and the company behind him to investigate if we can include a synchronous code path into that Python library to be used in a future revision of a oslo.messaging NATS driver. Hopefully we’ll have some more progress to report on that in the beginning of next year.
Feel free to reach out to me (here or on IRC, my nick is tobias-urdin) if you want to know more, have any questions or want to get involved in anything I’ve mentioned above!
Best regards Tobias
[1] https://meetings.opendev.org/meetings/oslo/2022/oslo.2022-09-19-15.12.log.tx... [2] https://github.com/nats-io/nats.py [3] https://github.com/Gr1N/nats-python [4] https://review.opendev.org/q/topic:asyncio-nats [5] https://review.opendev.org/c/openstack/oslo.messaging/+/862419
On 12 Sep 2022, at 09:05, Tobias Urdin tobias.urdin@binero.com wrote:
Hello,
Since I haven’t heard any objections we’ll move the meeting to the Oslo meeting on the 19th of September. See you there! [1]
[1] https://wiki.openstack.org/wiki/Meetings/Oslo
Best regards Tobias
On 8 Sept 2022, at 15:54, Herve Beraud <hberaud@redhat.commailto:hberaud@redhat.com> wrote:
WFM
Le jeu. 8 sept. 2022 à 15:16, Tobias Urdin <tobias.urdin@binero.commailto:tobias.urdin@binero.com> a écrit : Hello,
I don’t have anything against moving to the 19th of September meeting if that’s OK with everybody else, I propose we do that.
Let’s see if we get some feedback against the proposed new date otherwise we’ll move it.
Best regards
On 8 Sept 2022, at 08:41, Radosław Piliszek <radoslaw.piliszek@gmail.commailto:radoslaw.piliszek@gmail.com> wrote:
Hi Tobias et al,
That Monday I cannot attend due to other work commitments. Could we schedule it for the following week?
Cheers, Radek
On Tue, Sep 6, 2022, 18:33 Tobias Urdin <tobias.urdin@binero.commailto:tobias.urdin@binero.com> wrote: Great! I’ve updated the agenda with NATS driver and eventlet future topic https://wiki.openstack.org/wiki/Meetings/Oslo#Agenda_for_Next_Meeting
See you in the next meeting on 12th of September 15:00 UTC hopefully that will give us all some time to read through the following material:
- Current POC code https://review.opendev.org/c/openstack/oslo.messaging/+/848338 - Devstack plugin https://github.com/tobias-urdin/devstack-plugin-nats - Discussion on IRC on 2nd September about below links https://meetings.opendev.org/irclogs/%23openstack-oslo/%23openstack-oslo.202...https://meetings.opendev.org/irclogs/#openstack-oslo/%23openstack-oslo.2022-09-02.log.html
Links to old material: - The old spec that we can work upon https://review.opendev.org/c/openstack/oslo-specs/+/692784 - Old driver implementation for above old spec https://review.opendev.org/c/openstack/oslo.messaging/+/680629 - Old asyncio planning https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio - Old asyncio executor in oslo.messaging https://blueprints.launchpad.net/oslo.messaging/+spec/asyncio-executor - Old messaging API mail thread https://lists.openstack.org/pipermail/openstack-dev/2013-May/009784.html
Looking forward to talk to you all next monday!
Best regards Tobias
On 5 Sept 2022, at 12:48, Herve Beraud <hberaud@redhat.commailto:hberaud@redhat.com> wrote:
Le lun. 5 sept. 2022 à 10:00, Tobias Urdin <tobias.urdin@binero.commailto:tobias.urdin@binero.com> a écrit : Thanks! It seems that we have some traction here to get started in a more structured manner.
I’m on PTO today but back tomorrow, what do people feel about scheduling a recurring meeting about this topic?
The weekly Oslo meeting could host the conversation through a dedicated topic. https://wiki.openstack.org/wiki/Meetings/Oslo
You could modify the meeting agenda to add this topic. Daniel Bengtsson (damani) is our meeting facilitator. Do not hesitate to ping him to discuss the meeting management.
I still need to read through the old spec and old implementation of NATS, and I am a bit restricted on time currently but we could brainstorm in that meeting and get us started on a new/updated the other spec.
Should also note that, to anybody interested, that the topic is a little bit more broader than just a new messaging driver as it includes planning and/or working around the usage of eventlet etc, if you are interested in that topic please also reach out to us!
Best regards Tobias
On 2 Sept 2022, at 10:35, Felix Hüttner <felix.huettner@mail.schwarzmailto:felix.huettner@mail.schwarz> wrote:
+1, we would also be interested in helping out. Let me know where we can help out with reviews, writing code, testing, etc.
-- Felix Huettner
+1, I'm also interested in this topic. I did some tests with nats-py to see how it works and I'm volunteering to go further with this technology. Let me know if you need reviews, help etc...
Le jeu. 1 sept. 2022 à 15:50, Tobias Urdin <mailto:tobias.urdin@binero.commailto:tobias.urdin@binero.com> a écrit : Hello Kai,
Great to hear that you're interested! I'm currently just testing this as a POC directly in devstack so there is real-world testing (not should it yet I think, needs more development first).
Best regards Tobias
On 30 Aug 2022, at 18:32, Kai Bojens <mailto:kb@kbojens.demailto:kb@kbojens.de> wrote:
Am 29.08.22 um 15:46 schrieb Tobias Urdin:
If anybody, or any company, out there would be interested in collaborating in a project to bring this support and maintain it feel free to reach out. I'm hoping somebody will bite but atleast I've put it out there for all of you.
Hi, I am very much interested to take a closer look at your work and maybe contribute to it. Although I'm working with OpenStack for my employer at the moment, I'd do this in my spare time as I'm not sure that I can convince him to add another staging system with a full OpenStack installation just for developing a RabbitMQ replacement. That's not on our agenda as we are mostly users and not developers of OpenStack.
As I'm pretty new to the messaging topic and had just heard of NATS and your idea, I'll first would have to dive into NATS and then I would take a closer look at your code and maybe try to create some documentation or tests. So, I can't promise anything but as I said I'm very interested in your approach as I also see the massive load from RabbitMQ.
This brings me to the most important question: Where do you run and test your code?
Greetings, Kai
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarzhttps://www.datenschutz.schwarz/>.
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud