oslo.messaging integration development
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment.
Hey! It’s cool to see someone else talking about this. My team considered this, and personally I think it would work very well, we just didn’t have the development resources to throw at the problem on our own. I think it’s likely that we could participate in a larger discussion about it though, and possibly help here and there if someone else was driving it. —Adam On Mon, Feb 14, 2022 at 11:59 liyang <ly2008080808@yeah.net> wrote:
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment.
-- Thanks, --Adam
Hi, Thanks to rely. I found this link should be very helpful to us. https://streamnative.io/en/blog/tech/2020-06-15-announcing-aop-on-pulsar/ . 在 2022-02-14 14:55:55，"Adam Harwell" <flux.adam@gmail.com> 写道： Hey! It’s cool to see someone else talking about this. My team considered this, and personally I think it would work very well, we just didn’t have the development resources to throw at the problem on our own. I think it’s likely that we could participate in a larger discussion about it though, and possibly help here and there if someone else was driving it. —Adam On Mon, Feb 14, 2022 at 11:59 liyang <ly2008080808@yeah.net> wrote: Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment. -- Thanks, --Adam
I think if we deploy Apache pulsar with AOP in cluster & HA, the effect will be ok. I am going to ready practise this schema. 在 2022-02-14 15:18:25，"liyang" <ly2008080808@yeah.net> 写道： Hi, Thanks to rely. I found this link should be very helpful to us. https://streamnative.io/en/blog/tech/2020-06-15-announcing-aop-on-pulsar/ . 在 2022-02-14 14:55:55，"Adam Harwell" <flux.adam@gmail.com> 写道： Hey! It’s cool to see someone else talking about this. My team considered this, and personally I think it would work very well, we just didn’t have the development resources to throw at the problem on our own. I think it’s likely that we could participate in a larger discussion about it though, and possibly help here and there if someone else was driving it. —Adam On Mon, Feb 14, 2022 at 11:59 liyang <ly2008080808@yeah.net> wrote: Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment. -- Thanks, --Adam
I wonder if the limitation of forced durable queues will make things too slow? Right now I believe it’s recommended to not use queues in durable mode with openstack (if I recall correctly)? Also, it does add to the setup side of things, as it would require explicitly creating each vhost in Pulsar ahead of deployment, but that might not matter if people don’t use a ton of vhosts (we use one per service, but that’s not actually too many I suppose, and certainly isn’t required). I guess a lot of deployments may only use a single vhost? On Mon, Feb 14, 2022 at 16:21 liyang <ly2008080808@yeah.net> wrote:
I think if we deploy Apache pulsar with AOP in cluster & HA, the effect will be ok. I am going to ready practise this schema.
在 2022-02-14 15:18:25，"liyang" <ly2008080808@yeah.net> 写道：
Hi, Thanks to rely. I found this link should be very helpful to us. https://streamnative.io/en/blog/tech/2020-06-15-announcing-aop-on-pulsar/ .
在 2022-02-14 14:55:55，"Adam Harwell" <flux.adam@gmail.com> 写道：
Hey! It’s cool to see someone else talking about this. My team considered this, and personally I think it would work very well, we just didn’t have the development resources to throw at the problem on our own. I think it’s likely that we could participate in a larger discussion about it though, and possibly help here and there if someone else was driving it.
—Adam
On Mon, Feb 14, 2022 at 11:59 liyang <ly2008080808@yeah.net> wrote:
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment.
-- Thanks, --Adam
-- Thanks, --Adam
On 2/14/22 03:54, liyang wrote:
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment.
Between Erlang and Java, I very much prefer that Java stays away from my clusters. We already have Zookeeper and I'd very much love to get rid of it (but no choice...). Isn't there any better implementation of message queuing somewhere?!? Cheers, Thomas Goirand (zigo)
First off I will note that we've had very bad luck in the past with non-rabbit drivers getting traction either with developers or operators. Most of them have bit-rotted and been removed at this point. That said, we do have a list of requirements for oslo.messaging drivers in the documentation: https://docs.openstack.org/oslo.messaging/latest/contributor/supported-messa... We realize those are non-trivial to achieve for a new driver, so in discussions about other drivers we've pursued the option of developing the new driver out of tree as either a third-party library or a feature branch in oslo.messaging. For NATS[0] we ended up going with a feature branch, although to my knowledge that hasn't really gone anywhere (see above about having bad luck with this sort of thing). The other unfortunate reality is that we've lost the messaging experts who would ideally have reviewed a new driver. Working in a feature branch that doesn't affect the released library mitigates this somewhat because issues can be worked out without affecting production deployments, but unfortunately it is an obstacle. That said, if someone were to develop a new driver that would probably go a long way toward making them a messaging expert so this might be a good way to replenish our talent pool in this area. :-) I don't want to discourage anyone from writing a new driver (I think everyone would love to get rid of rabbit), but before starting down that path I think it's important to understand that there is a lot of inertia behind rabbit and messaging is a complex topic. This is not a project that will take a few weeks. Even a few months is a stretch. You're going to need a rock solid driver implementation and then be able to show that it's enough better than rabbit to convince people to switch. So, if after all this doom and gloom you still want to move forward, feel free to write up a spec similar to the NATS one and we can get you a feature branch or repo to work in. 0: https://review.opendev.org/c/openstack/oslo-specs/+/692784 On 2/13/22 20:54, liyang wrote:
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment.
I think that would apply if there was actually code being written, but it sounds like this would be a drop in, if Pulsar is able to use an AMQP compatibility plugin and essentially act as though it was RMQ. I’m curious to see if that works as expected. If not, then I agree it is a very complex thing to build a new messaging interface, and that is why I had previously expressed caution to our team when it was brought up as an option. I’m only cautiously optimistic now that someone else has broached the subject, and we do have at least some interest still in assisting. I’d just like to see if the “free” approach will bear fruit before starting down that treacherous path. 😅 On Tue, Feb 15, 2022 at 1:36 Ben Nemec <openstack@nemebean.com> wrote:
First off I will note that we've had very bad luck in the past with non-rabbit drivers getting traction either with developers or operators. Most of them have bit-rotted and been removed at this point.
That said, we do have a list of requirements for oslo.messaging drivers in the documentation:
https://docs.openstack.org/oslo.messaging/latest/contributor/supported-messa...
We realize those are non-trivial to achieve for a new driver, so in discussions about other drivers we've pursued the option of developing the new driver out of tree as either a third-party library or a feature branch in oslo.messaging. For NATS[0] we ended up going with a feature branch, although to my knowledge that hasn't really gone anywhere (see above about having bad luck with this sort of thing).
The other unfortunate reality is that we've lost the messaging experts who would ideally have reviewed a new driver. Working in a feature branch that doesn't affect the released library mitigates this somewhat because issues can be worked out without affecting production deployments, but unfortunately it is an obstacle.
That said, if someone were to develop a new driver that would probably go a long way toward making them a messaging expert so this might be a good way to replenish our talent pool in this area. :-)
I don't want to discourage anyone from writing a new driver (I think everyone would love to get rid of rabbit), but before starting down that path I think it's important to understand that there is a lot of inertia behind rabbit and messaging is a complex topic. This is not a project that will take a few weeks. Even a few months is a stretch. You're going to need a rock solid driver implementation and then be able to show that it's enough better than rabbit to convince people to switch.
So, if after all this doom and gloom you still want to move forward, feel free to write up a spec similar to the NATS one and we can get you a feature branch or repo to work in.
0: https://review.opendev.org/c/openstack/oslo-specs/+/692784
On 2/13/22 20:54, liyang wrote:
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment.
-- Thanks, --Adam
On Mon, 2022-02-14 at 10:32 -0600, Ben Nemec wrote:
First off I will note that we've had very bad luck in the past with non-rabbit drivers getting traction either with developers or operators. Most of them have bit-rotted and been removed at this point.
That said, we do have a list of requirements for oslo.messaging drivers in the documentation: https://docs.openstack.org/oslo.messaging/latest/contributor/supported-messa...
We realize those are non-trivial to achieve for a new driver, so in discussions about other drivers we've pursued the option of developing the new driver out of tree as either a third-party library or a feature branch in oslo.messaging. For NATS[0] we ended up going with a feature branch, although to my knowledge that hasn't really gone anywhere (see above about having bad luck with this sort of thing).
The other unfortunate reality is that we've lost the messaging experts who would ideally have reviewed a new driver. Working in a feature branch that doesn't affect the released library mitigates this somewhat because issues can be worked out without affecting production deployments, but unfortunately it is an obstacle.
That said, if someone were to develop a new driver that would probably go a long way toward making them a messaging expert so this might be a good way to replenish our talent pool in this area. :-)
I don't want to discourage anyone from writing a new driver (I think everyone would love to get rid of rabbit), but before starting down that path I think it's important to understand that there is a lot of inertia behind rabbit and messaging is a complex topic. This is not a project that will take a few weeks. Even a few months is a stretch. You're going to need a rock solid driver implementation and then be able to show that it's enough better than rabbit to convince people to switch.
So, if after all this doom and gloom you still want to move forward, feel free to write up a spec similar to the NATS one and we can get you a feature branch or repo to work in.
0: https://review.opendev.org/c/openstack/oslo-specs/+/692784 speaking of the NATS on is that actully moving forward. i had high hopes that NATS could be a scalable cloud native replecement for rabit im not familar with Apache Pulsar https://pulsar.apache.org/ perhaps it has a similar feature set to NATS https://nats.io/. lookign at it however Apache Pulsar seams to be written in java. not to be language snob but that might prove to be a barrier for deployment. that was an issue for monasca in the past. addign another language runtime to the requirement to you cluster like erlang is added by rabbit and java would be added by pulsar to me is a dissadvantage as it increase the treath domain form a secuirty perspecitve. https://github.com/apache/pulsar
NATS being written in go and therefor deployed a a single staticly linked binsary with no runtime deps is one of the advantages it brings https://github.com/nats-io/nats-server is apache/pulsar somthign that you specificly have exprenice with or wanted to use or just the first non rabit solution that seamed to fit the requireemtns that you came across. if so i think it would be worth your time asseing NATS too and perhaps collaberate with teh previous effort or take it over if you find it equally suitable.
On 2/13/22 20:54, liyang wrote:
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to oslo.messaging? Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may enforece the big scale in extension about Openstack deployment.
At least in our case, Pulsar was developed in house here at Yahoo originally and we have very large deployments and many experts (most original major contributors) available directly. So, I’m a little biased. :D On Tue, Feb 15, 2022 at 21:31 Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2022-02-14 at 10:32 -0600, Ben Nemec wrote:
First off I will note that we've had very bad luck in the past with non-rabbit drivers getting traction either with developers or operators. Most of them have bit-rotted and been removed at this point.
That said, we do have a list of requirements for oslo.messaging drivers in the documentation:
https://docs.openstack.org/oslo.messaging/latest/contributor/supported-messa...
We realize those are non-trivial to achieve for a new driver, so in discussions about other drivers we've pursued the option of developing the new driver out of tree as either a third-party library or a feature branch in oslo.messaging. For NATS[0] we ended up going with a feature branch, although to my knowledge that hasn't really gone anywhere (see above about having bad luck with this sort of thing).
The other unfortunate reality is that we've lost the messaging experts who would ideally have reviewed a new driver. Working in a feature branch that doesn't affect the released library mitigates this somewhat because issues can be worked out without affecting production deployments, but unfortunately it is an obstacle.
That said, if someone were to develop a new driver that would probably go a long way toward making them a messaging expert so this might be a good way to replenish our talent pool in this area. :-)
I don't want to discourage anyone from writing a new driver (I think everyone would love to get rid of rabbit), but before starting down that path I think it's important to understand that there is a lot of inertia behind rabbit and messaging is a complex topic. This is not a project that will take a few weeks. Even a few months is a stretch. You're going to need a rock solid driver implementation and then be able to show that it's enough better than rabbit to convince people to switch.
So, if after all this doom and gloom you still want to move forward, feel free to write up a spec similar to the NATS one and we can get you a feature branch or repo to work in.
0: https://review.opendev.org/c/openstack/oslo-specs/+/692784
speaking of the NATS on is that actully moving forward. i had high hopes that NATS could be a scalable cloud native replecement for rabit im not familar with Apache Pulsar https://pulsar.apache.org/ perhaps it has a similar feature set to NATS https://nats.io/. lookign at it however Apache Pulsar seams to be written in java. not to be language snob but that might prove to be a barrier for deployment. that was an issue for monasca in the past. addign another language runtime to the requirement to you cluster like erlang is added by rabbit and java would be added by pulsar to me is a dissadvantage as it increase the treath domain form a secuirty perspecitve. https://github.com/apache/pulsar
NATS being written in go and therefor deployed a a single staticly linked binsary with no runtime deps is one of the advantages it brings https://github.com/nats-io/nats-server
is apache/pulsar somthign that you specificly have exprenice with or wanted to use or just the first non rabit solution that seamed to fit the requireemtns that you came across. if so i think it would be worth your time asseing NATS too and perhaps collaberate with teh previous effort or take it over if you find it equally suitable.
On 2/13/22 20:54, liyang wrote:
Hi everyone, I find Apache Pulsar is very popular in MQ area, especially in clound native area. Is there any body integrate this MQ to
oslo.messaging?
Said to the truth. RabbitMQ developed in Erlang has very little source code level material, Qpid has very little operation practice in the internet. Kafka only support notification in oslo.messaging. I think the Apache Pulsar is clound native based. It may
enforece
the big scale in extension about Openstack deployment.
-- Thanks, --Adam
participants (5)
-
Adam Harwell
-
Ben Nemec
-
liyang
-
Sean Mooney
-
Thomas Goirand