Hello,
Quick reply, I agree with you on all points. Hopefully this design and collaboration discussion
and go on and reach a somewhat consensus on a path forward.
I should also be clear to points out that I’m with you, design matters, and there is a lot of things with this
design that has not been discussed or even scratched yet, but it’s compelling and I’m here trying to get the ball rolling :)
Best regards
Tobias
Hi
Tobias,
Thank
you for the detailed response. My query was to gather more
insight
on what your views/goals are and the responses do not
disappoint.
More
queries inline below.
On
Tue, 30 Aug 2022 at 11:15, Tobias Urdin <tobias.urdin@binero.com>
wrote:
I would like OpenStack design to more embrace the distributed, cloud-native approach that Ceph and Kubernetes brings, and the resiliency of Ceph (and yes, I’m a major Ceph enthusiast)
and there I’m seeing messaging and database as potential blockers to continue on that path.
We
both definitely agree that resiliency needs to be improved.
On Mon, 29 Aug 2022 at 15:47, Tobias Urdin <tobias.urdin@binero.com> wrote:
• Do retries and acknowledgements in the library (since NATS does NOT persist messages like RabbitMQ could)
What do you mean? Is NATS only a router? (I have not used this technology yet.)
It does not persist messages, if there is no backend to respond, the message will be dropped without any action hence why I
want the RPC layer in oslo.messaging (that already does acknowledge calls in the driver) to notify client side that it’s being processed
before client side waits for reply.
Ack,
that makes sense. To let the client know whether there is any
consumer
that accepted that message.
That
said, bear in mind the consumer might accept and then die. If
NATS
does not keep track of this message further, then the resilience
is
handicapped.
• Find or maintain a NATS python library that doesn't use async like the official one does
Why is async a bad thing? For messaging it's the right thing.
This is actually just myself, I would love to just being able to use the official that is async based instead it’s just
me that doesn’t understand how that would be implemented.
https://github.com/nats-io/nats.py instead of the one in POC https://github.com/Gr1N/nats-python which
has a lot of shortcomings and issues, my
idea was just to investigate if was even possible to implement in a feasible way.
Ack,
I see.
Finally, have you considered just trying out ZeroMQ?
Does not exist anymore.
I
think I might have been misunderstood as ZeroMQ still exists. ;-)
You
probably mean the oslo.messaging backend that it's gone.
I
meant that *maybe* it would be good to discuss a reimplementation of
that
which considers the current OpenStack needs.
I
would also emphasise that I imagine RPC and notification messaging
layers
to have different needs and likely requiring different
approaches.
I mean, NATS is probably an overkill for OpenStack services since the
majority of them stay static on the hosts they control (think
nova-compute, neutron agents - and these are also the pain points that
operators want to ease).
I don’t think it it, or even if it is, why not use a better solution or stable approach than RabbitMQ?
This is also the whole point, I don’t want OpenStack to become or be static, I want it to be more dynamic and
cloud-native in it’s approach and support viable integrations that takes it there, we cannot live in the past forever, let’s envision and dream of the future as we want it! :)
Ack,
you want it more dynamic and that's ok now that I understand your view.
That
said, my whole point regarding this boils down to the usual
design
principles that remind us that there are, more often than not,
some
tradeoffs that have been made to build some tech - NATS is likely
no
different: if it promises features A, B, C, D, and we need only A
and
B, then *maybe* it has some constraints on the A and B we want or
we
might miss that it lacks feature E or C/D add useless overhead. The
point
is to have that in mind before going too deep, try to spot and
tackle
such issues early on.
Finally, have you considered just trying out ZeroMQ?
ZeroMQ used to be supported in the past but then it was remvoed
if i understand correctly it only supprot notificaiton or RPC but not both
i dont recall which but perhapse im miss rememebrign on that point.
I believe it would be better suited for RPC than notifications, at
least in the simplest form.
As it’s advertised as scalable and performant I would argue that, why not use it for notifications as well? If anything according to
your observations above it’s more suited for that than RPC, even though request-reply (that we can use for RPC) is a strong first-class implementation in NATS as well.
Well,
that was about ZMQ. I mostly meant that synchronous RPC (that
happens
in OpenStack a lot) adapts very well to what can be achieved
with
ZeroMQ without a lot of fuss.
I mean, NATS is probably an overkill for OpenStack services since the
majority of them stay static on the hosts they control (think
nova-compute, neutron agents - and these are also the pain points that
operators want to ease).
its not any more overkill then rabbitmq is
True that. Probably.
I agree with that, also if you think about it, how many issues related to stability, performance and outages is related to RabbitMQ? It’s quite a few if you ask me.
Just the resource utilization and clustering in RabbitMQ makes me feel bad.
Here
we definitely agree.
As
we used to discuss this before in this community, we are not sure
if
this is RabbitMQ's fault of course or if we just don't know how to
utilise
it properly. ;-)
Anyhow,
RMQ being in Erlang does not help as it's more like a black
box
to most of us here I believe (please raise your hands if you can
debug
an EVM failure).
It’s here that I mean that the cloud-native and scalable implementation would shine, you should be able to rely on it, if sometimes dies so what, things should just
continue to work and that’s not my experience with RabbitMQ but it is my experience with Ceph because in the end the design really matters.
"Design
really matters" is something that I remind myself and others
almost
every day. Hence why this discussion is taking place now. :D
i also dont know waht you mean when you say
"majority of them stay static on the hosts they control"
NATS is intended a s a cloud native horrizontally scaleable message bus.
which is exactly what openstack need IMO.
NATS seems to be tweaked for "come and go" situations which is an
exception in the OpenStack world, not the rule (at least in my view).
I mean, one normally expects to have a preset number of hypervisors
and not them coming and going (which, I agree, is a nice vision, could
be a proper NATS driver, with more awareness in the client projects I
believe, would be an enabler for more dynamic clouds).
It could, but it also doesn’t have to be that. Why not strive for more dynamic? I don’t think anybody would argue that more dynamic is a bad thing
even if you were to have a more static approach to your cloud.
This
has been discussed already above - tradeoffs. One cannot just
make
up a hypervisor and need to spin up a nova-compute for it. It's a
different
story for non-resource-bound services that NATS is
advertised
for. You need more processing power? Sure, you spin another
worker
and connect it with NATS. That scalability might be coming at a
price
that we don't need to pay because OpenStack services are never
going
to scale with this level of dynamism.
Finally,
don't get me wrong. I love the fact that you are doing what
you
are doing. I just want to make sure that it goes in the right
direction.
Cheers,
Radek
-yoctozepto