Open Stack

Joshua Harlow wrote:
> Kevin Benton wrote:
>> >Timestamps are just one way (and likely the most primitive), using
>> redis (or memcache) key/value and expiry are another (and letting
>> memcache or redis expire using its own internal algorithms), using
>> zookeeper ephemeral nodes[1] are another... The point being that its
>> backend specific and tooz supports varying backends.
>>
>> Very cool. Is the backend completely transparent so a deployer could
>> choose a service they are comfortable maintaining, or will that change
>> the properties WRT to resiliency of state on node restarts,
>> partitions, etc?
>
> Of course... we tried to make it 'completely' transparent, but in
> reality certain backends (zookeeper which uses a paxos-like algorithm
> and redis with sentinel support...) are better (more resilient, more
> consistent, handle partitions/restarts better...) than others (memcached
> is after all just a distributed cache). This is just the nature of the
> game...
>

And for some more reading fun:

https://aphyr.com/posts/315-call-me-maybe-rabbitmq

https://aphyr.com/posts/291-call-me-maybe-zookeeper

https://aphyr.com/posts/283-call-me-maybe-redis

https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul

... (aphyr.com has alot of these neat posts)...

>>
>> The Nova implementation of Tooz seemed pretty straight-forward, although
>> it looked like it had pluggable drivers for service management already.
>> Before I dig into it much further I'll file a spec on the Neutron side
>> to see if I can get some other cores onboard to do the review work if I
>> push a change to tooz.
>
> Sounds good to me.
>
>>
>>
>> On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja at outlook.com
>> <mailto:harlowja at outlook.com>> wrote:
>>
>> Kevin Benton wrote:
>>
>> So IIUC tooz would be handling the liveness detection for the
>> agents.
>> That would be nice to get ride of that logic in Neutron and just
>> register callbacks for rescheduling the dead.
>>
>> Where does it store that state, does it persist timestamps to the DB
>> like Neutron does? If so, how would that scale better? If not,
>> who does
>> a given node ask to know if an agent is online or offline when
>> making a
>> scheduling decision?
>>
>>
>> Timestamps are just one way (and likely the most primitive), using
>> redis (or memcache) key/value and expiry are another (and letting
>> memcache or redis expire using its own internal algorithms), using
>> zookeeper ephemeral nodes[1] are another... The point being that its
>> backend specific and tooz supports varying backends.
>>
>>
>> However, before (what I assume is) the large code change to
>> implement
>> tooz, I would like to quantify that the heartbeats are actually a
>> bottleneck. When I was doing some profiling of them on the
>> master branch
>> a few months ago, processing a heartbeat took an order of
>> magnitude less
>> time (<50ms) than the 'sync routers' task of the l3 agent
>> (~300ms). A
>> few query optimizations might buy us a lot more headroom before
>> we have
>> to fall back to large refactors.
>>
>>
>> Sure, always good to avoid prematurely optimizing things...
>>
>> Although this is relevant for u I think anyway:
>>
>> https://review.openstack.org/#__/c/138607/
>> <https://review.openstack.org/#/c/138607/> (same thing/nearly same
>> in nova)...
>>
>> https://review.openstack.org/#__/c/172502/
>> <https://review.openstack.org/#/c/172502/> (a WIP implementation of
>> the latter).
>>
>> [1]
>> https://zookeeper.apache.org/__doc/trunk/__zookeeperProgrammers.html#__Ephemeral+Nodes
>>
>> <https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes>
>>
>>
>>
>> Kevin Benton wrote:
>>
>>
>> One of the most common is the heartbeat from each agent.
>> However, I
>> don't think we can't eliminate them because they are used
>> to determine
>> if the agents are still alive for scheduling purposes. Did
>> you have
>> something else in mind to determine if an agent is alive?
>>
>>
>> Put each agent in a tooz[1] group; have each agent periodically
>> heartbeat[2], have whoever needs to schedule read the active
>> members of
>> that group (or use [3] to get notified via a callback), profit...
>>
>> Pick from your favorite (supporting) driver at:
>>
>> http://docs.openstack.org/____developer/tooz/compatibility.____html
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html>
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html
>> <http://docs.openstack.org/developer/tooz/compatibility.html>>
>>
>> [1]
>> http://docs.openstack.org/____developer/tooz/compatibility.____html#grouping
>>
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping>
>>
>> <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
>> <http://docs.openstack.org/developer/tooz/compatibility.html#grouping>>
>> [2]
>> https://github.com/openstack/____tooz/blob/0.13.1/tooz/____coordination.py#L315
>>
>> <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315>
>>
>> <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
>>
>> <https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>>
>>
>> [3]
>> http://docs.openstack.org/____developer/tooz/tutorial/group_____membership.html#watching-____group-changes
>>
>> <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes>
>>
>> <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
>>
>> <http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>>
>>
>>
>>
>> __________________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.____openstack.org?subject:____unsubscribe
>> <http://openstack.org?subject:__unsubscribe>
>> <http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
>> http://lists.openstack.org/____cgi-bin/mailman/listinfo/____openstack-dev
>> <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev>
>> <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>
>>
>> ______________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>> ______________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.__openstack.org?subject:__unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> --
>> Kevin Benton
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev