Open Stack

Dolph,
Thanks!  It is good that python example is provided!
Qing

From: Dolph Mathews [mailto:dolph.mathews at gmail.com]
Sent: Thursday, October 31, 2013 9:25 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions


On Mon, Oct 28, 2013 at 5:46 PM, Qing He <Qing.He at radisys.com<mailto:Qing.He at radisys.com>> wrote:
In my hard drive-less use case, I need an in-core-db/cache that can be in the same db cluster with real db (with hard drive) with the same sql api so that the current openstack code do not need to be changed, instead, just a pluggin with some configurations.

This is pretty much the original use case that dogpile.cache grew out of, see:

  http://docs.sqlalchemy.org/en/rel_0_9/orm/examples.html#dogpile-caching


-----Original Message-----
From: Morgan Fainberg [mailto:m at metacloud.com<mailto:m at metacloud.com>]
Sent: Monday, October 28, 2013 10:22 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] distibuted caching system in front of mysql server for openstack transactions
In light of what Dolph said with regards to Keystone, we are using dogpile.cache to implement memoization in front of our driver calls.
It it has the ability to cache directly as well, but it has been effective (so far) for our use-case.

That being said, I am unsure if caching in front of MySQL is really what we want.  I believe that we should be caching after processing work (hence memoization mechanism) instead of at the SQL layer.  This also means we can be measured in what we cache (oh hey, it makes no sense to cache X because it needs to be "real time" or there isn't a performance issue with that query / call, but Y does a ton of processing and is an expensive join/temptable query).  In my experience, unless the whole application is designed with caching in mind, caching something as broad as MySQL calls (or any SQL store) is likely going to net exactly what Shawn Hartsock stated, adding a second performance issue.

--Morgan

On Mon, Oct 28, 2013 at 10:05 AM, Shawn Hartsock <hartsocks at vmware.com<mailto:hartsocks at vmware.com>> wrote:
>>
>> I once heard a quote.. "I had a performance problem, so I added caching.
>> now I have two performance problems."
>>
> this. 1,000 times this.
>
> Just to float this thought ... make sure it's considered...
>
> I've seen a *lot* of people misuse caching when what the really want is memoization.
>
> *
> http://stackoverflow.com/questions/1988804/what-is-memoization-and-how
> -can-i-use-it-in-python
> *
> http://stackoverflow.com/questions/10879137/how-can-i-memoize-a-class-
> instantiation-in-python
>
> ... I'm not sure what you're trying to do. So YMMV, TTFN, BBQ.
>
> # Shawn Hartsock
>
> ----- Original Message -----
>> From: "Clint Byrum" <clint at fewbar.com<mailto:clint at fewbar.com>>
>> To: "openstack-dev" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
>> Sent: Monday, October 28, 2013 12:12:49 PM
>> Subject: Re: [openstack-dev] distibuted caching system in front of mysql      server for openstack transactions
>>
>> Excerpts from Dolph Mathews's message of 2013-10-28 08:40:19 -0700:
>> > It's not specific to mysql (or sql at all), but keystone is using
>> > dogpile.cache around driver calls to a similar effect.
>> >
>> >   http://dogpilecache.readthedocs.org/en/latest/
>> >
>> > It can persist to memcache, redis, etc.
>> >
>>
>> I once heard a quote.. "I had a performance problem, so I added caching.
>> now I have two performance problems."
>>
>> Caching is unbelievably awesome in the jobs it can do well. When the
>> problem is straight forward and the requirements are few, it is just
>> the right thing to relieve engineering pressure to make an
>> application more scalable.
>>
>> However, IMO, more than narrow, well defined cache usage is a sign
>> that the application needs some reworking to scale.
>>
>> I like the principle of "let's use dogpile so we don't have to
>> reinvent multi-level caching". However, let's make sure we look at
>> each performance issue individually, rather than just throwing them
>> all in a cache box and waving the memcache wand.
>>
>> >
>> > https://github.com/openstack/keystone/blob/master/keystone/common/c
>> > ache/core.py
>> >
>> > On Fri, Oct 25, 2013 at 6:53 PM, Qing He <Qing.He at radisys.com<mailto:Qing.He at radisys.com>> wrote:
>> >
>> > >  All,****
>> > >
>> > > Has anyone looked at the options of putting a distributed caching
>> > > system in front of mysql server to improve performance? This
>> > > should be similar to Oracle Coherence, or VMware VFabric
>> > > SQLFire.****
>> > >
>> > > ** **
>> > >
>> > > Thanks,****
>> > >
>> > > ** **
>> > >
>> > > Qing****
>> > >
>> > > _______________________________________________
>> > > OpenStack-dev mailing list
>> > > OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> > >
>> >
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

-Dolph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131031/28e83c37/attachment.html>