Open Stack

>
> Also, every hour we are seeing a spike in our wsgi processes. If I
> understand,
> it takes about 10mins to re-read the data from memcached, does that sounds
> correct?


Spikes in uwsgi process occur when it retrieves updated records from
memcached and rebuilds indexes. During normal work (with processor running
every 8 hours) the number of new and updated records is quite low and
reading them should not impact performance significantly. However during
the initial load uwsgi process has to read all records and may hang for
several minutes.

For stackalytics.com we monitor that all requests are succeed (processed
within 60 seconds) and I'm really interested in more granulated data to
verify that there's no gaps in performance.

--Ilya

2015-11-10 17:30 GMT+03:00 Paul Belanger <pabelanger at redhat.com>:

> On Mon, Nov 09, 2015 at 06:06:47PM -0500, Paul Belanger wrote:
> > On Mon, Nov 09, 2015 at 12:28:27PM +0300, Ilya Shakhat wrote:
> > > Prod is based on nginx. It is configured to forward dynamic requests to
> > > uwsgi process, thus nginx itself does not consume much memory.
> > >
> > > The config is following:
> > > server {
> > > listen   80;
> > > server_name stackalytics.com;
> > > location / {
> > >         uwsgi_pass  unix:///tmp/stackalytics.sock;
> > >         include     uwsgi_params;
> > > }
> > > location  /static/ {
> > >         alias  /opt/stack/stackalytics/stackalytics/dashboard/static/;
> > > }
> > > }
> > >
> >
> > So, for people playing at home, nginx + uwsgi is more forgiving[1] then
> apache.
> >
> > [1] http://pastebin.com/3V7XEmZL
> >
> I made a mistake in reading my output, I was looking at virtual memory.
> Either
> way, do you have some time this morning to talk in openstack-infra?  I'd
> like to
> confirm some final numbers we are seeing after a full import.
>
> Also, every hour we are seeing a spike in our wsgi processes. If I
> understand,
> it takes about 10mins to re-read the data from memcached, does that sounds
> correct?
>
> How often are you refreshing the data on stackalytics.com? And are people
> actually getting hit with the 10min delay when accessing the page?
>
> > > --Ilya
> > >
> > > 2015-11-09 4:56 GMT+03:00 Paul Belanger <pabelanger at redhat.com>:
> > >
> > > > On Fri, Nov 06, 2015 at 06:59:15PM +0300, Ilya Shakhat wrote:
> > > > > I confirm that 8Gb is not enough. Currently the production is 16Gb
> - 3.5
> > > > > for memcached, 3.5 for every uwsgi process and up to 2 may be
> consumed by
> > > > > the processor.
> > > > >
> > > > Out of curiosity, are you using nginx or apache?  I've currently
> setup
> > > > Apache,
> > > > but might test nginx tomorrow. If you are using nginx how much RAM is
> > > > consumed
> > > > by each process.
> > > >
> > > > > --Ilya
> > > > >
> > > > > 2015-11-06 17:27 GMT+03:00 Paul Belanger <pabelanger at redhat.com>:
> > > > >
> > > > > > On Mon, Nov 02, 2015 at 08:25:19AM -0500, Paul Belanger wrote:
> > > > > > > On Thu, Oct 29, 2015 at 07:14:00AM +0300, Ilya Shakhat wrote:
> > > > > > > > Hi Paul,
> > > > > > > >
> > > > > > > > Using existing dump makes the cold start much more pleasant.
> Full
> > > > data
> > > > > > > > reload usually needed only when some incompatible changes are
> > > > > > introduced. I
> > > > > > > > mark such commits with the warning note and bump release
> number. To
> > > > > > help
> > > > > > > > you with the initial setup I can grab the dump from
> production and
> > > > > > share it
> > > > > > > > with you. The raw size is about 2G, hope it compressible.
> > > > > > > >
> > > > > > > Great. I'm sure if you jump into #openstack-infra we can find a
> > > > > > > location to upload it too.
> > > > > > >
> > > > > > > > This week I'm at the summit. If you are not, we can schedule
> a chat
> > > > > > next
> > > > > > > > week, e.g. on Thu would be ok.
> > > > > > > >
> > > > > > > Yup, that works for me too. Again, I'll be idling in
> #openstack-infra
> > > > > > when you
> > > > > > > have free time.
> > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ilya
> > > > > > > >
> > > > > > > > 2015-10-28 18:20 GMT+03:00 Paul Belanger <
> pabelanger at redhat.com>:
> > > > > > > >
> > > > > > > > > Greetings Ilya,
> > > > > > > > >
> > > > > > > > > I was hoping you could help save me some time doing the
> initial
> > > > > > import of
> > > > > > > > > stackalytics.o.o by using a stackalytics-dump from your
> > > > instance? If
> > > > > > I
> > > > > > > > > understand correctly, this is how you currently do backups
> /
> > > > > > persistent
> > > > > > > > > data
> > > > > > > > > from memcached?
> > > > > > > > >
> > > > > > > > > Additionally, maybe we could setup some time next week to
> chat
> > > > about
> > > > > > how to
> > > > > > > > > currently do maintenance on stackalytics.org, so I can
> include
> > > > that
> > > > > > > > > functionality
> > > > > > > > > into puppet-stackalytics[1]. Things like how often you dump
> > > > memcached
> > > > > > > > > data, if
> > > > > > > > > any existing scripts exists, current backup / restore
> procedures.
> > > > > > > > >
> > > > > > > > > PB
> > > > > > > > >
> > > > > > > > > [1] https://github.com/openstack-infra/puppet-stackalytics
> > > > > > > > >
> > > > > > >
> > > > > > Just a follow up to this thread, we've successfully merged 2
> patches
> > > > that
> > > > > > now
> > > > > > allow -infra to launch stackalytics.o.o[1][2].  At the moment,
> we are
> > > > > > holding
> > > > > > off on doing so because we'd like to confirm with you the current
> > > > system
> > > > > > requirements for stackalytics.com.  In my testing I found 8GB
> not to
> > > > be
> > > > > > enough
> > > > > > RAM for importing the stats and running uwsgi.  We are
> considering
> > > > > > launching a
> > > > > > node with 15GB of RAM however would like to hear back from you
> first.
> > > > > >
> > > > > > Also, do you mind connecting with on of the infra-roots to
> discuss an
> > > > > > upload
> > > > > > location for your current memcached dump?
> > > > > >
> > > > > > ---
> > > > > > [1] https://review.openstack.org/#/c/190765/
> > > > > > [2] https://review.openstack.org/#/c/190708/
> > > > > >
> > > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20151111/43efbbf8/attachment.html>