Open Stack

On Mon, Nov 09, 2015 at 06:06:47PM -0500, Paul Belanger wrote:
> On Mon, Nov 09, 2015 at 12:28:27PM +0300, Ilya Shakhat wrote:
> > Prod is based on nginx. It is configured to forward dynamic requests to
> > uwsgi process, thus nginx itself does not consume much memory.
> > 
> > The config is following:
> > server {
> > listen   80;
> > server_name stackalytics.com;
> > location / {
> >         uwsgi_pass  unix:///tmp/stackalytics.sock;
> >         include     uwsgi_params;
> > }
> > location  /static/ {
> >         alias  /opt/stack/stackalytics/stackalytics/dashboard/static/;
> > }
> > }
> > 
> 
> So, for people playing at home, nginx + uwsgi is more forgiving[1] then apache.
> 
> [1] http://pastebin.com/3V7XEmZL
> 
I made a mistake in reading my output, I was looking at virtual memory.  Either
way, do you have some time this morning to talk in openstack-infra?  I'd like to
confirm some final numbers we are seeing after a full import.

Also, every hour we are seeing a spike in our wsgi processes. If I understand,
it takes about 10mins to re-read the data from memcached, does that sounds
correct?

How often are you refreshing the data on stackalytics.com? And are people
actually getting hit with the 10min delay when accessing the page?

> > --Ilya
> > 
> > 2015-11-09 4:56 GMT+03:00 Paul Belanger <pabelanger at redhat.com>:
> > 
> > > On Fri, Nov 06, 2015 at 06:59:15PM +0300, Ilya Shakhat wrote:
> > > > I confirm that 8Gb is not enough. Currently the production is 16Gb - 3.5
> > > > for memcached, 3.5 for every uwsgi process and up to 2 may be consumed by
> > > > the processor.
> > > >
> > > Out of curiosity, are you using nginx or apache?  I've currently setup
> > > Apache,
> > > but might test nginx tomorrow. If you are using nginx how much RAM is
> > > consumed
> > > by each process.
> > >
> > > > --Ilya
> > > >
> > > > 2015-11-06 17:27 GMT+03:00 Paul Belanger <pabelanger at redhat.com>:
> > > >
> > > > > On Mon, Nov 02, 2015 at 08:25:19AM -0500, Paul Belanger wrote:
> > > > > > On Thu, Oct 29, 2015 at 07:14:00AM +0300, Ilya Shakhat wrote:
> > > > > > > Hi Paul,
> > > > > > >
> > > > > > > Using existing dump makes the cold start much more pleasant. Full
> > > data
> > > > > > > reload usually needed only when some incompatible changes are
> > > > > introduced. I
> > > > > > > mark such commits with the warning note and bump release number. To
> > > > > help
> > > > > > > you with the initial setup I can grab the dump from production and
> > > > > share it
> > > > > > > with you. The raw size is about 2G, hope it compressible.
> > > > > > >
> > > > > > Great. I'm sure if you jump into #openstack-infra we can find a
> > > > > > location to upload it too.
> > > > > >
> > > > > > > This week I'm at the summit. If you are not, we can schedule a chat
> > > > > next
> > > > > > > week, e.g. on Thu would be ok.
> > > > > > >
> > > > > > Yup, that works for me too. Again, I'll be idling in #openstack-infra
> > > > > when you
> > > > > > have free time.
> > > > > >
> > > > > > > Thanks,
> > > > > > > Ilya
> > > > > > >
> > > > > > > 2015-10-28 18:20 GMT+03:00 Paul Belanger <pabelanger at redhat.com>:
> > > > > > >
> > > > > > > > Greetings Ilya,
> > > > > > > >
> > > > > > > > I was hoping you could help save me some time doing the initial
> > > > > import of
> > > > > > > > stackalytics.o.o by using a stackalytics-dump from your
> > > instance? If
> > > > > I
> > > > > > > > understand correctly, this is how you currently do backups /
> > > > > persistent
> > > > > > > > data
> > > > > > > > from memcached?
> > > > > > > >
> > > > > > > > Additionally, maybe we could setup some time next week to chat
> > > about
> > > > > how to
> > > > > > > > currently do maintenance on stackalytics.org, so I can include
> > > that
> > > > > > > > functionality
> > > > > > > > into puppet-stackalytics[1]. Things like how often you dump
> > > memcached
> > > > > > > > data, if
> > > > > > > > any existing scripts exists, current backup / restore procedures.
> > > > > > > >
> > > > > > > > PB
> > > > > > > >
> > > > > > > > [1] https://github.com/openstack-infra/puppet-stackalytics
> > > > > > > >
> > > > > >
> > > > > Just a follow up to this thread, we've successfully merged 2 patches
> > > that
> > > > > now
> > > > > allow -infra to launch stackalytics.o.o[1][2].  At the moment, we are
> > > > > holding
> > > > > off on doing so because we'd like to confirm with you the current
> > > system
> > > > > requirements for stackalytics.com.  In my testing I found 8GB not to
> > > be
> > > > > enough
> > > > > RAM for importing the stats and running uwsgi.  We are considering
> > > > > launching a
> > > > > node with 15GB of RAM however would like to hear back from you first.
> > > > >
> > > > > Also, do you mind connecting with on of the infra-roots to discuss an
> > > > > upload
> > > > > location for your current memcached dump?
> > > > >
> > > > > ---
> > > > > [1] https://review.openstack.org/#/c/190765/
> > > > > [2] https://review.openstack.org/#/c/190708/
> > > > >
> > >