Open Stack

On Sun, 2014-06-08 at 13:32 -0400, Monty Taylor wrote:
> On 06/02/2014 06:32 PM, Jesus M. Gonzalez-Barahona wrote:
> > On Mon, 2014-06-02 at 14:31 -0700, Stefano Maffulli wrote:
> >> Hello folks
> >>
> >> I wanted to get your thoughts on the idea to move the whole Bitergia
> >> grimoire engine, scripts, database, etc to the openstack common
> >> infrastructure. I'm sure everybody will want this so the question is
> >> more about resources as in people willing to help Bitergia get their
> >> machinery puppetized the "OpenStack-CI way".
> >>
> >> At the moment the various spiders and scripts run on a machine on
> >> Bitergia's end and they drop the results of the elaboration (json, html,
> >> css files and sql dumps) on activity.openstack.org/dash/.
> >>
> >> While this setup is convenient and has been working so far, I think
> >> we've outgrown it. One of the areas that we want to do more with is the
> >> datawarehouse built by Bitergia, enable it to serve other purposes too
> >> and allow more collaboration from the community. For example, once the
> >> tools are all out in one place, interested parties could build some sort
> >> of service on top of your datawarehouse to export the data about the
> >> affiliation. Others could build tools to 'fix' such affiliation, pulling
> >> the various mailmap files used by gitdm and stackalytics.
> >>
> >> The question for OpenStack-CI is then: in the next weeks/months, is
> >> there going to be someone free, from the CI team or someone willing to
> >> join it (Dan?), to help Bitergia's team get their tools on our
> >> infrastructure?
> > 
> > Thanks for the introduction of the issue, Stefano.
> > 
> > Just to clarify a bit the needs to move all the Grimoire machinery to a
> > vm under openstack-ci, the software is split in three main parts:
> 
> Awesome.
> 
> > * MetricsGrimoire tools, which mine repositories (git, Launchpad, etc.),
> > and store the data into a MySQL database (well, in fact, one schema per
> > kind of repository).
> >
> > * The MySQL database itself.
> > 
> > * The vizGrimoire tools, that run the analysis, produce JSON files, and
> > the HTML/CSS/JavaScript files needed to serve the dashboard. All of this
> > is for now static, which means that you only need to serve those files
> > via HTTP, and you're done: no live queries to the database once the JSON
> > files are produced.
> > 
> > Right now, we produce the JSON files once a day. That means that
> > MetricsGrimoire is first run once a day (the tools know how to get
> > incremental information from repositories), data is updated in the
> > database, and then vizGrimoire analysis is run. All of this is
> > controlled by automator, the tool that is configured with the list of
> > repos to analyze, the analysis to run, etc.
> > 
> > MetricsGrimoire tools are written in Python. There are some Python
> > dependencies beyond Python 2.7, but I guess all of them are
> > straightforward from pypy.
> 
> Where are these stored/installed from? Is it "pip install
> MetricsGrimoire"? Is it in a git repo? Does this git repo want to move
> into openstack tooling or does it want to stay where it is and have
> openstack-infra consume releases?

Those are different tools in git repos. Providing them via pip could be
a part of the process, anyway: that's something we have in the roadmap
for a while. Right now, openstack-infra would consume the repositories.

More info about MetricsGrimoire, and its repos:

http://metricsgrimoire.github.io

> > The MySQL is a plain MySQL. It would be great having it in SSD, because
> > that speeds up queries a lot. But probably it will work with regular
> > disks.
> 
> We use cloud-based databases from Rackspace's Trove service. I have no
> idea what sort of disk is behind it - but MySQL databases are no problem.

ok. It's mainly a matter of performance, I guess Trove would be ok.

> > Most of vizGrimoire is also Python, but there is still some R code
> > (we're currently moving towards a pure-Python implementation). Python
> > and R dependencies are easy too (from pypy or CRAN).
> 
> Same questions as above on MetricsGrimoire. Do you have docs anywhere on
> installing these two things?

For MetricsGrimoire, it is very close to "just standard Python install".
But vizGrimoire is another story. Right now, it is a moving target, so
probably the best thing would be that we install it, and document the
procedure in detail for you. In the process, we could automate some
steps that are not automated yet.

In any case, some more info about vizGrimoire:

http://vizgrimoire.github.io

I guess the closer thing we have to a install doc is:

https://github.com/MetricsGrimoire/Automator/wiki/Install

(Automator is the script that runs all the party)

> > We're usually deploying on Debian or Ubuntu, but we have some experience
> > with other Linux-based OSs too.
> 
> Great. All of our services run on Ubuntu Precise right now. We'll be
> starting to think about an upgrade to trusty.
> 
> > We could do the whole deployment, if you can provide us with a
> > Debian/Ubuntu vm, including proper documentation to reproduce it if
> > needed.
> 
> All of our deployments are done in a fully automated manner driven from
> our puppet repo: https://git.openstack.org/cgit/openstack-infra/config.
> However - we'd be more than happy for you to write the puppet! (and
> happy to help with pointers)
> 
> I just wrote the puppet to run stackalytics in Infra:
> 
> https://review.openstack.org/98656
> 
> It's probably a great place to start in terms of looking at puppetizing
> activity board. If someone can write up something similar to:
> 
> https://wiki.openstack.org/wiki/Stackalytics/HowToRun
> 
> I'll be more likely to do what I did today which is wake up on a Sunday
> and lazily write some puppet while watching Rafael Nadal win his ninth
> French Open. :)

;-)

Thanks for the pointers, Monty. Therefore, I assume that the first thing
to do is to have a detailed installation information and/or Puppet
config files. We're not that familiar with Puppet, so maybe the best
thing for us would be to start writing the detailed docs, and after that
starting with the Puppet thing... Right?

Saludos,

	Jesus.

> > Please, let us know which kind of other details you may need.
> > 
> > 	Jesus.
> > 
> > 
> 

-- 
-- 
Bitergia: http://bitergia.com http://blog.bitergia.com