Open Stack

On 06/02/2014 06:32 PM, Jesus M. Gonzalez-Barahona wrote:
> On Mon, 2014-06-02 at 14:31 -0700, Stefano Maffulli wrote:
>> Hello folks
>>
>> I wanted to get your thoughts on the idea to move the whole Bitergia
>> grimoire engine, scripts, database, etc to the openstack common
>> infrastructure. I'm sure everybody will want this so the question is
>> more about resources as in people willing to help Bitergia get their
>> machinery puppetized the "OpenStack-CI way".
>>
>> At the moment the various spiders and scripts run on a machine on
>> Bitergia's end and they drop the results of the elaboration (json, html,
>> css files and sql dumps) on activity.openstack.org/dash/.
>>
>> While this setup is convenient and has been working so far, I think
>> we've outgrown it. One of the areas that we want to do more with is the
>> datawarehouse built by Bitergia, enable it to serve other purposes too
>> and allow more collaboration from the community. For example, once the
>> tools are all out in one place, interested parties could build some sort
>> of service on top of your datawarehouse to export the data about the
>> affiliation. Others could build tools to 'fix' such affiliation, pulling
>> the various mailmap files used by gitdm and stackalytics.
>>
>> The question for OpenStack-CI is then: in the next weeks/months, is
>> there going to be someone free, from the CI team or someone willing to
>> join it (Dan?), to help Bitergia's team get their tools on our
>> infrastructure?
> 
> Thanks for the introduction of the issue, Stefano.
> 
> Just to clarify a bit the needs to move all the Grimoire machinery to a
> vm under openstack-ci, the software is split in three main parts:

Awesome.

> * MetricsGrimoire tools, which mine repositories (git, Launchpad, etc.),
> and store the data into a MySQL database (well, in fact, one schema per
> kind of repository).
>
> * The MySQL database itself.
> 
> * The vizGrimoire tools, that run the analysis, produce JSON files, and
> the HTML/CSS/JavaScript files needed to serve the dashboard. All of this
> is for now static, which means that you only need to serve those files
> via HTTP, and you're done: no live queries to the database once the JSON
> files are produced.
> 
> Right now, we produce the JSON files once a day. That means that
> MetricsGrimoire is first run once a day (the tools know how to get
> incremental information from repositories), data is updated in the
> database, and then vizGrimoire analysis is run. All of this is
> controlled by automator, the tool that is configured with the list of
> repos to analyze, the analysis to run, etc.
> 
> MetricsGrimoire tools are written in Python. There are some Python
> dependencies beyond Python 2.7, but I guess all of them are
> straightforward from pypy.

Where are these stored/installed from? Is it "pip install
MetricsGrimoire"? Is it in a git repo? Does this git repo want to move
into openstack tooling or does it want to stay where it is and have
openstack-infra consume releases?

> The MySQL is a plain MySQL. It would be great having it in SSD, because
> that speeds up queries a lot. But probably it will work with regular
> disks.

We use cloud-based databases from Rackspace's Trove service. I have no
idea what sort of disk is behind it - but MySQL databases are no problem.

> Most of vizGrimoire is also Python, but there is still some R code
> (we're currently moving towards a pure-Python implementation). Python
> and R dependencies are easy too (from pypy or CRAN).

Same questions as above on MetricsGrimoire. Do you have docs anywhere on
installing these two things?

> We're usually deploying on Debian or Ubuntu, but we have some experience
> with other Linux-based OSs too.

Great. All of our services run on Ubuntu Precise right now. We'll be
starting to think about an upgrade to trusty.

> We could do the whole deployment, if you can provide us with a
> Debian/Ubuntu vm, including proper documentation to reproduce it if
> needed.

All of our deployments are done in a fully automated manner driven from
our puppet repo: https://git.openstack.org/cgit/openstack-infra/config.
However - we'd be more than happy for you to write the puppet! (and
happy to help with pointers)

I just wrote the puppet to run stackalytics in Infra:

https://review.openstack.org/98656

It's probably a great place to start in terms of looking at puppetizing
activity board. If someone can write up something similar to:

https://wiki.openstack.org/wiki/Stackalytics/HowToRun

I'll be more likely to do what I did today which is wake up on a Sunday
and lazily write some puppet while watching Rafael Nadal win his ninth
French Open. :)

> Please, let us know which kind of other details you may need.
> 
> 	Jesus.
> 
>