Open Stack

On Mon, 2014-06-02 at 14:31 -0700, Stefano Maffulli wrote:
> Hello folks
> 
> I wanted to get your thoughts on the idea to move the whole Bitergia
> grimoire engine, scripts, database, etc to the openstack common
> infrastructure. I'm sure everybody will want this so the question is
> more about resources as in people willing to help Bitergia get their
> machinery puppetized the "OpenStack-CI way".
> 
> At the moment the various spiders and scripts run on a machine on
> Bitergia's end and they drop the results of the elaboration (json, html,
> css files and sql dumps) on activity.openstack.org/dash/.
> 
> While this setup is convenient and has been working so far, I think
> we've outgrown it. One of the areas that we want to do more with is the
> datawarehouse built by Bitergia, enable it to serve other purposes too
> and allow more collaboration from the community. For example, once the
> tools are all out in one place, interested parties could build some sort
> of service on top of your datawarehouse to export the data about the
> affiliation. Others could build tools to 'fix' such affiliation, pulling
> the various mailmap files used by gitdm and stackalytics.
> 
> The question for OpenStack-CI is then: in the next weeks/months, is
> there going to be someone free, from the CI team or someone willing to
> join it (Dan?), to help Bitergia's team get their tools on our
> infrastructure?

Thanks for the introduction of the issue, Stefano.

Just to clarify a bit the needs to move all the Grimoire machinery to a
vm under openstack-ci, the software is split in three main parts:

* MetricsGrimoire tools, which mine repositories (git, Launchpad, etc.),
and store the data into a MySQL database (well, in fact, one schema per
kind of repository).

* The MySQL database itself.

* The vizGrimoire tools, that run the analysis, produce JSON files, and
the HTML/CSS/JavaScript files needed to serve the dashboard. All of this
is for now static, which means that you only need to serve those files
via HTTP, and you're done: no live queries to the database once the JSON
files are produced.

Right now, we produce the JSON files once a day. That means that
MetricsGrimoire is first run once a day (the tools know how to get
incremental information from repositories), data is updated in the
database, and then vizGrimoire analysis is run. All of this is
controlled by automator, the tool that is configured with the list of
repos to analyze, the analysis to run, etc.

MetricsGrimoire tools are written in Python. There are some Python
dependencies beyond Python 2.7, but I guess all of them are
straightforward from pypy.

The MySQL is a plain MySQL. It would be great having it in SSD, because
that speeds up queries a lot. But probably it will work with regular
disks.

Most of vizGrimoire is also Python, but there is still some R code
(we're currently moving towards a pure-Python implementation). Python
and R dependencies are easy too (from pypy or CRAN).

We're usually deploying on Debian or Ubuntu, but we have some experience
with other Linux-based OSs too.

We could do the whole deployment, if you can provide us with a
Debian/Ubuntu vm, including proper documentation to reproduce it if
needed.

Please, let us know which kind of other details you may need.

	Jesus.


-- 
-- 
Bitergia: http://bitergia.com http://blog.bitergia.com