Open Stack

On 06/09/2014 12:11 AM, Jesus M. Gonzalez-Barahona wrote:
> On Sun, 2014-06-08 at 13:32 -0400, Monty Taylor wrote:
>> On 06/02/2014 06:32 PM, Jesus M. Gonzalez-Barahona wrote:
>>> On Mon, 2014-06-02 at 14:31 -0700, Stefano Maffulli wrote:
>>>> Hello folks
>>>>
>>>> I wanted to get your thoughts on the idea to move the whole Bitergia
>>>> grimoire engine, scripts, database, etc to the openstack common
>>>> infrastructure. I'm sure everybody will want this so the question is
>>>> more about resources as in people willing to help Bitergia get their
>>>> machinery puppetized the "OpenStack-CI way".
>>>>
>>>> At the moment the various spiders and scripts run on a machine on
>>>> Bitergia's end and they drop the results of the elaboration (json, html,
>>>> css files and sql dumps) on activity.openstack.org/dash/.
>>>>
>>>> While this setup is convenient and has been working so far, I think
>>>> we've outgrown it. One of the areas that we want to do more with is the
>>>> datawarehouse built by Bitergia, enable it to serve other purposes too
>>>> and allow more collaboration from the community. For example, once the
>>>> tools are all out in one place, interested parties could build some sort
>>>> of service on top of your datawarehouse to export the data about the
>>>> affiliation. Others could build tools to 'fix' such affiliation, pulling
>>>> the various mailmap files used by gitdm and stackalytics.
>>>>
>>>> The question for OpenStack-CI is then: in the next weeks/months, is
>>>> there going to be someone free, from the CI team or someone willing to
>>>> join it (Dan?), to help Bitergia's team get their tools on our
>>>> infrastructure?
>>>
>>> Thanks for the introduction of the issue, Stefano.
>>>
>>> Just to clarify a bit the needs to move all the Grimoire machinery to a
>>> vm under openstack-ci, the software is split in three main parts:
>>
>> Awesome.
>>
>>> * MetricsGrimoire tools, which mine repositories (git, Launchpad, etc.),
>>> and store the data into a MySQL database (well, in fact, one schema per
>>> kind of repository).
>>>
>>> * The MySQL database itself.
>>>
>>> * The vizGrimoire tools, that run the analysis, produce JSON files, and
>>> the HTML/CSS/JavaScript files needed to serve the dashboard. All of this
>>> is for now static, which means that you only need to serve those files
>>> via HTTP, and you're done: no live queries to the database once the JSON
>>> files are produced.
>>>
>>> Right now, we produce the JSON files once a day. That means that
>>> MetricsGrimoire is first run once a day (the tools know how to get
>>> incremental information from repositories), data is updated in the
>>> database, and then vizGrimoire analysis is run. All of this is
>>> controlled by automator, the tool that is configured with the list of
>>> repos to analyze, the analysis to run, etc.
>>>
>>> MetricsGrimoire tools are written in Python. There are some Python
>>> dependencies beyond Python 2.7, but I guess all of them are
>>> straightforward from pypy.
>>
>> Where are these stored/installed from? Is it "pip install
>> MetricsGrimoire"? Is it in a git repo? Does this git repo want to move
>> into openstack tooling or does it want to stay where it is and have
>> openstack-infra consume releases?
> 
> Those are different tools in git repos. Providing them via pip could be
> a part of the process, anyway: that's something we have in the roadmap
> for a while. Right now, openstack-infra would consume the repositories.

git repos is fine - we install several things directly from git - just
was curious. I'm assuming based on this email so far that you want to
keep your git repos in github?

> More info about MetricsGrimoire, and its repos:
> 
> http://metricsgrimoire.github.io

Sweet. Thanks.

>>> The MySQL is a plain MySQL. It would be great having it in SSD, because
>>> that speeds up queries a lot. But probably it will work with regular
>>> disks.
>>
>> We use cloud-based databases from Rackspace's Trove service. I have no
>> idea what sort of disk is behind it - but MySQL databases are no problem.
> 
> ok. It's mainly a matter of performance, I guess Trove would be ok.
> 
>>> Most of vizGrimoire is also Python, but there is still some R code
>>> (we're currently moving towards a pure-Python implementation). Python
>>> and R dependencies are easy too (from pypy or CRAN).
>>
>> Same questions as above on MetricsGrimoire. Do you have docs anywhere on
>> installing these two things?
> 
> For MetricsGrimoire, it is very close to "just standard Python install".
> But vizGrimoire is another story. Right now, it is a moving target, so
> probably the best thing would be that we install it, and document the
> procedure in detail for you. In the process, we could automate some
> steps that are not automated yet.

Let's work through your script below and the puppet first. There's not
much value in having you install something on a server because then we
don't know anything about running it in the future, and if there is a
major failure and we have to rebuild, we won't be confident that our
automation can do the job. You have the code running currently and
publishing to activity - so I don't think there is an urgency in getting
the code running elsewhere- the real benefit is in getting it tied in to
normal process for us so we can run it together.

> In any case, some more info about vizGrimoire:
> 
> http://vizgrimoire.github.io
> 
> I guess the closer thing we have to a install doc is:
> 
> https://github.com/MetricsGrimoire/Automator/wiki/Install

Sweet. Putting it on my reading list.

> (Automator is the script that runs all the party)
> 
>>> We're usually deploying on Debian or Ubuntu, but we have some experience
>>> with other Linux-based OSs too.
>>
>> Great. All of our services run on Ubuntu Precise right now. We'll be
>> starting to think about an upgrade to trusty.
>>
>>> We could do the whole deployment, if you can provide us with a
>>> Debian/Ubuntu vm, including proper documentation to reproduce it if
>>> needed.
>>
>> All of our deployments are done in a fully automated manner driven from
>> our puppet repo: https://git.openstack.org/cgit/openstack-infra/config.
>> However - we'd be more than happy for you to write the puppet! (and
>> happy to help with pointers)
>>
>> I just wrote the puppet to run stackalytics in Infra:
>>
>> https://review.openstack.org/98656
>>
>> It's probably a great place to start in terms of looking at puppetizing
>> activity board. If someone can write up something similar to:
>>
>> https://wiki.openstack.org/wiki/Stackalytics/HowToRun
>>
>> I'll be more likely to do what I did today which is wake up on a Sunday
>> and lazily write some puppet while watching Rafael Nadal win his ninth
>> French Open. :)
> 
> ;-)
> 
> Thanks for the pointers, Monty. Therefore, I assume that the first thing
> to do is to have a detailed installation information and/or Puppet
> config files. We're not that familiar with Puppet, so maybe the best
> thing for us would be to start writing the detailed docs, and after that
> starting with the Puppet thing... Right?

Oh - you'll know puppet for sure by the time we get done with you. :)

Seriously though - more than happy to help drive the puppet side of
things if you have detailed instructions on how to install. I want you
to learn and be involved in that - but I also don't want you to be
totally blocked on the concept.