Open Stack

On Mon, 2014-06-09 at 09:31 -0700, Monty Taylor wrote:
> On 06/09/2014 12:11 AM, Jesus M. Gonzalez-Barahona wrote:
> > On Sun, 2014-06-08 at 13:32 -0400, Monty Taylor wrote:
> >> On 06/02/2014 06:32 PM, Jesus M. Gonzalez-Barahona wrote:
> >>> On Mon, 2014-06-02 at 14:31 -0700, Stefano Maffulli wrote:
> >>>> Hello folks
> >>>>
> >>>> I wanted to get your thoughts on the idea to move the whole Bitergia
> >>>> grimoire engine, scripts, database, etc to the openstack common
> >>>> infrastructure. I'm sure everybody will want this so the question is
> >>>> more about resources as in people willing to help Bitergia get their
> >>>> machinery puppetized the "OpenStack-CI way".
> >>>>
> >>>> At the moment the various spiders and scripts run on a machine on
> >>>> Bitergia's end and they drop the results of the elaboration (json, html,
> >>>> css files and sql dumps) on activity.openstack.org/dash/.
> >>>>
> >>>> While this setup is convenient and has been working so far, I think
> >>>> we've outgrown it. One of the areas that we want to do more with is the
> >>>> datawarehouse built by Bitergia, enable it to serve other purposes too
> >>>> and allow more collaboration from the community. For example, once the
> >>>> tools are all out in one place, interested parties could build some sort
> >>>> of service on top of your datawarehouse to export the data about the
> >>>> affiliation. Others could build tools to 'fix' such affiliation, pulling
> >>>> the various mailmap files used by gitdm and stackalytics.
> >>>>
> >>>> The question for OpenStack-CI is then: in the next weeks/months, is
> >>>> there going to be someone free, from the CI team or someone willing to
> >>>> join it (Dan?), to help Bitergia's team get their tools on our
> >>>> infrastructure?
> >>>
> >>> Thanks for the introduction of the issue, Stefano.
> >>>
> >>> Just to clarify a bit the needs to move all the Grimoire machinery to a
> >>> vm under openstack-ci, the software is split in three main parts:
> >>
> >> Awesome.
> >>
> >>> * MetricsGrimoire tools, which mine repositories (git, Launchpad, etc.),
> >>> and store the data into a MySQL database (well, in fact, one schema per
> >>> kind of repository).
> >>>
> >>> * The MySQL database itself.
> >>>
> >>> * The vizGrimoire tools, that run the analysis, produce JSON files, and
> >>> the HTML/CSS/JavaScript files needed to serve the dashboard. All of this
> >>> is for now static, which means that you only need to serve those files
> >>> via HTTP, and you're done: no live queries to the database once the JSON
> >>> files are produced.
> >>>
> >>> Right now, we produce the JSON files once a day. That means that
> >>> MetricsGrimoire is first run once a day (the tools know how to get
> >>> incremental information from repositories), data is updated in the
> >>> database, and then vizGrimoire analysis is run. All of this is
> >>> controlled by automator, the tool that is configured with the list of
> >>> repos to analyze, the analysis to run, etc.
> >>>
> >>> MetricsGrimoire tools are written in Python. There are some Python
> >>> dependencies beyond Python 2.7, but I guess all of them are
> >>> straightforward from pypy.
> >>
> >> Where are these stored/installed from? Is it "pip install
> >> MetricsGrimoire"? Is it in a git repo? Does this git repo want to move
> >> into openstack tooling or does it want to stay where it is and have
> >> openstack-infra consume releases?
> > 
> > Those are different tools in git repos. Providing them via pip could be
> > a part of the process, anyway: that's something we have in the roadmap
> > for a while. Right now, openstack-infra would consume the repositories.
> 
> git repos is fine - we install several things directly from git - just
> was curious. I'm assuming based on this email so far that you want to
> keep your git repos in github?
> 
> > More info about MetricsGrimoire, and its repos:
> > 
> > http://metricsgrimoire.github.io
> 
> Sweet. Thanks.
> 
> >>> The MySQL is a plain MySQL. It would be great having it in SSD, because
> >>> that speeds up queries a lot. But probably it will work with regular
> >>> disks.
> >>
> >> We use cloud-based databases from Rackspace's Trove service. I have no
> >> idea what sort of disk is behind it - but MySQL databases are no problem.
> > 
> > ok. It's mainly a matter of performance, I guess Trove would be ok.
> > 
> >>> Most of vizGrimoire is also Python, but there is still some R code
> >>> (we're currently moving towards a pure-Python implementation). Python
> >>> and R dependencies are easy too (from pypy or CRAN).
> >>
> >> Same questions as above on MetricsGrimoire. Do you have docs anywhere on
> >> installing these two things?
> > 
> > For MetricsGrimoire, it is very close to "just standard Python install".
> > But vizGrimoire is another story. Right now, it is a moving target, so
> > probably the best thing would be that we install it, and document the
> > procedure in detail for you. In the process, we could automate some
> > steps that are not automated yet.
> 
> Let's work through your script below and the puppet first. There's not
> much value in having you install something on a server because then we
> don't know anything about running it in the future, and if there is a
> major failure and we have to rebuild, we won't be confident that our
> automation can do the job. You have the code running currently and
> publishing to activity - so I don't think there is an urgency in getting
> the code running elsewhere- the real benefit is in getting it tied in to
> normal process for us so we can run it together.
> 
> > In any case, some more info about vizGrimoire:
> > 
> > http://vizgrimoire.github.io
> > 
> > I guess the closer thing we have to a install doc is:
> > 
> > https://github.com/MetricsGrimoire/Automator/wiki/Install
> 
> Sweet. Putting it on my reading list.
> 
> > (Automator is the script that runs all the party)
> > 
> >>> We're usually deploying on Debian or Ubuntu, but we have some experience
> >>> with other Linux-based OSs too.
> >>
> >> Great. All of our services run on Ubuntu Precise right now. We'll be
> >> starting to think about an upgrade to trusty.
> >>
> >>> We could do the whole deployment, if you can provide us with a
> >>> Debian/Ubuntu vm, including proper documentation to reproduce it if
> >>> needed.
> >>
> >> All of our deployments are done in a fully automated manner driven from
> >> our puppet repo: https://git.openstack.org/cgit/openstack-infra/config.
> >> However - we'd be more than happy for you to write the puppet! (and
> >> happy to help with pointers)
> >>
> >> I just wrote the puppet to run stackalytics in Infra:
> >>
> >> https://review.openstack.org/98656
> >>
> >> It's probably a great place to start in terms of looking at puppetizing
> >> activity board. If someone can write up something similar to:
> >>
> >> https://wiki.openstack.org/wiki/Stackalytics/HowToRun
> >>
> >> I'll be more likely to do what I did today which is wake up on a Sunday
> >> and lazily write some puppet while watching Rafael Nadal win his ninth
> >> French Open. :)
> > 
> > ;-)
> > 
> > Thanks for the pointers, Monty. Therefore, I assume that the first thing
> > to do is to have a detailed installation information and/or Puppet
> > config files. We're not that familiar with Puppet, so maybe the best
> > thing for us would be to start writing the detailed docs, and after that
> > starting with the Puppet thing... Right?
> 
> Oh - you'll know puppet for sure by the time we get done with you. :)
> 
> Seriously though - more than happy to help drive the puppet side of
> things if you have detailed instructions on how to install. I want you
> to learn and be involved in that - but I also don't want you to be
> totally blocked on the concept.
> 

Thanks a lot, Monty,

>From this discussion, I guess the first step is on our side, and it
would be producing detailed instructions on how we install the
machinery, so that those can be a first step for the puppetization,
right?

We're going to schedule this asap. I'll let you know when exactly, so
that we can start with the real game.

Saludos,

	Jesus.

-- 
-- 
Bitergia: http://bitergia.com http://blog.bitergia.com