Open Stack

Hi all, I'm sending this message again in plain text not html.

Hi Eoghan,

> 2. What technology is the metrics store based on, at a high level?
> (e.g. RDBMS accessed via sqlalchemy, noSQL store such as MongoDB,
> etc.)
>
> 3. How are you performing datapoint aggregation? (e.g. eagerly on
> metric ingestion, versus lazily on get)
>

In addition to Joonwon's,

Synaps basically stores metric data aggregated with 1 minute resolution in hybrid way, in-memory of storm workers and Cassandra database. And it rolls those aggregated datapoints up again when another resolutions of statistics are requested. 

Storm holds time-series datapoints within flexible sliding window per metric based on its longest period of alarm so that it can evaluate alarms without any reading operation from Cassandra database.

Synaps was designed with inspiration by following posts:
[1] https://www.cloudkick.com/blog/2010/mar/02/4_months_with_cassandra/
[2] http://pyvideo.org/video/675/storm-the-hadoop-of-realtime-stream-processing

Hi Endre,

> But how does Synaps then difffer from Ceilometer? It looks to be duplicated
> efforts here?
> 

I think I don't understand Ceilometer enough but as I know its main purpose is to be a billing system based on hourly collected data while Synaps is focusing to be a monitoring service which deals with more amount of data more frequently.

Thank you,
June Yi

------- Original Message -------
Sender : Endre Karlson<endre.karlson at gmail.com>
Date : 2012-10-11 06:04 (GMT+09:00)
Title : Re: [openstack-dev] Introducing Synaps project that provides AWS CloudWatch compatible API

But how does Synaps then difffer from Ceilometer? It looks to be duplicated efforts here?


Endre.


2012/10/10 Joonwon Lee <joonwon7.lee at samsung.com>

Hi, Eoghan,
1. No, we don't have any relation with ceilometer.
2. Cassandra and Storm (as explained in http://wiki.openstack.org/Synaps)
3. As far as I understand, aggregation is performed lazily on the request.
 (Metrics are collected every minute and aggregated according to the request.)

Deok-June Yi can explain with more details tomorrow, if you want.
Thanks,
Joonwon Lee


------- Original Message -------
Sender : Eoghan Glynn<eglynn at redhat.com>

Date : Oct 10, 2012 21:14 (GMT+09:00)

Title : Re: [openstack-dev] Introducing Synaps project that provides AWS CloudWatch compatible API



Hi Joonwon Lee,

> Unfortunately, we could not attend the Design Summit. We also need
> some time
> before opening our source codes, mostly for our internal process. We
> will try to shorten this time.

It's unfortunate you guys won't be at the summit, as it would have
been great to discuss in that forum.

But that said, the quicker you can push the code through your internal
process, the better.


> Now we're preparing the overview of our project and other
> documentation.
> If you have any questions, feel free to ask them at any time.

A few immediate questions that spring to mind:

1. Are you re-using any of the ceilometer[1] infrastructure for
   your metrics collection pipeline?

2. What technology is the metrics store based on, at a high level?
   (e.g. RDBMS accessed via sqlalchemy, noSQL store such as MongoDB,
    etc.)

3. How are you performing datapoint aggregation? (e.g. eagerly on
   metric ingestion, versus lazily on get)

Cheers,
Eoghan

[1] http://ceilometer.readthedocs.org/en/latest/architecture.html


> Here are answers of June Yi for Steven Hardy's questions below.
> > - Is the metric collection secure/authenticated?
> > - Are metrics collected from the hypervisors or in-instance?
> > - Can service monitoring (in instance) metrics be collected?
> > - Is the API authenticated (ie via keystone ec2 credentials?)
>
> All metrics are collected using the API. The API is authenticated via
> auth
> module came from Nova.Authenticate WSGI middleware which supports AWS
> signature v2. It works well with LdapDriver. But we did not tested if
> it is
> working with keystone yet.
>
> Default metrics (i.e. CPUUtilization, disk IO, network IO) are
> collected
> from hypervisors. Custom metrics can be collected from in-instance or
> any other source.
>
> Regards,
> --
> Joonwon Lee, principal software engineer, Samsung SDS
>
>
> ------- Original Message -------
> Sender : Eoghan Glynn

> Date : 2012-10-10 18:59 (GMT+09:00)
> Title : Re: [openstack-dev] Introducing Synaps project that provides
> AWS CloudWatch compatible API
>
>
>
> > > > I'm working for Synaps project that provides AWS CloudWatch
> > > > compatible API at Samsung SDS.
> > > > Currently, we have plan to open the project to the community.
> > > >
> > > > Here I introduce Synaps project:
> > > > http://wiki.openstack.org/Synaps
> > >
> > > Details are a bit scarce but at first glance there seem to be an
> > > overlap
> > > with the CloudWatch work in the Heat project ?
> >
> > There certainly seems to be from the description, but pretty hard
> > to
> > evaluate until we see some code ;)
>
> Yes, I would second that.
>
> Once the code is out the open, the community will be much better
> placed
> to judge which approach will make sense to concentrate efforts on.
>
> June Yi - are you planning on attending the design summit?
>
> Cheers,
> Eoghan
>
>
> > > Any chance the two groups could meet at the Design Summit next
> > > week
> > > and
> > > make sure we don't duplicate effort here ?
> > >
> > > Angus Saskeld has a session on CloudWatch in the Incubation room:
> > > http://openstacksummitfall2012.sched.org/event/60b0bcf1253efcb3dd72dff531d59d26
> >
> > Angus Salkeld and Steve Dake will be at the Summit and could
> > discuss,
> > also
> > possible overlap with metric collection in ceilometer (has been
> > discussed
> > previously).
> >
> > I have several questions which will be make/break for the heat
> > usage
> > of
> > Cloudwatch:
> >
> > - Is the metric collection secure/authenticated?
> > - Are metrics collected from the hypervisors or in-instance?
> > - Can service monitoring (in instance) metrics be collected?
> > - Is the API authenticated (ie via keystone ec2 credentials?)
> >
> > Many more questions, need to see code and evaluate if we could use
> > this for
> > heat or if we continue with our own implementation.
> >
> > --
> > Steve Hardy
> > Red Hat Engineering, Cloud
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev