Open Stack

On Wed, Oct 10, 2012 at 10:01 PM, Deok-June Yi <june.yi at samsung.com> wrote:

>  Hi Eoghan,
>
> > 2. What technology is the metrics store based on, at a high level?
> >    (e.g. RDBMS accessed via sqlalchemy, noSQL store such as MongoDB,
> >     etc.)
> >
> > 3. How are you performing datapoint aggregation? (e.g. eagerly on
> >    metric ingestion, versus lazily on get)
> >
>
> In addition to Joonwon's,
>
> Synaps basically stores metric data aggregated with 1 minute resolution in
> hybrid way, in-memory of storm workers and Cassandra database. And it rolls
> those aggregated datapoints up again when another resolutions of statistics
> are requested.
>
> Storm holds time-series datapoints within flexible sliding window per
> metric based on its longest period of alarm so that it can evaluate alarms
> without any reading operation from Cassandra database.
>
> Synaps was designed with inspiration by following posts:
>
> [1] https://www.cloudkick.com/blog/2010/mar/02/4_months_with_cassandra/
> [2]
> http://pyvideo.org/video/675/storm-the-hadoop-of-realtime-stream-processing
>
>
> Hi Endre,
>
> > But how does Synaps then difffer from Ceilometer? It looks to be
> duplicated
> > efforts here?
> >
>
> I think I don't understand Ceilometer enough but as I know its main
> purpose is to be a billing system based on hourly collected data while
> Synaps is focusing to be a monitoring service which deals with more amount
> of data more frequently.
>
>
Ceilometer collects data for use by a billing system, but it is not itself
a billing system. It sounds like Synaps collects metrics far more
frequently than ceilometer does.

Can you share a list of the things Synaps measures?

Doug


>
>
> Thank you,
> June Yi
>
>
>
>
>
> ------- *Original Message* -------
>
> *Sender* : Endre Karlson<endre.karlson at gmail.com>
>
> *Date* : 2012-10-11 06:04 (GMT+09:00)
>
> *Title* : Re: [openstack-dev] Introducing Synaps project that provides
> AWS CloudWatch compatible API
>
>
> But how does Synaps then difffer from Ceilometer? It looks to be
> duplicated efforts here?
>
> Endre.
>
> 2012/10/10 Joonwon Lee <joonwon7.lee at samsung.com>
>
>> Hi, Eoghan,
>> 1. No, we don't have any relation with ceilometer.
>> 2. Cassandra and Storm (as explained in http://wiki.openstack.org/Synaps)
>> 3. As far as I understand, aggregation is performed lazily on the request.
>>  (Metrics are collected every minute and aggregated according to the
>> request.)
>>
>> Deok-June Yi can explain with more details tomorrow, if you want.
>> Thanks,
>> Joonwon Lee
>>
>> ------- Original Message -------
>> Sender : Eoghan Glynn<eglynn at redhat.com>
>> Date : Oct 10, 2012 21:14 (GMT+09:00)
>> Title : Re: [openstack-dev] Introducing Synaps project that provides AWS
>> CloudWatch compatible API
>>
>>
>>  Hi Joonwon Lee,
>>
>> > Unfortunately, we could not attend the Design Summit. We also need
>> > some time
>> > before opening our source codes, mostly for our internal process. We
>> > will try to shorten this time.
>>
>> It's unfortunate you guys won't be at the summit, as it would have
>> been great to discuss in that forum.
>>
>> But that said, the quicker you can push the code through your internal
>> process, the better.
>>
>>
>> > Now we're preparing the overview of our project and other
>> > documentation.
>> > If you have any questions, feel free to ask them at any time.
>>
>> A few immediate questions that spring to mind:
>>
>> 1. Are you re-using any of the ceilometer[1] infrastructure for
>>    your metrics collection pipeline?
>>
>> 2. What technology is the metrics store based on, at a high level?
>>    (e.g. RDBMS accessed via sqlalchemy, noSQL store such as MongoDB,
>>     etc.)
>>
>> 3. How are you performing datapoint aggregation? (e.g. eagerly on
>>    metric ingestion, versus lazily on get)
>>
>> Cheers,
>> Eoghan
>>
>> [1] http://ceilometer.readthedocs.org/en/latest/architecture.html
>>
>>
>> > Here are answers of June Yi for Steven Hardy's questions below.
>> > > - Is the metric collection secure/authenticated?
>> > > - Are metrics collected from the hypervisors or in-instance?
>> > > - Can service monitoring (in instance) metrics be collected?
>> > > - Is the API authenticated (ie via keystone ec2 credentials?)
>> >
>> > All metrics are collected using the API. The API is authenticated via
>> > auth
>> > module came from Nova.Authenticate WSGI middleware which supports AWS
>> > signature v2. It works well with LdapDriver. But we did not tested if
>> > it is
>> > working with keystone yet.
>> >
>> > Default metrics (i.e. CPUUtilization, disk IO, network IO) are
>> > collected
>> > from hypervisors. Custom metrics can be collected from in-instance or
>> > any other source.
>> >
>> > Regards,
>> > --
>> > Joonwon Lee, principal software engineer, Samsung SDS
>> >
>> >
>> > ------- Original Message -------
>> > Sender : Eoghan Glynn
>>  > Date : 2012-10-10 18:59 (GMT+09:00)
>> > Title : Re: [openstack-dev] Introducing Synaps project that provides
>> > AWS CloudWatch compatible API
>> >
>> >
>> >
>> > > > > I'm working for Synaps project that provides AWS CloudWatch
>> > > > > compatible API at Samsung SDS.
>> > > > > Currently, we have plan to open the project to the community.
>> > > > >
>> > > > > Here I introduce Synaps project:
>> > > > > http://wiki.openstack.org/Synaps
>> > > >
>> > > > Details are a bit scarce but at first glance there seem to be an
>> > > > overlap
>> > > > with the CloudWatch work in the Heat project ?
>> > >
>> > > There certainly seems to be from the description, but pretty hard
>> > > to
>> > > evaluate until we see some code ;)
>> >
>> > Yes, I would second that.
>> >
>> > Once the code is out the open, the community will be much better
>> > placed
>> > to judge which approach will make sense to concentrate efforts on.
>> >
>> > June Yi - are you planning on attending the design summit?
>> >
>> > Cheers,
>> > Eoghan
>> >
>> >
>> > > > Any chance the two groups could meet at the Design Summit next
>> > > > week
>> > > > and
>> > > > make sure we don't duplicate effort here ?
>> > > >
>> > > > Angus Saskeld has a session on CloudWatch in the Incubation room:
>> > > >
>> http://openstacksummitfall2012.sched.org/event/60b0bcf1253efcb3dd72dff531d59d26
>> > >
>> > > Angus Salkeld and Steve Dake will be at the Summit and could
>> > > discuss,
>> > > also
>> > > possible overlap with metric collection in ceilometer (has been
>> > > discussed
>> > > previously).
>> > >
>> > > I have several questions which will be make/break for the heat
>> > > usage
>> > > of
>> > > Cloudwatch:
>> > >
>> > > - Is the metric collection secure/authenticated?
>> > > - Are metrics collected from the hypervisors or in-instance?
>> > > - Can service monitoring (in instance) metrics be collected?
>> > > - Is the API authenticated (ie via keystone ec2 credentials?)
>> > >
>> > > Many more questions, need to see code and evaluate if we could use
>> > > this for
>> > > heat or if we continue with our own implementation.
>> > >
>> > > --
>> > > Steve Hardy
>> > > Red Hat Engineering, Cloud
>> > >
>> > > _______________________________________________
>> > > OpenStack-dev mailing list
>> > > OpenStack-dev at lists.openstack.org
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121010/4cf9252e/attachment-0001.html>