Open Stack

Thanks for the quick response June Yi,

> Synaps basically stores metric data aggregated with 1 minute
> resolution in hybrid way, in-memory of storm workers and Cassandra
> database. And it rolls those aggregated datapoints up again when
> another resolutions of statistics are requested.

Are all aggregations (sum, min, max, average, sample count)
for the 60s metrics stored across all dimensions (i.e. instance ID,
instance type, image ID etc. for AWS/EC2 metrics)?

Or are some or all aggregation-over-dimensions only calculated lazily
if and when a corresponding call to GetMetricStatistics occurs?
 
> Storm holds time-series datapoints within flexible sliding window per
> metric based on its longest period of alarm so that it can evaluate
> alarms without any reading operation from Cassandra database.

How do you handle metric backfill? Say for example at time t0 a
delayed datapoint is received for t0-600s and you're evaluating
a monitor with say 3x5min evaluation periods. Is the in-memory
representation updated or somehow marked as stale so as to force
a re-read from Cassandra?

Cheers,
Eoghan


> Synaps was designed with inspiration by following posts:
> [1]
> https://www.cloudkick.com/blog/2010/mar/02/4_months_with_cassandra/
> [2]
> http://pyvideo.org/video/675/storm-the-hadoop-of-realtime-stream-processing
> 
> Hi Endre,
> 
> > But how does Synaps then difffer from Ceilometer? It looks to be
> > duplicated
> > efforts here?
> > 
> 
> I think I don't understand Ceilometer enough but as I know its main
> purpose is to be a billing system based on hourly collected data
> while Synaps is focusing to be a monitoring service which deals with
> more amount of data more frequently.
> 
> Thank you,
> June Yi
> 
> ------- Original Message -------
> Sender : Endre Karlson<endre.karlson at gmail.com>
> Date : 2012-10-11 06:04 (GMT+09:00)
> Title : Re: [openstack-dev] Introducing Synaps project that provides
> AWS CloudWatch compatible API
> 
> But how does Synaps then difffer from Ceilometer? It looks to be
> duplicated efforts here?
> 
> 
> Endre.
> 
> 
> 2012/10/10 Joonwon Lee <joonwon7.lee at samsung.com>
> 
> Hi, Eoghan,
> 1. No, we don't have any relation with ceilometer.
> 2. Cassandra and Storm (as explained in
> http://wiki.openstack.org/Synaps)
> 3. As far as I understand, aggregation is performed lazily on the
> request.
>  (Metrics are collected every minute and aggregated according to the
>  request.)
> 
> Deok-June Yi can explain with more details tomorrow, if you want.
> Thanks,
> Joonwon Lee
> 
> 
> ------- Original Message -------
> Sender : Eoghan Glynn<eglynn at redhat.com>
> 
> Date : Oct 10, 2012 21:14 (GMT+09:00)
> 
> Title : Re: [openstack-dev] Introducing Synaps project that provides
> AWS CloudWatch compatible API
> 
> 
> 
> Hi Joonwon Lee,
> 
> > Unfortunately, we could not attend the Design Summit. We also need
> > some time
> > before opening our source codes, mostly for our internal process.
> > We
> > will try to shorten this time.
> 
> It's unfortunate you guys won't be at the summit, as it would have
> been great to discuss in that forum.
> 
> But that said, the quicker you can push the code through your
> internal
> process, the better.
> 
> 
> > Now we're preparing the overview of our project and other
> > documentation.
> > If you have any questions, feel free to ask them at any time.
> 
> A few immediate questions that spring to mind:
> 
> 1. Are you re-using any of the ceilometer[1] infrastructure for
>    your metrics collection pipeline?
> 
> 2. What technology is the metrics store based on, at a high level?
>    (e.g. RDBMS accessed via sqlalchemy, noSQL store such as MongoDB,
>     etc.)
> 
> 3. How are you performing datapoint aggregation? (e.g. eagerly on
>    metric ingestion, versus lazily on get)
> 
> Cheers,
> Eoghan
> 
> [1] http://ceilometer.readthedocs.org/en/latest/architecture.html
> 
> 
> > Here are answers of June Yi for Steven Hardy's questions below.
> > > - Is the metric collection secure/authenticated?
> > > - Are metrics collected from the hypervisors or in-instance?
> > > - Can service monitoring (in instance) metrics be collected?
> > > - Is the API authenticated (ie via keystone ec2 credentials?)
> >
> > All metrics are collected using the API. The API is authenticated
> > via
> > auth
> > module came from Nova.Authenticate WSGI middleware which supports
> > AWS
> > signature v2. It works well with LdapDriver. But we did not tested
> > if
> > it is
> > working with keystone yet.
> >
> > Default metrics (i.e. CPUUtilization, disk IO, network IO) are
> > collected
> > from hypervisors. Custom metrics can be collected from in-instance
> > or
> > any other source.
> >
> > Regards,
> > --
> > Joonwon Lee, principal software engineer, Samsung SDS
> >
> >
> > ------- Original Message -------
> > Sender : Eoghan Glynn
> 
> > Date : 2012-10-10 18:59 (GMT+09:00)
> > Title : Re: [openstack-dev] Introducing Synaps project that
> > provides
> > AWS CloudWatch compatible API
> >
> >
> >
> > > > > I'm working for Synaps project that provides AWS CloudWatch
> > > > > compatible API at Samsung SDS.
> > > > > Currently, we have plan to open the project to the community.
> > > > >
> > > > > Here I introduce Synaps project:
> > > > > http://wiki.openstack.org/Synaps
> > > >
> > > > Details are a bit scarce but at first glance there seem to be
> > > > an
> > > > overlap
> > > > with the CloudWatch work in the Heat project ?
> > >
> > > There certainly seems to be from the description, but pretty hard
> > > to
> > > evaluate until we see some code ;)
> >
> > Yes, I would second that.
> >
> > Once the code is out the open, the community will be much better
> > placed
> > to judge which approach will make sense to concentrate efforts on.
> >
> > June Yi - are you planning on attending the design summit?
> >
> > Cheers,
> > Eoghan
> >
> >
> > > > Any chance the two groups could meet at the Design Summit next
> > > > week
> > > > and
> > > > make sure we don't duplicate effort here ?
> > > >
> > > > Angus Saskeld has a session on CloudWatch in the Incubation
> > > > room:
> > > > http://openstacksummitfall2012.sched.org/event/60b0bcf1253efcb3dd72dff531d59d26
> > >
> > > Angus Salkeld and Steve Dake will be at the Summit and could
> > > discuss,
> > > also
> > > possible overlap with metric collection in ceilometer (has been
> > > discussed
> > > previously).
> > >
> > > I have several questions which will be make/break for the heat
> > > usage
> > > of
> > > Cloudwatch:
> > >
> > > - Is the metric collection secure/authenticated?
> > > - Are metrics collected from the hypervisors or in-instance?
> > > - Can service monitoring (in instance) metrics be collected?
> > > - Is the API authenticated (ie via keystone ec2 credentials?)
> > >
> > > Many more questions, need to see code and evaluate if we could
> > > use
> > > this for
> > > heat or if we continue with our own implementation.
> > >
> > > --
> > > Steve Hardy
> > > Red Hat Engineering, Cloud
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>