Open Stack

I would like to follow this discussion so I picked up points.

- There are two way to collect info from swift, one is pollster and 
  the other is notification. And we discussed about how to solve the 
  performance degradation of swift_middleware here. 
  pollster:
   - storage.objects
   - storage.objects.size
   - storage.objects.containers
   - storage.containers.objects
   - storage.containers.objects.size
  notification:
   - storage.objects.incoming.bytes
   - storage.objects.outgoing.bytes
   - storage.api.request

- storage.objects.imcoming.bytes, storage.objects.outgoing.bytes and 
  storage.api.request are handled with swift_middleware because ceilometer 
  need to have the info with per-user and per-tenant basis.
- swift has statsd but there is no per-user and per-tenant related info 
  because to realize this swift has to have keystone-isms into core swift code.
- improves swift_middleware with stopping the 1:1 mapping b/w API calls and 
  notifications
- swift may consume 10s of thousands of event per second and this case is fairly 
  unique so far.

I would like to think this performance problem with the following point of view.
- need to handle 10s of thousands of event per second
- possibility to lost events (i.e. swift proxy goes down when events queued in a swift process)

With the notification style there are restriction for above points. Therefore I change the style 
to get storage.objects.imcoming.bytes, storage.objects.outgoing.bytes and 
storage.api.request from notification to pollster.
Here I met a problem that pointed out by Mr. Merritt, swift has dependency with keystone.
But I prefer to solve this problem rather than a problem for notification style. What do you think?

My rough idea to solve the dependency problem is 
- enable statsd (or similar function) in swift
- put a middleware in swift proxy
- this middleware does not have any communication with ceilometer but 
  put a mark to followed middleware or swift proxy
- store metrics with a tenant and a user by statsd if there is the mark
  store metrics by statsd if there is no mark
- Ceilometer (central agent) call APIs to get the metrics

Is there any way to solve the dependency problem?

Best Regards,
Hisashi Osanai