Open Stack

On 05/01/2012 09:57 PM, Andrew Clay Shafer wrote:
> I'm glad to see people championing the effort to implement metering. Is there someway to refocus the enthusiasm for solving the metering problem into engineering a general solution in OpenStack?
>
> I'm just going to apologize in advance, but I don't think this project is headed in the right direction.
>
> I believe metering should be a first class concern of OpenStack and the way this project is starting is almost exactly backwards from what I think a solution to metering should look like.
>
> The last thing I want to see right now is a blessed OpenStack metering project adding more agents, coupled to a particular db and making policy decisions about what is quantifiable.
>
> I think there are really three problems that need to be solved to do metering, what data to get, getting the data and doing things with the data.
>
> From my perspective, a lot if not all of the data events should be coming out of the services themselves. There is already a service that should know when an instance gets started by what tenant. A cross cutting system for publishing those events and a service definition for collecting them seems like a reasonable place to start. To me that should look awful lot like a message queue or centralized logging. Once the first two problems are solved well, if you are so inclined to collect the data into a relational model, the schema will be obvious.
>
> If the first two problems are solved well, then I could be persuaded that a service that provides some of the aggregation functionality is a great idea and a reference implementation on a relational database isn't the worst thing in the world. 
>
> Without a general solution for the first two problems, I believe the primary focus on a schema and db is premature and sub-optimal. I also believe the current approach likely results in a project that is generally unusable.
>
> Does anyone else share my perspective?
It would be better if all OpenStack core components agreed on unified interfaces / messages for metering that would be easy to harvest without installing agents on nodes. This is also true for many services outside of the OpenStack eco-system. However, much in the same way munin and nagios plugins are developped outside of the project for which they provide graphing and monitoring (for instance we recently published swift munin plugins in the repository where ops usually look for them : https://github.com/munin-monitoring/contrib/tree/master/plugins/swift and there is no glance plugin or up-to-date nova plugins yet ), metering agents will be developed separately, most of the time.

As I wrote in a previous mail, once we manage to provide an implementation that proves useful, we will be in a position to approach the core OpenStack components. Integrating the metering agents as part of the core component, much in the same way it's currently done in nova. That will reduce the overall complexity of deploying OpenStack with metering (which must not be mandatory). However, there is very little chance that all components developed around OpenStack are convinced and there will always be a need for a metering that is external to the component. Therefore, even if metering eventually manages to become a first class concern for the core OpenStack components, the proposed architecture of the metering project ( per node agents when necessary and a collector harvesting them into a storage ) will keep being used for other components.

Do you think I'm wrong ? We're at a very early stage and now is the time to question everything :-)