<br><br><div class="gmail_quote">On Thu, Aug 2, 2012 at 12:03 PM, Vishvananda Ishaya <span dir="ltr"><<a href="mailto:vishvananda@gmail.com" target="_blank">vishvananda@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">There are two possible solutions depending on what the goal of usage metering is.<div><br></div><div>Goal A: log usage of all created volumes regardless of attach status.</div><div><br></div>
<div>This should be done in cinder (or nova-volume/manager.py) side where the list of volumes exists.</div></div></blockquote><div><br></div><div>It seems we want all volumes, but we also want to know for each if it is attached (and possibly where). Does cinder have that information? If so, cinder could publish notifications similar to 'compute.instance.exists' which ceilometer could consume.</div>
<div><br></div><div>Doug</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Goal B: log usage of all attached volumes.</div>
<div><br></div><div>Compute keeps a record of volumes that are attached via the block_device_mapping table. The method for doing this is to get all instances belonging to the host, iterate through each instance and get a list of attached volues from block_device_mapping, and log usage for each one. You could do this inside the existing _instance_usage_audit task (which already grabs all active instances on the host) using self._get_instance_volume_bdms(instance_uuid).</div>
<div><br></div><div>Vish</div><div><br></div><div><div><div><div class="h5"><div>On Aug 2, 2012, at 4:43 AM, "O'Driscoll, Cian" <<a href="mailto:cian.o-driscoll@hp.com" target="_blank">cian.o-driscoll@hp.com</a>> wrote:</div>
<br></div></div><blockquote type="cite"><div lang="EN-US" link="blue" vlink="purple" style="font-family:Menlo;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div><div class="h5"><div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">Just wanted to start a discussion on the issue I hit yesterday when implementing volume usage metering (<a href="https://blueprints.launchpad.net/nova/+spec/volume-usage-metering" style="color:purple;text-decoration:underline" target="_blank">https://blueprints.launchpad.net/nova/+spec/volume-usage-metering</a>)<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">Here is what I found:<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">I create a periodic task in nova/compute/manager.py, the function looks like this<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> @manager.periodic_task<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> def _poll_volume_usage(self, context, start_time=None, stop_time=None):<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE"> if not start_time:<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> start_time = utils.last_completed_audit_period()[1]<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> curr_time = time.time()<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> if curr_time - self._last_vol_usage_poll > FLAGS.volume_poll_interval:<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE"> self._last_vol_usage_poll = curr_time<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> LOG.info(_("Updating volume usage cache"))<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> <span> </span><b>volumes = self.volume_api.get_all(context)<u></u><u></u></b></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> try:<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE"> vol_usage = self.driver.get_all_volume_usage(context, volumes,<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> start_time, stop_time)<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> except NotImplementedError:<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE"> return<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE"> for usage in vol_usage:<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> self.db.vol_usage_update(context, usage['volume'], start_time,<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> usage['rd_req'], usage['rd_bytes'],<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> usage['wr_req'], usage['wr_bytes'])<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">The issue is with the call “self.volume_api.get_all(context)”, in my blueprint I was going to use self.db.volume_get_all_by_host(ctxt, self.host) but because nova db doesn’t contain volume info anymore, we need to talk to cinder to get the info.<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">_poll_volume_usage is passed and admin context when ran. The admin context doesn’t contain a service_catalog or any authentication info from Keystone, bascically just a context that says is_admin=True.<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">So when “self.volume_api.get_all(context)” is called passing in the admin context, the instantiation of a python cinder client fails as the service catalog is empty.<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">Even if the service catalog was populated, I still think we’d fail as we wouldn’t have a Auth token to talk to cinder.<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">2012-08-01 14:05:01 ERROR nova.manager [-] Error during ComputeManager._poll_volume_usage: 'NoneType' object is not iterable<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager Traceback (most recent call last):<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager File "/opt/stack/nova/nova/manager.py", line 173, in periodic_tasks<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager task(self, context)<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager File "/opt/stack/nova/nova/compute/manager.py", line 2639, in _poll_volume_usage<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager volumes = self.volume_api.get_all(context)<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager File "/opt/stack/nova/nova/volume/cinder.py", line 125, in get_all<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager items = cinderclient(context).volumes.list(detailed=True)<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager File "/opt/stack/nova/nova/volume/cinder.py", line 45, in cinderclient<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager url = sc.url_for(service_type='volume', service_name='cinder')<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager File "/opt/stack/python-cinderclient/cinderclient/service_catalog.py", line 53, in url_for<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager for service in catalog:<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager TypeError: 'NoneType' object is not iterable<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">2012-08-01 14:05:01 TRACE nova.manager<u></u><u></u></span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-IE">One possible solution to this is to have a cinder admin user with read only(read all data from cinder db) access created in Keystone (Glance does something very similar when talking to swift)? It keeps the user data in a conf file on the glance nodes.<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">So before we call “self.volume_api.get_all(context)”, we can generate a cinder admin context using the info in a conf file.<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">I think this could be done in another blueprint as I feel there are other use cases where a cinder admin user is required (Any cinder auditing in nova would require the admin user).<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div><div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">For now I propose, to make progress in my implementation of just adding stats collection on detach and not implementing the periodic task for now.<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE">This would be volume usage metering part 1 say, as a proof of concept and when a general consensus/implementation is reached around the cinder admin user, I can implement the periodic task as part ?<u></u><u></u></span></div>
<div style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span lang="EN-IE"> </span></div></div></div></div>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" style="color:purple;text-decoration:underline" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="color:purple;text-decoration:underline" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br></div></blockquote>
</div><br></div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br>