Open Stack

Hello CloudKitters!

It was a great opportunity to be able to present this talk for you all
guys. It was a pleasure to see that so many people are interested in
CloudKitty, Ceilometer, and Gnocchi. I really hope that we can all
collaborate together to keep improving CloudKitty, and other related
systems.

Attached is the presentation we used during the presentation, so you all
can consult this material. Also, as always, if you need some guidance, let
us know.

Furthermore, as I mentioned in the presentation, I will provide a
step-by-step process on how to create a new metric using Ceilometer Dynamic
pollster that can collect/generate/derive (however you want to call it)
IOPs information for volumes. The new metric name is going to be called
"dynamic_pollster.volume.services.volume.iops". Also, the idea is that we
collect some attributes, and we use the attribute called "bootable", which
indicates if the volume is bootable or not. Then, if the volume is
bootable, we will charge for its IOPS; otherwise, we do not charge it. As I
mentioned, this is just a very simple use case. It is not targeting
anyone's need in production, but it can be used to get you going quite
easily with the pipeline.

Requirements:

   - Ceilometer up and running. Version 15.0.0 (or higher is recommended);
   - Gnocchi up and running.  Version 4.3.0 (or higher). If you want to run
   this in production, this patch might be very important (
   https://github.com/gnocchixyz/gnocchi/pull/1059); it has not been merged
   yet;
   - CloudKitty up and running. Version 10.0.0 (or higher).


With all systems up and running, you will need to do the following:

   - Ceilometer


   - Add the new metric collection in the "polling.yaml" file. You can add
      it in any of the sections you already have there. Or, you can create its
      own, as follows:

```
    - name: volume_iops_metric
      interval: 600
      meters:
        - dynamic_pollster.volume.services.volume.iops
```

   - After declaring the metrics there, you can create the pollster in the
      "pollster.d" folder in Ceilometer. The name of the file can be
anything, as
      long as it ends with ".yaml"; for instance,
"pollsters.d/volume-iops.yaml".
      Then, you will need to add the following content. Remember to replace the
      contents of "<>" with the values of your environment.

      ```
      - name: "dynamic_pollster.volume.services.volume.iops"
        response_entries_key: "volumes"
        next_sample_url_attribute: "volumes_links | filter(lambda v:
      v.get('rel') == 'next', value) | list(value) | value[0] |
value.get('href')"
        sample_type: "gauge"
        unit: "GB"
        value_attribute: "volume_type"
        endpoint_type: "volume"
        url_path:
      "/v3/<SERVICE_UUID_FOR_CEILOMETER_USER>/volumes/detail?all_tenants=true&limit=10"
        headers:
          "Openstack-API-Version": "latest"
        project_id_attribute: "os-vol-tenant-attr:tenant_id"
        metadata_fields:
          - "status"
          - "name"
          - "volume_type"
          - "availability_zone"
          - "user_id"
          - "bootable | str(value)"
          - "attachments | value or [ { 'server_id': '' } ] | value[0] |
      value['server_id']"
        metadata_mapping:
          "availability_zone": "dynamic_availability_zone"
          "name": "dynamic_display_name"
          "volume_type": "dynamic_volume_type"
          "attachments | value or [{ 'server_id': '' }] | value[0] |
      value['server_id']": "dynamic_instance_id"
          "bootable | str(value)": "bootable"
        value_mapping:
          "<VOLUME_TYPE_UUID_OR_VOLUME_TYPE_NAME>": "<THE IOPS THAT THIS
      VOLUME TYPE REPRESENTS>"

      ```


      - After creating that, you will need to customize the file
      "gnocchi_resources.yaml", which is the file that maps metrics to
resources,
      and attributes that are pushed to the backend. You can also
create your own
      "gnocchi_resources.yaml" file, and then use it in the pipeline
      configuration in the "pipeline.yaml" file, in the configuration of
      "publishers". Then, you would have something like
      "gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml",
      assuming the file is called "custom_gnocchi_resources.yaml". In
this file,
      you will need to add the new metric and attribute to be pushed to the
      metrics backend. The following is an example of "volume" resource
      declaration in the "gnocchi_resource.yaml" file with the new metric and
      attribute that we want to capture.

      ```
        - resource_type: volume
          metrics:
      <MANY OTHER METRICS HERE>
            dynamic_pollster.volume.services.volume.iops:
          attributes:
            display_name:
      resource_metadata.(dynamic_display_name|notification_display_name)
            bootable: resource_metadata.bootable
      <MANY OTHER ENTRIES HERE>

      ```

      - After the above configuration, you will need to create the new
      attribute in the Gnocchi resource type. To achieve that, you can
issue the
      following command.
      ```
      gnocchi resource-type update -a bootable:string:false volume
      ```
      - Then, you will need to restart the Ceilometer.
      - Now, you can issue "gnocchi resource show <VOLUME ID>", and you
      should see the new metric in the resource, and the new attribute as well.


With Ceilometer and Gnocchi configured for the new metric, we can move on
to CloudKitty.

   - The new metric needs to be declared in the "metrics.yml" file. The
   content should be something similar to the following (adapt as you need).

   ```
     dynamic_pollster.volume.services.volume.iops:
       unit: IOPS
       alt_name: disk-volume-iops-usage-hours
       groupby:
         - id
         - user_id
         - project_id
         - volume_type
         - display_name
         - revision_start
         - instance_id
         - availability_zone
         - bootable
       metadata: []
       extra_args:
         aggregation_method: max
         resource_type: volume
         use_all_resource_revisions: false
   ```
   - Restart CloudKitty processor
   - With the new metric introduced in CloudKitty processor, you will need
   to create the rating configuration. In our case, we wanted to rate the
   volumes marked as bootable for IOPs. The volumes that are not bootable
   would not be rated. The following are the steps one needs to execute to
   create the hashmap rating rules.
      - Create the service
      ```
      openstack rating hashmap service create disk-volume-iops-usage-hours
      ```
      - Then, create the field
      ```
      openstack rating hashmap field create <SERVICE_ID> <FIELD NAME WE
      WANT TO RATE, IN OUR CASE, IT IS bootable>
      ```
      - After that, we can create the rating rule
      ```
      openstack rating hashmap mapping  create --field <FIELD UUID THAT WE
      CREATE> --value 'true' 1 --type flat
      ```
      - Having done that, you should be able to rate all volumes by IOPS if
      they have the flag "bootable" set to true. Otherwise, they will not be
      rated.


That is all that one needs to execute to replicate the example I showed
during the presentation. If you guys have any doubts or difficulties when
using CloudKitty, do not hesitate to ping me.

-- 
Rafael Weingärtner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220609/15ddea14/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Gnocchi, Ceilometer, and CloudKitty, a dynamic and agnostic cloud monitoring and billing stack [OpenInfra summit 2022]..pdf
Type: application/pdf
Size: 761249 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220609/15ddea14/attachment-0001.pdf>