It sure is, we as well abandoned the MongoDB backend for Gnocchi which works pretty well.

Would be a shame if a migration back would be required, maybe we can get a discussion going on a more long-term solution as was discussed when talking about the future of Ceilometer.

Supporting Gnocchi or moving to another open source project as a storage backend that is stable and maintained. There were (and still is? Though unofficial out-of-tree) storage backends for Ceilometer that publishes to InfluxDB.

I were never able to follow-up on the meetings (I probably missed a lot of it) regarding the Ceilometer roadmap [1].

[1] https://etherpad.openstack.org/p/telemetry-train-roadmap

On 11/19/19 10:29 AM, Blom, Merlin, NMU-OI wrote:

Thanks for your work on ceilometer!

The gnocchi situation is realy sad.

We implemented solutions on Gnocchi and ceilometer.

In my opinion you abandoned the mongodb support for performance reasons and now you are going back to it?

Has mongodb made any significant performance improvements for time series data?

Best regards,

Merlin

*Von:*Lingxian Kong anlin.kong@gmail.com *Gesendet:* Dienstag, 19. November 2019 10:03 *An:* Radosław Piliszek radoslaw.piliszek@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Betreff:* Re: [gnocchi][telemetry][ceilometer][cloudkitty] Gnocchi unmaintained

We (ceilometer team) will probably add Ceilometer API and mongodb support back, considering the current Gnocchi project situation. However, Gnocchi will still be supported as a publisher in Ceilometer.

Best regards, Lingxian Kong

Catalyst Cloud

On Tue, Nov 19, 2019 at 9:54 PM Radosław Piliszek <radoslaw.piliszek@gmail.com mailto:radoslaw.piliszek@gmail.com> wrote:

Hello Folks,

It looks like gnocchi is "officially" marked as unmaintained:
https://github.com/gnocchixyz/gnocchi/issues/1049
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_gnocchixyz_gnocchi_issues_1049&d=DwMFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=hTUN4-Trlb-8Fh11dR6m5VD1uYA15z7v9WL8kYigkr8&m=czRC3qwwRqT3qKzfXMSVl78G4Sk8QVwT93okCgkBe34&s=Ob7yLjlWUAz9-8oMikC_QU9ivZBvtBKkqqFEvceGGM0&e=>

Has there been any discussion regarding how it affects OpenStack
projects? And/or are there any plans to amend this situation?

-yoctozepto

I am sorry to the telemetry project happened before, but now current telemetry core team had decided to add ceilometer api and mongodb support and cpu_utils support back. Gnoochi will still support as the backed. All the mentioned database (influxdb, ES....), we would happy everyone to submit patches to support as the database backed in ceilometer.

I created a storyboard to track ceilometer Ussuri release todo things in [0]. Free free to add things you want to do in Ussuri release.

Due to I will have a vacation this week, I can't hold this week's meeting, we can discuss more in the next irc meeting at 5 Dec 2:00 UTC.

[0] https://storyboard.openstack.org/#!/board/205

Luka Peschke luka.peschke@objectif-libre.com于2019年11月19日 周二18:24写道：

My two cents from my experience on cloudkitty: We had to implement several storage drivers, and faced more or less the same issues as the telemetry team did before us. We had a gnocchi driver at some point, which worked pretty well, but ended up being very hacky because gnocchi lacked flexibility for non-openstack metrics (ie. data models which aren't resource-based).

We ended up implementing a driver for InfluxDB which has relatively good perfs. But given that the open-source version of InfluxDB does not support HA/clustering, we also implemented an experimental Elasticsearch driver (which requires ES>=6.5).

The recent ES releases have really improved the support for timeseries, and it is the storage backend for elastic beats.

Given that many openstack deployments already have an Elasticsearch deployment for logs, and the large adoption of ES, it'd be my choice for a new Ceilometer storage driver.

However, Gnocchi is pretty stable in 4.3, and well integrated with Ceilometer. Wouldn't it be less effort to keep it functional for now (ie only bug/security fixes, no new features), instead of re-integrating deleted features to ceilometer ?

Cheers,

-- Luka Peschke (peschk_l)

Le 2019-11-19 10:51, Tobias Urdin a écrit :

...

It sure is, we as well abandoned the MongoDB backend for Gnocchi which works pretty well.

Would be a shame if a migration back would be required, maybe we can get a discussion going on a more long-term solution as was discussed when talking about the future of Ceilometer.

Supporting Gnocchi or moving to another open source project as a storage backend that is stable and maintained. There were (and still is? Though unofficial out-of-tree) storage backends for Ceilometer that publishes to InfluxDB.

I were never able to follow-up on the meetings (I probably missed a lot of it) regarding the Ceilometer roadmap [1].

[1] https://etherpad.openstack.org/p/telemetry-train-roadmap

On 11/19/19 10:29 AM, Blom, Merlin, NMU-OI wrote:

...

Thanks for your work on ceilometer!

The gnocchi situation is realy sad.

We implemented solutions on Gnocchi and ceilometer.

In my opinion you abandoned the mongodb support for performance reasons and now you are going back to it?

Has mongodb made any significant performance improvements for time series data?

Best regards,

Merlin

VON: Lingxian Kong anlin.kong@gmail.com GESENDET: Dienstag, 19. November 2019 10:03 AN: Radosław Piliszek radoslaw.piliszek@gmail.com CC: openstack-discuss openstack-discuss@lists.openstack.org BETREFF: Re: [gnocchi][telemetry][ceilometer][cloudkitty] Gnocchi unmaintained

We (ceilometer team) will probably add Ceilometer API and mongodb support back, considering the current Gnocchi project situation. However, Gnocchi will still be supported as a publisher in Ceilometer.

Best regards, Lingxian Kong

Catalyst Cloud

On Tue, Nov 19, 2019 at 9:54 PM Radosław Piliszek radoslaw.piliszek@gmail.com wrote:

...

Hello Folks,

It looks like gnocchi is "officially" marked as unmaintained: https://github.com/gnocchixyz/gnocchi/issues/1049 [1]

Has there been any discussion regarding how it affects OpenStack projects? And/or are there any plans to amend this situation?

-yoctozepto

Links:

[1]

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_gnocchixyz_g...

--

Thanks, Rong Zhu

On 19/11/2019 11:56, Rong Zhu wrote:

I am sorry to the telemetry project happened before, but now current telemetry core team had decided to add ceilometer api and mongodb support and cpu_utils support back. Gnoochi will still support as the backed. All the mentioned database (influxdb, ES....), we would happy everyone to submit patches to support as the database backed in ceilometer.

I created a storyboard to track ceilometer Ussuri release todo things in [0]. Free free to add things you want to do in Ussuri release.

Due to I will have a vacation this week, I can't hold this week's meeting, we can discuss more in the next irc meeting at 5 Dec 2:00 UTC.

[0] https://storyboard.openstack.org/#!/board/205

Luka Peschke <luka.peschke@objectif-libre.com mailto:luka.peschke@objectif-libre.com>于2019年11月19日 周二18:24写道：

Hi,

tbh, I am surprised to see the telemetry team trying to roll back to something, which is known to cause a lot of performance issues. There were good reasons for splitting ceilometer into several components.

There were lots of good ideas and suggestions already shared in this thread. My proposal here would be to keep ceilometer as is (as data collecting agent) and to write missing glue to digest or send data to a time-series database, like InfluxDB or Prometheus (plus many more options, not mentioned here).

If I remember correctly, MongoDB was deprecated because of the issues it caused and also because it was removed from Linux distributions, since there were the licensing issues.

Matthias

On Wed, Nov 20, 2019 at 9:39 AM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:

I second Matthias' stance - MongoDB does not look like the future-proof tool for the job, it's not really well suited for time-series data. I feel that would just be "wasted" work to bring it back, unless it is really like snapping fingers... Not to mention supporting it in the long run. Obviously not me making decisions. :-)

The other part, bringing API back, seems nice as there are ppl using Ceilometer with Monasca this way.

I am glad to hear that Monasca is finally getting official support as a publisher in Ceilometer. This should actually solve the main issue of lack of a reliable publisher

• Monasca already handles

modern persistent time-series databases.

I added [monasca] since we have the topic of Monasca as one of Ceilometer's new publishers. Feel free to remove.

As open source community, we don't force anyone to use some specific software or tooling, we support as more options as possible for real use cases but leave the choice to the users.

- Best regards, Lingxian Kong Catalyst Cloud

On Wed, Nov 20, 2019 at 9:28 PM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:

Sure, Lingxian. I have never negated that. To make sure I was not misunderstood, I will rephrase myself. Any kind of work requires effort which consists of human-spent time. If we are well-aware that MongoDB is not the best solution, the same time might be better spent integrating something else. Still, from my point of view, having Monasca as the backend is enough of a solution to the main issue in this thread which is Gnocchi not being maintained any longer.

Yeah, I'm pretty sure we are on the same page. Monasca is probably the best solution for you, but we (catalyst cloud) is still on the Ceilometer+MongoDB boat, and they work very well with us.

- Best regards, Lingxian Kong Catalyst Cloud

On Wed, Nov 20, 2019 at 8:33 AM Matthias Runge mrunge@matthias-runge.de wrote:

tbh, I am surprised to see the telemetry team trying to roll back to something, which is known to cause a lot of performance issues. There were good reasons for splitting ceilometer into several components.

I understand your concerns and the team will find a better way to deal with the rollback, the expected result is that it won't break anyone who are using Gnocchi or other storage backend but it means much to the cloud providers(like OVH and us) who are still using old version Ceilometer and relying on the API.

There were lots of good ideas and suggestions already shared in this thread. My proposal here would be to keep ceilometer as is (as data collecting agent) and to write missing glue to digest or send data to a time-series database, like InfluxDB or Prometheus (plus many more options, not mentioned here).

Does InfluxDB or Prometheus support to store samples that could be leveraged for auditing or billing purpose? I'm not familiar with TSDB but I got the answer NO according to the chatting with some other people.

If I remember correctly, MongoDB was deprecated because of the issues it caused and also because it was removed from Linux distributions, since there were the licensing issues.

I don't think the license change will affect the cloud that only uses MongoDB as internal service backend storage unless I'm missing something.

- Best regards, Lingxian Kong Catalyst Cloud

On 19/11/2019 21:55, Lingxian Kong wrote:

On Wed, Nov 20, 2019 at 8:33 AM Matthias Runge <mrunge@matthias-runge.de mailto:mrunge@matthias-runge.de> wrote:

tbh, I am surprised to see the telemetry team trying to roll back to
something, which is known to cause a lot of performance issues. There
were good reasons for splitting ceilometer into several components.

I understand your concerns and the team will find a better way to deal with the rollback, the expected result is that it won't break anyone who are using Gnocchi or other storage backend but it means much to the cloud providers(like OVH and us) who are still using old version Ceilometer and relying on the API.

I assume, you haven't seen any performance issues with both Ceilometer or Gnocchi?

There were lots of good ideas and suggestions already shared in this
thread. My proposal here would be to keep ceilometer as is (as data
collecting agent) and to write missing glue to digest or send data to a
time-series database, like InfluxDB or Prometheus (plus many more
options, not mentioned here).

Does InfluxDB or Prometheus support to store samples that could be leveraged for auditing or billing purpose? I'm not familiar with TSDB but I got the answer NO according to the chatting with some other people

Billing is a complicated beast for multiple reasons. Let's look only at ingestion. First of all, you are right, e.g Prometheus should not be used in billing situations, see[1], also on [2] (same source as [1]).

However, the same is true for Gnocchi and also for Ceilometer. You are going to loose metrics if your collection is faster than your backend can digest it, for Gnocchi is such behaviour documented. If you are looking at speed, I'd consider prometheus to be much faster than gnocchi and ceilometer and thus better suited for handling metrics.

[1] https://github.com/prometheus/docs/blob/master/content/docs/introduction/ove... [2] https://prometheus.io/docs/introduction/overview/#when-does-it-not-fit

If I remember correctly, MongoDB was deprecated because of the issues it
caused and also because it was removed from Linux distributions, since
there were the licensing issues.

I don't think the license change will affect the cloud that only uses MongoDB as internal service backend storage unless I'm missing something.

You can read a bit of story on MongoDB relicensing at [3].

The license change was regarding commercial use. You can find the full text at [4], depending on your use case, it may be valuable to ask a lawyer, especially, when in doubt.

[3] https://hub.packtpub.com/mongodb-withdraws-controversial-server-side-public-... [4] https://www.mongodb.com/licensing/server-side-public-license

From my POV (and apparently others here in this thread as well), I'd

take a step back and revisit https://storyboard.openstack.org/#!/board/205 again, in order to run into the same issues as in Newton cycle. I mean, if you'd want ceilometer from Newton, it is still there.

Matthias

On Wed, Nov 20, 2019 at 11:11 AM Sam Morrison sorrison@gmail.com wrote:

We too use ceilometer and gnocchi and it works great. The old way with mongo simply didn't work. Our environment is around 10k running instances and gnocchi currently has data for 1,667,072 resources with 6,879,910 metrics.

We are happy with gnocchi and so far haven't needed to change anything but would help contribute if there were things we needed. Would be great if we could just maintain gnocchi as opposed to making yet another thing.

Sam

Glad to hear that you are going well with Gnocchi, and I also hope Gnocchi could be maintained as expected.

Just to be clear, adding MongoDB support back doesn't mean you have to install and maintain that once you upgrade Ceilometer in the future, we will guarantee the existing deployments won't be affected, it's optional and you are the boss.

- Best regards, Lingxian Kong Catalyst Cloud

On Thu, 2019-11-21 at 13:29 +0100, Thomas Goirand wrote:

On 11/19/19 9:55 PM, Lingxian Kong wrote:

...

I don't think the license change will affect the cloud that only uses MongoDB as internal service backend storage unless I'm missing something.

What you are probably missing, is that none of the downstream distribution will continue to package MongoDB. That, for sure, will have an effect on what people will use.

If an operator decides to still continue to use these back-end, probably it's going to be using outside of the distro packages, which leads to many problems, including:

• inferior quality packages.
• availability of the repositories (ie: not enough mirror, just the one

of upstream).

• impossibility to redistribute the packages (for example: on an ISO

image, or in a repository), and it may even be forbidden to publish a public repository with it.

• probably, folks contributing to config management project will be

reluctant to offer compatibility for these back-ends.

With my Debian OpenStack package maintainer hat on: I will certainly ignore any backend that would be using MongoDB or InfluxDB, as these cannot be used without non-debian packages.

influxdb is mit licensed so im not sure why you would not be able to package or redistribute it in debian. https://github.com/influxdata/influxdb/blob/master/LICENSE mongodb is a different story but we shoudl not paint influx with the same brush and it should be a valid alternitive to Gnocchi as it is a time serise database.

I also will do my best to convince everyone that using non-free software is a bad idea, and that these company who broke the free software license promise cannot be trusted anymore.

if that is your goal then you should advocate for influxDB then since its under an even more liberial license then gnocchi was.

So definitively, the change of license is problematic in many ways.

Cheers,

Thomas Goirand (zigo)

On Thu, Nov 21, 2019 at 03:29:01PM +0000, Jeremy Stanley wrote:

On 2019-11-21 13:57:56 +0000 (+0000), Sean Mooney wrote:

...

On Thu, 2019-11-21 at 13:29 +0100, Thomas Goirand wrote:

[...]

...

...

With my Debian OpenStack package maintainer hat on: I will certainly ignore any backend that would be using MongoDB or InfluxDB, as these cannot be used without non-debian packages.

influxdb is mit licensed so im not sure why you would not be able to package or redistribute it in debian.

[...]

And indeed, it's been packaged in Debian for years, and still is:

https://packages.debian.org/influxdb

The main concern about it is the open-core development model where "advanced" features like stability and redundancy require you to purchase their proprietary enterprise version instead of the less full-featured (but freely-licensed) community version:

https://www.influxdata.com/blog/update-on-influxdb-clustering-high-availabil...

I know several very large sites (ingesting billions of records per day) that run community InfluxDB and they get HA by putting influx-proxy [1] in front of it. I've evaluated it for large scale uses before as well, and with influx-proxy I found no need for the clustering option.

Nate

[1] https://github.com/shell909090/influx-proxy

On Fri, Nov 22, 2019 at 3:50 AM Witek Bedyk witold.bedyk@suse.com wrote:

On 11/21/19 9:53 PM, Nate Johnston wrote:

...

I know several very large sites (ingesting billions of records per day) that run community InfluxDB and they get HA by putting influx-proxy [1] in front of it. I've evaluated it for large scale uses before as well, and with influx-proxy I found no need for the clustering option.

Similar architecture is followed by Monasca. InfluxDB instances can be assigned to different Kafka consumer groups and consume messages independently from the message queue. In case one of the instances is down all the measurements are still buffered and get persisted as soon as the instance is available again.

I'm curious on what the final decision of the Ceilometer team regarding this discussion?

Best greetings Witek

On 11/21/19 4:29 PM, Jeremy Stanley wrote:

On 2019-11-21 13:57:56 +0000 (+0000), Sean Mooney wrote:

...

On Thu, 2019-11-21 at 13:29 +0100, Thomas Goirand wrote:

[...]

...

...

With my Debian OpenStack package maintainer hat on: I will certainly ignore any backend that would be using MongoDB or InfluxDB, as these cannot be used without non-debian packages.

influxdb is mit licensed so im not sure why you would not be able to package or redistribute it in debian.

[...]

And indeed, it's been packaged in Debian for years, and still is:

https://packages.debian.org/influxdb

The main concern about it is the open-core development model where "advanced" features like stability and redundancy require you to purchase their proprietary enterprise version instead of the less full-featured (but freely-licensed) community version:

https://www.influxdata.com/blog/update-on-influxdb-clustering-high-availabil...

And we also should remember one of the reason that spawned OpenStack into existence: some other cloud solution was open-core, and there was multiple conflict of interest where upstream made it difficult to contribute (because they needed to differentiate commercially). That very much is the main issue of any software adopting the open-core model: conflict of interest by the company behind it.

Cheers,

Thomas Goirand (zigo)

Another approach could be to use Monasca as the back end. The publisher has been recently added upstream [1].

It uses InfluxDB as the time series DB. The message queue between the API and TSDB adds resiliency and allows setting up InfluxDB in HA.

What you get on top is a generic, multi-tenant monitoring solution. Cloud users can install their own agents, push own application metrics and set up own alerting per project. Support for auto-scaling with Heat templates is included.

Greetings Witek

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-system-architec...

On 11/19/19 10:51 AM, Tobias Urdin wrote:

It sure is, we as well abandoned the MongoDB backend for Gnocchi which works pretty well.

Would be a shame if a migration back would be required, maybe we can get a discussion going on a more long-term solution as was discussed when talking about the future of Ceilometer.

Supporting Gnocchi or moving to another open source project as a storage backend that is stable and maintained. There were (and still is? Though unofficial out-of-tree) storage backends for Ceilometer that publishes to InfluxDB.

I were never able to follow-up on the meetings (I probably missed a lot of it) regarding the Ceilometer roadmap [1].

[1] https://etherpad.openstack.org/p/telemetry-train-roadmap

On 11/19/19 10:29 AM, Blom, Merlin, NMU-OI wrote:

...

Thanks for your work on ceilometer!

The gnocchi situation is realy sad.

We implemented solutions on Gnocchi and ceilometer.

In my opinion you abandoned the mongodb support for performance reasons and now you are going back to it?

Has mongodb made any significant performance improvements for time series data?

Best regards,

Merlin

*Von:*Lingxian Kong anlin.kong@gmail.com *Gesendet:* Dienstag, 19. November 2019 10:03 *An:* Radosław Piliszek radoslaw.piliszek@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Betreff:* Re: [gnocchi][telemetry][ceilometer][cloudkitty] Gnocchi unmaintained

We (ceilometer team) will probably add Ceilometer API and mongodb support back, considering the current Gnocchi project situation. However, Gnocchi will still be supported as a publisher in Ceilometer.

Best regards, Lingxian Kong

Catalyst Cloud

On Tue, Nov 19, 2019 at 9:54 PM Radosław Piliszek <radoslaw.piliszek@gmail.com mailto:radoslaw.piliszek@gmail.com> wrote:

Hello Folks,

It looks like gnocchi is "officially" marked as unmaintained:
https://github.com/gnocchixyz/gnocchi/issues/1049
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_gnocchixyz_gnocchi_issues_1049&d=DwMFaQ&c=vo2ie5TPcLdcgWuLVH4y8lsbGPqIayH3XbK3gK82Oco&r=hTUN4-Trlb-8Fh11dR6m5VD1uYA15z7v9WL8kYigkr8&m=czRC3qwwRqT3qKzfXMSVl78G4Sk8QVwT93okCgkBe34&s=Ob7yLjlWUAz9-8oMikC_QU9ivZBvtBKkqqFEvceGGM0&e=>

Has there been any discussion regarding how it affects OpenStack
projects? And/or are there any plans to amend this situation?

-yoctozepto