[kolla] openstack centralized logging
Hi,
I'm trying to find out how the "Venus" project could work. Unfortunately, there's not much to find yet. There's no installation guide as for other projects although it already exists since Zed release. I tried enabling it in my kolla all-in-one test deployment (2023.1), the api and manager container are running, but I have no idea how to get some data into or out of it. My conclusion is that it's not a mature project yet, could anyone comment on that?
As an alternative, I enabled centralized logging in the same deployment as well, but unfortunately I can't use the dashboard because of [1]. So I don't really know what I can get out of opensearch in order to have a centralized logging. If possible, I would like to adapt that to a non-containerized openstack deployment. How can I change the opensearch version in my kolla deployment to be able to login and inspect what I got?
Any pointers and comments are highly appreciated! Thanks! Eugen
On 11/04/2024 13:38, Eugen Block wrote:
Hi,
I'm trying to find out how the "Venus" project could work. Unfortunately, there's not much to find yet. There's no installation guide as for other projects although it already exists since Zed release. I tried enabling it in my kolla all-in-one test deployment (2023.1), the api and manager container are running, but I have no idea how to get some data into or out of it. My conclusion is that it's not a mature project yet, could anyone comment on that?
It looks like it uses ElasticSearch as the backend, and Kolla is substituting OpenSearch[1]. That will probably bring issues, since the APIs have diverged. Something to consider in your evaluation.
As an alternative, I enabled centralized logging in the same deployment as well, but unfortunately I can't use the dashboard because of [1]. So I don't really know what I can get out of opensearch in order to have a centralized logging. If possible, I would like to adapt that to a non-containerized openstack deployment. How can I change the opensearch version in my kolla deployment to be able to login and inspect what I got?
For now, we will revert back to the 2.12.0 release of OpenSearch which should work fine. 2.13.0 was recently released. Kolla is not currently pinning OpenSearch which leads to these surprise breakages. There is a patch to change that here: https://review.opendev.org/c/openstack/kolla/+/915322
Centralized logging in Kolla is well tested, widely used and generally stable.
Any pointers and comments are highly appreciated! Thanks! Eugen
[1] https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/venus/t...
Hi,
thanks for your response.
It looks like it uses ElasticSearch as the backend, and Kolla is substituting OpenSearch[1]. That will probably bring issues, since the APIs have diverged. Something to consider in your evaluation.
I'll keep that in mind.
For now, we will revert back to the 2.12.0 release of OpenSearch which should work fine. 2.13.0 was recently released. Kolla is not currently pinning OpenSearch which leads to these surprise breakages. There is a patch to change that here: https://review.opendev.org/c/openstack/kolla/+/915322
Is there any way to update my current deployement to use OpenSearch version 2.12.0? I'm not very familiar with kolla yet. Or should I just wait until the patch is merged and redeploy? There's nothing important in that deployment so I wouldn't mind.
Centralized logging in Kolla is well tested, widely used and generally stable.
Thanks for the reassurance! Eugen
Zitat von Doug Szumski doug@stackhpc.com:
On 11/04/2024 13:38, Eugen Block wrote:
Hi,
I'm trying to find out how the "Venus" project could work. Unfortunately, there's not much to find yet. There's no installation guide as for other projects although it already exists since Zed release. I tried enabling it in my kolla all-in-one test deployment (2023.1), the api and manager container are running, but I have no idea how to get some data into or out of it. My conclusion is that it's not a mature project yet, could anyone comment on that?
It looks like it uses ElasticSearch as the backend, and Kolla is substituting OpenSearch[1]. That will probably bring issues, since the APIs have diverged. Something to consider in your evaluation.
As an alternative, I enabled centralized logging in the same deployment as well, but unfortunately I can't use the dashboard because of [1]. So I don't really know what I can get out of opensearch in order to have a centralized logging. If possible, I would like to adapt that to a non-containerized openstack deployment. How can I change the opensearch version in my kolla deployment to be able to login and inspect what I got?
For now, we will revert back to the 2.12.0 release of OpenSearch which should work fine. 2.13.0 was recently released. Kolla is not currently pinning OpenSearch which leads to these surprise breakages. There is a patch to change that here: https://review.opendev.org/c/openstack/kolla/+/915322
Centralized logging in Kolla is well tested, widely used and generally stable.
Any pointers and comments are highly appreciated! Thanks! Eugen
[1] https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/venus/t...
On 12/04/2024 14:26, Eugen Block wrote:
Hi,
thanks for your response.
You're welcome.
It looks like it uses ElasticSearch as the backend, and Kolla is substituting OpenSearch[1]. That will probably bring issues, since the APIs have diverged. Something to consider in your evaluation.
I'll keep that in mind.
For now, we will revert back to the 2.12.0 release of OpenSearch which should work fine. 2.13.0 was recently released. Kolla is not currently pinning OpenSearch which leads to these surprise breakages. There is a patch to change that here: https://review.opendev.org/c/openstack/kolla/+/915322
Is there any way to update my current deployement to use OpenSearch version 2.12.0? I'm not very familiar with kolla yet. Or should I just wait until the patch is merged and redeploy? There's nothing important in that deployment so I wouldn't mind.
Your options are something like:
1. Directly downgrade the OpenSearch package in the containers and then restart the containers. This is strictly for testing purposes.
2. Build your own Kolla OpenSearch image and deploy it (you can override the package install, see: https://docs.openstack.org/kolla/latest/admin/image-building.html#building-k...), or pull in the patch above
3. Wait for the patch above to merge and use the new Docker image associated with that
I'm not sure if you will have issues with downgrading from 2.13.0 -> 2.12.0 yet. There is a (small?) chance that OpenSearch data created in 2.13.0 is not backwards compatible with 2.12.0. You could just clear the docker volume in that case (given that you don't care about the data).
Centralized logging in Kolla is well tested, widely used and generally stable.
Thanks for the reassurance! Eugen
Zitat von Doug Szumski doug@stackhpc.com:
On 11/04/2024 13:38, Eugen Block wrote:
Hi,
I'm trying to find out how the "Venus" project could work. Unfortunately, there's not much to find yet. There's no installation guide as for other projects although it already exists since Zed release. I tried enabling it in my kolla all-in-one test deployment (2023.1), the api and manager container are running, but I have no idea how to get some data into or out of it. My conclusion is that it's not a mature project yet, could anyone comment on that?
It looks like it uses ElasticSearch as the backend, and Kolla is substituting OpenSearch[1]. That will probably bring issues, since the APIs have diverged. Something to consider in your evaluation.
As an alternative, I enabled centralized logging in the same deployment as well, but unfortunately I can't use the dashboard because of [1]. So I don't really know what I can get out of opensearch in order to have a centralized logging. If possible, I would like to adapt that to a non-containerized openstack deployment. How can I change the opensearch version in my kolla deployment to be able to login and inspect what I got?
For now, we will revert back to the 2.12.0 release of OpenSearch which should work fine. 2.13.0 was recently released. Kolla is not currently pinning OpenSearch which leads to these surprise breakages. There is a patch to change that here: https://review.opendev.org/c/openstack/kolla/+/915322
Centralized logging in Kolla is well tested, widely used and generally stable.
Any pointers and comments are highly appreciated! Thanks! Eugen
[1] https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/venus/t...
Thank you. I will probably just wait a bit until the pinned version is available. I tried a couple of things but none of them worked.
Zitat von Doug Szumski doug@stackhpc.com:
On 12/04/2024 14:26, Eugen Block wrote:
Hi,
thanks for your response.
You're welcome.
It looks like it uses ElasticSearch as the backend, and Kolla is substituting OpenSearch[1]. That will probably bring issues, since the APIs have diverged. Something to consider in your evaluation.
I'll keep that in mind.
For now, we will revert back to the 2.12.0 release of OpenSearch which should work fine. 2.13.0 was recently released. Kolla is not currently pinning OpenSearch which leads to these surprise breakages. There is a patch to change that here: https://review.opendev.org/c/openstack/kolla/+/915322
Is there any way to update my current deployement to use OpenSearch version 2.12.0? I'm not very familiar with kolla yet. Or should I just wait until the patch is merged and redeploy? There's nothing important in that deployment so I wouldn't mind.
Your options are something like:
- Directly downgrade the OpenSearch package in the containers and
then restart the containers. This is strictly for testing purposes.
- Build your own Kolla OpenSearch image and deploy it (you can
override the package install, see: https://docs.openstack.org/kolla/latest/admin/image-building.html#building-k...), or pull in the patch above
- Wait for the patch above to merge and use the new Docker image
associated with that
I'm not sure if you will have issues with downgrading from 2.13.0 -> 2.12.0 yet. There is a (small?) chance that OpenSearch data created in 2.13.0 is not backwards compatible with 2.12.0. You could just clear the docker volume in that case (given that you don't care about the data).
Centralized logging in Kolla is well tested, widely used and generally stable.
Thanks for the reassurance! Eugen
Zitat von Doug Szumski doug@stackhpc.com:
On 11/04/2024 13:38, Eugen Block wrote:
Hi,
I'm trying to find out how the "Venus" project could work. Unfortunately, there's not much to find yet. There's no installation guide as for other projects although it already exists since Zed release. I tried enabling it in my kolla all-in-one test deployment (2023.1), the api and manager container are running, but I have no idea how to get some data into or out of it. My conclusion is that it's not a mature project yet, could anyone comment on that?
It looks like it uses ElasticSearch as the backend, and Kolla is substituting OpenSearch[1]. That will probably bring issues, since the APIs have diverged. Something to consider in your evaluation.
As an alternative, I enabled centralized logging in the same deployment as well, but unfortunately I can't use the dashboard because of [1]. So I don't really know what I can get out of opensearch in order to have a centralized logging. If possible, I would like to adapt that to a non-containerized openstack deployment. How can I change the opensearch version in my kolla deployment to be able to login and inspect what I got?
For now, we will revert back to the 2.12.0 release of OpenSearch which should work fine. 2.13.0 was recently released. Kolla is not currently pinning OpenSearch which leads to these surprise breakages. There is a patch to change that here: https://review.opendev.org/c/openstack/kolla/+/915322
Centralized logging in Kolla is well tested, widely used and generally stable.
Any pointers and comments are highly appreciated! Thanks! Eugen
[1] https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/venus/t...
On 15.04.2024 08:55, Eugen Block wrote:
Thank you. I will probably just wait a bit until the pinned version is available. I tried a couple of things but none of them worked.
Hi,
notice that the actual problem was pinned down to be a permission issue in opensearch-dashboards upstream
which suddenly installed files with ownership root, we have a workaround here:
https://review.opendev.org/c/openstack/kolla/+/915981
The upstream bugreport is here:
https://github.com/opensearch-project/opensearch-build/issues/4643
HTH
Sven
Hi Sven,
thanks for pointing that out. Unfortunately, I'm still not too familiar with kolla to be able to change the ownership in a running system. If I enter the opensearch container I don't have any sudo permissions. If you have any tip how to do that in a running environment, I would greatly appreciate it!
Thanks! Eugen
Zitat von Sven Kieske kieske@osism.tech:
On 15.04.2024 08:55, Eugen Block wrote:
Thank you. I will probably just wait a bit until the pinned version is available. I tried a couple of things but none of them worked.
Hi,
notice that the actual problem was pinned down to be a permission issue in opensearch-dashboards upstream
which suddenly installed files with ownership root, we have a workaround here:
https://review.opendev.org/c/openstack/kolla/+/915981
The upstream bugreport is here:
https://github.com/opensearch-project/opensearch-build/issues/4643
HTH
Sven
Hi Eugen,
If I enter the opensearch container I don't have any sudo permissions. If you have any tip how to do that in a running environment, I would greatly appreciate it!
You can try to exec bash on a running container as root user like in the following example:
$ sudo docker exec -u root -ti elasticsearch bash (elasticsearch)[root@control01 /]# id uid=0(root) gid=0(root) groups=0(root) (elasticsearch)[root@control01 /]# exit
$ sudo docker exec -ti elasticsearch bash (elasticsearch)[elasticsearch@ control01 /]$ id uid=42412(elasticsearch) gid=42412(elasticsearch) groups=42412(elasticsearch),42400(kolla)
Best, Matteo
Thank you so much! I will try that tomorrow.
Zitat von Matteo Piccinini matteo.piccinini@gmail.com:
Hi Eugen,
If I enter the opensearch container I don't have any sudo permissions. If you have any tip how to do that in a running environment, I would greatly appreciate it!
You can try to exec bash on a running container as root user like in the following example:
$ sudo docker exec -u root -ti elasticsearch bash (elasticsearch)[root@control01 /]# id uid=0(root) gid=0(root) groups=0(root) (elasticsearch)[root@control01 /]# exit
$ sudo docker exec -ti elasticsearch bash (elasticsearch)[elasticsearch@ control01 /]$ id uid=42412(elasticsearch) gid=42412(elasticsearch) groups=42412(elasticsearch),42400(kolla)
Best, Matteo
Hi Matteo,
that worked just fine, I can browse the logs now. :-) Thanks a lot (again)!
Zitat von Matteo Piccinini matteo.piccinini@gmail.com:
Hi Eugen,
If I enter the opensearch container I don't have any sudo permissions. If you have any tip how to do that in a running environment, I would greatly appreciate it!
You can try to exec bash on a running container as root user like in the following example:
$ sudo docker exec -u root -ti elasticsearch bash (elasticsearch)[root@control01 /]# id uid=0(root) gid=0(root) groups=0(root) (elasticsearch)[root@control01 /]# exit
$ sudo docker exec -ti elasticsearch bash (elasticsearch)[elasticsearch@ control01 /]$ id uid=42412(elasticsearch) gid=42412(elasticsearch) groups=42412(elasticsearch),42400(kolla)
Best, Matteo
participants (4)
-
Doug Szumski
-
Eugen Block
-
Matteo Piccinini
-
Sven Kieske