[all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM
Liye Pang(逄立业) wrote:
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects [...] OpenStack-aware centralized log management sounds very interesting to me...
If others are interested in collaborating on producing that component, I personally think it would be a great fit for the "operations tooling" section of the OpenStack Map[1]. [1] https://www.openstack.org/software/ -- Thierry Carrez (ttx)
On Mon, 11 Jan 2021 at 13:13, Thierry Carrez <thierry@openstack.org> wrote:
Liye Pang(逄立业) wrote:
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects [...] OpenStack-aware centralized log management sounds very interesting to me...
If others are interested in collaborating on producing that component, I personally think it would be a great fit for the "operations tooling" section of the OpenStack Map[1].
Let's not forget that Monasca has a log aggregation API [1] [1] https://wiki.openstack.org/wiki/Monasca/Logging
-- Thierry Carrez (ttx)
On Mon, 11 Jan 2021 at 13:13, Thierry Carrez <thierry@openstack.org> wrote:
Liye Pang(逄立业) wrote:
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects [...] OpenStack-aware centralized log management sounds very interesting to me...
If others are interested in collaborating on producing that component, I personally think it would be a great fit for the "operations tooling" section of the OpenStack Map[1].
Let's not forget that Monasca has a log aggregation API [1]
The major work of monasca project about logs is indexing logs for users to retrieve. The venus project is based on indexed data and provides more functions, such as correlation analysis, error alarms,problem location , etc.
-- Thierry Carrez (ttx)
-----邮件原件----- 发件人: Thierry Carrez [mailto:thierry@openstack.org] 发送时间: 2021年1月11日 21:12 收件人: openstack-discuss@lists.openstack.org 主题: Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
Liye Pang(逄立业) wrote:
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects [...] OpenStack-aware centralized log management sounds very interesting to me...
If others are interested in collaborating on producing that component, I personally think it would be a great fit for the "operations tooling" section of the OpenStack Map[1].
Yes, after Inspur did a 1,000-nodes OpenStack single-cluster large-scale test, I was more convinced of the benefits Venus can bring to operation and maintenance. By Venus, we can quickly locate and find problems with the OpenStack platform, which can bring great convenience to operation and maintenance. https://mp.weixin.qq.com/s/RSrjjZjVFn086StNLV1Ivg This is the article of 1000-nodes test, but it's wrote by Chinese, don't worry ^^, we will publish the English article in future. This is the demo for Venus, hope that can help you to know what it can be done: https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM
-- Thierry Carrez (ttx)
Original ML: http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019748.h... brinzhang
---- On Tue, 12 Jan 2021 02:45:10 -0600 Brin Zhang(张百林) <zhangbailin@inspur.com> wrote ----
-----邮件原件----- 发件人: Thierry Carrez [mailto:thierry@openstack.org] 发送时间: 2021年1月11日 21:12 收件人: openstack-discuss@lists.openstack.org 主题: Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
Liye Pang(逄立业) wrote:
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects [...] OpenStack-aware centralized log management sounds very interesting to me...
If others are interested in collaborating on producing that component, I personally think it would be a great fit for the "operations tooling" section of the OpenStack Map[1].
Yes, after Inspur did a 1,000-nodes OpenStack single-cluster large-scale test, I was more convinced of the benefits Venus can bring to operation and maintenance. By Venus, we can quickly locate and find problems with the OpenStack platform, which can bring great convenience to operation and maintenance.
https://mp.weixin.qq.com/s/RSrjjZjVFn086StNLV1Ivg This is the article of 1000-nodes test, but it's wrote by Chinese, don't worry ^^, we will publish the English article in future.
This is the demo for Venus, hope that can help you to know what it can be done: https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM
Thanks Liye, Brin for the details, I also things this is a valuable project for day-to-day operation on a large-scale cloud or even small scale to automate the log to alter, etc. Just one question, can we configuration the particular error log msg string/pattern for raising alarm? and different levels of alarm (critical, high priority etc)? For example: If there is a known limitation on my cloud (due to RBAC or backend deps) and requests end up in error so I do not want to raise an alarm for those. -gmann
-- Thierry Carrez (ttx)
Original ML: http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019748.h...
brinzhang
---- On Tue, 12 Jan 2021 02:45:10 -0600 Brin Zhang(张百林) <zhangbailin@inspur.com> wrote ---- > >-----邮件原件-----
发件人: Thierry Carrez [mailto:thierry@openstack.org] > >发送时间: 2021年1月11日 21:12 > >收件人: openstack-discuss@lists.openstack.org 主题: Re: [all]Introduction to venus which is the project of log management > and has been contributed to the OpenStack community > > >Liye Pang(逄立业) wrote:
Hello everyone, after feedback from a large number of operations and > >> maintenance personnel in InCloud OpenStack, we developed the log > >> management project “Venus” for the OpenStack projects [...] > > OpenStack-aware centralized log management sounds very interesting to > me...
If others are interested in collaborating on producing that component, I > personally think it would be a great fit for the > > "operations tooling" section of the OpenStack Map[1].
Yes, after Inspur did a 1,000-nodes OpenStack single-cluster large-scale test, I was more convinced of the benefits Venus can bring to operation and maintenance. By Venus, we can quickly locate and find problems with the OpenStack platform, which can bring great convenience to operation and maintenance.
https://mp.weixin.qq.com/s/RSrjjZjVFn086StNLV1Ivg This is the article of 1000-nodes test, but it's wrote by Chinese, don't worry ^^, we will publish the English article in future.
This is the demo for Venus, hope that can help you to know what it can be done: https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM > > > [1] https://www.openstack.org/software/
Thanks Liye, Brin for the details, I also things this is a valuable project for day-to-day operation on a large-scale cloud or even small scale to automate the log to alter, etc. Just one question, can we configuration the particular error log msg string/pattern for raising alarm? and different levels of alarm (critical, high priority etc)? For example: If there is a known limitation on my cloud (due to RBAC or backend deps) and requests end up in error so I do not want to raise an alarm for those.
-gmann
At present, the alarm task for the error log is placed in our monitoring system, and the data is obtained from venus. Only the alarms for the number of error logs and the regular matching of typical error log are realized,and alarm level can be defined by yourself. In the future, we will migrate the alarm notification function to venus, and at the same time, we will comprehensively organize the matching mode of the error log to form a configuration template. Everyone is welcome to join us.
-- Thierry Carrez (ttx)
Original ML: http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019748.h...
brinzhang
This seems really interesting. Tracing events with request-ids is something that is quite useful. What is the current state? Can it be deployed by a third party? On Sun, Jan 10, 2021 at 4:01 PM Liye Pang(逄立业) <pangliye@inspur.com> wrote:
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background
In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered：
l Time-consuming for log querying while the server increasing to thousands.
l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services.
l The large amount and dispersion of log make faults are difficult to be discovered.
l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus
According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced *Venus *project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform.
Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario
Venus played a key role in the following scenarios：
l *Retrieval:* Provide a simple and easy-to-use way to retrieve all log and the context.
l *Analysis*: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports.
l *Alerts*：Convert retrieval into active alerts to realize the error finding in massive logs.
l *Issue location*: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure
The architecture of log management system based on Venus and elastic search is as follows:
Diagram 0: Architecture of Venus
*venus_api*: API module，provide API、rest-api service.
*venus_manager*: Internal timing task module to realize the core functions of the log system. Current progress
The current progress of the Venus project is as follows:
l Collection：Develop *fluentd* collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc.
l Index：Dealing with multi-dimensional index data in *elasticsearch*, and provide more concise and comprehensive authentication interface to return query results.
l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors.
l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services.
l Location：Develop the call chain analysis function based on *global_requested* series, which can show the execution sequence, time and error information, etc., and provide the export operation.
l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples
Two examples of Venus application scenarios are as follows.
1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully.
First, we can find the request id of the operation and jump to the virtual machine creation call chain page.
Then, we can query the calling process, view and download the details of the log of the call.
2. In the cloud platform, the error log of each module can be converted into alarms to remind the users.
Further, we can retrieve the details of the error log and error log statistics.
Next step
The next step of the Venus project is as follows:
l *Collection*：In addition to fluent, other collection plugins such as logstash will be integrated.
l *Analysis*: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data.
l *display*: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin.
l *location*: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry
*Venus library*: https://opendev.org/inspur/venus
You can grab the source code using the following git command:
git clone https://opendev.org/inspur/venus.git
Venus Demo
*Youtu.be*: https://youtu.be/mE2MoEx3awM
Laurent Dumont wrote:
This seems really interesting. Tracing events with request-ids is something that is quite useful.
What is the current state? Can it be deployed by a third party?
I see code up at https://opendev.org/inspur/ but I haven't tried deploying it. If it gathers momentum, I suspect it will be proposed as a new official OpenStack project, and if the Technical Committee approves it, it will be moved under the openstack/ namespace on opendev.org. It already follows our usual repository structure (venus, python-venusclient, venus-tempest-plugin...) -- Thierry
---- On Wed, 13 Jan 2021 04:59:47 -0600 Thierry Carrez <thierry@openstack.org> wrote ----
Laurent Dumont wrote:
This seems really interesting. Tracing events with request-ids is something that is quite useful.
What is the current state? Can it be deployed by a third party?
I see code up at https://opendev.org/inspur/ but I haven't tried deploying it.
If it gathers momentum, I suspect it will be proposed as a new official OpenStack project, and if the Technical Committee approves it, it will be moved under the openstack/ namespace on opendev.org. It already follows our usual repository structure (venus, python-venusclient, venus-tempest-plugin...)
+1, I agree that this is ready to apply as an official project and then we can start the discussion in TC about checking the requirement. Please propose the patch to governance and I am adding it to the next TC meeting agenda too. - https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestio... -gmann
-- Thierry
---- On Mon, 16 Aug 2021 10:06:18 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Wed, 13 Jan 2021 04:59:47 -0600 Thierry Carrez <thierry@openstack.org> wrote ----
Laurent Dumont wrote:
This seems really interesting. Tracing events with request-ids is something that is quite useful.
What is the current state? Can it be deployed by a third party?
I see code up at https://opendev.org/inspur/ but I haven't tried deploying it.
If it gathers momentum, I suspect it will be proposed as a new official OpenStack project, and if the Technical Committee approves it, it will be moved under the openstack/ namespace on opendev.org. It already follows our usual repository structure (venus, python-venusclient, venus-tempest-plugin...)
+1, I agree that this is ready to apply as an official project and then we can start the discussion in TC about checking the requirement.
Please propose the patch to governance and I am adding it to the next TC meeting agenda too.
- https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestio...
Thanks to the 'Venus' project team for all the discussion and welcome to the OpenStack governance projects. TC has merged the project application, you can start the process of moving it under openstack/ namespace: https://review.opendev.org/c/openstack/governance/+/804824 -gmann
-gmann
-- Thierry
Thanks TC team and all contributors. We hope that more contributors can join Venus. If you have any questions, you can contact us via ML ([venus]) or #openstack-venus channel. brinzhang -----邮件原件----- 发件人: Ghanshyam Mann [mailto:gmann@ghanshyammann.com] 发送时间: 2021年9月10日 0:47 收件人: openstack-discuss <openstack-discuss@lists.openstack.org> 主题: [lists.openstack.org代发]Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community ---- On Mon, 16 Aug 2021 10:06:18 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ---- > ---- On Wed, 13 Jan 2021 04:59:47 -0600 Thierry Carrez <thierry@openstack.org> wrote ---- > > Laurent Dumont wrote:
This seems really interesting. Tracing events with request-ids is > > > something that is quite useful.
What is the current state? Can it be deployed by a third party?
I see code up at https://opendev.org/inspur/ but I haven't tried > > deploying it.
If it gathers momentum, I suspect it will be proposed as a new official > > OpenStack project, and if the Technical Committee approves it, it will > > be moved under the openstack/ namespace on opendev.org. It already > > follows our usual repository structure (venus, python-venusclient, > > venus-tempest-plugin...) > > +1, I agree that this is ready to apply as an official project and then we can start > the discussion in TC about checking the requirement.
Please propose the patch to governance and I am adding it to the next TC meeting agenda too.
- https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestio...
Thanks to the 'Venus' project team for all the discussion and welcome to the OpenStack governance projects. TC has merged the project application, you can start the process of moving it under openstack/ namespace: https://review.opendev.org/c/openstack/governance/+/804824 -gmann
-gmann
-- Thierry
---- On Fri, 10 Sep 2021 00:35:34 -0500 Brin Zhang(张百林) <zhangbailin@inspur.com> wrote ----
Thanks TC team and all contributors. We hope that more contributors can join Venus. If you have any questions, you can contact us via ML ([venus]) or #openstack-venus channel.
Hi Brin, frickler noticed that #openstack-venus channel is not registered in OFTC network, is it on freenode or another network? -gmann
brinzhang
-----邮件原件----- 发件人: Ghanshyam Mann [mailto:gmann@ghanshyammann.com] 发送时间: 2021年9月10日 0:47 收件人: openstack-discuss <openstack-discuss@lists.openstack.org> 主题: [lists.openstack.org代发]Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
---- On Mon, 16 Aug 2021 10:06:18 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ---- > ---- On Wed, 13 Jan 2021 04:59:47 -0600 Thierry Carrez <thierry@openstack.org> wrote ---- > > Laurent Dumont wrote:
This seems really interesting. Tracing events with request-ids is > > > something that is quite useful.
What is the current state? Can it be deployed by a third party?
I see code up at https://opendev.org/inspur/ but I haven't tried > > deploying it.
If it gathers momentum, I suspect it will be proposed as a new official > > OpenStack project, and if the Technical Committee approves it, it will > > be moved under the openstack/ namespace on opendev.org. It already > > follows our usual repository structure (venus, python-venusclient, > > venus-tempest-plugin...) > > +1, I agree that this is ready to apply as an official project and then we can start > the discussion in TC about checking the requirement.
Please propose the patch to governance and I am adding it to the next TC meeting agenda too.
- https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestio...
Thanks to the 'Venus' project team for all the discussion and welcome to the OpenStack governance projects.
TC has merged the project application, you can start the process of moving it under openstack/ namespace:
https://review.opendev.org/c/openstack/governance/+/804824
-gmann
-gmann
-- Thierry
Sorry to reply you so late, venus could already be installed through devstack, you can try it. In addition, we will also consider it in kolla-ansible. 发件人: Laurent Dumont <laurentfdumont@gmail.com> 发送时间: 2021年1月12日 12:57 收件人: Liye Pang(逄立业) <pangliye@inspur.com> 抄送: openstack-discuss@lists.openstack.org 主题: Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community This seems really interesting. Tracing events with request-ids is something that is quite useful. What is the current state? Can it be deployed by a third party? On Sun, Jan 10, 2021 at 4:01 PM Liye Pang(逄立业) <pangliye@inspur.com <mailto:pangliye@inspur.com> > wrote: Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM
This looks cool. One question about the Venus api, does it support full Elasticsearch DSL or just a subset of queries On Mon, Jan 11, 2021 at 4:59 AM Liye Pang(逄立业) <pangliye@inspur.com> wrote:
Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background
In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered：
l Time-consuming for log querying while the server increasing to thousands.
l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services.
l The large amount and dispersion of log make faults are difficult to be discovered.
l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus
According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced *Venus *project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform.
Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario
Venus played a key role in the following scenarios：
l *Retrieval:* Provide a simple and easy-to-use way to retrieve all log and the context.
l *Analysis*: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports.
l *Alerts*：Convert retrieval into active alerts to realize the error finding in massive logs.
l *Issue location*: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure
The architecture of log management system based on Venus and elastic search is as follows:
Diagram 0: Architecture of Venus
*venus_api*: API module，provide API、rest-api service.
*venus_manager*: Internal timing task module to realize the core functions of the log system. Current progress
The current progress of the Venus project is as follows:
l Collection：Develop *fluentd* collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc.
l Index：Dealing with multi-dimensional index data in *elasticsearch*, and provide more concise and comprehensive authentication interface to return query results.
l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors.
l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services.
l Location：Develop the call chain analysis function based on *global_requested* series, which can show the execution sequence, time and error information, etc., and provide the export operation.
l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples
Two examples of Venus application scenarios are as follows.
1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully.
First, we can find the request id of the operation and jump to the virtual machine creation call chain page.
Then, we can query the calling process, view and download the details of the log of the call.
2. In the cloud platform, the error log of each module can be converted into alarms to remind the users.
Further, we can retrieve the details of the error log and error log statistics.
Next step
The next step of the Venus project is as follows:
l *Collection*：In addition to fluent, other collection plugins such as logstash will be integrated.
l *Analysis*: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data.
l *display*: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin.
l *location*: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry
*Venus library*: https://opendev.org/inspur/venus
You can grab the source code using the following git command:
git clone https://opendev.org/inspur/venus.git
Venus Demo
*Youtu.be*: https://youtu.be/mE2MoEx3awM
Not all es DSL Some venus api will be directly converted to es api, some will query es data and return the result after calculation, and some will query mysql data, such as alarms. 发件人: Lei Zhang <lei12zhang12@gmail.com> 发送时间: 2021年1月15日 11:26 收件人: Liye Pang(逄立业) <pangliye@inspur.com> 抄送: openstack-discuss@lists.openstack.org 主题: Re: [all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community This looks cool. One question about the Venus api, does it support full Elasticsearch DSL or just a subset of queries On Mon, Jan 11, 2021 at 4:59 AM Liye Pang(逄立业) <pangliye@inspur.com <mailto:pangliye@inspur.com> > wrote: Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM
participants (7)
-
Brin Zhang(张百林)
-
Ghanshyam Mann
-
Laurent Dumont
-
Lei Zhang
-
Liye Pang(逄立业)
-
Mark Goddard
-
Thierry Carrez