[tripleo][openstack-ansible] Integrating ansible-role-collect-logs in OSA
Hello,
With os_tempest project, TripleO and Openstack Ansible projects started collaborating together to reuse the tools developed by each other to avoid duplicates and enable more collaboration.
During Denver Train 2019 PTG, we decided to unifying the CI logs for both the projects by providing a unified experience to the developers while browsing the CI logs making sure we have a similar logs structure for both the projects, so that one can easily navigate through the logs without scratching the heads and also the logs tree structure should say what logs are present where.
In TripleO, we have ansible-role-collect-logs[1.] role for the same and in OSA we have logs_collect.sh[2.] script for the same. But once the logs gets collected at each other projects, It is very hard to navigate and find out where is the respective files.
A little about ansible-role-collect-logs roles: * A role for aggregating logs from different nodes * Provide a list for files to collect at one places in playbook [3] * Hyperlink the logs with description about log files * Once logs collection is done, it pushes to a particular log server
For example, tempest.html or stestr_results.html is the common file for viewing tempest results. In TripleO, we keep it under logs folder but in OSA logs/openstack/aio1-utility/stestr_results.html. If a new user contributes to another project, he tries to follow the same pattern for find logs as he seen in current project. By Keeping the same structure at all places It would be easier.
So moving ahead what we are going to do: * Refactor collect-logs role to pass defaults list of files at one place * Pass the list of different logs files based on deployment tools * Put system/containers related commands at one place * Replace the collect_logs.sh script with playbook in OSA and replace it.
Thanks for reading, We are looking for the comments on the above suggestion.
Links: [1.] https://opendev.org/openstack/ansible-role-collect-logs/ [2.] https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/lo... [3.] https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/c...
Thanks,
Chandan Kumar
On Fri, 2019-06-28 at 16:30 +0530, Chandan kumar wrote:
With os_tempest project, TripleO and Openstack Ansible projects started collaborating together to reuse the tools developed by each other to avoid duplicates and enable more collaboration.
... And that's amazing!
In TripleO, we have ansible-role-collect-logs[1.] role for the same and in OSA we have logs_collect.sh[2.] script for the same. But once the logs gets collected at each other projects, It is very hard to navigate and find out where is the respective files.
Agreed.
By Keeping the same structure at all places It would be easier.
Agreed again
So moving ahead what we are going to do:
- Refactor collect-logs role to pass defaults list of files at one
place
It seems the role is doing a lot of things, but things can be conditionally triggered. Wondering if this role shouldn't be split into multiple roles... But that's a different story.
- Pass the list of different logs files based on deployment tools
I think this doesn't need to be in the role. Make the role the simplest as possible, and flexible enough to get passed the list of things to log/not log, and the destination. OSA can pass a list of files it wants to gather. But isn't that what the role already does? Or did I misunderstood?
- Put system/containers related commands at one place
Can we simply rely on ansible inventory, and running a play with the role (targetting all) would gather logs for all systems (aio, multinodes, containers), each system could go into their own directory (log folder would be <something>/{{ inventory_hostname }}/...) for example:
aio1/<files gathered> aio1-nova/<files gathered> machine2/<files gathered>
It simple enough. But I am happy to see a different approach.
- Replace the collect_logs.sh script with playbook in OSA and replace
it.
:thumbsup:
Thanks for reading, We are looking for the comments on the above suggestion.
Thanks for tackling that up! I am looking forward a simple common file gathering :)
If you need to do changes in the role (to implement new features), maybe it can help you if I give you a prio list :)
What I am _generally_ looking for in the logs: - The ara reports - The tempest report - The /etc/openstack_deploy/ - The /var/log/ for containers/hosts
What I am _sometimes_ looking for in the logs (so less important/more contextual for me): - ram/disk usage per host - NICs details - cpu features (but I am not sure we really need this anymore) - host details (generally zuul does that for me, so no need to implement something there)
Regards, Jean-Philippe Evrard (evrardjp)
On Wed, Jul 10, 2019 at 2:23 AM Jean-Philippe Evrard < jean-philippe@evrard.me> wrote:
On Fri, 2019-06-28 at 16:30 +0530, Chandan kumar wrote:
With os_tempest project, TripleO and Openstack Ansible projects started collaborating together to reuse the tools developed by each other to avoid duplicates and enable more collaboration.
... And that's amazing!
Good stuff :) Agreed.
In TripleO, we have ansible-role-collect-logs[1.] role for the same and in OSA we have logs_collect.sh[2.] script for the same. But once the logs gets collected at each other projects, It is very hard to navigate and find out where is the respective files.
Agreed.
By Keeping the same structure at all places It would be easier.
Agreed again
So moving ahead what we are going to do:
- Refactor collect-logs role to pass defaults list of files at one
place
It seems the role is doing a lot of things, but things can be conditionally triggered. Wondering if this role shouldn't be split into multiple roles... But that's a different story.
- Pass the list of different logs files based on deployment tools
I think this doesn't need to be in the role. Make the role the simplest as possible, and flexible enough to get passed the list of things to log/not log, and the destination. OSA can pass a list of files it wants to gather. But isn't that what the role already does? Or did I misunderstood?
The TripleO team passes various config files to the collect roles depending on what the needs and requirements are. Some of these config files are public some are not.
upstream config https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/c...
default config https://github.com/openstack/ansible-role-collect-logs/blob/master/defaults/...
These are of course just passed in as extra-config. I think each project would want to define their own list of files and maintain it in their own project. WDYT?
- Put system/containers related commands at one place
Can we simply rely on ansible inventory, and running a play with the role (targetting all) would gather logs for all systems (aio, multinodes, containers), each system could go into their own directory (log folder would be <something>/{{ inventory_hostname }}/...) for example:
aio1/<files gathered> aio1-nova/<files gathered> machine2/<files gathered>
It simple enough. But I am happy to see a different approach.
Yes, this is exactly how it works today. We don't break out which files should be collect for each host, but that is just our preference.
- Replace the collect_logs.sh script with playbook in OSA and replace
it.
:thumbsup:
Thanks for reading, We are looking for the comments on the above suggestion.
Thanks for tackling that up! I am looking forward a simple common file gathering :)
If you need to do changes in the role (to implement new features), maybe it can help you if I give you a prio list :)
What I am _generally_ looking for in the logs:
- The ara reports
- The tempest report
- The /etc/openstack_deploy/
- The /var/log/ for containers/hosts
Agree, having a table of contents in the footer is nice for users as well. https://github.com/openstack/tripleo-ci/blob/master/docs/tripleo-quickstart-... Which is added by infra via https://opendev.org/opendev/system-config/src/branch/master/modules/openstac...
What I am _sometimes_ looking for in the logs (so less important/more contextual for me):
- ram/disk usage per host
- NICs details
- cpu features (but I am not sure we really need this anymore)
- host details (generally zuul does that for me, so no need to
implement something there)
AFAICT, if we were to organize the role more aggressively via the tasks we can easily enable or disable features as needed per project. The majority of the work would around the reorganization to better suit various projects.
Any thoughts on additional work that I am not seeing?
Thanks for responding! I know our team is very excited about the continued collaboration with other upstream projects, so thanks!!
Regards, Jean-Philippe Evrard (evrardjp)
Sorry for the late answer...
On Wed, 2019-07-10 at 12:12 -0600, Wesley Hayutin wrote:
These are of course just passed in as extra-config. I think each project would want to define their own list of files and maintain it in their own project. WDYT?
Looks good. We can either clean up the defaults, or OSA can just override the defaults, and it would be good enough. I would say that this can still be improved later, after OSA has started using the role too.
It simple enough. But I am happy to see a different approach.
Simple is good!
Any thoughts on additional work that I am not seeing?
None :)
Thanks for responding! I know our team is very excited about the continued collaboration with other upstream projects, so thanks!!
Likewise. Let's reduce tech debt/maintain more code together!
Regards, Jean-Philippe Evrard (evrardjp)
Hello,
I've started to split the logs collection tasks in small tasks [1] in order to allow other users to choose what exactly they want to collect. For example, if you don't need the openstack information, or if you don't care about networking, etc.
Please take a look.
I'll also add it on the OSA agenda for tomorrow's meeting.
Kind regards,
1 - https://review.opendev.org/#/c/675858/
On Mon, Jul 22, 2019 at 8:44 AM Jean-Philippe Evrard < jean-philippe@evrard.me> wrote:
Sorry for the late answer...
On Wed, 2019-07-10 at 12:12 -0600, Wesley Hayutin wrote:
These are of course just passed in as extra-config. I think each project would want to define their own list of files and maintain it in their own project. WDYT?
Looks good. We can either clean up the defaults, or OSA can just override the defaults, and it would be good enough. I would say that this can still be improved later, after OSA has started using the role too.
It simple enough. But I am happy to see a different approach.
Simple is good!
Any thoughts on additional work that I am not seeing?
None :)
Thanks for responding! I know our team is very excited about the continued collaboration with other upstream projects, so thanks!!
Likewise. Let's reduce tech debt/maintain more code together!
Regards, Jean-Philippe Evrard (evrardjp)
participants (4)
-
Arx Cruz
-
Chandan kumar
-
Jean-Philippe Evrard
-
Wesley Hayutin