Open Stack

On Thu, Jan 4, 2018, at 6:46 AM, Matt Riedemann wrote:
> I've talked to a few people on the infra team about this but I'm not 
> sure what is temporary and transitional and what is permanent and needs 
> to be fixed, and how to fix it.
> 
> The main issue is for newer jobs like tempest-full, the logs are under 
> controller/logs/ and we lose the log analyze formatting for color, being 
> able to filter on log level, and being able to link directly to a line 
> in the logs.
> 
> Should things be like logs/controller/* instead? If not, can someone 
> point me to where the log analyze stuff runs so I can see if we need to 
> adjust a path regex for the new structure?

I don't think that is necessary, instead the next item you noticed is related to the issue.

> 
> The other thing is zipped up files further down the directory structure 
> now have to be downloaded, like the config files:
> 
> http://logs.openstack.org/69/530969/1/check/tempest-full/223c175/controller/logs/etc/nova/

The issue is that the wsgi os-loganalyze application is only applied to .txt log files if they are also gzipped:
https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/templates/logs.vhost.erb#n95

As you've noticed the new job processes log files differently. In the case of etc contents they are now zipped when they weren't before and in the case of the service logs themselves are no longer gzipped when they were gzipped before.

So if we want os-loganalyze to annotate these log files they should be gzipped by the job before getting copied to the log server (this also helps quite a bit with disk usage on the log server itself so is a good idea regardless).

> 
> I think that's part of devstack-gate's post-test host cleanup routine 
> where it modifies gz files so they can be viewed in the browser.

It was, but this new job does not use devstack-gate at all, there is only devstack + job config. Fixes for this will need to be applied to the new job itself rather than to devstack-gate.

I've pushed up https://review.openstack.org/531208 as a quick check that this is indeed the general problem, but for longer term fix I think we want to update our log publishing ansible roles to compress everything that isn't already compressed.

> 
> Please let me know if there is something I can help with here because I 
> really want to get the formatting back to help with debugging CI issues 
> and I've taken for granted how nice things were for oh these many years.

Please check that the above change results in the os-loganalyze behavior that you expect and if adventurous you can help us updating the generic publishing role.

Clark