Hi!

OSProfiler enables tracing only requests with a special header. The header is not embedded in each request (even if you configure osprofiler in your system), you need to use a special CLI argument to set it. So even if the tracing of one particular request slows done the flow (which should not happen), it should give zero impact on the performance of the whole system.

As for Rally (in particular, the task component), it creates resources with a special naming format that allows to filter out only these resources during the cleanup process. We are using Rally in production as a part of the monitoring for ~5 years or so.