Open Stack

2015-01-07 13:07 GMT+01:00 Ritesh Raj Sarraf <rrs at researchut.com>:
> On 01/07/2015 04:32 PM, Tim Bell wrote:
>
>> The profile of this hardware looks virtualized. That makes me ask if you
>> have
>> installed the para-virtualized drivers ?
>>
>
> Out of interest, does using para-virtualized drivers reduce the chance of
> soft lockups ? I can understand that it is more efficient but I do not know
> if the soft lockups are more likely when using emulated hardware.
>
>
> Please see below for more details. You have a high change to run into soft
> lockups if you have no CPU cycles available. When I/O intensive tasks run,
> most of your CPU cycles are blocked contending to get an ack for the write()
> call.
>
>
> On 01/07/2015 04:43 PM, Matej Mailing wrote:
>
> Hi, we are using virtio
> <target dev='vda' bus='virtio'/>
> (from the instance's xml on the compute node)
>
> Should we use any particular other drivers on the instance as well?
>
>
> No. You should always use virtio drivers for both, network and block.
>
> But, your CPU consumptions were still blocked. My initial suspicion was that
> they were blocked because you weren't using virtio drivers. When you don't
> use virtio drivers, the Host OS spends more time in translating the
> instructions. By using virtio, you instruct right inside the Guest OS, the
> capabilities of the Host OS and the underlying physical hardware. Thus
> resulting is much better performance.
>
> That said, since you are using virtio for both net/block, it would be
> interesting to determine the cause of your setup. What kind of backend
> storage do you have ? What is the latency on it ?
>
> It could genuinely be one of the following case:
>
> * Slow storage
> * Slow storage driver (can be knocked off, given that you confirmed you use
> virtio)
> * Memory contention - If you do too much buffered I/O, without writing the
> dirty pages back.
>
>
> PS: It'd be easier if you pasted your logs, formatted, somewhere else. The
> mail client screws up the logs completely. :-(
>
> --
> Ritesh Raj Sarraf
> RESEARCHUT - http://www.researchut.com
> "Necessity is the mother of invention."

Hi,

we have tweaked some NFS parameters for our backend storage, including
packet sizes and now the load is constantly slower, there is much less
I/O waiting. There were no lock-ups after that. So it really seems it
was slow storage that was to be blamed. I will certainly report back
if the failures occur again.

Thanks everyone for your ideas and time.
Regards,
Matej