Open Stack

On 09/11/2013 01:22 PM, James E. Blair wrote:
> Hi,
> 
> Thanks Russel, Paul, and everyone else who helped with the Asterisk
> testing on Friday.  Here's how I see the outcome:
> 
> The main problem we're trying to solve is that the audio is sometimes
> choppy.
> 
> We found that moving the PBX to some specific Rackspace vms improved the
> situation.  Moving to an 8g vm in the same dc did not improve the
> situation; moving to a 30g vm in the same dc did improve it.
> 
> Moving to a small vm in a new dc which is lightly loaded also did
> improve it.  And finally, moving to a small vm in HPCloud also improved
> it.
> 
> This leads me to believe that the load on the host hardware affects this
> problem, possibly due to additional latency in providing cpu time
> slices.  The fact that both using a larger VM (which reserves more of
> the host's resources) and a less heavily loaded data center (less
> competition for host resources) improved things points in this
> direction.
> 
> After this test, Paul disabled the silence suppression, and when we
> moved back to our original vm, the problem did not manifest.  Combining
> this result with the above suggests to me that perhaps the silence
> suppression algorithm is particularly sensitive to cpu latency.
> 
> Regardless, here are some possible next steps:
> 
> * Move Asterisk to a 30g vm in Rackspace (in either our current or the
>   new data center).  This appears to reserve a large enough CPU capacity
>   to help with the problem, though it's possible future host load
>   changes may still affect us.
> 
> * Move Asterisk to a vm in HPCloud.  This has the additional downsides
>   that we can provide no IPV6 service, the Asterisk (and network)
>   configuration becomes more complicated due to NAT, and HPCloud still
>   occasionally terminates vms without warning.  It is also possible here
>   that future host load changes may affect us (but that is more
>   speculative since we haven't seen evidence of that so far).
> 
> * Disable silence suppression in the conference server.  This appears to
>   have a much smaller effect on load than we anticipated (we tested with
>   60 participants and did not observe a problem).  However, silence
>   suppression is useful on large calls to reduce background noise from
>   non-speakers.
> 
> Does anyone have any other observations or suggestions?

I think it's worth trying with the existing smaller VM with that option
turned off to see if it helps long term and just monitor the load.

The option in question is:

;dsp_drop_silence=yes  ; This option drops what Asterisk detects as
silence from
                       ; entering into the bridge.  Enabling this option
will drastically
                       ; improve performance and help remove the buildup
of background
                       ; noise from the conference. Highly recommended
for large conferences
                       ; due to its performance enhancements.


What this does is have Asterisk monitor all audio streams and detect who
is actually talking.  It only mixes the audio streams of the active
talkers into the bridge.  That means doing audio mixing of a few
participants vs all of them at any given time.  It can be a significant
improvement, but only if load is a problem in the first place.  We may
not be loading this up enough for it to matter.

As for why it could sound choppy with this option, it's possible that
it's being too aggressive (not detecting talking soon enough).  There
are a couple of options that could be tweaked for this:

;dsp_talking_threshold=128  ; The time in milliseconds of sound above
what the dsp has
                            ; established as base line silence for a
user before a user
                            ; is considered to be talking.  This value
affects several
                            ; operations and should not be changed
unless the impact on
                            ; call quality is fully understood.
                            ;
                            ; What this value affects internally:
                            ;
                            ; 1. Audio is only mixed out of a user's
incoming audio stream
                            ;    if talking is detected.  If this value
is set too
                            ;    loose the user will hear themselves
briefly each
                            ;    time they begin talking until the dsp
has time to
                            ;    establish that they are in fact talking.
                            ; 2. When talk detection AMI events are
enabled, this value
                            ;    determines when talking has begun which
results in
                            ;    an AMI event to fire.  If this value is
set too tight
                            ;    AMI events may be falsely triggered by
variants in
                            ;    room noise.
                            ; 3. The drop_silence option depends on this
value to determine
                            ;    when the user's audio should be mixed
into the bridge
                            ;    after periods of silence.  If this
value is too loose
                            ;    the beginning of a user's speech will
get cut off as they
                            ;    transition from silence to talking.
                            ;
                            ; By default this value is 160 ms. Valid
values are 1 through 2^31

;dsp_silence_threshold=2000 ; The time in milliseconds of sound falling
within the what
                            ; the dsp has established as baseline
silence before a user
                            ; is considered be silent.  This value
affects several
                            ; operations and should not be changed
unless the impact
                            ; on call quality is fully understood.
                            ;
                            ; What this value affects internally:
                            ;
                            ; 1. When talk detection AMI events are
enabled, this value
                            ;    determines when the user has stopped
talking after a
                            ;    period of talking.  If this value is
set too low
                            ;    AMI events indicating the user has
stopped talking
                            ;    may get falsely sent out when the user
briefly pauses
                            ;    during mid sentence.
                            ; 2. The drop_silence option depends on this
value to
                            ;    determine when the user's audio should
begin to be
                            ;    dropped from the conference bridge
after the user
                            ;    stops talking.  If this value is set
too low the user's
                            ;    audio stream may sound choppy to the
other participants.
                            ;    This is caused by the user
transitioning constantly from
                            ;    silence to talking during mid sentence.
                            ;
                            ; The best way to approach this option is to
set it slightly above
                            ; the maximum amount of ms of silence a user
may generate during
                            ; natural speech.
                            ;
                            ; By default this value is 2500ms. Valid
values are 1 through 2^31


If background noise is a concern, there are other ways to help that
problem.  There's the classic "tell everyone to mute themselves", but
there's also this option:

;denoise=yes ; Sets whether or not a denoise filter should be applied
             ; to the audio before mixing or not.  Off by default. Requires
             ; codec_speex to be built and installed.  Do not confuse
this option
             ; with drop_silence.  Denoise is useful if there is a lot
of background
             ; noise for a user as it attempts to remove the noise while
preserving
             ; the speech.  This option does NOT remove silence from
being mixed into
             ; the conference and does come at the cost of a slight
performance hit.


The denoise audio filter works quite well in my experience.  As the
description suggests, it does consume CPU, but it may be worth a shot.
Note that it requires a module that we don't have (func_speex, not
codec_speex, that's a typo ... I should go fix that).

-- 
Russell Bryant