Open Stack

On Wed, Sep 11, 2013 at 1:59 PM, Russell Bryant <rbryant at redhat.com> wrote:
> On 09/11/2013 01:22 PM, James E. Blair wrote:
>> Hi,
>>
>> Thanks Russel, Paul, and everyone else who helped with the Asterisk
>> testing on Friday.  Here's how I see the outcome:
>>
>> The main problem we're trying to solve is that the audio is sometimes
>> choppy.
>>
>> We found that moving the PBX to some specific Rackspace vms improved the
>> situation.  Moving to an 8g vm in the same dc did not improve the
>> situation; moving to a 30g vm in the same dc did improve it.
>>
>> Moving to a small vm in a new dc which is lightly loaded also did
>> improve it.  And finally, moving to a small vm in HPCloud also improved
>> it.
>>
>> This leads me to believe that the load on the host hardware affects this
>> problem, possibly due to additional latency in providing cpu time
>> slices.  The fact that both using a larger VM (which reserves more of
>> the host's resources) and a less heavily loaded data center (less
>> competition for host resources) improved things points in this
>> direction.
>>
>> After this test, Paul disabled the silence suppression, and when we
>> moved back to our original vm, the problem did not manifest.  Combining
>> this result with the above suggests to me that perhaps the silence
>> suppression algorithm is particularly sensitive to cpu latency.
>>
>> Regardless, here are some possible next steps:
>>
>> * Move Asterisk to a 30g vm in Rackspace (in either our current or the
>>   new data center).  This appears to reserve a large enough CPU capacity
>>   to help with the problem, though it's possible future host load
>>   changes may still affect us.
>>
>> * Move Asterisk to a vm in HPCloud.  This has the additional downsides
>>   that we can provide no IPV6 service, the Asterisk (and network)
>>   configuration becomes more complicated due to NAT, and HPCloud still
>>   occasionally terminates vms without warning.  It is also possible here
>>   that future host load changes may affect us (but that is more
>>   speculative since we haven't seen evidence of that so far).
>>
>> * Disable silence suppression in the conference server.  This appears to
>>   have a much smaller effect on load than we anticipated (we tested with
>>   60 participants and did not observe a problem).  However, silence
>>   suppression is useful on large calls to reduce background noise from
>>   non-speakers.
>>
>> Does anyone have any other observations or suggestions?
>
> I think it's worth trying with the existing smaller VM with that option
> turned off to see if it helps long term and just monitor the load.
>
> The option in question is:
>
> ;dsp_drop_silence=yes  ; This option drops what Asterisk detects as
> silence from
>                        ; entering into the bridge.  Enabling this option
> will drastically
>                        ; improve performance and help remove the buildup
> of background
>                        ; noise from the conference. Highly recommended
> for large conferences
>                        ; due to its performance enhancements.
>
>
> What this does is have Asterisk monitor all audio streams and detect who
> is actually talking.  It only mixes the audio streams of the active
> talkers into the bridge.  That means doing audio mixing of a few
> participants vs all of them at any given time.  It can be a significant
> improvement, but only if load is a problem in the first place.  We may
> not be loading this up enough for it to matter.
>
> As for why it could sound choppy with this option, it's possible that
> it's being too aggressive (not detecting talking soon enough).  There
> are a couple of options that could be tweaked for this:
>
This was my thought also, and even suggest that Asterisk might be too
aggressive.  Since I haven't played much with ConfBridge in Asterisk
11, I couldn't think of how to test the different settings during the
window.

> ;dsp_talking_threshold=128  ; The time in milliseconds of sound above
> what the dsp has
>                             ; established as base line silence for a
> user before a user
>                             ; is considered to be talking.  This value
> affects several
>                             ; operations and should not be changed
> unless the impact on
>                             ; call quality is fully understood.
>                             ;
>                             ; What this value affects internally:
>                             ;
>                             ; 1. Audio is only mixed out of a user's
> incoming audio stream
>                             ;    if talking is detected.  If this value
> is set too
>                             ;    loose the user will hear themselves
> briefly each
>                             ;    time they begin talking until the dsp
> has time to
>                             ;    establish that they are in fact talking.
>                             ; 2. When talk detection AMI events are
> enabled, this value
>                             ;    determines when talking has begun which
> results in
>                             ;    an AMI event to fire.  If this value is
> set too tight
>                             ;    AMI events may be falsely triggered by
> variants in
>                             ;    room noise.
>                             ; 3. The drop_silence option depends on this
> value to determine
>                             ;    when the user's audio should be mixed
> into the bridge
>                             ;    after periods of silence.  If this
> value is too loose
>                             ;    the beginning of a user's speech will
> get cut off as they
>                             ;    transition from silence to talking.
>                             ;
>                             ; By default this value is 160 ms. Valid
> values are 1 through 2^31
>
> ;dsp_silence_threshold=2000 ; The time in milliseconds of sound falling
> within the what
>                             ; the dsp has established as baseline
> silence before a user
>                             ; is considered be silent.  This value
> affects several
>                             ; operations and should not be changed
> unless the impact
>                             ; on call quality is fully understood.
>                             ;
>                             ; What this value affects internally:
>                             ;
>                             ; 1. When talk detection AMI events are
> enabled, this value
>                             ;    determines when the user has stopped
> talking after a
>                             ;    period of talking.  If this value is
> set too low
>                             ;    AMI events indicating the user has
> stopped talking
>                             ;    may get falsely sent out when the user
> briefly pauses
>                             ;    during mid sentence.
>                             ; 2. The drop_silence option depends on this
> value to
>                             ;    determine when the user's audio should
> begin to be
>                             ;    dropped from the conference bridge
> after the user
>                             ;    stops talking.  If this value is set
> too low the user's
>                             ;    audio stream may sound choppy to the
> other participants.
>                             ;    This is caused by the user
> transitioning constantly from
>                             ;    silence to talking during mid sentence.
>                             ;
>                             ; The best way to approach this option is to
> set it slightly above
>                             ; the maximum amount of ms of silence a user
> may generate during
>                             ; natural speech.
>                             ;
>                             ; By default this value is 2500ms. Valid
> values are 1 through 2^31
>
>
> If background noise is a concern, there are other ways to help that
> problem.  There's the classic "tell everyone to mute themselves", but
> there's also this option:
>
> ;denoise=yes ; Sets whether or not a denoise filter should be applied
>              ; to the audio before mixing or not.  Off by default. Requires
>              ; codec_speex to be built and installed.  Do not confuse
> this option
>              ; with drop_silence.  Denoise is useful if there is a lot
> of background
>              ; noise for a user as it attempts to remove the noise while
> preserving
>              ; the speech.  This option does NOT remove silence from
> being mixed into
>              ; the conference and does come at the cost of a slight
> performance hit.
>
>
> The denoise audio filter works quite well in my experience.  As the
> description suggests, it does consume CPU, but it may be worth a shot.
> Note that it requires a module that we don't have (func_speex, not
> codec_speex, that's a typo ... I should go fix that).
>
This was the only side effect of the silence change, we did see more
'hot mic' and background noise.  I'd vote for another round of testing
toggling some settings to see if we can find the sweet spot.

> --
> Russell Bryant
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra



-- 
Paul Belanger | PolyBeacon, Inc.
Jabber: paul.belanger at polybeacon.com | IRC: pabelanger (Freenode)
Github: https://github.com/pabelanger | Twitter: https://twitter.com/pabelanger