Well, Viavoice in my case acted strange using Alsa, and so I just ended up configuring my soundcard using the kernel drivers and the problem went away. So I haven't tested it in a while to see if it has changed, and haven't felt like trying it again.