I had that too for the first week of release! I didn't even know what was causing it, and the game audio was the last thing I checked! Works fine for me now, though.
It's the ambient sounds that kills my framerates. If I have any other sounds up, including music, my sounds or the voice over... it only slows it slightly. Turn ambient sounds off.
