If they aren't on the call, they won't be able to hear what you're saying until someone on the call records it and gives it to them. They can also join the voice chat without you knowing, but they'd have to be muted and you wouldn't hear the "ding" when they did.