You can do this and also listen to conversations you aren’t present for from the active character portraits without changing settings. There’s even a prompt for ongoing conversations to join in on them as a listener remotely.