Priority Speaker

I have lowered the priority speaker dimm so that when I am speaking everyone else gets muted. But for the life of me I can't figure out how to make it so I can't hear them as well. I use my TS for 40+ person raids and if 2 or 3 people start yelling it's very distracting for the leaders.

Any help would be greatly appreciated.

Priority Speaker