Prevent server group to move users from some channels

I've got a server group which grants you access to join an admin-only channel. The problem is, the holders of the server group can move non-holders into the channel, which I'd like to block, how do I do that?

Prevent server group to move users from some channels