When removing/adding a server group that has any talk power at all, to any user, their actual talk power only updates after the user re-joins the channel they were in.
This affects the talk power, and the order that the clients are sorted in a channel, of course until they rejoin.