A team has at most 8 members, a clan at most 25. Once your group is full, nobody else can be invited and pending invites can't be accepted – you get a notice instead.
In the "Clan" tab you see next to the member count how full your group is, e.g. "Team 6/8" or "Clan 18/25" (yellow = full, red = over the limit).
If a group is over the limit from earlier times, nobody is removed. New members can only join again once you are below the limit. The leader gets a notice in the mailbox and when logging in.