Hello,
You should have the same privileges as the main organizer's account, try removing the "consent" feature from his account before trying to remove him from the family group.
Here's a video you can use as a guide.
https://www.youtube.com/watch?v=1XI__wWPkTU
I've seen the same issue happen to other users when they're trying to remove their child from the family group to remove the restrictions on their account.
Hope this helps,
JC