When you remove a unit or subunit from an event, it removes ALL invitees, not just those only associated with the (sub)unit.
When you add a unit to an event, it does not change the invitees list.
When you add a subunit to an event, it automatically invites all youth, leaders, and parents associated with that sub-unit to the event.
Ideally, only invitees that would be impossible to invite after removing the (sub)unit should become uninvited.
Ideally, adding a (sub)unit should never impact the existing invitee list.