Do all the settings in one Customer Group need to be shared?

Yes, there is currently no granularity for the settings. All members of the Customer Group will have to inherit all the settings specified.