I agree with you that CORs are not expected to wear unit numbers on their uniforms shirts, and they are probably not supposed to do so. In places where I’ve been active in Scouting, there were very few CORs that were engaged with their units, and, consequently, very few who ever appeared in uniform. When a COR is frequently seen in uniform, s/he is usually deeply involved with his or her unit(s), and all the ones I can recall seeing or knowing wore the unit numbers on their uniforms. While these individuals are members of the council that governs the council as well as members of the district committee, their emotional bonds are usually to their units, and, right or wrong, they consider that their first responsibility.
The COR position patch is found in the universal and nonunit insignia section of the Guide to Awards and Insignia. Under the description it says it is to be worn by a “council Scouter.” That’s a pretty strong indication that unit numbers are not expected to be worn with that position patch, but I haven’t seen anything that bluntly says doing so is forbidden.
If I were to encounter a uniformed COR who is obviously actively engaged with his or her unit(s), I wouldn’t dream of suggesting that the unit numerals do not belong on the uniform. (I never bring up points about other adults’ uniforms anyway, but I’ll discuss it, if they open the door to the conversation.) I would not want to disturb any positive energy a COR is bringing to his or her unit(s).
An indication that the COR may wear unit numerals is the fact that the COR appears on the unit roster. So, while a COR is a council Scouter, s/he is also a unit Scouter. On page 16 of the Chartered Organization Representative Guidebook (see https://filestore.scouting.org/filestore/pdf/511-421(16)_WEB.pdf) shows an organization chart for a Cub Scout pack. It is impossible to tell in the chart where the chartered organization ends and where the Cub Scout pack begins. Studying the chart may lead some to conclude that the COR is part of both and thus both a unit and a council Scouter.
CORs that have multiple units with different numbers likely would not wear unit numerals. In the past, it hasn’t been uncommon for LDS chartering organizations to have three Boy Scout troops, and each would typically have the same COR.