Why would you reasonably expect cadmium to be a contaminant in zinc but not in silver?

Cadmium is an element that usually forms a Cd2 + ion just like Zn2+.

Silver commonly forms the univalent, Ag + ion, so it could not directly replace Zn2 + in ores and you will have to read up on this, most heavy metal mining firms have silver, lead, zinc operations; these are very common mixtures of ores. So, the cadmium is more likely to be the contaminant in zinc than in silver.