Images only for variation with assigned image

Variation Swatches:
I only want to see images for items that have been assigned an image for the individual variations. The others may be displayed as buttons, but currently, I see the generic product image there, times the variations. For example (see attached file), a crochet needle with different sizes must be displayed with buttons (for sizes). But a crochet kit with different colors en sizes, must be displayed with images (for the colors) and buttons (for sizes). What’s the correct setting?