Well, technically
they're there. They just don't have names or a specific number, nor do they arrive at the manger, but rather sometime later after the birth.
My mother-in-law sets up her nativity each year with the wise men figurines set up way off to the side, to indicate that they were traveling, not there yet. Gotta settle that cognitive dissonance somehow...