Same Butterfly Weed, Different Butterfly

Maybe they put the word out when they find a good garden? If it should happen that you have too many caterpillars at one time and you don't want to kill them but you don't want to sacrifice your flowers, you could always move them manually to another part of the garden that isn't so noticeable. You can move them with your hand (they don't bite or sting) or cut off the stem and carry them with it.