Mildew, dirt, and such get between paint and the material. This makes the paint crack and peel more quickly. This means you’ll have to paint it again sooner. Loose paint has the same effect.
Bare wood needs priming so the paint can stick to it better.
If the replacement boards are wood, prime on all six sides. If the boards you need to replace are in a problem area and likely to rot again in the future, try to replace them with a product other than wood. Azek and James Hardi trim are good examples.
You’ll use one or two coats depending on the condition of the wood and if there is a color change happening. Two coats are always a good idea when in doubt.