Break it down into cases. Is the coordinate closest to a corner, or to some point on an edge? You can determine that without actually working out any distance - play around with pen and paper and hopefully that will become clear. Then calculate the distance to either the relevant corner, or the relevant edge.
It's easiest to think about an axis-aligned rectangle, and you can convert any problem to have an axis-aligned rectangle by rotating everything.
Quite a few pitfalls there, though. If I was doing computer graphics often, the geometry wouldn't be a problem I guess. But I've not done any such thing in decades.
If the job needs the geometry, then it might be a useful question to ask.
It's easiest to think about an axis-aligned rectangle, and you can convert any problem to have an axis-aligned rectangle by rotating everything.