It can happen without neutron absorption, the neutrons simply knock the atoms out of alignment, and they bounce around a bit (so each neutron can cause more than 1 defect).

(if it doesn't follow why that would matter, look into the heat treatment of steel)