> What you say makes logical sense, but is hard to square against the fact that, as a nobody, I can borrow $1M at under 3% fixed interest for 30 years to buy a house.

that's canceled out by prices being higher because every other buyer has access to the same rates. Your monthly payments works out to be the same in the end because everybody bids up to the max they can afford.