I can afford to buy a luxury townhome across the street in a neighborhood I love, but I won’t for 4 reasons