After selling his company for $6.5 billion and doing 40 angel investments, Ross Mason is joining an unusual quant-like VC firm