According to Palace insiders, a royal Fab Four never would have worked