Meghan Markle was apparently put off a birthday party by her baby shower backlash