Fractals are math’s trippiest concept, and they get even weirder when used to solve a puzzle involving the British coast