Convicted pedophile once managed all the funds of the billionaire owner of Victoria's Secret, his only known client, before falling-out in 2007