Former prime minister had business dealings with the disgraced billionaire and was a frequent visitor to his homes, but denies he ever knew of any his sex trafficking activities