Former president claims Americans know him well so there's no need for a public showdown with his rivals for the White House