Former US president's advanced age and diminished political influence could also stand between him and the White House