Case relates to 1988 killing of at least 5,000 prisoners allegedly ordered by Ayatollah Khomeini in order to avenge attacks carried out by exiled opposition group