Judge accuses director of failing to investigate claims in widely discredited 2002 documentary that alleged the IDF carried out a massacre in the West Bank