Yaakov Litzman, now housing minister, says he was kept in the loop 'here and there' as experts presented opinions to decision makers