Mann says will he devote more time to role as anti-Semitism czar, accuses far-left leader of giving 'green light' to hatred of Jews