Ex-ministers reportedly say opposition leader has failed to create genuine relationships with party MKs, basing them only on 'give and take,' which he can't do now that he's not PM