Move reportedly coordinated with Bennett and will last two weeks, in bid to ease pressure on the party, while preventing permanent break with government