Move will see Kahana returning to Knesset as an MK and bumping out Yomtob Kalfon amid fears he would be the latest party member to quit teetering government