Party officials feel the PM's political career is nearing its end, but they're keeping quiet, at least for now