One minister lambastes the figures as fear-mongering that is 'detached from reality,' and other ministries are said to predict far lower numbers