Officials express concern that social distancing measures and lack of visits could cause psychological hardships