Conservative governments have cracked down on refugee entry, replacing permanent visas with temporary ones and refusing asylum to those who arrive by boat