'Our parents or grandparents, neighbors or community members once faced a similarly callous and unwelcoming world, with fatal consequences,' groups say in letter to Pompeo