About half of those surveyed say US president's actions have made things worse for African Americans, Muslims, Hispanics and women