Some conservatives are beginning to ring the alarm over the issue, arguing it is increasingly costing the party votes