Polling may signal trouble for Trump as key group slips away