Several assailants tore down flags and banners outside caravan, scuffled with group of men inside, then fled from scene by car