Exiles say little to gain from ties with North African country, but others see opening for access to long-forsaken heritage sites for community forced to flee decades ago