Justices say in 5-4 opinion that flagship Modern Orthodox university in New York must exhaust other appeals before turning to nation's highest court