Reports say PM's concerns over judicial appointments, top court possibly blocking him from forming government are sticking points between parties