A sprawling compound shows that the area's (sometimes gruesome) history, tied to an ancient Jewish sect, extends well beyond the caves made famous in 1947