Current, former residents tell Daily Beast they often saw ex-PM's security detail in building allegedly linked to sex trafficking ring