PM's office says 'distance was kept' from his adviser on ultra-Orthodox affairs, who met with him on Thursday