Former chief of equal rights watchdog said it’s disrespectful to want them to ‘be like us’