OpenAI's Models Found to Perpetuate Caste Bias in India
In a disturbing revelation, researchers have discovered that OpenAI's language models are perpetuating caste bias in India, a country with a complex social hierarchy. The models, which are widely used for tasks such as language translation and text generation, were found to replace the surname of a Dalit applicant with one associated with high-caste Indians.
According to Dhiraj Singha, a postdoctoral sociology fellow who encountered this issue firsthand, the chatbot's behavior was not an isolated incident. "The experience of AI actually mirrored society," he said in an interview. "It reminded me of the sorts of microaggressions I've encountered when dealing with people from more privileged castes."
Singha, who grew up in a Dalit neighborhood in West Bengal, India, had used ChatGPT to refine his application for postdoctoral fellowships in March. The chatbot not only smoothed out his language but also replaced his surname "Singha" with "Sharma," which is associated with high-caste Indians. This incident sparked concerns about the potential perpetuation of caste bias through AI models.
Caste, a social hierarchy that dates back centuries, remains a significant issue in India. The country's constitution prohibits discrimination based on caste, but the practice persists in various forms. Dalits, who are at the bottom of the caste hierarchy, continue to face marginalization and exclusion.
The use of AI models in India has been growing rapidly, with many companies and institutions adopting these technologies for tasks such as language translation, customer service, and content generation. However, the recent discovery raises questions about the potential consequences of perpetuating social biases through technology.
Experts say that this incident highlights the need for greater awareness and regulation of AI development in India. "This is a wake-up call for the AI community to recognize the potential impact of their models on marginalized communities," said Dr. Nishant Shah, a researcher at the Centre for Internet and Society.
OpenAI has not publicly commented on the issue, but the company has acknowledged that its models can perpetuate biases if they are trained on biased data.
The Indian government has also been criticized for its handling of caste-related issues. In 2018, the Supreme Court ruled that the Scheduled Castes and Scheduled Tribes (Prevention of Atrocities) Act, which protects Dalits from violence and harassment, was being misused by non-Dalit individuals to settle personal scores.
As AI continues to transform industries and societies around the world, this incident serves as a reminder of the need for greater accountability and transparency in AI development. "We must ensure that our technologies do not perpetuate existing social inequalities," said Dr. Shah.
Background
Caste is a complex social hierarchy in India that dates back centuries. The country's constitution prohibits discrimination based on caste, but the practice persists in various forms. Dalits, who are at the bottom of the caste hierarchy, continue to face marginalization and exclusion.
Additional Perspectives
The incident has sparked concerns about the potential consequences of perpetuating social biases through technology. Experts say that this is a wake-up call for the AI community to recognize the potential impact of their models on marginalized communities.
Current Status and Next Developments
OpenAI has not publicly commented on the issue, but the company has acknowledged that its models can perpetuate biases if they are trained on biased data. The Indian government has been criticized for its handling of caste-related issues, and experts say that greater awareness and regulation of AI development in India are needed to prevent similar incidents in the future.
Sources
Dhiraj Singha, postdoctoral sociology fellow
Dr. Nishant Shah, researcher at the Centre for Internet and Society
OpenAI
Indian government
*Reporting by Technologyreview.*
