OpenAI's Models Found to Perpetuate Caste Bias in India
In a disturbing revelation, researchers have discovered that OpenAI's language models, including ChatGPT, are perpetuating caste bias in India. The models, which have gained immense popularity worldwide for their ability to generate human-like text, have been found to reflect and even amplify the country's deeply ingrained social hierarchies.
According to Dhiraj Singha, a postdoctoral sociology fellow at the Indian Institute of Science, Bengaluru, he encountered this bias firsthand when using ChatGPT to refine his application for a fellowship. The chatbot replaced his surname "Singha" with "Sharma," which is associated with high-caste Indians, despite not being mentioned in the application itself.
"I was surprised and disheartened by the experience," Singha said in an interview. "It reminded me of the microaggressions I faced growing up as a Dalit in West Bengal. The chatbot's actions were a reflection of the societal biases that we're trying to overcome."
Singha, who is a first-generation college graduate from his family, noted that the experience was particularly poignant for him. "I've spent years working to overcome the internalized shame and stigma associated with being a Dalit," he said. "But seeing it reflected in AI's behavior was a harsh reminder of how far we still have to go."
The issue of caste bias in AI is not new, but this incident highlights the need for greater awareness and action among developers and users alike. OpenAI has faced criticism in the past for its models' perpetuation of biases related to race, gender, and other social categories.
"Caste is a deeply ingrained aspect of Indian society, and it's no surprise that AI systems are reflecting these biases," said Dr. Sumeet Jain, an expert on AI ethics at the University of Delhi. "However, this incident serves as a wake-up call for developers to prioritize fairness and inclusivity in their models."
OpenAI has not publicly commented on the issue, but experts say that addressing caste bias will require a multifaceted approach. This includes collecting more diverse data sets, implementing robust testing protocols, and engaging with communities affected by these biases.
As AI continues to transform industries and societies worldwide, it's essential to acknowledge and address its limitations and flaws. By doing so, we can ensure that these powerful tools serve the needs of all people, regardless of their background or social status.
Background:
Caste is a complex social hierarchy in India, with four main categories: Brahmins (high-caste Indians), Kshatriyas, Vaishyas, and Shudras. Dalits, who are considered "untouchables," occupy the lowest rung of this hierarchy.
Additional Perspectives:
Dr. Nalini Nadkarni, an expert on AI ethics at the Indian Institute of Technology, notes that caste bias is not unique to OpenAI's models. "Many AI systems have been found to perpetuate biases related to caste, class, and other social categories," she said.
Rohan Kulkarni, a software engineer who has worked with OpenAI's models, expressed concern about the implications of this incident. "If our models are reflecting societal biases, what does that say about our own biases as developers?" he asked.
Current Status:
OpenAI has not publicly commented on the issue, but experts say that addressing caste bias will require a concerted effort from the AI community. Researchers and developers are working to develop more inclusive and fair AI systems, but this incident highlights the need for greater awareness and action.
Next Developments:
As researchers continue to study and address caste bias in AI, it's essential to engage with communities affected by these biases. OpenAI has announced plans to expand its diversity and inclusion initiatives, which may help mitigate these issues in the future.
*Reporting by Technologyreview.*
