Trust in AI Depends on Peer Review, Not Expert Opinions

As the world becomes increasingly reliant on artificial intelligence (AI), a pressing question arises: how can we ensure that AI systems are trustworthy and unbiased? The answer lies not in the expertise of a select few, but in the collective wisdom of peers. In a world where AI is rapidly transforming industries and societies, it's time to rethink the way we evaluate the trustworthiness of AI models. Behind the scenes of AI development, experts like Vinay Chaudhri propose using expert interviews and Sunstein tests to gauge an AI model's true level of understanding. These methods are effective in assessing technical proficiency, but they also risk perpetuating power structures that favor a select group of elites. This is a concern highlighted by Cathy O'Neil in her book review, where she notes that AI systems often reflect the goals of the select few people who build and control them. Consider the case of AI-powered hiring tools, which have been shown to discriminate against certain groups of people. These tools are often developed and controlled by a small group of experts who may not fully understand the nuances of bias and fairness. As a result, the AI systems they create can perpetuate existing power structures and reinforce systemic inequalities. This is not just a technical issue; it has significant implications for society. As AI becomes increasingly pervasive, it's essential that we have a robust system in place to ensure that AI systems are trustworthy and unbiased. This is where panels of peers come in – a diverse group of individuals from various backgrounds and expertise who can provide a more comprehensive evaluation of an AI model's trustworthiness. Panels of peers can bring a range of perspectives and experiences to the table, allowing for a more nuanced understanding of an AI model's strengths and weaknesses. This approach also acknowledges that AI is not just a technical issue, but a social and cultural one as well. By involving a diverse group of peers in the evaluation process, we can ensure that AI systems are developed with the needs and values of all stakeholders in mind. Experts like Dr. Kate Crawford, a leading researcher on AI and bias, agree that panels of peers are essential for evaluating AI trustworthiness. "We need to move beyond the idea that AI is just a technical issue, and instead recognize that it's a social and cultural one," she says. "Panels of peers can provide a more comprehensive understanding of an AI model's trustworthiness, and help us develop AI systems that are fair and unbiased." In addition to providing a more nuanced understanding of AI trustworthiness, panels of peers can also help to identify and mitigate potential biases in AI systems. By involving a diverse group of individuals in the evaluation process, we can ensure that AI systems are developed with the needs and values of all stakeholders in mind. As AI continues to transform industries and societies, it's essential that we have a robust system in place to ensure that AI systems are trustworthy and unbiased. Panels of peers offer a promising solution to this challenge, providing a more comprehensive evaluation of an AI model's trustworthiness and helping to identify and mitigate potential biases. In the words of Dr. Crawford, "We need to think about AI as a social and cultural issue, and involve a diverse group of peers in the evaluation process. This is the only way we can ensure that AI systems are developed with the needs and values of all stakeholders in mind." By embracing this approach, we can create a more equitable and just AI future, where AI systems are developed with the needs and values of all stakeholders in mind.