Consultants Stumped by AI, Scoring 95% Until They Knew the Answer Was Machine-Generated

SAP conducted an internal experiment to gauge consultant attitudes toward AI, revealing striking results. Five teams were asked to validate answers to more than 1,000 business requirements completed by SAP's AI co-pilot, Joule for Consultants, a workload that would normally take several weeks. Four teams were told the analysis had been completed by junior interns fresh out of school, and they reviewed the material, finding it impressive, with a rating of about 95% accuracy. The fifth team, however, was informed that the answers had come from AI, and they rejected almost everything. According to Guillermo B. Vazquez Mendez, chief architect of RI business transformation and architecture at SAP America Inc., the consultants' initial skepticism was a result of how they were communicated with about the AI's capabilities. "We need to be very cautious as we introduce AI, especially in how we communicate with senior consultants about its possibilities and how to integrate it into their workflows," Mendez said. The experiment highlighted the importance of clear communication and understanding of AI's capabilities in order to effectively integrate it into professional workflows. The experiment was conducted as a means to understand consultant attitudes toward AI and its potential impact on their work. The results showed that when consultants were informed that the analysis had been completed by AI, they were less likely to accept the accuracy of the results. However, when they were asked to validate each answer one by one, they discovered that the AI was highly accurate, surfacing detailed insights that they had initially dismissed. The overall accuracy of the AI's answers was again about 95%. The experiment has significant implications for the integration of AI into professional workflows. As AI becomes increasingly prevalent in various industries, it is essential to understand how professionals will react to its capabilities and how it can be effectively integrated into their work. The results of the experiment suggest that clear communication and understanding of AI's capabilities are crucial in order to ensure its successful integration. The experiment has also sparked discussions about the potential benefits and challenges of AI in professional settings. Some experts argue that AI can augment human capabilities, freeing up professionals to focus on higher-level tasks and decision-making. Others caution that AI can also lead to job displacement and changes in the nature of work. The experiment was conducted internally by SAP, and the results have not been publicly disclosed until now. The company has not announced any plans to implement the findings of the experiment into its business practices. However, the results of the experiment are likely to have implications for the broader industry, as companies continue to explore the potential of AI in their workflows.