Friday, 14 March 2025
28.3 C
Singapore
34.9 C
Thailand
21.8 C
Indonesia
27.8 C
Philippines

New research highlights ChatGPT’s struggles in helping with coding tasks

New research from Purdue University reveals significant errors in ChatGPT's programming assistance, emphasising caution and calling for further study.

According to recent research, ChatGPT is still grappling with effectively assisting with programming issues despite becoming an overnight sensation. While many developers have turned to generative AI tools like GitHub’s Copilot to streamline their workflow and free up time for more productive tasks, a new study from Purdue University sheds light on significant shortcomings in ChatGPT’s performance.

Study reveals widespread errors

Researchers at Purdue University analysed 517 questions from Stack Overflow, comparing ChatGPT’s answers to those provided by human experts. The findings were startling: more than half (52%) of the responses generated by ChatGPT were incorrect. The breakdown of errors is as follows: 54% were conceptual misunderstandings, 36% were factual inaccuracies, 28% were logical mistakes in code, and 12% were terminology errors.

The study also highlighted that ChatGPT often produced unnecessarily lengthy and complex responses. This overabundance of detail can lead to confusion and distractions for developers seeking straightforward answers. Despite these issues, an ultra-small-scale poll involving 12 programmers revealed that one-third preferred ChatGPT’s articulate, textbook-like responses. This preference underscores how easily the AI’s seemingly authoritative tone can mislead coders.

Implications for the coding community

The implications of these findings are significant. Errors in coding can cascade, potentially causing problems across multiple departments or even entire organisations. The researchers emphasise the importance of caution when using ChatGPT for programming tasks.

They state, “Since ChatGPT produces many incorrect answers, our results emphasise the necessity of caution and awareness regarding the usage of ChatGPT answers in programming tasks.” This caution is vital to prevent minor coding errors from escalating into more significant, complex issues.

Call for further research and transparency

Beyond urging caution, the researchers advocate for further studies to identify and mitigate these errors. They also call for greater transparency and communication regarding the potential inaccuracies in ChatGPT’s responses. This openness is crucial for developers to make informed decisions about when and how to use AI tools in their workflows.

As the coding community continues to integrate AI into its practices, these findings serve as a reminder of the limitations and risks associated with relying too heavily on automated tools. While ChatGPT and similar technologies offer exciting possibilities, their current capabilities require scrutiny and responsible use to ensure they genuinely enhance productivity without introducing significant errors.

Hot this week

ASUS launches global giveaway with gaming hardware worth US$13,000

ASUS launches a global giveaway worth US$13,000, offering motherboards, CPUs, graphics cards, and more. Enter by 17 April 2025.

Google launches free AI prompting course to boost workplace efficiency

Google launches a free AI prompting course on Coursera to help professionals use AI effectively and improve workplace efficiency. Available in Singapore.

More Singapore organisations turning to AI to tackle cyber threats

More Singaporean organisations are using AI for cybersecurity as phishing threats grow, but investment in essential security training is declining.

Signal president warns AI agents pose serious privacy and security risks

Signal President Meredith Whittaker warns at SXSW that AI agents pose serious risks to privacy and security by requiring deep system access.

Oracle could be the leading choice to manage TikTok in the US

Oracle is reportedly the top choice to manage TikTokโ€™s US operations as ByteDance seeks a deal before the April deadline. Uncertainty remains.

Blueskyโ€™s CEO trolls Mark Zuckerberg with a viral T-shirt that sells out in minutes

Blueskyโ€™s CEO Jay Graber trolled Mark Zuckerberg with a Latin T-shirt at SXSW, selling out replicas in 30 minutes. Here's why it struck a chord.

Oracle could be the leading choice to manage TikTok in the US

Oracle is reportedly the top choice to manage TikTokโ€™s US operations as ByteDance seeks a deal before the April deadline. Uncertainty remains.

Android introduces Auracast support for hearing aids in public audio broadcasts

Android 16 will add Auracast support, allowing hearing aids to connect directly to public audio broadcasts.

AI startup Sesame unveils base model for its voice assistant

AI startup Sesame has released CSM-1B, the base model behind its voice assistant Maya, raising concerns over voice cloning risks and safeguards.

Related Articles