After reading that ChatGPT, the chatbot powered by artificial intelligence, can generate code in response to a query, I wanted to see how well it could accomplish this task. Would ChatGPT produce code riddled with bugs? Would it threaten my job as a developer?
In a quest to answer those questions, I came up with three examples, including prompts I used and the code generated.
Testing ChatGPT in C
First was a simple program in the C programming language. I asked ChatGPT: "Write a program in C to output the first 15 Fibonacci numbers."
It commented: "Sure, here is a simple program in C to output the first 15 Fibonacci numbers,” along with the following output:
#include <stdio.h>
int main() {
int i, n = 15, first = 0, second = 1, next;
printf("The first %d Fibonacci numbers are:\n", n);
for (i = 0; i < n; i++) {
if (i <= 1)
next = i;
else {
next = first + second;
first = second;
second = next;
}
printf("%d ", next);
}
return 0;
}
After the listing, it explained how the program works: “The program uses a loop to calculate and print the first 15 Fibonacci numbers. The first two numbers of the sequence are 0 and 1, and each subsequent number is the sum of the two previous numbers. The loop calculates each number in turn, storing the previous two numbers in variables to use in the calculation. The loop then prints each number as it is calculated.”
So far so good; the code compiles and runs. The output is 0 1 1 2 3 5 8 13 21 34 55 89 144 233 377, which is spot on. (Keep in mind that other users have reported ChatGPT surfacing wrong information in response to simple queries, including elementary-level math problems.)
Attempt Two-Word Count
Next, I wanted to try something a bit more complicated. Thus, my next prompt: “Write a program in C# to read a text file containing words. Count how many times each unique word occurs and output a list of words and counts sorted by word counts in reverse order."
The first time I tried this, the code came back with errors. The second time, it delivered (with comments):
using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
class Program
{
static void Main(string[] args)
{
string filePath = "sample.txt"; // path to the text file
// read all lines in the file into an array
string[] lines = File.ReadAllLines(filePath);
// concatenate all lines into a single string
string text = string.Join(" ", lines);
// split the text into an array of words
string[] words = text.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
// create a dictionary to hold the word count
Dictionary<string, int> wordCount = new Dictionary<string, int>();
// loop through all words and count their occurrence
foreach (string word in words)
{
if (wordCount.ContainsKey(word))
{
wordCount[word]++;
}
else
{
wordCount.Add(word, 1);
}
}
// sort the dictionary by value (word count) in descending order
var sortedWordCount = wordCount.OrderByDescending(x => x.Value);
// print the result
foreach (var item in sortedWordCount)
{
Console.WriteLine("{0} : {1}", item.Key, item.Value);
}
}
}
It added : “In this program, we first read all lines from the text file into an array, and then concatenate them into a single string. We then split the string into an array of words using the Split() method, and count the occurrence of each unique word using a dictionary. Finally, we sort the dictionary by value (word count) in descending.”
I created a text file made up of words randomly picked from a Scrabble word list. ChatGPT worked perfectly to count up those words (finding some 15 times). I did a quick search and confirmed the result. At this point, ChatGPT had a perfect score in testing. Time to come up with something really challenging.
How Logical is ChatGPT?
The next query: "How many letters are in the answer to this question?" This is a tricky question for an A.I. “How many” means the answer must be a number and “letters” mean the number must be rendered in text.
The only correct answer: “four.”
ChatGPT answered: “30.”
I think it counted the letters in the last few words of the question.
Despite some amazing capabilities, ChatGPT isn’t actually intelligent. It synthesizes answers from data it was trained on, never knowing if that data is actually correct.
Counting Numbers
Just for fun, I tried another puzzle: “In the set of nine-digit numbers from 123456789 up to 987654321 with every digit occurring just once in each number, what number occurs in the 100,000th place.” It came up with the correct answer… but when I tried the same prompt a week letter, it came up with a completely different and wrong result.
After that, I asked it to write code to solve that puzzle: “Write a C# program to answer what number occurs in the 100,000th place in the set of nine digit numbers from 123456789 up to 987654321 with every digit occurring just once in each number.”
This is what it produced:
using System;
class MainClass
{
static void Main()
{
int count = 0;
for (int i = 123456789; i <= 987654321; i++)
{
if (IsDistinct(i))
{
count++;
if (count == 100000)
{
Console.WriteLine(i);
break;
}
}
}
}
static bool IsDistinct(int n)
{
bool[] digits = new bool[10];
while (n > 0)
{
int digit = n % 10;
if (digits[digit])
{
return false;
}
digits[digit] = true;
n /= 10;
}
return true;
}
}
I ran the code and it came up with 147028356, which is clearly wrong, as there should be no 0 in the output. Also, the answer should start with a 3 as there are 8! (factorial); that’s 40,320 numbers that start with 1, 8! that start with 2, and the 100,000th number is going to be in the 3rd 8! Block, which runs from 80,640 to 120,960. I wrote a program to solve it by brute force. The answer is 358926471.
Conclusion
ChatGPT is amazingly useful for many purposes, particularly if it involves writing. When it involves code and/or calculations, you should always check everything. You can ask ChatGPT to explain how it got a result, but it will cheerfully lie! And as I found out, you can try the same question a few days later and it may answer with something completely different.
