Less than a month ago, OpenAI released a language model that can generate human-like text based on a given input, which is being talked about everywhere. Have you seen it? Tried it? It certainly can create a standard job description in seconds that looks like every other one. What kind of questions have you asked the tool?
ChatGPT (Chat Generative Pre-Trained Transformer) is an AI bot that can provide text answers in a conversational and consistent style. And It didn't take long for the tech world to discover it also writes Coding content in different programming languages. The volume of AI driven content has grown dramatically. In fact, according to the Verge, Stack Overflow, the go-to question-and-answer site for coders and programmers, has temporarily banned users from sharing responses generated by AI chatbot ChatGPT.
This capability to enhance a developer’s coding quickly can be a really good aspect, if you think of a partner to complement or enhance your code, for example. But how good is the code? Part of the concern that StackOverflow identified is that the ChatGPT answers/ content may not be correct. ChatGPT makes it easy to generate responses that appear to be correct, but on further review, are wrong.
Testing the ChatGPT Developer
We decided to test the capability and quality of ChatGPT by asking it to answer some of our favorite/ famous retired questions from our library of over 40,000 technical questions, then comparing them to our "Ideal answer" guidelines. We asked the ChatGPT developer 15 of our retired questions and put it through a FloCareer interview. Did the ChatGPT developer pass. Would it be recommended to our clients?
The ChatGPT Developer as envisioned by ChatGPT's cousin, DALL-E
Here's what we found (3 sample questions):
Retired Question: “Van Eck sequence”
(Note: Candidate will be able to see the following description, and a YouTube video about the Van Eck sequence at the time of the Interview).
Write code in your favorite programming language to print the "Van Eck Sequence" for the first 100 positions.
Sequence is defined as below:
You start with the number 0 at the first position.
For the next number, if you have seen the number at the current position before, the next number is the distance between the current position and the position where you saw this number before.
If you haven't seen this number before, print 0.
So, you start with 0. You haven't seen 0 before, so the next number is also 0. But, now the next number is 1 -- because you saw 0 at position 1 and position 2. You haven't seen 1 before, so the next number is 0. You saw 0 at position 2 and position 4, so the next number is 2. And so on...
0, 0, 1, 0, 2, 0, 2, 2, 1, 6, 0, 5, 0, 2, 6, 5, 4, 0, 5, 3, 0, 3, 2, 9, 0, 4, 9, 3, 6, 14, 0, 6, 3, 5, 15, 0,...
FloCareer’s Ideal Answer:
FloCareer’s Ideal Answer:
ChatGPT Answer:
ChatGPT Answer Outcome (compiled code):
ChatGPT Answer (Second option):
ChatGPT Answer Outcome (compiled code):
FloCareer’s Verdict: Interesting approach, but provides wrong answers.
Retired Question: “Josephus' Problem”
There was a group of 41 Jewish soldiers surrounded by the Roman army, and they didn't want to get caught. So, they sat down in a circle and came up with an algorithm. Everybody had a sword, and starting from person #1 in the circle, everybody will kill the next living person on the left. So, #1 will kill #2. #3 will kill #4, #5 will kill #6 and so on. The last living person will have to commit suicide to avoid getting caught by the Romans.
The soldier called Josephus preferred to be caught over committing suicide. So, in the group of 41 soldiers, he chose the location where he will be the last person living.
Write a program to figure out, in a group of given N people, where should Josephus sit to live at the end of all internal killing.
There is a mathematical solution to this problem (check out www.youtube.com/watch?v=uCsD3ZGzMgE). But, your program should use the brute force method to find the position. The output of the program may look like this:
FloCareer’s Ideal Answer:
ChatGPT Answer:
ChatGPT Answer Outcome (compiled code):
ChatGPT Answer (Second option):
ChatGPT Answer Outcome (compiled code):
FloCareer’s Verdict: Looks interesting, but not quite our ideal answer.
Retired Question: “Recaman's Sequence”
Recaman's sequence is defined as follows:
The simplest way to explain the sequence is: "go back, and if you can't go back, go forward".
Write a program to print the first N number of Recaman's sequence.
The first few numbers would be:
0, 1, 3, 6, 2, 7, 13, 20, 12, 21, 11, 22, 10, 23....
Here's the Numberphile video about the same: www.youtube.com/watch?v=FGC5TdIiT9U
Interesting Tidbits:
Neil Sloane (creator of the On-Line Encyclopedia of Integer Sequences (OEIS)) has conjectured that every number eventually appears, but it has not been proved.
Allan Wilks, Nov 06, 2001, computed 10^15 terms of this sequence. At this point, all the numbers below 852655 had appeared, but 852655 =
5*31*5501 was missing.After 10^25 terms of A005132 the smallest missing number is still - Benjamin Chaffin, Jun 13 2006.
Even after 7.78*10^37 terms, the smallest missing number is still - Benjamin Chaffin, Mar 28 2008.
Even after 4.28*10^73 terms, the smallest missing number is still - Benjamin Chaffin, Mar 22 2010.
Even after 10^230 terms, the smallest missing number is still 852655. - Benjamin Chaffin, 2018
FloCareer’s Ideal Answer:
ChatGPT Answer:
ChatGPT Answer Outcome (compiled code):
ChatGPT Answer (Second option):
ChatGPT Answer Outcome (compiled code):
FloCareer’s Verdict: Looks good, but provides wrong answers.
FloCareer Conclusions on ChatGPT for Coding
When we compile the results, some of them are found accurate, some presented a different result from their own described outcome, and some presented errors. All of them were different from our ideal answer examples. This shows us that Artificial Intelligence writing codes isn’t a threat to good code development. Although the current version of ChatGPT is fast, it is not quite accurate. In fact, the dialogue with the ChatGPT developer required more coaching/ help from the interviewer. Thus, as you can see from the above answers, the ChatGPT developer would not pass the FloCareer interview process and be recommended to our clients. In our next post, we will discuss how ChatGPT can be used for Interview Frauds to augment their responses and potentially game the system.
