Embed Notice

HTML Code

<blockquote style="position: relative; padding-left: 55px;"><section><a href="https://mastodon.social/users/danluu/statuses/111758882204729038">Dan Luu (danluu@mastodon.social)'s status on Monday, 15-Jan-2024 17:01:39 JST</a><a href="https://mastodon.social/@danluu" title="danluu@mastodon.social"><img src="https://gnusocial.jp/avatar/35339-48-20221122073124.webp" width="48" height="48" alt="Dan Luu" style="position: absolute; left: 0; top: 0;">Dan Luu</a></section><article><p>How good is ChatGPT at solving Advent of Code?</p><p><a href="https://www.themotte.org/post/797/chatgpt-vs-advent-of-code" rel="nofollow noreferrer">https://www.themotte.org/post/797/chatgpt-vs-advent-of-code</a></p><p>Not very good, it turns out.</p></article><footer><a rel="bookmark" href="https://gnusocial.jp/conversation/2590291#notice-5130277">In conversation</a><time datetime="2024-01-15T17:01:39+09:00" title="Monday, 15-Jan-2024 17:01:39 JST">about 10 months ago</time> <span>from <span><a href="https://mastodon.social/@danluu/111758882204729038" rel="external" title="Sent from mastodon.social via ActivityPub">mastodon.social</a></span></span><a href="https://mastodon.social/@danluu/111758882204729038">permalink</a><h4>Attachments</h4><ol><li><label><a rel="external" href="https://gnusocial.jp/attachment/2114660">... GPT enthusiast "Advent of AI" (henceforth enthusiast) who tried to solve the problems using ChatGPT Plus and then also wrote a blog about it using ChatGPT Plus. Since the blog is generated by ChatGPT it's absolute shit and potentially contains hallucinations, however the github repo with the transcripts is valuable.The enthusiast turned out to be completely useless: it resorted often to babystepping ChatGPT through to the result and he stopped on day 6 anyway.The youtuber was much more informative, for the most part he stuck to letting ChatGPT solve the problem on its own. However he did give it, on a few occasions, some big hints, either by debugging ChatGPT's solution for it or explaining it how to solve the problem. I have noted this in the results.
Results
	part 1	part 2 	notes
day 1	OK 	FAIL 	
day 2	OK 	OK 	
day 3	FAIL 	N/A 	
day 4	OK 	OK 	Uses brute force solution for part 2
day 5	OK 	FAIL 	
day 6	FAIL 	N/A 	ChatGPT Plus solves both parts
day 7	FAIL 	N/A 	
day 8	OK 	FAIL 	ChatGPT Plus solves part 2 if you tell it what the solution is
day 9	FAIL 	N/A 	ChatGPT Plus solves both parts
day 10	FAIL 	N/A 	
day 11	FAIL 	N/A 	ChatGPT Plus could solve part 1 with a big hint
day 12	FAIL 	N/A 	The perofrmance of GPT-4 this year was a bit worse than GPT-3.5 last year. Last year GPT-3.5 could solve 3 days on its own (1, 2 and 4) while GPT-4 this year could only solve 2 full days (2 and 4).ChatGPT Plus however did a bit better, solving on its own 4 days (2, 4, 6 and 9).</a></label><br><a href="https://files.mastodon.social/media_attachments/files/111/758/879/780/328/903/original/5773a1bf81e320b9.png" rel="external">https://files.mastodon.social/media_attachments/files/111/758/879/780/328/903/original/5773a1bf81e320b9.png</a></li></ol></footer></blockquote>

Corresponding Notice

Embed this notice
Dan Luu (danluu@mastodon.social)'s status on Monday, 15-Jan-2024 17:01:39 JST Dan Luu
How good is ChatGPT at solving Advent of Code?
https://www.themotte.org/post/797/chatgpt-vs-advent-of-code
Not very good, it turns out.
In conversationabout 10 months ago from mastodon.socialpermalink
Attachments
1. ... GPT enthusiast "Advent of AI" (henceforth enthusiast) who tried to solve the problems using ChatGPT Plus and then also wrote a blog about it using ChatGPT Plus. Since the blog is generated by ChatGPT it's absolute shit and potentially contains hallucinations, however the github repo with the transcripts is valuable. The enthusiast turned out to be completely useless: it resorted often to babystepping ChatGPT through to the result and he stopped on day 6 anyway. The youtuber was much more informative, for the most part he stuck to letting ChatGPT solve the problem on its own. However he did give it, on a few occasions, some big hints, either by debugging ChatGPT's solution for it or explaining it how to solve the problem. I have noted this in the results. Results part 1 part 2 notes day 1 OK FAIL day 2 OK OK day 3 FAIL N/A day 4 OK OK Uses brute force solution for part 2 day 5 OK FAIL day 6 FAIL N/A ChatGPT Plus solves both parts day 7 FAIL N/A day 8 OK FAIL ChatGPT Plus solves part 2 if you tell it what the solution is day 9 FAIL N/A ChatGPT Plus solves both parts day 10 FAIL N/A day 11 FAIL N/A ChatGPT Plus could solve part 1 with a big hint day 12 FAIL N/A The perofrmance of GPT-4 this year was a bit worse than GPT-3.5 last year. Last year GPT-3.5 could solve 3 days on its own (1, 2 and 4) while GPT-4 this year could only solve 2 full days (2 and 4). ChatGPT Plus however did a bit better, solving on its own 4 days (2, 4, 6 and 9).
  https://files.mastodon.social/media_attachments/files/111/758/879/780/328/903/original/5773a1bf81e320b9.png

Public

Embed Notice

HTML Code

Corresponding Notice