100 Days of Gen AI: Day 39

Today I’m calling the Siri Playlist Actions project done. It was a fun week-long project.

Today I added polish—updating the readme, improving how token refresh handling works, and making some additional refactors for readability/maintainability.

Here’s my take on how working with ChatGPT as an assistant worked out.

ChatGPT functioned well as a prolific junior engineer. The generated Go code compiled every time without issue, and early on while the app was small, it pretty much got everything right on the first try. Though as the size and complexity of the app grew, its abilities started to break down. There were two examples in particular that stood out.

Example 1. The doom loop

I’ve seen this pattern before where the generated solution “A” returns an error. I share the error with ChatGPT, and it generates a new solution “B.” This hits a new error, which I again share. ChatGPT generates a new solution, which is just another variation on “A.” I then feed the error back and it generates another variation of “B.” And so it repeats in a doom loop. It seems unable to pull itself out of this loop, even though the earlier solutions and their corresponding errors are certainly still within the context window.

Example 2. Locally optimal, but globally nonsensical

As the size and complexity of the app grows, ChatGPT seems to lose context on the other parts of the project (even though I’m not sure I actually exceeded the context window). Maybe those earlier conversations are outside the context window, maybe it struggles with recency bias, or maybe my prompting wasn’t well-directed enough.

For example, at one point on this project I needed to add the ability to check if a Spotify token was expired, and if so, refresh it. When prompted, ChatGPT built this functionality into the GetCurrentlyPlayingSong function. This is all well and good for this particular function, but it’s only one of six functions that interact with spotify, so the other five functions would still lack this functionality, leaving us in a place where we could duplicate this functionality five more times for each of the other functions, or we could build in this functionality in a much more obvious place, a place where it could be shared.

The logical place would be to handle token refresh at the time the spotify auth credentials are pulled from the database. I prompted it with this suggestion, and it got back on track making sensible suggestions.

Overall thoughts

Overall, ChatGPT was a huge help in this project. I’d estimate it cut down total development time by somewhere in the range of 50-75%. And it was a great way to gain more exposure to Go!