ChatPRD vs. Top LLMs: A Comparative Analysis
I compared the PRDs from the top LLMs and ChatPRD. ChatPRD emerged as a clear favorite.
Well, technically, it tied with a 4o Mega-Prompt — both getting B+. But here’s why I prefer ChatPRD overall.
(You can see all the PRDs for yourself here.)
Simple Prompt Results
It’s common knowledge at this point:
You can’t prompt a generic LLM with something simple like ‘I work on Gmail. Create a PRD for Cmd+K functionality.’
The resulting PRD scores anywhere from a C to an F:
• GPT-4o: Much better than GPT-4, scored a solid C
• Gemini Advanced: Disappointingly poor, expected more from the #2 on LLMSys, scored a D
• Claude Opus: Slightly better than Gemini, but still lacking in content quality, scored a D+
• LLama-3: The worst performer, with truly garbage content, scored an F
Basically, if you just simply prompt LLMs you’re going to think they’re garbage for PRDs.
Mega-Prompt Results
But you can definitely optimize these LLMs. Since ChatGPT-4o came out on top, I focused on optimizing that.
The secret is: you need to create a massive mega-prompt with stylistic advice, detailed PRD instructions, and specific section guidance.
It works. The result is:
• Quality of Content: Rivals ChatPRD’s, both scoring B-
• Usability of Writing: Below ChatPRD, scoring B compared to ChatPRD’s A-
• Completeness: Can be elite, scoring A- versus ChatPRD’s B
But that’s with a heavily tuned mega-prompt just for the task. It takes significant time and effort to craft these prompts. And they’re not easily reusable for other PRD tasks.
ChatPRD Can Take Simple
This is where ChatPRD shines. You can create a similar quality PRD overall in much less time:
1. You start with the same simple prompt we started with for all LLMs, a single line
2. Then you just answer the 3–5 followup questions ChatPRD has
3. And, optionally, further fine-tune (which I recommend)
Like with any AI generated content, you need to edit it to sound human and ensure it didn’t hallucinate.
But the amount of editing is much less for ChatPRD. The usability of the writing is that much higher. As Alisa Haman said, “it actually sounds like a PM.”
So that’s where, in the end, you end up saving a lot of time.
Time Saved Is Worth It For PMs
That’s why I recommend ChatPRD.
And also worked with the team to build an ultimate guide to using it.
As a 3x CPO herself, understands that for PMs, time saved writing PRDs is:
• Time available for building relationships
• The option to talk to more customers
• Or simply not burning out
AI isn’t replacing PMs anytime soon.
But PMs who embrace AI will certainly replace those who don’t.
