I have the same experience. I have my own benchmark, I take a relatively complex project like FreeSWITCH on github, which is part of the training set for all coding LLMs anyway so they should know it and I ask the AI to code small snippets, tests, suggestions and fixes to see how well it understands the codebase and the architecture.
I just tried the latest Cursor + Sonnet and it failed in every task. The problem is that there is no way to understand the code without either complete understanding of the domain and the intents or running it in some context.
Telecom and media domains in particular are well documented in specs and studied in forum discussions. I am sure they are part of the training data, because I can get most answers if asked directly. So far the LLMs fail to reason about anything useful for me.
I just tried the latest Cursor + Sonnet and it failed in every task. The problem is that there is no way to understand the code without either complete understanding of the domain and the intents or running it in some context.
Telecom and media domains in particular are well documented in specs and studied in forum discussions. I am sure they are part of the training data, because I can get most answers if asked directly. So far the LLMs fail to reason about anything useful for me.