I experience this with many of the "advanced" models. I find they're actually the most efficient on their low/medium settings, occasionally high. Anything higher than that, and they start inventing more task list items than they check off. They seem to think that every personal project needs extensive adversarial analysis and guardrails and will invent non-issues without being asked.
Unfortunately, I think it's just a continuation of a trend that already started pre-covid. I was surprised that most juniors and interns I worked with seemed to lack the ability to "self-help" and Google basic questions they had. I think it reflects a schooling system in the US that doesn't teach any real critical thinking - just studying until you "know" the answer, without actually teaching students the intuition of how to get there. The result is that when they encounter something not even novel, but slightly unfamiliar they don't even make an attempt to start or immediately tunnel vision down the wrong path without taking a first step back. Funnily, the latter approach feels similar to how LLMs perform on truly novel problems, too.
Claude has always noticeably degraded under heavy load. Opus goes from "decent to work with" to "dumb intern" depending on whether you're working at 3 AM west coast or 10 AM - 5 PM. It's part of why I cancelled my subscription - "max" plans and "extra high" effort are meaningless when there's so much variability between model availability, model performance, and harness bugs every day and every week.
/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
> 3. /goal Spend X minutes from $time writing a technical design doc on $feature.
Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.
I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.
So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
Leaving to run unhindered makes sense when the agents understand both the complete domain context and can be supplied with a strong exit criteria that they understand well.
Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.
Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
I think like the other commentator noted, I don't do this blindly, as I also wouldn't if I was instructing a junior engineer. I have a rough sense of how long it would take me to research all angles of the task. 5 minutes for a complicated feature is too short for a comprehensive plan. Letting it run for a few hours is also probably unproductive. But 10-60 minutes depending on complexity with followup review is a nice sweet spot. Just my $0.02.
The hypothesis generation phase is looking like the most critical part of having an agent reliably hit your targets.
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks.
Can we get a count of people that have had Claude read irrelevant documents or perform unnecessary web searches even when told not to from the beginning?
I'm starting to wonder if this increased token usage is inadvertently bleeding into how Anthropic actually trains their model, especially leading up to IPO. As older models are deprecated and users are forced onto newer models, if the default is less efficient and more token expensive that directly results in higher "profit" for Anthropic in terms of the consumption their users have to tolerate - lest they jump to a competitor.
I've had no problems like the ones you've mentioned while using Opus 4.8. It does overthink stuff with higher effort levels but that's kind of expected.
> I feel like this benchmark reiterates my disbelief that anyone uses the latest Anthropic models for any productive work. They seem to be the best at burning tokens and spawning unnecessary subagents even for well-defined and tightly scoped tasks.
I keep Claude around for some specific tasks:
- Linked up to Figma MCP to implement front-end stuff
- Data analysis, in the "Connect AI to a data source and ask questions" way. I've tried both Opus 4.8 high and GPT 5.5 high for this and Opus is stronger because it gets the intent in the question better
I used to keep it around for planning too, but the 4.8 plans have had more holes than swiss cheese.
Now that enterprise customers are pay-as-you-go with tokens I suspect we'll see renewed interest in OpenAI and their focus on token efficiency. At least I hope so if the alternative is abandoning the tools entirely.
I switched from Anthropic to OpenAI after spending ~$40K in equivalent token costs using Claude over 3 months.
I found Opus 4.7 to be slow and wasteful with token usage. It's shocking how inefficient it is with tasks like bash tool usage and web searching, delegating them to a dozen subagents only to get stuck and never return until you esc and intervene. That, in addition to all of the broken tooling Anthropic built in to limit token usage like the broken monitoring tool made managing Claude a chore. I was happy to pay $200/month for Opus 4.5 when they had more capacity, but 4.7 felt like a huge step back and no longer worth the price and inconvenience.
I remember an OpenAI employee comment on the GPT5.5 release post about how they specifically geared it towards long-horizon tasks and its been a breathe of fresh air in that regard. I have five two-week long sessions going right now and there's been no degradation in performance or efficiency. It's much better at carrying rules/learnings forward even in long-running sessions and grounding/refreshing itself in verified facts when it loses context.
Its funny because in two weeks I've gotten way more done with GPT5.5 with way fewer tokens and way less handholding. I think this goes to show how important tooling and the harness is and how a capable model like Opus 4.7 can be severely handicapped by bad product decisions.
Being able to mange context over long running sessions is a function of the harness, not the model. Are you using Claude Code with GPT5.5? Codex? piclaw? They’ll all have different context management strategies to let you keep going when you would otherwise have filled up context and be forced to stop.
It doesn’t matter how good the harness is if the model does a bad job of planning and continuing from long context. A good harness cannot overcome a weak model.
The longest route on the NYC subway is precisely when you have an appointment and the train decides to stop because there's electrical issues, someone jumped on the track, "there's a train stopped ahead of us", the express decides to go local instead, someone is holding the door, your route involves the F/G or any line that serves less affluent neighborhoods...
Are you saying F/G AND any line serving less affluent areas? Because if not, the G (save for maybe 2-3 stops in Bed-Stuy) is all affluent neighborhoods.
Pfft, there's no G train, it's just a psych experiment to see how long people will wait at a fake train station before giving up and seeking alternative transportation.
On the one hand, this sort of thing feels inexcusable to me. In my mind, the subway is the most reliable mode of city transportation.
On the other, the NYC subway is one of the few 24/7 subway systems in the world, which makes me envious somewhat. Ours closes at midnight and opens around 5 AM.
IMO the subway reliability issues are overblown these days. In the 2010s it was in a really dark spot but it's doing a lot better these days.
> In my mind, the subway is the most reliable mode of city transportation.
Even with its problems the NYC subway still is. Traffic is a nightmare. I have friends who insist on taking a cab to the airport because it's more reliable then end up complaining because they're still on the Belt Parkway or whatever.
I spent a week in NYC in 2014. Used the subway a lot, but mostly within Manhattan. Don't remember having any issues except a few stations smelling of shit and the card readers on the turnstiles being very picky about the speed of your swipe.
In the 2010s it was so bad that I gave up on the subway and either walked, Ubered or took the bus everywhere.
When the MTA bus is better than the subway, things in NY are grim -- it was like the 80s & early 90s again.
The reliability has improved a bit, but the subway crime is also way, way up. So yeah, still a hard pass.
In the mid 2010s i had a reverse commute from Manhattan to Brooklyn and there was a few months where 3 days a week my full commute would take 2hr+ (midtown to sunset park) because the train just sat multiple times not moving. Especially on the bridge where it could be up to an hour just stopped.
The absolute worst time that I ever remember though was maybe in '89 or '90. The Lex-53rd st E/F station had a ton of ongoing construction and on weekends it was being used as a transfer station only that summer. All the staircases were closed and you could only get in/out via train -- this idea seems insane to me but that's NYC sometimes and especially in that era.
Anyway, my family and I were on a train passing through that station one Saturday or Sunday morning on the way to Queens and they made us exit the train inside the station as it was going out of service. Apparently a pipe had burst (I think?) and no trains were in service. The AC in the station was not working and it was maybe 90+ degrees underground. Plus water was leaking from everywhere. And there were hundreds of us trapped on a crowded, wet, dirty platform for like 4-6 hours while no trains were running and there was no way to get out.
Aside: that Belt Parkway story is why people from NY who live/work in Manhattan try to never fly out of JFK. That's the real solution. So much easier to get to EWR/LGA.
> The reliability has improved a bit, but the subway crime is also way, way up.
Eh, it’s all relative, I think you’re in less danger of experiencing issues on the subway than you are getting hit on the road. I take the subway all the time and have never had any problems.
Subway assaults went up 3x between 2009 and 2025 and violent crime in general in the subway has had nearly a 20% spike just in the first two months of this year alone. Assaults by repeat offenders are up 2x from 2019 to 2025.
I'll take a minor fender bender every now and again over someone hitting me in the head with a brick ever.
I can't believe the suggestion is that property damage and/or low-speed collisions would be preferable to being assaulted. And I've been tapped by cars and walked away from it several times. Plus the "well it never happened to me" is just survivorship bias. Over enough decades and enough rides something fucked up on the subway _will_ happen to you eventually.
> I'll take a minor fender bender every now and again over someone hitting me in the head with a brick ever
You're more than 7x more likely to die in a car in New York than on the subway. If you're the kind of idiot that voluntarily trespasses onto the tracks, you're 2x more likely.
In 2023, 112 motor-vehicle occupants died in New York [1] "97 people were fatally struck by subway trains" [2], nearly half (49%) of which are suicides and 33% of which are accidents, almost all of which involve voluntarily trespassing onto the tracks [3]. Five people were killed by assault [4].
In 2023 exactly two Uber passengers died in NYC. One from jumping out of their moving vehicle onto the LIE and the other from being rear-ended by a drag racer on the Whitestone Expressway.
The alternative to taking the subway for the vast majority of NYC residents is not driving to work.
And as an Uber passenger living in NYC I overwhelmingly spent my time on roads that were not highways.
I can't find exact taxi passenger deaths but between 2019 & 2023 there were 23 passenger fatalities across all rideshare services and taxis. At least one of those was a fatal drug overdose (2022).
> between 2019 & 2023 there were 23 passenger fatalities across all rideshare services and taxis
Which is way more than the total number of homicides on the subway system. All of this is before adjusting for trip frequency. (Uber and Lyft do about a fifth of the trips as the subway.)
I lived in New York for 10 years and go back frequently. I take Ubers and cabs (and Blade) all the time. It's convenient. And sometimes, yes, I just want a quiet space in which to relax. But pretending it's safer is simply untrue.
Wait we went from simple assault to comparing it just to homicides now? I just don't want to get attacked or slashed by somebody.
Okay you lived in NY for 10 years, I lived there for over 40. The subway is shit compared to where it was only a handful of years ago. In the last years that I was there, Uber was way safer.
Also I said "fatalities", which isn't just murder and isn't even necessarily a crime. There were 39 homicides in the subway[1] during that same period. So it's not less. But also those are murders whereas the 23 were mostly from accidents.
2009 was a historic low point for subway crimes. Only looking at relative numbers from then is misleading.
There were 573 assaults on the subway in 2024, up from something like 150 in 2009. There were something like 1.9 billion journeys taken that year. Avoiding the subway because of the 1 in 2,000,000 danger of assault is not rational.
Exaggerated paranoid thinking is indeed the thing that leads a lot of people to unnecessarily leave NYC.
573 _felony_ assaults. Misdemeanor assault is still a thing. That includes people fighting you with their fists or groping you.
Also anyone in NY with a functioning brain knows just how underreported subway crime is. You can either get where you're going and go about your day or end it by trying to find a cop and then trying to get them motivated to even take your report.
2009 being a low point for crime should just be normal. I lived through all the bad decades in NY and you're not there yet but certainly trending towards it.
I like how multiple years of "hey why does all of this fucked up shit keep happening to me?" is "exaggerated paranoid thinking". Truly stunning and brave. Being repeatedly victimized by random crime is just a mindset, bruh.
> you’re in less danger of experiencing issues on the subway than you are getting hit on the road
For what it's worth, I lived in New York for ten years and was in one car accident (cabbie, distracted by whatever phone all they're all constantly dialled into, blew through a stop sign) and zero even closer calls on the subway.
Idk, my recent experience with Claude is that 4.7 barely knows how to use basic bash tools - how to properly check when programs have finished running, even basic stuff like how to run pytest suites and read the failed tests from the output without re-running the suite to specifically look for them. It's shockingly dumb for all of the tooling they've built into Claude Code (the useless Monitoring tool that blocks bash polling/sleeping that actually works, etc.).
I finally get fed up and started using GPT 5.5 the past 4 days and its a breath a fresh air despite feeling much more minimal. With Claude I had to write so many hooks to enforce behaviors it wouldn't remember and it lacked common sense on. GPT 5.5 does a much better job with things like knowing the AWS CDK CLI can hang on long CloudFormation deployments and it should actively check the deployment status using CloudFormation API rather than hanging for 30+ minutes - and it does this all without asking.
Maybe there's better tooling built into Codex too, but at least on the surface level it seems like how smart the model is makes a significant difference because Claude has more tools than I can count and still struggles to use "grep".
Edit: Like just now - I can't tell you how many times I day I see this sequence:
"Sorry, I'll run in parallel"
"Error editing file"
"File must be read first"
Repeat 10x for the 10 subagents Claude spawned and then it gets stuck until you press escape and it says "You rejected the parallel agents. Running directly now"
reply