Qwen 3.8 27B is around Opus 4.8 level of capability on the Agentic Intelligence Index (52 vs 57). In my testing the locally hosted Qwen is good enough that looking at a given piece of work output I couldn't tell you which model was behind it.
Lately I've been throwing tasks at Qwen and a frontier or recently-frontier model (as well as Kimi, GLM, etc) and the smaller parameter models are not really comparable to Opus when it comes to making intelligent decisions about greyer areas of good software architecture.
Amazing results for open weight and that size, but a really long way off, and I'm extremely skeptical of benchmarks that show these smaller models as being anywhere close to Opus 4.8 (or even earlier Opus's).
I’ve been doing the same thing, giving the same tasks to Qwen 3.8 27B and Opus, and the main difference is that Qwen does not consider edge cases which Opus catches. It’s good at the happy path, but even when hinting that there are uncovered edge cases and gotchas it’s oblivious to it. So I feel like I need a bigger model to do planning/review.
I got so much better experience LLM-Chunking(think RAG) with qwen-38 27B ONCE i move the thinking effort to HIGH vs XHIGH (i think is the default on Open Router).
Not really compatible on all fronts, it's very capable especially with tool calling, workflows, logic and its base coding ability, but it's only a 27b model so it does not have anywhere near the level of knowledge baked in as larger models. This does not mean that it's not a good or useful model - it is on both accounts and very efficient, but it's not similar to a large model generally speaking.
As a counter to that - I've tried various flavors/quants/full weights and Qwen 3.8 27B has been entirely useless at anything non-trivial. Sure - it can do some boilerplate work (though, even armed with a well written spec and working within a very well known framework it went off the rails and did things in a way that were... um... questionable at best) but I don't see it as anything more than a personal assistant style model. Zero chance I'd "work" with it, I spent days trying to get it to do something for me that was usable that I didn't have to have reviewed and refined by a frontier level model or myself. Couldn't do it. The idea that qwen 3.8 27b is _anywhere near_ Opus 4.8 is laughable. Pure benchmaxxing.
DS4 Flash 0731, on the other hand, wildly opposite experience. Would recommend.
GLM 5.2 - even quanted down to a hybrid 4/3 bit setup is amazing for everything but the hardest/most complex stuff in the same projects/realm.
I've had the exact opposite experience. I've been using 3.8 for my daily driver since last week, and I've gradually been giving it more and more complex tasks as it continues to deliver high quality results. Now I am basically handing off large complex features, and 3.8 is doing the planning, task breakdown, implementation and review with just a few notes from my side.
The tradeoff is time (especially on RDMA4 hardware) - it does take a long time and spend a lot of tokens to get to the result, but I've found I can trust the results enough that I can queue a lot of work, essentially have it running all the time and achieve a decent velocity.
It's the first small local model I've felt like I can do real work with.
I can't get 3.8 to exit thinking loops. It will just think and think and think on the most trivial topics. I wanted it to port a speed test powershell script to c#. Claude opus 5 completes it under 60 seconds. I let 3.8 churn about 6 different times for 30+ minutes and it never wrote a single line of code to disk. It wrote lots of lines in thinking.
I've found it tends toward long thinking loops even for simple tasks (and any quantization seems to increase their length), but those do exit eventually, unlike with Qwen 3.6.
I use the Unsloth UD_Q2_K_XL GGUF with default parameters, along with that custom template linked elsewhere in the thread, and no K/V cache quantization.
For smaller models, you'll probably find anything below Q4 will need handholding. Check Unsloth's graphs at the different quantisations VS error rates and you'll see why.
Thank you, I know about those. And I'll stick with this quant. Normally I'd be with yout there, but Qwen 3.8 is turning out to be good at self-correcting, and the free memory I can use for extra context is worthwhile.
That sounds like something is off - I'm using UD-Q4_K_XL on pi with xhigh thinking, and unless I'm vastly underestimating the complexity of the script that's the kind of task I would expect to take a couple of minutes (getting ~30t/s decode). What server are you running, and are you using the recommended parameters from qwen/unsloth?
My experience has been that anything less than a 4-bit quant has a tendency to go off the rails. There’s a threshold of coherency that is being crossed somewhere internal to the model I guess.
Try the same prompt with a larger quant (even if it runs very slowly because the model no longer fits in VRAM) & see if Qwen does better - if so, there’s your answer.
What harness? I've had similar results as _Implicated_ said above - it's not done well in any of the tests I've tried with it. I currently have it hung off DS4Flash as a pseudo-vision tool and subagent only because of this.
I'm using pi inside a self-made harness. I've found going super lightweight with context (AGENTS.md is maybe 20 lines) and letting the model discover what it needs to gives the best results.
Both of you should mention what quant you're using. And as another comment said, what tasks you're doing, i.e. coding, classification, summarizing etc.
My use-case is coding, currently working on a project with a Rust backend and TS/React/Vite frontend, with probably tens of thousands of lines of code total (including tests).
Fascinating that after years of for me this for me that, literally no one writes 2-3 other words like “i do react frontend” or whatever just for us to know why the results are different
I'm using the unsloth dynamic Q4 and getting good results. I was running Q5, but Q4 gives more context headroom so I can run two agents in parallel with ~100k context each with 32GB vram.
Interestingly, I've found the output of Qwen 3.8 27B to be competitive with Opus (4.6-ish anyway, not quite 4.8), but the experience is very different.
Where Opus has seen it before and knows how to do it, Qwen knows how to work it out. It turns out it's surprisingly capable at working things out. The obvious drawback is that it takes tokens and time.
All the same, getting to run something this capable locally is momentous, and suggests to me that streaming tokens from colossal data centers might not be the long term path forward.
I gave Qwen 3.8 27B and Opus 4.8 the same task in the same codebase. They both came up with the same diff. It wasn't a particularly challenging task (removing a feature flag and updating applicable specs), but it was character for character.
Given it was probably one of the simplest things you could change in the codebase, the kind of stuff you give a new developer on the project, I'm not sure it's so telling, there is usually just about one way to remove a feature flag.
That was actually Watchdogs 2. Great game, basically GTA but you are a hacker in the Bay Area. The SF portion of the map was accurate down to every building.
If that second link is an in-game map, it is far from accurate to the real world. Unless it’s from a timeline in which Golden Gate Park is connected to the Presidio, and half of SF has fallen into the ocean.
yeah, I was confused during the whole thing, i get 70 t/s on a 3090, which evens out around 50 t/s at 128k+ , have been running 3.6 and now 3.8 (both iq4_nl at 256k q4 kv) on the 3090 for months. I am confused as to what we 'discovered' here, it's a common config. and at less than 1/2 the price of the gpu (and double the bandwidth, though no fp4 cores to be fair).
A 3090 has 936 GB/s of bandwidth vs 432 GB/s on the 70W RTX PRO 4000 SFF, so 70 t/s there is not surprising. The interesting constraint here was fitting a workload-tuned 5.01 BPW quant + 256K + MTP into 24 GB while working with less than half the memory bandwidth
Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.
If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).
Looks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers.
Or just 100 token input benchmarks.
Or just failed ones as actual measures.
You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.
Humans naturally want SOMEONE to be the good guy! Sad story, in this instance.
reply