Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.


It takes time and effort to build such benchmark. It works at Databricks scale, I'm not sure smaller companies are ready to invest on internal benchmarks.

But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: