Could you explain how you define hallucination in the context of creativity? Lack of consistency aside, the two look quite similar to me - or would you consider the existence of orcs and elves in LOTR to be hallucinations, too?
> Could you explain how you define hallucination in the context of creativity
Mostly as you've mentioned, "lack of consistency", which manifests in variety of ways - presence of cellphones in historical settings (I'd call them "global inconsistency") and a character for whatever reason suddenly smoking a cigarette ("local inconsistency") I've never mentioned nor present in the context.
The noise is a huge problem, indeed. Still, a panel of review agents using models and harnesses different from the one implementing a set of changes has proven immensely useful for myself. The panel is basically an n×m matrix of agents and highly specific review prompts, i.e.:
- review for intent fulfillment: is the ticket done?
- review for correctness: race condition bugs, ...
- review for security: check against this list of sources and best practices
- review for api conformity: identify all surfaces of systems outside this codebase touched by the code changes and check against their docs
- etc. pp., same for maintainability, observability & analytics, test coverage, usage of feature flags
The matrix is sparse, so not every model is used with each of the review categories. Effort levels vary, too. The next stage does a consolidation across all findings, then another stage spins up one agent per finding and investigates the whole codebases for identical / similar instances of the finding; finally, it suggests a fix.
This works extremely well for finding deficits, but the amount of noise drives me insane, too. Lots of feedback is technically correct and "by the book", but pretty useless in practical terms – or even detrimental because the amount of code written and thus the size of the change set explodes. I'm not yet sure how to tackle this problem, any suggestions are welcome!
You need to literally review the review with another llm pass to push back on the first. Ask it to do something like reassess the severity claims and only surface real P0 to P2 issues.
Just came across this and it saved not only all of my email, but also my day! Awesome stuff!
"Currently supports Gmail, Google Calendar, Microsoft Teams, Discord, Slack, CardDAV, Granola, Circleback, Notion AI Meeting Notes, Beeper Desktop, and IMAP sync, plus offline imports from Slackdump, MBOX exports, Apple Mail (.emlx) directories, PST archives, and common chat/text export formats."
Reminds me of how my wife, at the start of the pandemic, beat me TWELVE TIMES IN A ROW in rock-paper-scissors for going first playing Azul. That's like... anyways, she lost all of the Azul games, so I guess we're even.
I had a buddy like you. When he was drunk, I could look in his eyes and think, "I chose rock last, so now he'll choose paper to beat rock again." And I won the vast majority of the time (in a drinking game that involved dice).
Also, there's a Simpsons episode: Bart thinks "I'll pick rock. NOTHING beats rock!", and Lisa thinks "He ALWAYS picks rock...".
I know an e-commerce company where the staging environment was completely hijacked by product managers to "stage" their data. They've even convinced management to ask IT to build a tool for migrating data from staging to production. All of this just to avoid building a proper release flow for (product-)data.
reply