They are repeating a million times on their huggingface page that the thinking output should be included in the conversation history for multiturn use. That makes me wonder, is this generally needed for LLMs? Because that implies that they only really function well on typicial multiturn flows; I'm experimenting with a completely different approach: there is still the main message stream in the context, but the agent can use structured means to exchange messages and interact with terminals and the file system in a statefull manner. The state is rendered to the context on every cycle, with the message history just being a "panel". I'm still in the middle of trying this out so I can't say yet if it will work. But I hope the models are flexible enough for this.
This non-Preview release scored 16/25. Probably the same model as the preview, or at least not particularly improved if you want agentic performance.
Good to see more options for large open models though!
It's hard to point definitively to a reason it underperforms
but generally models that perform well at agentic tasks were trained on very large numbers of tokens (Qwen, frontier models) or were heavily post trained for reasoning (see eg Nemotron-Cascade-2-30B-A3B at 21/25 vs the base model Nemotron-3-Nano-30B-A3B-Base at 12/25 )
They are repeating a million times on their huggingface page that the thinking output should be included in the conversation history for multiturn use. That makes me wonder, is this generally needed for LLMs? Because that implies that they only really function well on typicial multiturn flows; I'm experimenting with a completely different approach: there is still the main message stream in the context, but the agent can use structured means to exchange messages and interact with terminals and the file system in a statefull manner. The state is rendered to the context on every cycle, with the message history just being a "panel". I'm still in the middle of trying this out so I can't say yet if it will work. But I hope the models are flexible enough for this.
Trinity Large Preview managed 17/25 on my agentic SQL benchmark: https://sql-benchmark.nicklothian.com/?#all-data which is a fairly mediocre score for a large model (Qwen 27B managed 23/25)
This non-Preview release scored 16/25. Probably the same model as the preview, or at least not particularly improved if you want agentic performance.
Good to see more options for large open models though!
It's hard to point definitively to a reason it underperforms but generally models that perform well at agentic tasks were trained on very large numbers of tokens (Qwen, frontier models) or were heavily post trained for reasoning (see eg Nemotron-Cascade-2-30B-A3B at 21/25 vs the base model Nemotron-3-Nano-30B-A3B-Base at 12/25 )
This is one of the first high-performing fully open weight American models to my knowledge. Congrats! (insert American flag here)
The OG Lllama 3? GPT-OSS? NVidia Nemotron 3?
Maybe a better link: https://www.arcee.ai/blog/trinity-large-thinking
The weights are on huggingface, surprisingly: https://huggingface.co/arcee-ai/Trinity-Large-Thinking
That's crazy affordable. Promising! Maybe gonna give today's submission StepFun 3.5 Flash a run perhaps, who knows. https://news.ycombinator.com/item?id=47602879 https://app.uniclaw.ai/arena?tab=costEffectiveness&via=hn