My test ranked the models backwards

17 November 2026

Since early November I have been testing whether a model on my laptop can sort my mail. It doesn’t have to answer anything. It only has to tell me which mails need a reply today, which this week, and which none. If a laptop can do that, no second company has to read my inbox for it.

So I tested three models. Claude Sonnet runs in Anthropic’s cloud and was my reference. Gemma 4 26B is an open model from Google that runs on my MacBook Pro with 32 GB. Gemma 4 E4B is its small sibling, which also fits on a laptop with 16 GB. I’ll call it the small model.

Made-up mails

On 8 November I started with 20 mails I wrote myself: a client asking for a quote, a newsletter, a question from a colleague. Claude and the bigger Gemma sorted all 20 correctly, the small model 18. That is the order I expected.

My real inbox

Then I took 200 mails from June to October and let my own behaviour grade the models. If I replied the same day, the mail needed a reply today. Within a week, this week. Never, none. I answer only about one mail in a hundred, so I picked 80 I had answered and 120 I had not.

One number matters most here: of the mails I answered the same day, how many did a model miss? A miss means someone waits for me. A false alarm only costs me a glance.

All three models missed about half of them, Claude too. The misses showed why. A short mail from a client looks like a newsletter if you don’t know who sent it. I know my clients. The models didn’t.

Three lines of context

So before each mail, every model got three lines from my own setup: the Gmail category, how often I had written to the sender before, and the person’s role if I keep a note on them.

With that, the small model missed 37% of my same-day replies. Claude still missed 51%. In one case its context said I had written to the sender 28 times, and it still decided the mail could wait. The bigger Gemma didn’t get better either.

Then I changed one sentence. I had asked whether a mail “needs Ben’s answer”. Now I asked whether Ben would reply, and added that I reply to clients, collaborators and people I know. Claude improved and missed 43%. The small model got worse and missed 46%, so the prompt has to be tuned per model.

The trade-off

In those 200 mails, 40% were mails I had answered. In my real inbox it is about 1%. So I ran both models on 300 newer mails from early November, as they came in. I had answered 3 of them, and both models caught all 3, which with 3 mails proves little. But the small model flagged 50 mails as needing a reply, Claude 19. It misses less and raises more false alarms.

The largest Gemma, 31B, didn’t make it. It took over a minute per mail and ran out of memory once it got the context.

What I do now

Since 16 November the small model sorts my inbox in the background, next to my current setup, without acting on anything, at about 5 seconds per mail. In early December I check how often it was right. If it holds, it moves to a 16 GB MacBook that runs all day, and my mail stays on a machine I own.

For sorting, context mattered more than the model. Three lines of it moved the numbers more than going from a small laptop model to Claude. My made-up test favoured the big models because made-up mails carry all their context in the text. Real mail doesn’t.

Sorting is the easy part

To sort mail, a model mostly needs to know who wrote. To answer a question, it needs to know the answer, and there the model matters. For a practice I built an agent that is always on and helps the team with admin. Claude answered 27 of 28 test questions correctly. The open models at IONOS, a German cloud, got between 4 and 16. In a later run Mistral Large came closest, 24 against 26 for Claude Sonnet, and Mistral is French. Claude Opus graded the answers without knowing which model wrote them, which may favour Claude; I haven’t checked by hand yet. The bigger open models that might close the gap, GLM and Kimi, I couldn’t find at any German provider. That is the next post.

Why can a German company that wants its data in Germany run a model on a laptop that sorts its mail, but not rent one in Germany that answers its customers?


I write What Actually Works: real cases where AI made work or life easier, experiments I build myself, and what I learned from both.

Free. One click to leave. You will get one email asking you to confirm.

Or follow along on LinkedIn and Instagram.