Showerthought: I tried both a big open model and a small fine-tuned one for a customer service bot, and the small one won

Last month, my team built a test bot using GPT-4 for a client's online help page. It was smart but slow and gave weirdly long answers. Then we fine-tuned a much smaller Llama model on just their past support tickets for about $200 in compute costs. The small model answered in under 2 seconds every time and used the client's exact wording, which customers liked way more. The big model felt like showing off, but the small one just solved the problem. Has anyone else found that smaller, focused AI works better for specific business jobs?

3 comments

3 Comments

annawebb7d ago

Your test only shows what works for one simple task. Big models can handle surprise questions a small bot would fail at completely. That two second speed won't matter when a confused customer asks something not in your old tickets. You're basically building a fancy search engine, not real intelligence. Relying on that small box could really backfire when you need to understand new problems.

luna8247d ago

Ugh, my friend's small bot totally froze when a client asked about their weird new error code.

rileyp496d ago

Wait, you hit the exact same wall we did last year. Our small bot crashed on a weird shipping question too. So we set up a simple rule: if the bot's confidence score drops below 80%, it instantly hands off to a human agent with the full chat history. You keep the fast answers for common stuff but never leave a customer stuck. Isn't that better than one slow, expensive model trying to do everything?