Yuchen He 中文

Notes

The wrong answers that raise no error

A fine-tuned 0.5B model writes the right SQL 82.6% of the time. What worries me is the 16.4% that runs cleanly and returns the wrong rows.

I fine-tuned a 0.5B model to write SQL and scored it by running the SQL. Right answers went from 42.7% to 82.6%, the number you would put on a slide. The number I care about is 16.4%: answers that run cleanly and return the wrong rows. Only 1.0% fail to run, so a schema check, the obvious guardrail, catches almost none of what goes wrong.

Qwen2.5-0.5B-Instruct, no fine-tuning (Q4_K_M)
  • 42.7% right
  • 41.9% run, wrong rows
  • 15.3% did not run
Fine-tuned with LoRA (Q4_K_M)
  • 82.6% right
  • 16.4% run, wrong rows
  • 1.0% did not run
391 held-out questions from b-mc2/sql-create-context, each scored by running the generated SQL on three generated SQLite databases and comparing the rows with the reference query. Source: llm-finetune-lab, results/.

Reading 30 of the wrong answers, the most common problem was a changed string constant: “nouvelair” became “nouvelle-air”. A rule that flags constants that are not in the question fires on 18.8% of the wrong answers and 0.3% of the right ones, and costs no extra inference. Adding a five-sample agreement check brought silent errors from 16.4% to 9.1% while still answering 76.2% of questions. Better, and still above the 5% target, which was my own assumption, and the repository says so.

So the decision I wrote down is: do not ship it as an autonomous answerer; ship it as a tool that suggests SQL for a person to review. The lesson is that accuracy tells you how often a model is right, and the product decision depends on how it fails. Measure the failure you cannot see.

The scoring method, the audit of the 30 cases, the guardrail table and what would change the decision are in the llm-finetune-lab README.