Experiments
Tests cover CAD operations, reconstruction quality and processing time.
Independent interpretations: three views, four methods
Twelve routes began from original rasters with no intermediate geometry or evaluator feedback shared between methods. All saved native files passed persistence checks. One Robust candidate was corrected and reevaluated. Internal scores were too generous on some mechanically wrong nut contours.
| Method | Calls | Total seconds | Input tokens | Cached input | Output tokens |
|---|---|---|---|---|---|
| QCAD | 9 | 1,069.59 | 21,834 | 0 | 46,695 |
| FreeCAD direct | 9 | 1,181.51 | 22,213 | 9,636 | 48,119 |
| FreeCAD Neka | 9 | 1,168.56 | 22,271 | 9,636 | 45,385 |
| FreeCAD Robust | 11 | 1,326.41 | 32,004 | 9,636 | 52,658 |
Cached input is included in input; reasoning is included in output. Fixed order and cache effects confound rankings. Provider usage is not an invoice; dollar costs were not verified.
Neka and Robust MCP
Both passed 3/3 trials on the same 140-entity model. Median time including startup, rendering and cleanup: Neka 23.33 s, Robust 30.39 s.
This compares execution, not model interpretation. Six trials are too few to establish reliability.
Full-sheet reconstruction
| Campaign | Astra accepted | Sol accepted |
|---|---|---|
| FreeCAD Neka | 4/10 | 5/10 |
| Native QCAD | 4/10 | 3/10 |
Accepted by the common Astra evaluator, not by a human reviewer. Campaign conditions differed, so these are not backend accuracy rankings.
Where the QCAD campaign spent its time
Twenty whole-sheet routes took 9 h 43 m 34 s wall time. CAD accounted for about 6 m 31 s; the model/correction workflow was the bottleneck.
Matrix: backend, drafter model and strategy
Three drawings (06, 03, 10) were rebuilt by FreeCAD and QCAD, with Astra and Sol as drafter, each with the edit-capable strategy and with regeneration from scratch: 24 final outputs. Each drawing, backend and drafter combination began with one shared attempt; each strategy then added two attempts. The evaluator, the blind overlay judge, reasoning effort and limits were the same everywhere. FreeCAD has no hatch entity, so it was told to omit hatching and was not penalized for it.
| Group | Internally accepted | Mean internal score | Judge adherence / coverage | Ink precision | Input / output tokens |
|---|---|---|---|---|---|
| FreeCAD | 7 / 12 | 84.7 | 88.4 / 88.8 | 77.1% | 376,379 / 614,602 |
| QCAD | 3 / 12 | 81.4 | 87.4 / 86.1 | 74.9% | 569,949 / 862,726 |
| Astra drafter | 7 / 12 | 84.9 | 88.5 / 87.7 | 78.4% | 434,389 / 729,641 |
| Sol drafter | 3 / 12 | 81.2 | 87.3 / 87.2 | 73.5% | 511,939 / 747,687 |
| Edit-capable | 5 / 12 | 83.0 | 88.0 / 87.6 | 76.1% | 496,951 / 692,872 |
| From scratch | 5 / 12 | 83.1 | 87.8 / 87.3 | 75.9% | 449,377 / 784,456 |
| Astra, edit-capable | 4 / 6 | 86.0 | 88.7 / 88.5 | 78.7% | 219,923 / 329,149 |
| Astra, from scratch | 3 / 6 | 83.8 | 88.3 / 86.8 | 78.1% | 214,466 / 400,492 |
| Sol, edit-capable | 1 / 6 | 80.0 | 87.3 / 86.7 | 73.5% | 277,028 / 363,723 |
| Sol, from scratch | 2 / 6 | 82.3 | 87.3 / 87.8 | 73.6% | 234,911 / 383,964 |
- Strategy: overall a tie, 5/12 each. With the stronger Astra drafter the edit-capable arms were accepted slightly more often and scored higher, and editing used about 12% fewer output tokens. It remains the default revision mode.
- Drafter: Astra did better than Sol on every aggregate and was faster.
- Backend: FreeCAD was accepted more often than QCAD, but QCAD was also judged on hatching, which FreeCAD omits, so the comparison is not like-for-like.
Open the full-page overlay viewer: toggle the original and every version per drawing, filter by backend, drafter or strategy.
One route per cell; differences of a few points are within judge and route variation. The judge and drafters share a provider. The client switched to streamed responses mid-run after network drops, and two FreeCAD bases lost their first attempt to an MCP connection timeout. No output has human approval; the overlay is not metrology.
Next tests
- Check the evaluator against known defects.
- Repeat the backend, drafter and strategy matrix with several routes per cell and human review.
- Compare reasoning settings with the PC kept awake.
- Test checkpoint recovery before remote hosting.
Test details and prerequisites
Selected drawing examples are published with authorization. Other originals, raw responses and private configuration remain local.