Maquete.ai turns 3D models into photorealistic renders for architects
This month, my focus with Maquete has been implementing sound evaluations (referred to as evals). Previously, I would be left bewildered as to why blindly upgrading models didn’t produce better results. I was testing without a scientific foundation.
Deciding what to test
To set up an eval, first you must analyse failures within your application. What are users complaining about? What can you clearly see is underperforming? Each day, I analyse outputs in Maquete and classify failures into groups. For example, I may use the below classifications.
CGI appearance
Incorrect material
Geometry failure
Once I have discovered a key failure, I analyse the potential causes in the pipeline. This can be a model mistake, a prompt issue, or a code bug. It’s easier to analyse failures when the outputs are inherently visual, like image and video generation. Using an AI-as-a-judge helps to review a large number of results. Reviewing 10,000 cases would be too laborious.
Performing the evaluation
Let’s assume that I’ve discovered a model is writing poor descriptions. As a result, I compare our current model with a different model such as Grok 4.7, Opus 5.5, or GPT 6.1 Sol.
The standard test I run is a paired comparison. For example, I could test the same source image and prompt through model A and model B. We randomise the left/right placements of the results and display them blind (without a label) so as not to bias the reviewer. Additionally, we drop ties from the result so we only report clear winners.
Deciding the right number of fixtures
A fixture is a real customer case (source image and prompt). I carefully select a mix of production failures and good results to ensure we catch regressions as well as improvements.
Too few fixtures and you won’t get robust results. Too many and your R&D cost surges. Ultimately, the number of fixtures you include is a business decision.
Decisions taken from experiments without a scientific basis could harm the business. As such, we require ‘significant’ results. When comparing two equally good models, one model may win more by chance. Flipping an unbiased coin 20 times won’t always create an even distribution of heads and tails. In our case, we call a result significant if two equally good models would produce a result that lopsided (due to randomness) less than 5% of the time.
Perhaps budget only allows for a 20-fixture test. To call the result significant, the new model would need 15 wins out of 20 (75%). Despite a win rate of 70% (14-6) sounding convincing, randomness would cause that result 1 in 9 times. A win rate of 75% would only be caused by randomness 4% of the time (under our 5% threshold). More fixtures mean a lower win rate is required to be classed as significant. For example, a 50-fixture test would need 66% wins and a 100-fixture test would need only 61%.
Realistically, a 20-fixture test is only useful for weeding out poor models that lose by wide margins. The downside is you’d be likely to miss switching to a genuinely better model. A model winning 14 times out of 20 isn’t a significant result. However, it may be the better model. The fewer fixtures you run, the more randomness is dictating your decisions. That’s why I think a 50-fixture test is a sensible minimum for a switching decision.
How to view failures
If an image generation has a 5% chance of producing a bad result, there is a 64% chance that at least 1 result will fail in a 20-fixture test. Therefore, experiencing one failure does not automatically mean that the model isn’t viable.
Running a 1-fixture test won’t tell you which model is stronger. At best, it is a smoke test for determining whether a model is viable at all. Having a larger dataset reduces that variance so that you aren’t deterred by failures.
AI as a judge
Human review can be costly and time-consuming. AI-as-a-judge has become a popular method for reviewing outputs quickly and cheaply. Some are concerned that having an AI review AI output is tautological. Additionally, models typically prefer their own outputs, e.g. GPT preferring GPT outputs compared to Claude and vice versa. Using a different model like Gemini to review a Claude vs GPT test may be advisable.
Prompt your AI judge with clearly defined scoring criteria, such as a clear pass/fail if certain thresholds are not met.
An AI judge can provide instantaneous results. Before trusting an AI judge at scale, I cross-check it against my architect cofounder’s judgement on a few dozen results. Human review verifies alignment with your AI judge. Misalignment means you must make adjustments.
Shipping the change
We gradually roll out the change when the results show that it is producing better results and we are confident in its statistical significance.
On a regular cadence, I review failures and create an eval with the aim of improving the product. This iterative, scientific approach leads to faster improvements than blind guesswork. It is this iterative approach, assisted with the speed of AI, that produces a true moat for AI startups.