How to Measure Run-to-Run Variance in AI 3D: Seeds, Reruns and What Vendors Document

What is documented, and what is not

The sources we read say little. Meshy’s comparison post (updated 27 October 2026; it discloses that Meshy is one of the tools compared) lists output consistency across repeated runs as not measured. This page is one of the protocols in the tests hub. We looked for a published run-to-run variance test and found none among the pages we read, which is a limit of our reading, not proof that none exists.

Seeds are the usual way to make a generative model repeatable, and whether you get one depends on the tool and the surface. This is what the reference pages showed in November 2026:

Where a row says none, check the current reference and UI yourself, and record “not exposed” only if it is absent there.

Retries are a separate feature that is easy to confuse with repeatability: Meshy’s pricing page lists free retries per task in the web app on paid plans (4 on Pro, 12 on Premium, 40 on Ultra), and does not explain what a retry does. If it is a reroll inside the same task, it is a way to take more samples, and a reason to log how many you used.

What a seed does not promise

A documented seed is a claim to verify, and it only covers one exact configuration. Four things sit outside it.

• The model version. Runware, an inference vendor documenting its own platform, says two different models given the same seed produce completely different output. Meshy’s ai_model defaults to latest, which currently resolves to Meshy 7.1, so an unpinned call can change under you. Tripo publishes dated snapshots such as v3.1-20260211; send one explicitly.
• Every random stage. Tripo’s text route has three seeds. Our reading of its parameter notes is that a text-route run needs all three fixed, because model_seed promises a repeat only for “the same input” and image_seed controls the image that becomes that input.
• The sampler. Runware says stochastic schedulers add randomness at each step, so even a fixed seed can differ slightly between runs. That page is about its model catalog in general. Whether a given 3D pipeline uses such a step is something to read in its code or measure.
• Hardware and libraries. PyTorch’s documentation says “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms”, and that results may differ between CPU and GPU “even when using identical seeds”. TRELLIS.2 seeds through PyTorch, and Hunyuan3D-2.1’s demo seeds a PyTorch generator. That is a statement about the framework, not a measurement of either model, but it means a local seed is safest treated as valid for one machine and one environment. Log GPU, driver and PyTorch version.

Repeatability is also not archiving. Meshy’s Terms say API output for non-Enterprise customers is deleted three days after generation (its pricing table agrees for Pro, Premium and Ultra but lists 14 days for Studio), and that it may remove web-app output over a storage limit or tied to an inactive account. Keep the raw file of anything you approve, whatever the seed says.

Decide what “the same” means first

Reproduced is a claim with three possible strengths, and each needs a different check.

GeometryOS, in a March 2026 article written from a pipeline vendor’s position, argues a production gate should use a “byte-level or geometry-structure diff tolerance”. That suits automated pipelines. Most solo developers buying credits care about the third row, and should not pay for a seeded workflow to get the first.

What you are measuring

The protocol has two arms, because they answer different questions.

Arm A, unseeded variance. One prompt, five runs, default settings, same tool and pinned model version, same day if possible. Leave seeds unset. Compare the runs to each other. This is what a normal user gets, and it feeds cost planning.

Arm B, seeded reproduction. Only where a seed is exposed. Pick one value, run the same input three times with every seed field, setting and model string fixed, and grade the results against the levels above. On Tripo, run a further pair that keeps model_seed and changes only texture_seed, since its docs describe that as the way to get new textures on the same geometry. If the geometry moves, the documented separation did not hold on your input.

Derived values to compute after the runs: the spread of triangle count (maximum minus minimum, and as a share of the median), the number of runs out of five that pass the gate, and the gap between the best run and the median run.

The decisions it informs

• Pass rate across five. If p of five runs pass the gate, you paid for 5 ÷ p runs per usable asset on that prompt, plus your selection time. That arithmetic belongs in cost per usable asset; this protocol gives you the rate to feed it.
• Best versus median. A big gap means marketing pages and reviews that show the best run overstate typical output. Compare the median run across tools, not the best.
• Style stability. For a set of assets that must look related, layout_diff and score variance tell you whether the tool can hold a look; see keeping a consistent style.
• Whether a seed is worth building around. If Arm B reproduces at the level you need, you can store a seed and model string instead of relying on luck during iteration. If it does not, treat every approved output as unrepeatable and archive it.

Pass or fail

Set the threshold before running: a tool passes a prompt if at least four of five runs pass the class gate and the triangle-count spread stays within a range you declare for your engine budget. Report prompts where it fails rather than averaging them away. Run ten prompts across classes (the standard set has five classes) so a tool that is steady on props but erratic on creatures shows up.

When the runs will not line up

Most surprises trace to something that changed besides the seed.

Limits

Five runs is a small sample. It can reveal a tool that is wildly inconsistent but cannot give a reliable rate. The binomial distribution shows how little it pins down: five passes out of five is compatible with a true pass rate as low as about 55% at 95% one-sided confidence (0.05 raised to the power 1/5), and twenty straight passes only lifts that floor to about 86%. That is arithmetic about sample size, not a result for any tool. Use more runs for a decision that costs real money. Variance can also change after a model release, so retesting is covered in model version changes. Failure and queue behavior is logged separately in the failed runs and queue logging.