Artifact Withholding
Code, data, or environment deliberately kept unavailable — scrutiny avoided, moat preserved
Competitive advantage, reduced embarrassment from failed replication, and lower immediate scrutiny — while claims remain in the literature and in production.
Artifact withholding is a dark pattern: teams successfully keep code, data, model weights, or environment specifications private or incomplete while still publishing or deploying claims that depend on those artifacts — “available upon request,” “proprietary,” “too large to share,” or simply absent.
The pattern works in the short term: competitors cannot easily replicate or audit; failed replications are harder to execute; reviewers and regulators see the claim without the chain. It pairs with publication over provenance under misaligned incentives — sharing pays reputational and competitive costs; withholding pays until the field catches up or harm surfaces.
The pattern
Claims enter the permanent literature or product surface; artifacts stay behind NDAs, broken links, or “internal only” buckets. Requesters face friction, delay, or silence. When code is released, it is often insufficient to reproduce reported numbers — wrong commit, missing preprocessing, undisclosed leakage (Kapoor & Narayanan, 2022).
The claimed effect
- IP and first-mover protection
- Avoidance of public failed replication
- Faster publication without packaging cost
- Regulatory and partner audits that stop at documentation, not reruns
In software systems
Production models served via API only; training pipelines “trade secret.” Evaluation harnesses not shipped with releases. “Reproducibility” means a static PDF supplement, not a pinned container and dataset hash in a registry.
In human organisations
Labs treat datasets as career assets. Industry teams block external audit of training data for liability reasons. “We’ll open-source after publication” becomes never — provenance deferred indefinitely.
In socio-technical systems
Closed benchmarks and leaderboards reward participants who withhold preprocessing tricks. Vendor “accuracy” claims without reproducible evaluation harness. The socio-technical norm accepts performance marketing as science until independent groups invest heroic effort to replicate.
What it violates
Values: Truth and accountability — others cannot verify or contest. Coherence — public epistemic status exceeds private evidence access.
Principles: Document before apply and contestability — meaningful challenge requires artifact access.
How to protect the organisation
Bind claims to artifact commits. No benchmark number in a paper or press release without registry refs — same discipline as Event Sourced Science.
Replication as first-class review. Fund and credit independent reruns; treat withholding as negative evidence in track record.
Honest alternatives: Open run IDs and pinned environments in epistemic harness; experiment architecture with pre-registered data and code availability before results are inspected.
Related architecture
Withholding makes ruptured provenance chain inevitable for outsiders; combined with non-propagating refutation when claims eventually fail.