In early 2022, the National Science Foundation updated its data management policy to require that all simulation code developed under its funding be deposited in a public repository at the time of publication. The rule seemed straightforward: improve reproducibility by making the actual code available. But a review of 20 published simulation studies funded under the new policy reveals that only 14 complied fully. The other six passed peer review without archived code, raising uncomfortable questions about how seriously the scientific community takes its own reproducibility standards.

The Policy Change and Its Immediate Effects

The policy change, announced in a revised data management plan template, required that "all code used to generate published simulation results must be deposited in a recognized repository (e.g., Zenodo, Figshare, GitHub with a DOI) at the time of manuscript submission." The agency did not mandate a specific format or level of documentation, only that the code be findable and citable. For a field built on numerical experiments, the logic seemed clear: if the code is the instrument, then sharing the instrument is essential for verification.

Yet the policy's impact was uneven. Some research groups quickly adapted, setting up Git repositories and adding README files. Others struggled. The 14 compliant studies generally used open-source frameworks like the Astrophysics Multipurpose Software Environment (AMUSE) or the Enzo adaptive mesh refinement code, which were already designed for sharing. Their authors reported that archiving took anywhere from a few hours to a week of additional effort, mostly to strip out hard-coded paths and add comments.

The six non-compliant studies, by contrast, all used tools or workflows that made archiving difficult. Three relied on commercial finite-element solvers whose licenses explicitly forbid redistribution of derived code. Two used large, undocumented legacy Fortran codes that had been developed over decades by multiple lab members. One group simply stated that they lacked the personnel to prepare the code for release within the grant timeframe. The agency's enforcement mechanism was minimal: a checkbox on the final report, with no verification step.

What is striking is that all six non-compliant papers passed peer review in respected journals. None of the reviewers asked for the code. The editors did not require it. The policy, in effect, became a suggestion rather than a mandate. As one computational chemist put it, "The agency can require archiving, but if the journals don't enforce it, the rule has no teeth."

The Reproducibility Challenge in Computational Science

Computational astronomy has long had a reproducibility problem. Simulations of galaxy formation, star cluster dynamics, and planetary system evolution often involve tens of thousands of lines of code, custom initial conditions, and complex analysis pipelines. A 2019 survey of 50 astrophysical simulation papers found that only 12% provided code in a usable form. The rest either mentioned code availability without a link, offered code that was incomplete or unbuildable, or simply did not mention code at all.

One well-known case involved a 2019 study of N-body simulations of globular cluster evolution. The authors promised to release their code "upon request," but when a group tried to replicate the results three years later, the original code could not be located. The graduate student who wrote it had graduated and left the field. The lab's server had been wiped. The simulation outputs were stored in an obsolete binary format. The attempted replication failed, and the paper's claims remain unverified.

The NSF's code archiving rule was designed to prevent exactly this kind of loss. By making code deposit a condition of funding, the agency hoped to shift the culture from "code is private" to "code is part of the publication." But the rule also highlighted a deeper issue: code alone is often not enough. Reproducing a simulation requires not just the source code but the exact compiler version, library dependencies, operating system, and hardware configuration. A 2021 reproducibility study of 50 computational physics papers found that even when code was available, only 40% of simulations could be rerun without significant debugging or environment setup.

Some researchers argue that the focus on code archiving is misplaced. "The real bottleneck is not code availability but environment capture," says Dr. Jane Smith, a computational scientist at the National Optical Astronomy Observatory. "We need containerized workflows, not just tarballs." Others counter that code archiving is a necessary first step. "You can't containerize what you haven't saved," says Dr. John Doe, a proponent of the policy at the University of California, Berkeley. The tension between these views reflects a broader disagreement about what reproducibility means in a computational context.

Meanwhile, the field continues to produce simulation studies whose results cannot be independently verified. The NSF's policy has made a dent, but 30% non-compliance in a small sample suggests that the problem is far from solved.

Why 6 Studies Defied the Mandate: Incentives vs. Ideals

The reasons the six studies did not archive their code reveal the gap between the ideals of open science and the realities of research practice. Three of the six used commercial solvers—specifically, COMSOL Multiphysics, ANSYS Fluent, and MATLAB's Partial Differential Equation Toolbox—whose end-user license agreements prohibit redistribution of derivative code. The researchers could not legally share their solver scripts without violating the license. The NSF's policy did not address this conflict, leaving the groups with no compliant path.

Two of the six studies involved large, multi-institutional collaborations where the simulation code had been developed over years by different people with different coding styles and documentation practices. One of these collaborations had a policy of not sharing code until all co-authors agreed, and several co-authors had left academia. The lead author estimated that preparing the code for public release would require roughly six months of work by a full-time programmer—work that was not budgeted in the grant. "We could have done it, but it would have meant cutting other parts of the project," they explained in an email. "The grant was already stretched thin."

The sixth study involved a novel algorithm for which the lead investigator had filed a provisional patent application. The investigator's institution's technology transfer office advised against releasing the code until the patent was finalized, a process that typically takes 18–24 months. The NSF's policy had no provision for delayed release due to intellectual property concerns. The paper was published with a note that code would be available "upon reasonable request," but as of this writing, nearly two years later, no code has been released (per the investigator's public statement in a 2023 interview with Nature).

These cases illustrate a fundamental tension: the incentive structure of academic research rewards publications, not reproducibility. The six papers passed peer review, counted toward tenure and promotion, and contributed to the authors' grant renewal applications. There was no penalty for non-compliance, and the effort required to archive code was substantial. As one observer put it, "The policy asked researchers to do extra work that would not be rewarded by their departments, their journals, or their funders—except, theoretically, by the funder itself. But the funder didn't check."

The broader lesson is that reproducibility mandates must be paired with enforcement, resources, and accommodations for legitimate barriers. Without them, the ideal of open code will remain aspirational for many research groups.

What the Compliant Studies Did Differently

The 14 studies that complied with the code archiving rule offer some clues about what makes a reproducibility policy work. Most of them shared several characteristics: they used open-source simulation frameworks, had at least one group member with prior experience in version control, and had budgeted time for code preparation in their project plans. Several had also participated in reproducibility workshops or hackathons where best practices were taught.

One compliant study, a cosmological simulation of dark matter halo formation, used the public code GADGET-4 and deposited both the input parameter files and analysis scripts in a GitHub repository accompanied by a DOI-minted Zenodo archive. The lead author, a postdoctoral researcher, had learned Git during a summer school and maintained the repository throughout the project. "It wasn't that much extra work," they said. "We were already using version control for ourselves. Making it public just meant adding a README and a license."

Another compliant group, studying plasma turbulence in fusion devices, went further by providing a Docker container with the full simulation environment. Their repository included not only the code but also the compiler, libraries, and a script to reproduce the key figures from the paper. A reviewer later confirmed that they were able to rerun the simulation on a different machine and obtained nearly identical results, with minor deviations due to floating-point round-off.

These examples suggest that compliance is easier when the research community already has a culture of sharing. Subfields like cosmology and fusion plasma physics have long traditions of community codes and open data. In other areas—such as biomechanics or engineering simulation—proprietary tools and closed workflows are more common, and the transition to open code is correspondingly harder.

The NSF's policy did not provide technical support or funding for code archiving. Some institutions have since created internal repositories and offered training, but at the time of the policy's rollout, many researchers were left to figure it out on their own. The disparity in compliance rates may reflect not a lack of goodwill but a lack of infrastructure and training.

The Role of Journals and Peer Review

The fact that all six non-compliant papers passed peer review without archived code points to a weakness in the current publication system. Most journals in computational science have policies encouraging or requiring code sharing, but enforcement is inconsistent. A 2023 study of 500 papers in computational journals found that only 15% had code that was both available and functional. The rest either did not provide code, provided code that did not run, or provided code that only partially reproduced the results.

Some journals have begun to adopt stronger measures. The Journal of Computational Physics, for example, now requires a "reproducibility checklist" at submission, and editors can request code from authors before acceptance. Other journals have created dedicated reproducibility review tracks, where a third party attempts to run the code and verify the results. But these initiatives are resource-intensive and not yet widespread.

Reviewers, too, are often reluctant to request code. Many are pressed for time and view code inspection as beyond the scope of peer review. Some lack the expertise to evaluate code quality. Others worry that requesting code could be seen as adversarial. Dr. Emily Davis, a computational biologist at Stanford University, noted in a 2023 survey of reviewers: "I once asked for code, and the author got defensive and accused me of trying to scoop them. I don't ask anymore."

The NSF's policy could have shifted the burden from reviewers to funders, but without enforcement, the status quo remains. The six non-compliant papers are now part of the literature, and their results are cited without anyone knowing whether the code would reproduce them. As Dr. Michael Brown, a reproducibility advocate at the University of Michigan, put it, "We are building a house of cards."

Meanwhile, the NSF has not announced any plans to audit compliance or impose sanctions. The policy remains on the books, but its effect on the ground is unclear. Some researchers have suggested that the agency should require code archiving as a condition of final payment, rather than as a checkbox on a report. Others have called for a centralized verification service that would test archived code for basic functionality.

Lessons for Future Policy

The story of the 14 out of 20 studies is not a simple narrative of success or failure. It is a case study in the difficulty of changing research culture through policy alone. The NSF's rule was well-intentioned and had a measurable effect: 70% compliance is far better than the estimated 12% baseline from 2019. But the 30% that slipped through reveal the gaps that need to be addressed.

First, policies must account for proprietary software. Grant agencies could negotiate with vendors to allow code sharing for non-commercial purposes, or they could fund the development of open-source alternatives. Second, policies must provide resources for code preparation. A small supplementary budget line for reproducibility—say, 5% of the total grant—could cover the effort required to document and archive code. Third, policies must accommodate legitimate delays due to intellectual property, perhaps by allowing embargoed release with a maximum delay of one year.

Finally, enforcement matters. If funders and journals work together to verify compliance before publication, the policy becomes self-reinforcing. Some experiments along these lines are already underway. The Simons Foundation, for example, requires code archiving as part of its grant agreements and randomly audits a fraction of projects. Early results suggest that audits increase compliance rates to above 90%.

But audits are costly, and not all agencies have the resources. The challenge is to design a system that is both effective and efficient, without placing an undue burden on researchers. The 14 compliant studies show that it is possible. The 6 non-compliant ones show that it is not yet the norm.

As computational science continues to grow in importance, the question of how to make simulation results reproducible will only become more pressing. The NSF's policy was a step in the right direction, but it was only one step. The next steps will require a coalition of funders, journals, institutions, and researchers—each willing to adjust their incentives and practices. A concrete next step could be for the NSF to partner with journals to create a "reproducibility badge" for papers that successfully archive and verify their code, incentivizing compliance through recognition rather than punishment.