Categorical Integrity in Non-AI Domains
The citation-index forensic and a channel contrast
1 Introduction
At the documented market’s 2016 endpoint, one article in a top journal could earn its authors up to US$165,000 in cash at some Chinese universities (Quan, Chen, & Shu, 2017). The price was set by a category — the journal’s tier — and the tier was computed from a mean that describes almost none of the articles it prices (Seglen, 1997). A number built to select source journals had become a wage scale. This note asks whether the route from tool to wage scale is the one the series has been mapping.
The third note named academic citation indices, standardized testing, performance review, and scientific peer review as candidate domains (Ortulanu, 2026c §7, §8). This note asks whether reflexive contamination, derived from one financial and one AI case, is visible with forensic specificity in records that share neither substrate.
The domains supply variation the original cases lacked. Journals, schools, and manuscripts differ in use weight, anchor structure, judge lineage, and access to external evidence. That variation can separate a traveling mechanism from a traveling vocabulary.
The primary forensic is the journal impact factor, where the use channel has been surveyed, sanctioned, and debated for decades. Standardized testing then supplies a within-domain contrast between a high-stakes instrument and a lower-stakes external reference. Peer review supplies the intent-dominant boundary. The reform record tests whether changing institutional use can weaken a loop even when the measurement substrate stays in place.
Scope. The analysis reads documented records and time-fixes their quantities. Its comparisons are structural and longitudinal rather than newly estimated.
Methodological commitment. The apparatus enters the cases with its elements fixed: category, reliance, recursive channel, anchor, and external divergence. The cases can then fail the mapping rather than inherit it by description.
2 The Citation Index as a Categorical Benchmark
Garfield and Sher created the journal impact factor to help select source journals for the Science Citation Index (Garfield, 2006). The ratio divides citations received in a year to the preceding two years’ items by the substantive articles and reviews published in those years. Its later uses extended to library decisions and researcher evaluation. Between tool and gate, the object of judgment changed. Its output is a scalar. Nothing in the scalar is categorical, and on the third note’s distinction that is exactly the point: the impact factor is the cleanest available instance of a benchmark that is scalar by output and categorical by use (Ortulanu, 2026c §2).
The categorical conversion is institutional and explicit. Journals are sorted into quartiles — Q1 through Q4 within a subject category — and the quartile, not the decimal, is what evaluation systems consume. Hiring and promotion files count Q1 publications; national assessment exercises and institutional reward schedules rank by quartile; “published in a high-impact journal” functions as a category judgment about the article and, by extension, its authors. The conversion has carried cash denominations: a survey of Chinese university reward schedules found per-paper bonuses pegged to journal tier and indexing, reaching US$165,000 for a single top-journal article at the survey’s 2016 endpoint (Quan, Chen, & Shu, 2017). The reform literature calls this an incentive problem; the price list calls it a market. The audit relation is layered in a way the series’ earlier instances were not: the index audits journals, and institutions then use the journal-level judgment as a proxy audit of researchers. The proxy step was diagnosed inside the domain decades ago — the distribution of citations across a journal’s articles is so skewed that the journal mean represents almost none of them, with the most-cited minority of articles supplying most of the citations (Seglen, 1997) — and the diagnosis changed little. In the first note’s vocabulary, the question “how good is this work?” migrated into “is it in a Q1 journal?”; proxy displacement, operating in plain sight, with the displacement’s invalidity published in the domain’s own flagship venues (Ortulanu, 2026a §2).
The relation carries the institutional elements of the series’ apparatus. A finite judgment set in actual use — the quartile, the high-impact designation. Institutional reliance — third parties acting on the judgment without independent inquiry, the second note’s condition for an audit cycle (Ortulanu, 2026b §6). And a use channel: the next section’s subject.
3 The Recursive Use Channel, Documented
What makes this domain the right primary forensic is that the feedback channel — the benchmark’s judgments entering the production of the population it will judge next — does not have to be inferred from structure. It has been surveyed, sanctioned, and editorialized about for years.
Editors optimize the metric directly. Coercive citation — the practice of editors pressuring authors to add citations to the editor’s journal before acceptance, without indicating the manuscript needs them — was surveyed across economics, sociology, psychology, and business: roughly one in five respondents reported having been coerced, and coercion concentrated in higher-ranked journals (Wilhite & Fong, 2012). Citation stacking among cooperating journals — the cartel pattern — is recognized by the index’s own publisher, which suppresses titles from the rankings each year for citation distortion; the sanction list is the domain’s running ledger of recognized events, an institutional acknowledgment that the channel exists and is used (Clarivate, 2026). Composition is engineered at the journal level: review-heavy front matter to harvest citations, negotiation over which items count in the denominator — practices aired by editors themselves (PLoS Medicine Editors, 2006). None of this requires dishonesty about any article’s content. It is the benchmark’s own arithmetic, optimized by the population it measures.
Authors complete the loop. Venue choice tracks the quartile because careers are paid in it; the population of submissions reshapes toward what high-quartile journals are understood to accept; the journals’ intake, and hence their citation profile, and hence the next year’s quartiles, absorb the reshaping. In the fifth note’s terms the channel weight is high and institutionalized: the benchmark’s judgments enter the update of the journal population, the article population, and the author population at once, openly, on an annual clock (Ortulanu, 2026e §2). The loop is self-feeding in that note’s sense: submissions shaped by the metric become the citation base from which the next metric is computed, the population’s output re-entering as the instrument’s input (Ortulanu, 2026e §4). Gaming is abundant here, and the third note’s qualification carries: the mechanism does not require the gaming, and the gaming accelerates it (Ortulanu, 2026c §2). Coercive citation is intent. The migration of submissions toward the categories, the slow matching of article form to what the metric rewards — that is use, and it could proceed with every editor honest.
This channel belongs to a broader record of ranking reactivity. Law-school rankings have been shown to reorganize resource allocation, admissions, and presentation around the published measure (Espeland & Sauder, 2007). The citation case adds a layered category: journals are ranked, articles inherit the rank, and careers inherit it again. The neighboring literature establishes that rankings remake populations; the present question is whether the series’ discrimination and anchor conditions separate the resulting trajectories.
4 The Markers, Observed
The second note proposed three markers of an audit cycle losing informativeness; the third operationalized them; the fifth gave them an order of assembly under stated assumptions (Ortulanu, 2026b §7; 2026c §5; 2026e §6). Three evidential objects must be kept apart here. Seglen’s skewness result is static invalidity. Coercion, stacking, and composition engineering document the channel. A study comparing 1902–2009 supplies the trajectory: the relationship between journal impact factors and their papers’ citations strengthened through most of the twentieth century and weakened after 1990 (Lozano, Larivière, & Gingras, 2012). The trajectory is real; its attribution to recursive use is not identified by that study.
Marker 2 is partially visible. Agreement on a contracting task requires Aₜ holding or rising while τₜ contracts and Rₜ fails to improve. The citation record supplies the scope component: the quartile judgment is consumed as if it ranked article quality, while the journal-level number weakly constrains the article-level property (Seglen, 1997), and that relation weakened after 1990 (Lozano et al., 2012). It does not supply a longitudinal analogue of inter-evaluator agreement or rerun stability. The mapping is therefore τ-like, not the full marker. Its decadal chronology is consistent with scope loss preceding later recognition, below the resolution the fifth note’s order requires.
Marker 3 is the domain’s strongest exhibit. Divergence between evaluation results and downstream consequence has been measured here with a directness the AI case has not yet permitted. Journal rank is a weak-to-moderate predictor of an article’s eventual citation impact, and on the reliability dimension the sign runs the other way: rank associates with higher retraction rates and with inflated effect sizes — “a moderate to strong predictor of both intentional and unintentional scientific unreliability,” in the assessment of the review (Brembs, Button, & Munafò, 2013). The category that institutions consume as a quality assurance correlates, on the available record, with the failure mode it is consumed to exclude. The association is divergence evidence, not a causal estimate: visibility, scrutiny, and detection effort rise with rank, and the source literature flags the confounders itself (Brembs et al., 2013). What the divergence supports is the narrower reading — the consumed category does not behave as a clean proxy for the article-level property it stands in for. The 2008 analogue is close in structure, though not in tempo or causal closure: the highest category, relied on most heavily, carrying the concentrated risk (Ortulanu, 2026b §6).
Marker 1 has a possible domain-specific phenotype. Its definitional core is loss of discrimination at the relied-upon frontier together with rising cost of restoring an independent distinction. Saturation is one phenotype; inflation can be another only if those two components are separately shown. Here the journal–article citation relation weakens over time (Lozano et al., 2012). Article-level, post-publication assessment supplies an independent check, but its cost trajectory is not measured in the assembled record. The inflation itself is documented and decomposed: lengthening reference lists are its predominant source (Althouse, West, Bergstrom, & Bergstrom, 2008). That analysis measures citation accounting, not article quality. Inflation alone would not count.
The domain also supplies two kinds of recognized event. Annual suppression lists recognize channel manipulation: specific citation patterns have distorted the metric enough to trigger a sanction (Clarivate, 2026). DORA recognizes a different object — the misuse of journal-level categories in individual decisions — and recommends against journal metrics in hiring, promotion, and funding (DORA, 2012). The Leiden Manifesto extends the second recognition into principles for metric use (Hicks, Wouters, Waltman, de Rijcke, & Rafols, 2015). Sanction acknowledges channel abuse; reform acknowledges category–referent detachment. Their coexistence is evidence at two layers, not one event class.
5 The Reform Record, Read Through the Fourth Note
What happened after recognition is more informative than a verdict of reform failure. DORA accumulated thousands of institutional signatories; funders and universities revised assessment language and formally discouraged journal metrics in hiring, promotion, and funding. The intervention targets reliance. In the fifth note’s notation it attempts to lower β, the weight of a benchmark judgment in the next production cycle (Ortulanu, 2026e §2). If implemented to the limit, that change closes the recursive channel even while the citation substrate remains unchanged. Governance can therefore interrupt this mechanism without reconstituting the instrument.
The unresolved question is implementation. Quartiles still gate careers in many systems, stated policy and local practice diverge, and signatory status records a commitment rather than the weight applied in a promotion file. Where use persists, the fourth note’s profile sorts the remaining repairs (Ortulanu, 2026d §3, §5). Article-level citation metrics relocate the category boundary while staying inside the citation graph the population can optimize. Alternative indices and competing databases remain close in lineage. The domain has not deployed at scale a task reference independent of citation and review behavior, with verification access attached. DORA addresses the channel; metric redesign addresses the instrument; neither should be judged as the other.
The distinction yields two predictions. The manipulation check is whether the weight of quartile status in career allocation actually falls. Conditional on that change, the substantive prediction is weaker subsequent optimization of journals toward the index. Institutions that replace one citation metric with another while keeping the career weight fixed should show re-targeting instead. The comparison requires behavior, not policy language. DORA’s uneven adoption supplies the variation for a matched study.
6 A Contrast and a Boundary Case
6.1 High-stakes tests and NAEP
Campbell’s law names the corruption of indicators under decision use; the present mechanism names one channel of it (Campbell, 1976; Ortulanu, 2026c §3). High-stakes state tests placed proficiency bands and progress thresholds inside school rewards. Instruction, scheduling, and student selection reshaped toward the tested form. The benchmark’s judgment entered production with high weight.
NAEP sampled students across many of the same jurisdictions without attaching local rewards to school-level results. Its use channel was weaker and its item ecology sat farther from classroom optimization. The population-level trends supply a within-domain scissors: state scores rose while the lower-stakes audit measure moved much less (Koretz, 2017). The comparison concerns overlapping jurisdictions, not matched tests of the same pupils. Genuine domain-wide learning predicts movement in both. Instrument-specific optimization predicts a widening gap.
The contrast does not isolate the verdict channel. High-stakes tests differ from NAEP in local consequence, item exposure, format predictability, and the intensity of test-specific preparation. Some score inflation can therefore run through exposure and leakage rather than through recursive use of verdicts. NAEP is neither timeless nor physically external, and national curricular attention can reach it. Its advantage is relational: local institutions have less power and less reason to inhabit it. The case supports the fourth note’s graded profile, but the observed gap reflects the bundle rather than identifying the verdict-mediated share.
The Atlanta investigation documented coordinated answer-sheet tampering across dozens of schools (Bowers, Wilson, & Hyde, 2011). Fraud made the incentive channel visible. Lawful teaching to the tested form can combine the same institutional incentive with exposure to recurrent item formats; the record used here does not separate their shares. The scandal was an event. The test contrast locates the bundle.
6.2 Peer review at the boundary
The verdict population — accept, reject, revise — gates careers, and the gate’s pressure travelled back up the channel: fabricated reviewer identities, author-controlled reviews, and organized rings surfaced in retraction waves when publishers audited submission systems (Ferguson, Marcus, & Oransky, 2014). Intent dominates this case. It documents the channel’s value to the population but does not establish the slow erosion observed in citation indexing. Where the index changes through ordinary optimization, the review scam enters by intrusion. The common door does not make the mechanisms identical.
Testing supplies the within-domain contrast. Peer review supplies the boundary. Citation indexing remains the documented-channel forensic with a longitudinal discrimination trace: the channel is observed, the journal–article relation weakens over time, divergence appears in other outcomes, and institutional intervention targets channel weight. The record does not by itself assign the trajectory to the channel.
7 Limitations
The treatments are readings of existing records, and the limits are those of the records.
Causal attribution is weaker here than in the series’ financial case. The 2008 episode ended in an external bill that settled what the categories had concealed; the citation-index case has no terminal event, and its marker-3 evidence — the rank-retraction association — is correlational, with confounders the source literature itself flags: detection effort, scrutiny, and visibility all rise with rank (Brembs et al., 2013). The longitudinal weakening of the journal–article citation relation is also multiply caused; digital access is the mechanism Lozano et al. (2012) advance. The note reads the first as divergence evidence and the second as a trajectory, not either as a causal estimate of recursive erosion.
The domain figures are time-fixed and coarse. The coercion rate is a 2012 survey of four disciplines (Wilhite & Fong, 2012); suppression-list composition changes annually; the post-DORA record is institutional and uneven across countries. None of the quantities is current measurement, and the note’s claims are calibrated to their vintage.
The instance base remains shallow outside the primary forensic. Standardized testing is treated through one synthesis and one investigation; peer review through one episode class; performance review — the third note’s fourth candidate — is not treated at all, for want of a public record of comparable quality. The generality the note supports is conditional on the depth actually supplied: one documented-channel domain with a longitudinal trace, two shallow contrasts, one untouched.
The NAEP contrast partially stress-tests the channel claim inside one domain, but does not supply a fully uninhabitable anchor or identify verdict use separately from exposure. Machine-checked proof and world-adjudicated outcomes remain the stronger negative controls. The closing section turns them into a survey design.
8 Open Questions
Four questions remain.
The Resistant Domains. Mathematics with machine-checked proof and competitive domains adjudicated by the world supply stronger negative controls. The fifth note predicts resistance where the anchor is uninhabitable, and the fourth adds the need for evaluator access to it (Ortulanu, 2026d §3; 2026e §7). A survey of heavily used systems that retain discrimination would test both conditions.
The Comparative Clock. The domains differ in cycle time — annual for the index, seasonal for testing, continuous for AI evaluation — and the fifth note’s model predicts erosion rates ordered by channel weight and influx, not by calendar. Whether the cross-domain record can be put on the model’s common scale, the way the AI saturation timings were sketched onto it, is open (Ortulanu, 2026e §7).
The Reform Test. DORA yields a paired expectation. Material reduction in career use should weaken metric-directed optimization; metric replacement at constant career weight should redirect it. Signatory status is insufficient. Hiring, promotion, and funding behavior must supply the treatment measure.
The Missing Fourth. Performance review in corporate governance was named with the others and left untreated here. Its records are private, which is itself a datum — the third note’s marker-3 protocol requires observable downstream consequences (Ortulanu, 2026c §5).
The non-AI record supplies three different weights of evidence: a documented channel with a longitudinal discrimination trace in citation indexing, a channel–exposure contrast in testing, and an intent-dominant boundary in peer review. DORA adds an intervention whose effect should be read from behavior rather than signatures. Nearly thirty years separate Seglen’s demonstration from this paragraph. The category it indicted still prices careers; the next test is whether institutions that actually stop paying in it change the population it produces.
References
Althouse, B. M., West, J. D., Bergstrom, T., & Bergstrom, C. T. (2008). Differences in impact factor across fields and over time. arXiv:0804.3116.
Bowers, M. J., Wilson, R. E., & Hyde, R. L. (2011). Special investigation into test tampering in Atlanta’s school system. Office of the Governor, State of Georgia.
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience, 7, 291.
Campbell, D. T. (1976). Assessing the impact of planned social change. Occasional Paper Series No. 8. Dartmouth College, Public Affairs Center.
Clarivate. (2026). Journal Citation Reports: Title-suppression notices. Journal Citation Reports support.
DORA. (2012). San Francisco Declaration on Research Assessment. Developed at the Annual Meeting of the American Society for Cell Biology, San Francisco. sfdora.org.
Espeland, W. N., & Sauder, M. (2007). Rankings and reactivity: How public measures recreate social worlds. American Journal of Sociology, 113(1), 1–40. doi:10.1086/517897.
Ferguson, C., Marcus, A., & Oransky, I. (2014). Publishing: The peer-review scam. Nature, 515(7528), 480–482.
Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431.
Koretz, D. (2017). The testing charade: Pretending to make schools better. University of Chicago Press.
Lozano, G. A., Larivière, V., & Gingras, Y. (2012). The weakening relationship between the impact factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology, 63(11), 2140–2145. doi:10.1002/asi.22731.
Ortulanu, G. (2026a). A category-level failure mode not captured by distribution metrics. osservadore. doi:10.5281/zenodo.19627563.
Ortulanu, G. (2026b). The audit asymmetry problem: Capacity conditions for substrate-independent audit. osservadore. doi:10.5281/zenodo.20092930.
Ortulanu, G. (2026c). The reflexive contamination of categorical benchmarks. osservadore. doi:10.5281/zenodo.21252512.
Ortulanu, G. (2026d). Substrate independence as a structural requirement. osservadore. doi:10.5281/zenodo.21802089.
Ortulanu, G. (2026e). Recursive evaluation collapse. osservadore. doi:10.5281/zenodo.22743710.
PLoS Medicine Editors. (2006). The impact factor game. PLoS Medicine, 3(6), e291.
Quan, W., Chen, B., & Shu, F. (2017). Publish or impoverish: An investigation of the monetary reward system of science in China (1999–2016). Aslib Journal of Information Management, 69(5), 486–502. doi:10.1108/AJIM-01-2017-0014.
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ, 314(7079), 498–502.
Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335(6068), 542–543.