From AI Governance to Human Agency: What Should Humans Remain Capable of Doing?

AI governance has become urgent across scholarly publishing. Publishers, universities, libraries, funders, societies, and technology providers are writing policies on privacy, security, intellectual property, transparency, bias, accountability, and research integrity. These responses are necessary. They address only part of the problem.

AI is embedded in how researchers search, interpret evidence, write, review, decide, and communicate. Governance must consider more than what an AI system is permitted to do. It must also consider what people stop doing because the system can do it for them.

This changes the governance question. We must ask where AI should be used, where it should be restricted, and what human cognition should remain active. What should researchers continue to practice? What effort is necessary for expertise to develop? What happens when convenience removes the effort that forms judgment?

These questions extend AI governance into what might be called cognitive governance. Governance determines where AI operates in the research process. Cognitive governance asks how that allocation affects human thinking and capability over time.

The temptation to outsource thinking

Much of the discussion about AI focuses on the mechanics of adoption: prompts, workflows, models, interfaces, automation, productivity, output quality. These questions matter, but they can distract from a more basic one: what should the human mind be doing?

Generative AI can draft, summarize, translate, compare, classify, analyze, code, retrieve information, identify patterns, and synthesize material. It can participate in several stages of a research task: finding the literature, summarizing it, identifying the main arguments, suggesting a research question, outlining, drafting, critiquing, revising. Each step may seem reasonable alone. Together, they can remove much of the intellectual engagement through which a researcher builds understanding.

This is cognitive outsourcing: a tool performing the work through which expertise develops. Researchers have always used tools to extend their capabilities, but scholarly work depends on a further distinction, easy to state and easy to blur in practice: performing a task is not the same as exercising judgment about it.

AI can perform this	This still requires human judgment
Formatting references	Deciding which literature matters
Flagging a statistical inconsistency	Deciding whether it changes the interpretation of a study
Summarizing an argument	Understanding its significance
Matching a manuscript with potential reviewers	Deciding whether those reviewers bring the right intellectual perspective

The concept of cognitive debt makes the same point: outsourcing these capacities before they are exercised can trade a short-term gain in efficiency for a longer-term loss in capability.

The danger grows when output quality becomes the measure of success. A polished answer can conceal weak thinking. A well-written review can conceal that the reviewer never formed an independent assessment. A sophisticated summary can create the appearance of understanding without the experience that produces it.

Have we misunderstood learning?

This raises a more fundamental question about learning.

Formal education has been organized around acquiring, organizing, and applying knowledge. We read, remember, consolidate, and reproduce information, and knowledge becomes the measure of expertise. AI can now store, retrieve, compare, synthesize, and recombine information at a scale no individual can match.

If knowledge is the endpoint of learning, AI is a problem: machines can now perform many of the functions through which knowledge has traditionally been demonstrated.

The more useful distinction is between knowledge and learning. Knowledge is a product of learning. Learning is a process of engagement.

Learning occurs through observation, questioning, experimentation, interpretation, reflection, and experience. A researcher learns a field by engaging with its evidence, disagreements, methods, anomalies, and unresolved questions. A reviewer develops judgment by reading manuscripts carefully and weighing evidence, significance, and limitations. An editor develops judgment by confronting many forms of scholarship and learning to distinguish what matters from what merely appears convincing.

Knowledge can be transferred. Learning requires direct engagement with a subject. AI makes knowledge more available while making that engagement optional. An AI-generated synthesis may accurately account for a body of literature, but it cannot guarantee that the researcher understands why the literature developed as it did, which assumptions remain contested, or where the important questions lie. The researcher may receive the answer without the process that would have produced understanding.

A recent discussion of AI-mediated open science makes the same point: greater access to information does not produce greater understanding. As AI makes discovery and retrieval easier, the ability to interpret evidence, recognize uncertainty, ask meaningful questions, and apply knowledge becomes more important.

Learning in an AI-rich environment cannot simply mean knowing more. It must build the capacity to engage directly with a subject and form judgments about it.

The cost of removing productive effort

There is a difference between unnecessary effort and productive effort.

No researcher benefits from formatting references by hand. Automating that work frees attention for more important work. There is little educational value in repeating a mechanical task once its logic is understood.

Other effort is different. Reading a difficult paper, working through an unfamiliar method, comparing conflicting explanations, testing an assumption, writing an argument, receiving criticism, revising one’s position: these are cognitively demanding. They can be uncomfortable. They are also how expertise develops.

The challenge for AI governance is to distinguish between effort that can be removed and effort that should be preserved.

Cognitive debt is the useful idea here. Capacities develop through use. Discovery develops through engagement with evidence. Discernment develops through repeated comparison and evaluation. Judgment develops through decisions made under incomplete information. If AI performs these activities before a person has exercised them, that person becomes more efficient and less capable.

The risk is greatest for early-career researchers and students. Someone who has spent years learning to formulate a research question, examine evidence, interpret results, and defend a conclusion has built a cognitive framework through practice. Someone who relies on AI for those steps may produce competent work without the same depth of judgment. Delegating the work that builds judgment is most consequential while that judgment is still forming.

The answer is to decide where the effort AI saves should go. If AI removes routine work, reinvest the capacity in deeper inquiry, interpretation, experimentation, and judgment. Otherwise efficiency becomes an end in itself.

When AI becomes a validation machine

Cognitive outsourcing has another dimension that is easy to overlook. People do not always use AI to avoid work. Sometimes they use it to avoid uncertainty.

Scholarly work is full of uncertainty. A researcher may question whether an idea is original. A reviewer may be unsure whether a methodological weakness is serious enough to affect the conclusions. An editor may struggle with a difficult publication decision. A student may doubt whether an argument is sound.

These experiences are uncomfortable, but uncertainty is part of intellectual work. It creates the conditions for questioning, investigation, and revision.

AI can remove that discomfort fast. A researcher presents an idea and receives an immediate assessment. The system can strengthen the argument, supply evidence, produce counterarguments, or simply reassure the user that the idea is sound. The interaction becomes a form of intellectual validation.

This creates what might be called a validation trap. The more readily AI provides confirmation, the less reason a person may feel to sit with uncertainty long enough to investigate it. Instead of asking, “What am I missing?” the user may ask, “Is this good?” Instead of testing an assumption, the user may ask the system to strengthen it. Instead of engaging with disagreement, the user may ask AI to resolve it.

The problem is subtle because the resulting work may improve. The writing may be clearer. The argument may be more coherent. The researcher may feel more confident. But confidence and understanding are different things.

Emotional Fitness is relevant here: the capacity to recognize one’s reactions, stay steady under pressure, and exercise grounded judgment. In knowledge work, it includes the ability to stay with uncertainty without turning discomfort into a verdict on one’s own ability or the quality of an idea.

A researcher who can tolerate uncertainty uses AI differently: to test thinking rather than certify it. They can ask for counterarguments, competing interpretations, weaknesses, missing evidence, alternative explanations, letting AI add friction where it matters instead of removing it.

The question for responsible AI use is simple: are we using AI to examine our thinking, or to make uncertainty disappear?

Human-in-the-loop is not enough

This distinction exposes a limitation in the familiar idea of the “human in the loop.”

A workflow can include a human at the final approval stage while most of the thinking happens elsewhere. If AI summarizes a manuscript, identifies its weaknesses, evaluates their significance, recommends a decision, and the editor simply approves it, the human remains formally in the loop. The judgment happened somewhere else.

What matters is human-in-thought.

Human-in-thought means people stay engaged where interpretation, discernment, and judgment are required. An editor may use AI to identify patterns across thousands of submissions, then examine the cases that matter. A reviewer may use AI to flag statistical issues, then judge whether they affect the study’s validity. An author may use AI to improve language, then keep responsibility for the research question, argument, interpretation, and conclusions.

The question is always: who performed the judgment?

This is a more useful test for governance than asking whether AI was used. An earlier discussion of AI disclosure reaches the same conclusion: as AI assistance becomes routine, whether a tool was used matters less than whether the author remains responsible for the ideas, claims, interpretations, and conclusions.

What cognitive governance would require

Cognitive governance begins with purpose.

The first question is what AI should serve in research and scholarly communication. Earlier work on AI governance set out a sequence: intention precedes purpose, strategy, guardrails, accountability. Skip that sequence, and organizations optimize individual processes while losing sight of what they are for.

The second question is cognitive. Organizations should examine each significant AI-enabled workflow and identify the kind of cognition it involves. Mechanical cognition can often be automated. Discovery, interpretation, discernment, and consequential judgment require more human involvement.

The third question is developmental. What capabilities does the workflow help people build, and which does it let them bypass? This should be part of AI evaluation. An AI tool that saves ten hours a week may look successful. If it also removes the activities that build essential expertise, the organization is creating cognitive debt while reporting productivity gains.

The fourth question is emotional. Why are people using AI? Efficiency is one reason. So are anxiety, fear of making mistakes, pressure to perform, and the desire for reassurance. Governance that ignores these motivations misses part of how AI is actually used.

The fifth question is accountability. When an AI-assisted decision has consequences, responsibility must remain traceable. What did the system do? What did the human do? What remained the human’s responsibility? These questions matter more than a generic declaration that AI was “involved.”

Together, these five questions make human capability part of AI governance.

Question	What it asks
Purpose	What is AI intended to serve in research and scholarly communication?
Cognitive	What kind of cognition does this workflow involve, and how much of it should remain human?
Developmental	What capabilities does the workflow help people build, and which does it let them bypass?
Emotional	Why are people actually using AI here: efficiency, anxiety, uncertainty, or the desire for reassurance?
Accountability	When an AI-assisted decision has consequences, what did the system do, what did the human do, and what remained the human’s responsibility?

Cognitive AI literacy

The research community needs a broader idea of AI literacy, one that includes how AI affects the process of thinking. Researchers need to know which tasks they can delegate, which they should keep doing themselves, which capabilities they are building through practice, and which may weaken through repeated outsourcing.

A cognitively AI-literate researcher can recognize when AI is helping with mechanical work and when it is beginning to shape their judgment. They should know when to seek a synthesis and when to read the underlying literature, and when the process of finding an answer is itself part of learning.

This matters especially in education. Teaching students how to prompt an AI system is useful. Teaching them when not to is more important.

The same principle applies to researchers, reviewers, and editors. Training should include practices that preserve independent judgment: forming an initial view before consulting AI, reading primary evidence rather than relying entirely on summaries, testing AI-generated interpretations against source material, seeking counterarguments, and recognizing when uncertainty should remain unresolved.

Libraries and publishers have a particular role here. Libraries have long supported information literacy. Publishers steward the scholarly record and the processes through which knowledge is evaluated and disseminated. Universities build research capability. Together, these institutions can define what responsible AI use means at each stage of intellectual development.

This extends the argument made in Open Science 2.0: deep reading and sustained engagement are part of the development of expertise, not merely ways of retrieving information.

Governance must protect capability

The implications extend beyond individual researchers to the organizations that shape scholarly work.

Publishers already use AI for submission screening, integrity checks, reviewer identification, language assistance, and workflow management. These applications reduce mechanical work substantially. The governance question is what happens to the human capabilities around those workflows.

An adaptive publisher learns from AI-assisted operations. AI can identify patterns across submissions, reviews, editorial decisions, and production processes, but people must interpret those patterns and decide what they mean. The value lies in converting operational data into organizational learning. The adaptive publisher framework makes this explicit: AI expands institutional memory and surfaces patterns, while editorial leadership remains responsible for interpreting their significance and changing practice.

The same principle applies across the research ecosystem. Publishers, societies, libraries, institutions, funders, researchers, and technology providers each control different parts of the environment. Their policies will differ, but researchers operate across all of them. Fragmented expectations create friction and encourage people to treat governance as compliance rather than a coherent system.

The goal is coordination around a few durable principles: AI should serve research. Mechanical work can be automated where appropriate. Human judgment should remain visible and accountable. Learning should not be displaced. Uncertainty should remain part of intellectual work. The capabilities that underpin scholarship should be deliberately maintained.

The research community has already begun moving toward ecosystem-level AI governance. The next step is to include cognitive capability in that conversation.

The question governance should ask

AI will keep changing what researchers, reviewers, editors, and publishers can accomplish. Tasks that once required significant time and expertise will become easier. That is an opportunity, and a responsibility.

As machines grow more capable of retrieving knowledge, processing information, generating language, identifying patterns, and producing plausible interpretations, human value depends increasingly on capabilities that cannot be reduced to output. The ability to ask a meaningful question, engage directly with evidence, recognize significance, tolerate uncertainty, form an independent judgment, and take responsibility for a conclusion matters more as AI grows more capable.

This gives AI governance a wider horizon.

The question is not how powerful AI will become, or how much work it can automate. It is what we want researchers, students, reviewers, editors, and institutions to remain capable of doing after years of working alongside it.

That is the question cognitive governance must answer.