The Semantic Reader Project

Making sense of inline citations with personalized context and visual cues.  While most prior work on supporting research paper discovery has focused on developing bespoke interfaces for recommender systems or visualizations based on the citation graph¹⁵ and paper content,⁸ research paper discovery via inline citations in a reading interface is important yet under-explored. One study estimates that reading and exploring inline citations accounts for around one in five research paper discoveries during active research.²¹ However, while all inline citations are relevant to the current research paper, it is likely that some are more relevant to the current reader than others. For example, a reader reading papers about aspect extraction of online product reviews to learn more about natural language processing techniques would be less interested in citations to research papers around e-commerce and marketing. In addition, citations to the same research papers often have different surface forms across papers (that is, reference numbers), making it all the more difficult for readers to keep track of all the inline citations they should explore or have already explored during literature reviews.

Intelligent reading interfaces have the potential to address these issues by augmenting inline citations with personalized visual cues and context to help a reader spot and make sense of relevant work amid a sea of encountered citations. For example, CiteSee⁶ highlights inline citations with different colors to signal to a reader whether a paper has been previously encountered or saved, as well as how it might be relevant to their interests based on their reading history and publication record. Additionally, CiteSee⁶ imbues the Paper Cards with personalized context to explain how cited works relate to the reader, such as citing contexts from other papers that were familiar to the reader (Figure 2). While CiteSee focused on surfacing structured signals (for example, citations between familiar and unfamiliar encountered papers), as the capability of AI methods increases, an exciting future direction is to provide additional personalized explanations based on the content of papers; for example, generating a description of how an encountered citation may be relevant to one of the reader’s publications or how it compares and contrasts with a familiar paper saved in the reader’s library.

Surfacing incoming citations to enable awareness of follow-on work.  While augmenting inline citations can help readers better prioritize the most relevant prior work,⁶ many relevant research papers are not cited in the first place—in particular, follow-on work published after the current paper is not cited. To address this, reading interfaces could bring additional citations into the current paper as annotations, so that readers can become aware of relevant work not cited in the current paper. For example, taking inspiration from social document annotation systems,⁴⁰ CiteRead³³ creates margin notes in the current paper with citation sentences, or citances, from citing papers published afterwards, as a form of commentary on the current paper (Figure 3).

To produce these annotations automatically, reading interfaces could leverage AI to determine which papers have the most relevant commentary to surface, so as to avoid overwhelming the reader with excessive annotation. CiteRead, for example, determines relevant work using a trained model that derives signal from citational discourse and textual similarity, that is, from scientific paper embeddings.⁸ Reading interfaces should also determine how to best localize margin annotation to a specific position in the paper, a particularly challenging task because citations do not typically reference specific locations or passages in a cited paper. To address this, CiteRead determines when it is feasible to localize to particular spans of text in the paper being read, and when to fall back to coarser document units (for example, sections). While CiteRead focused on a paper’s citances as relevant commentary, another exciting direction is to augment the current paper with passages from relevant follow-on work that may fail to cite the current paper or even with broader AI generated commentary.

Guided skimming with faceted highlights.  Scholarly reading is often a sensemaking process involving interleaved foraging for relevant passages and comprehension of found information to integrate it into one’s existing knowledge. Readers spend a significant amount of time foraging for relevant passages of papers. This is particularly true when they skim. A time-pressed reader might attempt to learn more from a paper in less time by skimming its abstract, section headers, text styling, and/or visuals to identify its most significant information. That said, skimming in this way may not connect a reader with the breadth of significant ideas in a paper.

Reading interfaces can help readers encounter more important ideas in a paper, in less time. In the Scim¹⁰ project, we augmented the reading application in a way that helped expose readers to the breadth of important ideas in a paper with faceted (that is, multi-category) highlights. Scim uses a tuned language model with custom post-processing to identify a set of highlights for passages meriting reader attention. These highlights are approximately evenly distributed throughout a paper to encourage examination of major ideas not just in the front matter of the paper, but in all sections. The highlights are tuned to be sparse enough that they can be rapidly reviewed, and dense enough so as to avoid the perception of a tool that “missed” a passage. Furthermore, they are faceted, representing four kinds of major paper findings—research objectives, novel aspects of the research, methodological aspects, and results. Finally, the highlights are controllable: Readers can tune the skimming experience by using controls that alter the density of highlights in the paper as a whole, or within individual passages. This project represents a vision of AI as a helper in foraging for relevant information in a paper. Tools like Scim could be yet more powerful with improved AIs for identifying relevant passages, personalization to the information needs of individual readers, and the ability to point out not just significant textual content, but also significant visual content (for example, important aspects of figures and tables).

Reader-sourced hyperlinks for low-vision navigation support.  The task of navigating between sections and retrieving content can be particularly challenging for blind and low-vision readers due to limitations in auditory information access or small viewports under high magnification.³⁶ A small viewport can make navigation difficult and necessitate scrolling,³¹ which is problematic when the reader needs to jump back and forth to understand the content. In-document hyperlinks, like those in tables of content or inline references, can help but are typically unidirectional and do not exist (for example, jumping between a results section and relevant experimental design). Most existing tools do not address such challenges associated with low-vision and magnification. Reading interfaces might minimize scrolling requirements for low-vision readers by augmenting the paper with new hyperlinks. For example, Ocean³¹ provides bi-directional hyperlinks that enable navigating to and from associated content without disrupting the viewport. These links allow for easy revisiting of portions of the paper with tabbed reading. AI automation may play an important role in scalable creation of links to power such interfaces, though naive application may not yield desired results. For example, Ocean found in an exploratory field-deployment study with mixed-ability groups of low-vision and sighted readers that readers placed greater value in links curated by teammates over crowd or machine generations. In lieu of automation, Ocean includes an authoring interface that allows readers to create and share paper links during reading, thereby enabling readers to build shared interpretations of a paper through collaboration. Of course, as the reliability of AI improves, interfaces can employ it as a means to suggest or even fully automate scalable link creation.

Understanding terms and symbols with on-demand, in situ definitions.  Understanding a paper requires understanding the vocabulary it uses, including acronyms, symbols, and invented terms. However, this is by no means an easy task; a typical paper may contain dozens of such terms. Ideally, readers would be able to summon informative definitions with little effort. In the context of scientific papers, familiar gloss designs work less well at helping readers understand terms. One reason is that terms can have multiple senses in a single paper, so a gloss would ideally need to select the sense that matches the context. Another reason is that a reader may rely on usages of a term, and not just its definitions, to understand its meaning. This is especially the case when terms lack explicit definitions.

Intelligent reading interfaces might bring about more effective glosses for scientific papers by providing context-relevant explanations that provide access to the sum of information about a term. ScholarPhi,¹² for example, favors definitions that appear just before a term’s usage when a term has multiple definitions. Furthermore, ScholarPhi’s glosses consolidate information of all kinds, providing access to all definitions, descriptions, and in-context usages in a compact widget. Finally, to define complex mathematical formulas, ScholarPhi presents its explanations with high economy, showing definitions for all symbols (and nested sub-symbols) at once, automatically placed adjacent to them in the formula’s margins (see Figure 5). Additional advances in AI could lead to even better experiences for understanding terms. For instance, AI could be used to generate definitions of terms and symbols even when no explicit definitions are supplied in the text.²⁶ Furthermore, AI could be used to identify and prioritize which terms to define for users, and how to explain them in approachable terms, should they have an accurate model of what the reader knows.

Simplifying complex papers with passage-level plain language summaries.  Helping a reader understand individual terms and phrases only addresses part of the problem. Papers often contain passages so dense and complex that individual definitions are not enough to help someone read.¹¹ Can AI be brought into reading applications to make such passages understandable? One solution is to incorporate techniques for plain-language summarization to remove jargon and make passages more approachable. Yet, while modern AI can produce plain-language summaries of long texts, it is not clear how these can enhance the reading experience of the original document. For example, simply attaching a plain-language summary to a paper does not help as papers are read non-linearly; such summaries can even detract from reading the original paper.

Intelligent reading interfaces can grant readers access to plain-language summarization for any passage when and where they need them. For example, in Paper Plain,¹ when a reader encounters a difficult section, clicking on a button adjacent to the section header brings up a generated summary of that section in the paper margin (see Figure 6). To help readers who are so overwhelmed that they do not even know where to begin reading, Paper Plain greets them with a sidebar of questions (for example, What did the paper find? or What were the limitations?), links to automatically extracted answering passages, and language-model-generated plain-language summaries of those answering passages. Taken together, these passage-level summaries provide an “index” into the paper’s text, helping readers understand the “gist” of complex passages. For a tool like Paper Plain to see widespread use, several issues in AI need to be addressed. First, how can simplifications be generated without hallucinations, so that a reader can be confident that a simplification accurately reflects the findings of the underlying paper? This is particularly important for domains such as biomedicine, where an inaccurate simplification could lead to a reader making dangerous health decisions. Second, how can AI be optimized to support rapid, on-demand simplification of any passage, large or small, that a reader selects?

Fusing papers and presentation videos to create engaging multimodal experiences.  Sometimes, the best explanation of an idea is non-textual. For example, an algorithm might be better explained through an animation, and a user interface might be better showcased through a screen recording, as opposed to the prose of a paper. Instead of consuming the two formats independently, could interactive reading interfaces offer readers access to these alternative, more powerful descriptive forms as they read? One approach is for reading interfaces to align external media with the paper text and allow reader traversal in one medium to automatically trigger traversal in the other. For example, in Papeo,²⁰ paper passages are linked to excerpts of talk videos, and readers skimming through the paper will see corresponding jumps in the video (and vice versa). Unlike text-skimming with Scim and Paper Plain, video-skimming in Papeo combines multiple modalities to explain complex information. For example, instead of reading a long text description of a complex, dynamic system, readers can see the system’s behavior in a video recording or animation along with the author’s commentary. As observed in Papeo’s studies, readers can use these interactions to fluidly transition between watching video and reading text, using video to quickly get an overview and then selectively descend into the text when they desire a more detailed understanding of the paper. Finally, AI can be an effective means to automate alignment between paper passages and video excerpts; in fact, Papeo developed an AI-supported authoring interface in which these links are derived using a pretrained language model and surfaced as suggestions for an author to interactively confirm or refine.

Collecting and curating research threads by clipping and synthesizing across papers.  Saving clips and organizing them is one common approach to supporting synthesis across multiple documents. This is especially important during literature review, in which scholars often save clips from related work sections that organize and summarize different relevant papers. Prior work has pointed to the importance of tightly integrating clipping and synthesis support in the reading process, and how incurring significant context-switching costs can be detrimental to sense-making. Therefore, recent work has developed tools aimed at reducing the cognitive and interaction costs of clipping and triaging information ²³ to support everyday online researchers. However, designing clipping and synthesis support tools for research papers is relatively under-explored and introduces exciting new research opportunities.

Better clipping and synthesis support has the potential to lower interaction and cognitive costs, as well as improve awareness and discovery during literature review (for example, to help readers form their view of the research landscape). Systems such as Threddy¹⁷ and Synergi¹⁹ allow readers to clip sentences from different papers and organize them into a hierarchy of “threads.” These reading interfaces maintain rich context for each clipped snippet, keeping track of its provenance and resolving any inline citations that appear in the snippet to their corresponding papers. Based on the text of the clips and their inline citations, Synergi¹⁹ traverses thousands of neighboring papers in the citation graph and generates summaries of relevant research threads (and their associated papers) to expand the readers’ exploration and coverage. As AI capabilities improve, there is a corresponding opportunity for human-AI interaction research to develop novel systems and interactions that can better support complex synthesis tasks by allowing readers to express their research interests via natural language (for example, clipped sentences) and to generate richer summaries over larger collections of documents based on reader interests.

Understanding research landscapes by reading and exploring related work sections across papers.  In contrast to providing synthesis support for individual papers, a complementary approach allows readers to directly search and extract related work passages across many papers. The intuition here is that while any single related work section may be missing important citations relevant to a reader’s research topic, piecing together related work passages across many papers in a single reading interface can provide readers with a rich and comprehensive overview of a research topic. However, unlike reading full papers, reading a collection of extracted passages presents novel research challenges due to their lack of navigational structures (for example, sections). Further, related work passages relevant to the same research topic often cite overlapping prior work, making them hard to explore and read while keeping track of which papers are new versus already explored.

Intelligent reading interfaces can empower scholars to efficiently explore research landscapes by directly supporting this novel reading process over extracted related work passages. For example, systems such as Relatedly²⁹ provide readers with the ability to organize passages into meaningful topics and subtopics by augmenting each passage with language-model-generated descriptive titles and organizing them using a diversity-based ranking algorithm that highlights different research threads. For example, when exploring related work paragraphs about “misinformation”, the first few passages returned by Relatedly may be titled “Fact Checking Datasets”, “Social Media and Misinformation”, and “Fake News Detection Techniques” as opposed to highly similar passages all titled “Related Work” or “Background.” Additionally, when scholars explore inline citations across many passages, Relatedly provides cross-referencing support by keeping track of passages and references visited by the reader which, in turn, allows the reading interface to dynamically re-rank passages to promote unexplored threads.