Augmenting Scientific Creativity with an Analogical Search Engine Augmenting Scientific Creativity with an Analogical Search Engine

2.1.1 Overview of Modeling. In the first stage of the system, purpose and mechanism tokens are identified from article abstracts (Figure 2, ①). Research article abstracts often include descriptions of the most important purpose or the core problem addressed in an article and the proposed mechanism or the approach taken to address the problem, making them good candidates for identification and extraction of tokens corresponding to them. For example, for a similar problem of facilitating heat transfer, Article A may propose an approach that modifies the structure of the material used at the interface between crystalline silicon (semiconductor material) and the substrate, while Article B may propose a more distant mechanism (due to the mismatch on scale) of fin-based heat sinks commonly used for electronic devices. The goal of this first stage is to automatically identify and extract tokens that correspond to the similar purpose (e.g., “facilitate heat transfer”) as well as the mechanisms (e.g., “modifying the structure of the material used at the interface between crystalline silicon” vs. “fin-based heat sinks”) from the abstracts A and B.

One relevant automated approach for identifying purposes and mechanisms from scientific abstracts is DISA [63], which formulates the task as supervised sentence classification. However, we found that many key sentences in abstracts include both purpose and mechanism, breaking the assumptions of a sentence-level classifier (e.g., “In this article, [a wavelet transforms-based method] for [filtering noise from images] is presented.”). To overcome this limitation we follow [62] and frame purpose and mechanism identification as a sequence-to-sequence (Seq2Seq) learning task [5, 101] and develop deep neural networks with inductive biases capable of learning token-level patterns in the training dataset. Our dataset consists of crowdsourced annotations from Chan et al. (the dataset is constructed via application of [21] to a larger corpus of around 2,000 article abstracts largely in computer science domains) (Table 1). We train the models to classify input features (tokens or spans of tokens) as either purpose (PP), mechanism (MN) or neither.

We train two deep neural networks (Models 1 and 2), achieving increasing accuracy of classification. The first model is based on a Bi-directional LSTM (BiLSTM) architecture for sequence tagging [56, 64], in which the forward (the beginning of the sequence to the end) and the backward passes condition each token position in the text with its left and right context, respectively. A main source of improvement of Model 2 over Model 1 is the ability to more selectively attend to informative tokens in a sentence rather than treating each token in a sequence as independent of each other (as a hypothetical example, an extremely effective model based on this approach may assign more weights to the tokens “selectively attend to informative tokens”, as they represent the core mechanism described in the previous sentence) and to leverage the regularities of co-occurrence with surrounding words through the self-attention mechanism [108].

2.1.2 Seq2Seq Model Implementation Details. We implement the BiLSTM architecture of Model 1 in PyTorch [87]. We use pre-trained GloVe [88] word embeddings with 300 dimensions, consistent with prior work [11, 78, 88] to represent each token in the sequence as 300-dimensional input vectors for the model. We train the model with a cross entropy loss objective for per-token classification in the three (PP, MN, Neither) token classes.

For Model 2, we adapt the SpanRel [67] architecture and implement it on AllenNLP [41]. We implement a self attention mechanism that tunes weights for the core word in each span as well as the boundary words that distinguish the context of use, consistent with [79]. We use the pre-trained ELMo 5.5B [90] embeddings for token representation following the near state-of-the-art performance reported in [67] on the scientific Wet Lab Protocol dataset. We train the model using a similar procedure as Model 1. We leave detailed training parameters for Models 1 and 2 to the Appendix.

2.1.3 Introducing Human-in-the-loop Filtering for Model 1. The final classification performance (F1-scores) of Model 1 on the validation set is 0.509 (Purpose), 0.497 (Mechanism), and 0.801 (neither). We found that the limited accuracy contributed to how the system retrieves irrelevant search results. Because reactions to obviously irrelevant results are not useful, we added a human-in-the-loop [31] filtering stage. The filtering proceeded as follows: members from the research team inputted problem queries received from study participants into the system. Once the model produced matches, they went over from the top of the sorted list and removed only those that are irrelevant to the problem context. They continued filtering until at least 30 articles with reasonable purpose similarity were collected. After Winsorizing at top and bottom 10% [115], the human filterers reviewed 45 articles per query (SD: 27.6, min: 6, max: 138) for 5 queries (SD: 2.4, min: 2, max: 9) to collect 33 (SD: 3.5, min: 30, max: 40) purpose-similar articles (about 12/45 = 26% error rate). In Study 1 we show that the limited retrieval accuracy of Model 1 is sufficient for use as a probe with this additional human-in-the-loop filtering. In Study 2 and case studies, we demonstrate how this filtering can be removed with Model 2 while achieving a similar accuracy.

2.1.4 Scaling Model Inference. In order to have sufficient coverage to return diverse results, we collected an initial corpus of 2.8 million research articles from Springer Nature.² After deduplication (based on Digital Object Identifier using BigQuery³) and filtering only articles with at least 50 characters in the abstract we were left with 1.7 million articles in four subjects (Table 2). We stored the resulting corpus in Google Cloud storage buckets.⁴ To scale the classification of the Seq2Seq models we used the Apache Beam API⁵ on Google Cloud Dataflow⁶ to parallelize the operation.

2.2.1 Overview. In the second stage, the identified purpose texts are incorporated into the system to enable search-by-analogy of articles that solve similar problems using different mechanisms, at an interactive speed (Figure 2, ②). Relevant previous approaches include Hope et al. [61] which first clusters similar purposes (through k-means with pruning) and subsequently samples within each cluster of similar purposes to maximize the diversity of mechanisms (via a GMM approximation algorithm [92]), or [62] which employs similarity metrics to balance the similarity to a purpose query and the distance to a mechanism query (and vice versa). In contrast, from pilot tests in our corpus we discovered that even close purpose matches of scientific articles already had high variance in terms of the mechanisms they propose. We hypothesize that this may be the case due to the enormous span of possible research topics and the relative sparseness of their coverage in our corpus, and/or due to the emphasis on novelty in scientific research that discourages future articles which might contribute relatively small variations to an existing mechanism. We leave exploration of these hypotheses for future work and simplify our sampling of the scientific articles to the one based solely on the similarity of purpose, sufficient for ensuring diversity.

In order to support fast retrieval (e.g., sub-second response time) of articles with similar purposes at scale (e.g., millions of articles), we pre-train Spotify's Annoy ⁷ indices of nearest neighboring purposes. Annoy trains a neural network to assign an embedding vector corresponding to a purpose to an index in the high-dimensional space that brings it close to other indices of purpose vectors that have similar meaning (see Section 2.2.3 for details of the metric used for the similarity of meaning). Annoy uses random projection and tree-building (see [1, 2]) to create read-only, file-based indices. Because it decouples creation of the static index files from lookup, it enables efficient and flexible search by utilizing many parallel processes to quickly load and map indices into memory.

2.2.2 Interactive Speed. Additionally Annoy minimizes its memory footprint in the process. This efficiency, critical for real-time applications such as ours, was further validated during our test of the end-to-end latency on the Web, with the average response taking 2.4 s (SD = 0.56 s).⁸ The level of latency we observed was sufficiently low to enable interactive search by end users (both human-in-the-loop filterers in Study One and researcher participants in case studies).

2.2.3 Implementation Details. To construct the similarity space, we first encode the purpose texts into high-dimensional embedding vectors which then can be used to compute pairwise semantic similarity. Here, the choice of an encoding algorithm depends on three main constraints. First, the pairwise similarity, when computed, should correlate well with the human-judged semantic similarity between the purposes. Second, similarity calculation between varying lengths of texts should be possible because extracted purposes can differ in length. Third, computationally efficient methods are preferred for scaling. To meet these requirements, we chose Universal Sentence Encoder (USE)⁹ to encode purposes into fixed 512-dimensional vectors. USE trains a transformer architecture [108] on a large corpus of both unsupervised (e.g., Wikipedia) and supervised (e.g., Stanford Natural Language Inference dataset [13]) data to produce a neural network that can encode text into vectors that meaningfully correlate with human judgment (e.g., evaluated on the semantic textual similarity benchmark [19]). USE can handle texts of varying lengths (e.g., from short phrases to sentences to paragraphs), and with high efficiency [20], thereby making it suitable for our system.

We pre-compute pairwise similarity of the purpose embeddings and store the indices in neighborhoods of high similarity for fast retrieval of similar purposes. As mentioned before, we train the Annoy indices on Google Cloud AI Platform.¹⁰ We use 1—the Euclidean distance of normalized vectors (i.e., given two vectors $\mathbf {u}$ and $\mathbf {v}$, $\text{distance}(\mathbf {u}, \mathbf {v}) = \sqrt {\left(2\left(1 - \text{cos}\left(\mathbf {u}, \mathbf {v}\right)\right)\right)}$) as a similarity metric (using a Euclidean distance based metric for nearest neighbor clustering shows good performance, see [4] for a related discussion on the impact of the distance metric on the retrieval performance). We set the hyper-parameter k specifying the number of trees in the forest to 100 (larger k’s result in more accurate results but also decreases performance; see [2] for further details). Empirically, 100 seemed to strike a good balance between the precision-performance tradeoff, thus we did not experiment with this parameter further.

3.2.1 Participants. We recruited eight graduate (four women) researchers in the fields of sciences and engineering via email advertisement at a private R1 U.S. institution. Four were senior PhD students (3rd year or above and one recently defended their thesis) and the rest was 2nd year or below. Disciplinary backgrounds of the participants included: Mechanical (3), Biomedical (2), Environmental (1), Civil (1), and Chemical Engineering (1). Once a participant signed up for the study, we asked them to describe their research problems and send the research team search queries they use to look for inspirations on popular search engines such as Google Scholar.¹¹ Members of the research team screened articles with relevant purposes using these queries on the filtering interface (Figure 4, left). Despite our efforts to collect articles over diverse topical areas, the search engine did not contain enough articles for two of the participants who work on relatively novel fields (e.g., “machine learning methods of 3D bioprinting”). These participants were interviewed on their current practices for reviewing prior works and coming up with new ideas for research and were not included in the subsequent analyses.

3.2.2 Study Procedure and Keyword-search Control. The rest of the participants were then invited to in-person interviews. To ensure that participants would be exposed to a sufficiently diverse set of analogical mechanisms and to maximize our power to observe the ideation process, we generated a list of top 30 results from the analogical search engine using the search queries provided by the study participants. As a control condition we also included top 15 results from a keyword-based search engine using the standard Okapi BM25 algorithm [82] ($k_1 = 1.2, b = 0.75$) using the same search queries as the analogical search engine. The order of results in the list was randomized and participants were blind to condition. To account for the difference in the quantity of exposure in the analysis, we normalized the ideation outcomes by the number of results returned in each condition. Using this list we employed a think-aloud protocol [80, 107] in which participants were presented with the title, abstract, and other metadata of articles and asked to think aloud as they read through them with the goal of generating ideas useful for their research using our Web-based interface (Figure 4, right). Although time consuming, this approach allowed us to capture rich data on participants’ thought process and how those processes changed and evolved as participants considered how an article might relate to their own research problems. In addition, we asked the participants to make a judgment on the novelty of each article on a 3-point Likert-scale. After participants finished reviewing the 45 articles, we interviewed them about their overall thoughts on the results’ relevance and novelty and whether there were any surprising or unique results. Each interview lasted about one and a half hours and the participants were compensated $15/hr for their participation.

3.2.3 Data and Coding. In total, our data consisted of 267 article recommendations for six participants and their Likert-scale questionnaire responses measuring the content novelty, after removing 3 within-condition duplicates (these articles included cosmetic changes such as different capitalization in the title or abstract). One participant ran out of time towards the end of the interview and only provided novelty measures for the last 17 article recommendations in the randomized list. Thus, 250 transcripts of participants’ think-aloud ideation after reading each article were used for analyzing ideation outcomes. To code the distance between the Creative Adaptation and Direct Application types of ideation outcomes, the coders took into consideration (1) the verbs used to describe the ideas (e.g., “design”, “develop”, or “invent” were generally associated more with distant ideas compared to “apply”, “use”, “adopt”; see Table 3); (2) the context of ideas such as participants’ expression of unfamiliarity or uncertainty of the domain involved (e.g., “I'm not really sure” vs. “I'm familiar with this domain”); and (3) participants’ perceived immediacy of the idea's applicability (i.e., ideas perceived by participants as more immediately applicable were associated with direct application but not creative adaptation ideas). Two of the authors coded a fraction of the data together (13/250, 5.2%) and then independently coded the rest blind-to-condition, using the four ideation outcomes types described in Section 3.1 and with the following protocol: The coders first judged the existence of an idea. If there was, then its type was further distinguished between Creative Adaptation and Direct Application using the linguistic and contextual descriptions described above (e.g., Creative Adaptation ideas were more frequently associated with the “design” words, higher unfamiliarity and uncertainty of the domains, and less immediate applicability, compared to Direct Application ideas). In case there was no concrete idea in the data, coders further distinguished between the Background vs. None cases.

The agreement between coders was significant, with Cohen's $\kappa = 0.89$ (near perfect agreement) for the four categories of ideation outcome. Given the high level of agreement between the coders, any disagreements were resolved via discussion on a case-by-case basis.

3.2.4 Apparatus 1: The Human-in-the-loop Filtering Interface. In Study 1, members of the research team first received search queries from study participants and reviewed the model-produced purpose matches to filter irrelevant articles using a filtering interface (Figure 4, left). This additional step was introduced to ensure that articles with obviously dissimilar purposes are not returned to study participants. Reviewers determined whether each article contained a clearly irrelevant purpose in which case it was removed by clicking the Dissimilar button at the bottom of the article. On the other hand when the Similar button was clicked it turned the background of the article green in the interface and increased the number of the articles collected so far. Reviewers continued the screening process until at least 30 articles with reasonable purpose similarity were collected.

3.2.5 Apparatus 2: The Ideation Task Interface. The filtered articles were then displayed as a randomized list of articles to study participants (Figure 4, right). In addition to the content and metadata of articles (e.g., authors, publication date, venue), each article was presented with a Likert-scale question for measuring content novelty and a text input for ideation.

3.2.6 Limitations. To reduce potential biases, our coders were blind to experimental conditions and relied on participants’ statements of ideas’ novelty and usefulness (e.g., “I've never seen something like this before,” “this is not a domain I would've searched if I used Google Scholar”), and achieved a high inter-rater reliability. We believe coders had a reasonable understanding of how participants arrived at specific ideas from descriptions of their current and past research topics, think-alouds, and end-of-experiment discussions. Despite this, we also acknowledge the limitations of this approach and discuss how future research may improve upon it (see Section 7.2.1).

3.2.7 On Reporting the Results. We report the result of our studies below. To denote statistical significance we use the following notations: $^{*} (\alpha = 0.05)$, $^{**} (\alpha = 0.01)$, $^{***} (\alpha = 0.001)$, $^{****} (\alpha = 0.0001)$. Alpha levels were adjusted when appropriate in post-hoc analyses using Bonferroni correction.

3.3.1 Analogy Articles Differed from Keyword Articles and were Judged more Novel. The viability of our approach is based on the assumption that the analogy search pipeline returns a different distribution of results than a keyword-based baseline. This assumption appeared to hold true: the keyword-search and analogy-search conditions resulted in almost completely disjoint sets of article recommendations. Out of the total 267 articles, the overlap between analogy and keyword articles was only one. Analogy articles appeared to represent a complementary set of results users would be unlikely to encounter through keyword-based search.

To further examine this assumption we had participants rate the novelty of the results by asking them “have you seen this article before?” on a 3-point Likert scale response options of 1: “Yes, I have seen this article before”, 2: “Yes, not exactly this article but I have seen similar ideas before”, and 3: “No, I have not seen anything like this before”. Participants found articles recommended in the analogy condition to contain significantly more novel ideas (2.7, SD: 0.48) compared to the keyword condition (2.3, SD: 0.55) (Welch's two-tailed t-test, $t = -5.53, p = 1.33\times 10^{-7}$) (Figure 5, left). Participants thought the “variance in results is much higher than using other search engines” (P5) and “there're a lot of bordering domains$\ldots$ which can be useful if I want to get ideas in them” (P4).

This difference was also reflected in the content of articles, with keyword articles having significantly more overlapping terms with participant-provided query terms (4.1, SD: 1.74) than analogy articles (1.6, SD: 1.42) (Welch's two-tailed t-test, $t(145.27) = 11.70, p = 1.10\times 10^{-22}$) (Figure 5, right).¹² More occurrences of familiar query terms in keyword articles’ titles and abstracts may have led participants to perceive them as more familiar.

3.3.2 Analogy Articles Resulted in more Creative Adaptation Ideas than Direct Application Ideas. We found that the distribution of ideation outcome types differed significantly between analogy and keyword articles ($\chi ^2(3) = 52.12, p \lt 1.0\times 10^{-10}$). Participants came up with more creative adaptation ideas (N = 53; 32% of total) over direct application ideas (N = 3; 2%) using analogy articles. In contrast, keyword articles resulted in more direct application ideas (N = 16; 19%) than creative adaptation ideas (N = 10; 12%) (Figure 6). The difference between creative adaptation and direct application was significant ($\chi ^2(1) = 28.41, p = 9.84\times 10^{-8}$).

To illustrate more concretely the divergent patterns of ideation leading to Creative Adaptation and Direct Application ideas, we describe vignettes from three participants (Table 3). While Direct Application ideas represented close-knit techniques and mechanisms directly useful for the source problem (described with verbs such as apply and adopt), Creative Adaptation type ideas were more distant from the source problem and could be characterized with the use of different verbs associated with significant adaptation (design and invent). For example, P1’s research focused on the methods for improving nanoscale heat transfer in semiconductor materials. Previously he developed mechanisms for manipulating the thermal conductivity at solid-solid interfaces, specifically by adjusting the semiconductor wall structures. Thus, an article reporting experimental results of manipulating thermal conductance on planar metallic contact points was deemed a directly useful article that might contain helpful techniques. On the other hand, an analogy article which dealt with the heat transfer phenomenon at a macroscale, using fin-based heat sink designs for electronic devices, gave him a new inspiration: to adapt fins for nanoscale heat transfer in semiconductors to not only transfer heat but also convert it into a useful form of mechanical energy. Despite the mismatch on scale ([macroscale] $\nleftrightarrow$ [microscale]), challenging the assumption of the typical size of a fin-based design engendered an idea to creatively adapt it to convert heat into energy through an array of tiny fins, rather than merely dissipating it into space as in the original formulation of the problem. P1 also found another analogy article focused on thermal resistance at a liquid-solid interface useful for future ideation because despite its surface dissimilarities, there was a potential mapping that may open up a new space of ideas (e.g., [liquid] $\nleftrightarrow$ [polymer substrate], [solid] $\nleftrightarrow$ [germanium], yet the pairwise relation [liquid:solid] $\leftrightarrow$ [polymer substrate:germanium] may be analogous and interesting): “This is liquid$\ldots$ but it's about liquid-solid interface which can be useful$\ldots$ because for the substrate that sits on top of silicon or germanium you use polymers which have liquid-like properties” (P1).

In the case of P2, an article focused on computational methods for toxicity prediction was deemed directly helpful because “if certain nanomaterials are toxic to certain microorganisms that eat plants or kill them but safe for the plant, we can target these organisms using the nanomaterials as pesticide. Another way this can be helpful is in predicting the chance of toxicity of the nanoparticles in our fertilizers” (P2). Whereas an analogy article that uses image analysis for plant identification reminded her of “hyperspectral imaging in plants, like a CT scan for plants. So making a hyperspectral 3D model using something like this$\ldots$ to optically sense and trace plant cells (such that the entry of fertilizer nanoparticles into plant cells can be monitored, a sub-problem of P2’s research problem) would be pretty cool.”

As a third example, P6’s research focused on recording and simulating electrical activity using microelectrode arrays. To him, an analogy article about printing sensors for electrocardiogram (ECG) recording seemed to present an interesting idea despite its mismatch in terms of scale ([nanoscale] $\nleftrightarrow$ [macroscale]) and manufacturing mechanism ([fabrication] $\nleftrightarrow$ [printing]), because the pairwise relation between [nanoscale:fabrication] $\leftrightarrow$ [macroscale:printing] engendered a reflection on the relative advantages of different methods and future research directions): “Interesting idea! Instead of nanoscale fabrication, printing can be a good alternative for example for rapid prototyping. But I think the resolution won't be enough (for use) in nanoscale$\ldots$ works for this particular article's goal, but an idea for future research is whether we can leverage the benefit of both worlds—rapid printing and precision of nanoscale fabrication” (P6).

3.3.3 The Level of Purpose-match had Different Effects on the Ideation Outcome. Suggested in these examples is a certain kind of distance the ideas in analogy articles maintain in order to spur creative adaptation. We hypothesize that some amount of difference in purpose facilitates creative adaptation. This process may involve a curvilinear relationship between the degree of purpose mismatch and the resulting ideation outcome, with too much or too little deviation leading to a little-to-no benefit or even an adverse effect on the ideation outcome, a pattern that is consistent with the findings in the literature of creativity and learning outcomes (e.g., Csikszentmihalyi's optimal difficulty [25]). For this analysis, we coded each article based on three levels of purpose-match to the source problem:

• Full: Both high- and low-level purposes match.
  
• Part: Only the high-level abstract purpose matches. Explicit descriptions of the high-level purpose exist in either title and abstract of the article. At the same time, certain low-level aspects of the participant's research problem are mismatched as evidenced by relevant comments from the participant.
  
• None: Neither high- nor low-level purposes match.
  

Examples of these types of purpose-match are provided in Table 4. High-level match can be considered as a first-order criterion of purpose match and low-level match as a second-order criterion: If the article does not have overlapping terms in terms of its purpose with the user query cast at a high level (e.g., transfer heat, grow plants) then the low-level match does not matter, but if the article's purpose matches at the high level, its low-level alignment (e.g., specific aspects of the purpose, such as its scale or materialistic phase) will additionally determine full (i.e., aligned in both high- and low-level aspects of the purpose) vs. partial match (i.e., aligned only in the high-level but not low-level aspects of the purpose). Therefore, the coding procedure was symmetrical to the procedure described for coding four types of ideation outcome, with the high-level purpose match deciding between {Full, Part} and None match types, while the low-level purpose further distinguishing between Full vs. Partial match. Following this procedure, two independent coders achieved an inter-rater reliability Cohen's $\kappa = 0.72$ (substantial agreement) and disagreements were resolved with case-by-case discussion.

We used the mediation package¹³ [105] to conduct a mediation analysis between the condition, the kind of purpose-match, and the binary Creative Adaptation ideation outcome. The analysis showed that the effect of condition (Keyword vs. Analogy) on the binary outcome of creative adaptation was mediated by the degree of purpose-match, but not by the novelty of content, suggesting that the difference between full vs. partial matching on purpose is much more significant than the variance in the content novelty. We come back to this result in the discussion (Section 7.2.3). Table 5 presents the result of the mediation analyses. The regression coefficient between creative adaptation and condition was significant as was the regression coefficient between the degree of purpose match and creative adaptation. The indirect effect was $(-0.42)\times (0.21) = -0.09$. We tested the significance of this indirect effect using a bootstrapping procedure [91] ($p \lt 2 \times 10^{-16}$), by computing the unstandardized indirect effects for each of 1,000 bootstrapped samples as well as the 95% confidence interval (CI).¹⁴

Partial purpose matches in both keyword and analogy articles led to creative adaptation, but the rate was significantly higher with analogy articles. As expected, the ratio of direct application decreased from the keyword articles that fully match in purpose (Keyword Full, 68%) to the keyword articles that partially match in purpose (Keyword Part, 6%) (Figure 8). At the same time, the rate of creative adaptation increased from the keyword articles that fully match in purpose (Keyword Full, 0%) to the keyword articles that partially match in purpose (Keyword Part, 21%). However, the rate of creative adaptation differed significantly between the keyword and analogy articles, with the rate more than doubling among the analogy articles over keyword articles (Analogy Part 47% vs. Keyword Part 21%). Homing in on the partial matches, these articles led to creative adaptation ideas significantly more often in analogy search (47%) than keyword search (21%) (Welch's two-tailed t-test, $t(112.22) = -3.40, p = 9.0\times 10^{-4}$, Figure 7, left). While the partial purpose mismatch was highly associated with creative adaptation ideas, it could be a double-edged sword. Among the analogy articles, 38% of the partial mismatches resulted in no useful ideation outcome as opposed to the 47% that resulted in creative adaptation (Figure 8, Analogy Part). Therefore, knowing what mismatches are beneficial to creative adaptation has important implications for facilitating generative misalignment for ideation.

4.2.1 Improved Token-level Prediction of a Span-based Model. First we compared the span-based Model 2 with five other baselines to evaluate the token-level classification performance (Table 6). Model 2’s overall F1 score was the highest at 0.65 (Purpose; PP: 0.65, Mechanism; MN: 0.64, an 0.14- and 0.14-absolute-point increase from Model 1, respectively) on the validation set which represents an overall 0.15-absolute-point increase from Model 1 used for the initial human-in-the-loop analogical search engine.

4.2.2 Pipeline with a Span-based Model Reflected Human Judgment for Ranking the Results. The improved token-level prediction performance materialized as an increase in the pipeline's ability to judge the degree of purpose match. For this evaluation, we first recorded every query provided by Study 1 participants that human-in-the-loop filterers used to search and filter the relevant articles. Then, we simulated the search condition of the filterers for the automated pipeline by providing it input as the exact queries they used. We capped the number of top search results sufficiently large at 1,000 for each query. From these top 1,000 results, we selected articles that also appeared in the human-in-the-loop system and collected the corresponding human-vetted judgments of high or low purpose-match. For each of these articles, we also collected its corresponding rank positions on the new (automated) pipeline's list of results.

We compared the mean ranks of articles that are judged by human filterers as high purpose match to those of low purpose matches. The result showed that the new pipeline indeed was able to distinguish between the two groups of articles; low purpose matches (i.e., articles that were deemed not relevant and subsequently filtered by the judges in Study 1) were placed at significantly lower positions on the list than high purpose matches (i.e., unfiltered articles in Study 1). The mean rank for low purpose matches was 465 while for high purpose matches it was 343 (Figure 9). This difference was significant ($t(192.49) = 3.29, p = 0.0012$. Welch's two-tailed t-test.).

4.2.3 Different Model Performance on Finding Articles that Fully or Partially Match on Purpose. Data and coding. In addition to the overall rankings reflecting human-vetted judgments we also found that the proportion of partial purpose matches was significant among the top-ranked results. We sourced top 20 results for each participant's research problem with the automated system (Model 2) using the exact queries and order used by the human-in-the-loop filterers in Study 1. We compared this to four other approaches: (1) the human-in-the-loop system in Study 1 (BiLSTM with filtering), (2) a BiLSTM-based system excluding the human-in-the-loop from 1 (BiLSTM), (3) randomly sampled articles (Random), and (4) a keyword-based search results, which was used as control in Study 1 (Keyword). There were no overlapping articles between Model 2 and other conditions except for the Keyword condition which had 1 overlapping article. To code the degree of purpose match, we blended the results of Model 2, biLSTM, and Random conditions. Two of the authors coded a fraction of the data together blind-to-condition (7.4%, $N = 20/270$) following the same procedure used in Study 1. Then they independently coded the rest blind-to-condition achieving an inter-rater agreement of $\kappa = 0.80$ (substantial agreement). We resolved any disagreement through discussion on an individual case basis.

Result. We found that the Model 2-based system achieved a parity with the human-in-the-loop system (Study 1) for finding purpose matches (Figure 10), with more than half of the system's top 20 results judged to be partial purpose matches. In contrast, when human-in-the-loop filtering was removed from the BiLSTM-based system, the frequency of partial purpose matches decreased from 58% to 37% while the frequency of no matches increased from 40% to 59%. Random sampling resulted in mostly irrelevant results, with no alignment on purpose with the source problem. An interesting point of comparison is between the keyword-based and Model 2-based search results. Keyword search mostly outperformed Model 2-based system by finding full purpose matches at a much higher rate (23% in keyword search vs. 4% in the Model 2-based system), with similar rates of partial purpose matches (58% vs. 55%), and significantly less no purpose matches (19% vs. 41%). On average the purpose match score was the highest in keyword-search followed by the Model 2-based and the human-in-the-loop systems (Figure 11). Combined with the results of Study 1, this suggests the complementary value of analogical search: The higher rate of full-matches in keyword-search may be good when searchers know what they are looking for, such as in direct search tasks and foraging from familiar sources of ideas. Nonetheless, because analogy articles were both deemed significantly more novel by the scientists and had little-to-no overlap with keyword-search articles, they augmented keyword-based search results with a complementary set of articles that introduce useful mismatches in their purposes. This set of articles may open up new domains of ideas that scientists may not have been aware of, and encourage creative adaptation.

5.2.1 Overall Impressions. Overall participants described their experience with the analogical search engine in a positive light (e.g., “helps me think at a broad topic or a big picture level”—P2; “find some very interesting and useful ideas, the design is also very simple, good when focusing on key areas of research”—P5; and “very interested now what the future of this engine would look like”—P3), but a deeper look suggested that the success of ideation depended on how well searchers were able to engage with analogical results that deviate from their expectations: “It's surprising that the engine recommends examples like these”—P3; “If I input the same search queries on Google Scholar it'd not normally return these things$\ldots$ this search engine works in a different way”—P1.

5.2.2 “Not the Kind of Article I'd Look for but$\ldots$ ”: The Challenge of Early Rejections. Unlike similarity-maximizing search engines, the diversity in analogical search results can lead to premature rejection of alternative mechanism ideas. One of the factors contributing to premature rejection of alternatives may be the tendency for adherence to a set of existing ideas or concepts, as studied in the literature of design fixation (e.g., [66]). In our study, the participants found the variety of domains featured in search results confusing, and it sometimes prevented them from engaging with the ideas therein. For example, P3, whose research studies ways to manage or reduce task complexity for nuclear power plant operators, expected to see results similar to Google Scholar which are typically in the domains of operational and managerial sciences, but was surprised by unfamiliar domains represented in search results: “These (distributed networked systems design or path planning for automated robots) are not the kinds of fields that I normally read in, if I found them elsewhere I would've probably thought they're irrelevant and skipped” (P3). Ranging from unfamiliar terms (P1, P4, P5) to unfamiliar categories of approaches (e.g., “Not sure what ‘Gauss–Newton approach for solving constrained optimization’ is”—P6), or high-level research directions (e.g., “this is different from my research direction, people who work on this direction might find it interesting, though”—P1), participants saw the diversity of results as a challenge for engagement. P1 pointed out a perceived gap between the expectation of least effort and the cognitive processing required when engaging with analogical ideas and adapting them:

“As I understand it, I think this search engine is trying to present articles from related but different fields to let people make connections. But people expect less friction. (The result is) something interesting but I can't directly write it into a project proposal$\ldots$ I think it would be challenging to make people get interested in investing time to read the articles in depth to come up with connections. I wonder what would happen if this was hosted just as an online website (instead of the study context)”—P1

On the other hand, analogs that did get examined more deeply could ultimately lead to creative adaptation. For example, P3 mapped task scheduling among computer processes to task assignment among the nuclear power plant operators, and came up with an idea to adapt algorithmic scheduling used in real-time distributed systems to a scheduling mechanism that could be useful in her research context. Represented symbolically this process was akin to ideating what might best fill in the “?” in the relational structure [scheduling algorithm:processes in distributed systems] $\leftrightarrow$ [?:nuclear power plant operators]: “I think the algorithms proposed in this article could be useful for calculating the operator task execution time, the power plant system's response time, and the time margin between the execution time and the system response time$\ldots$ so that the next task assignment can factor in these margins and things related to workers’ well-being like rest and time required between switching tasks” (P3).

Participants seemed to recognize a small number of core relations as kernel for creative adaptation. In the example of P3, scheduling processes in the distributed systems article piqued her interest and led her to connect them with similar concepts in the literature she was already familiar with: “You need to make that connection$\ldots$ I saw parallels between (distributed systems domain) concepts like [scheduling] and [tasks] and [scheduling tasks for the operators]” (P3). Similarly, P5 recognized a similarity between [monitoring people's performance] in fitness training and [monitoring whether construction workers are wearing personal protective equipment] in construction sites. He then adapted the idea of tracking heat emission in the fitness context to his own: “I like the idea of [monitoring heat emissions] in fitness training$\ldots$ maybe I can also detect heat emissions from construction workers to see if they are wearing the safety vests or masks while also monitoring the site conditions and worker efficiency. It also gives me an idea to monitor the $\text{CO}_2$ emissions from workers so as to improve the robustness of detection” (P5). In this case, monitoring and the physical nature of the activities involved helped P5 see the connection useful for creatively adapting the source idea.

5.2.3 “I don't Know What to Type in”: The Challenge of Query (Re-)Formulation. Another challenge participants faced was that they were not used to formulating their search queries in terms of high level purposes of their research. On average participants entered 5.2 queries (Min: 1, Max: 18, SD: 5.87), 87% (27) of which were in the form of a single noun phrase (e.g., “heat pipe evaporation,”—P1, “task complexity”—P3, “theoretical optimization convergence for non-convex functions”—P6) or a comma-separated set of multiple noun phrases (e.g., “heat transfer, nanoscale, fluid”—P2) that represented specific aspects related to research purposes rather than the core purposes themselves. For example, the purpose of “heat pipe evaporation” may be to transfer heat, and the purpose of searching for “theoretical optimization convergence for$\ldots$” may be to detect when optimization converges or diverges, or to effectively sample unknown (non-convex) distributions.

One of the reasons why participants formulated search queries in this way may be wrongly assuming that the search engine used keyword matching to find results. For example, extensive prior experience with search engines that highlight matching keywords in abstracts (e.g., Google Scholar) in response to users’ search queries can reinforce such assumptions among the users. In addition, participants’ domain knowledge useful for judging which of the returned articles are relevant may have led them to notice a set of keywords the inclusion of which strongly signifies the relevance of a article. In contrast, the analogical search results often seemed to not feature such directly similar terms and this contributed to the difficulty of judging whether a result is relevant and how: “I find these articles not very related to my search query at first. It'd be better if you can use some graph or some pictures to indicate how these articles can relate to my keywords” (P5); “I'd not consider$\ldots$ (because) they are totally different, right? They look irrelevant$\ldots$ until I think about it I can realize that it's useful. But if you give me the article, at first I don't realize that” (P3).

While it may not feel as compelling or natural to participants, formulating and abstracting queries at a high level may lead to searching more distant results that are analogous at a higher level. For example, by querying “detect personal protective equipment” instead of “personal protective equipment construction,” P5 found novel mechanisms of detection, such as general image segmentation algorithms or an approach to monitoring heat in the context of fitness training not specific to construction sites and personal protective equipment but nonetheless useful for creative adaptation. Querying “scheduling tasks” instead of “task complexity” for P3 resulted in finding scheduling algorithms in distributed computer systems that otherwise P3 would not have encountered, while “assigning tasks” led to novel auction mechanisms which made her think about a system in which each power plant operator can bid for a task as opposed to being assigned one. Schematically, Figure 13 shows how formulating queries at a higher level of abstraction than specifying the problem context in full details (Ⓐ $\rightarrow$ Ⓑ) may lead to discovering novel mechanisms that are relevant at the high level of abstraction, and in more distant ways from the original problem formulation (Ⓑ $\rightarrow$ Ⓒ).

7.2.1 Experimental Design and Improving its Validity. Our findings have several limitations. First the design of our studies may be improved to increase the experimental validity. We believe that our coders of the ideation outcomes had a reasonable understanding of participants’ research context from descriptions of current and past research topics, think-alouds with 45 articles, and end-of-experiment discussions, and that the procedure of coding reduced potential biases (e.g., the coders were blind to experimental conditions, relied on participants’ statements of novelty and distance). Despite this, it is possible that they judged ideas differently from domain experts, for example coding more or fewer ideas as creative adaptation, or pre-filtering useful ideas in the human-in-the-loop stage. In addition, other quality dimensions such as potential for impact or domain-expert-judged idea quality are largely inaccessible within the studies presented here. Future research may improve on these limitations by iterating on the experimental design, collecting data for triangulating the results and capturing other quality dimensions of the generated ideas.

Additionally, future work may add ablation studies to quantify the effects of human filtering in Study 1 on the ideation outcome as well as sensitivity studies to relate how much the increased token-level classification performance of trained models may reduce the burden of human filtering. Furthermore, additional experiments with baselines other than keyword-based search using the whole abstract will help pinpoint the potential advantages of representing and matching articles using extracted purposes and mechanisms. For example, Chan et al. [21] found that embedding all words from an abstract (using GloVe embeddings) resulted in retrieval of fewer analogical items than when extracted purposes and mechanisms were used. Replicating this result with additional approaches such as contextualized word embeddings and pre-trained language models (e.g., ELMo [90], BERT [29], and SciBERT [7]) will be valuable.

7.2.2 Potential Sampling Bias. The sampling strategy in Study 1 was purposefully unbalanced, where analogical articles were sampled twice as much as keyword articles to ensure participants’ exposure to sufficiently diverse results. This was crucial for uncovering potential benefits and challenges of our analogical search engine and investigating its viability. This ratio was chosen purposefully, to balance the statistical power for detecting potentially significant differences between the conditions, while also limiting the number of articles that each participant had to review. Given the cognitive burden of reviewing an article while thinking aloud, we decided on 45 in total with the 2:1 ratio to fit the practical time limits of interviews. However, this may have led to unanticipated effects on ideation outcomes despite having accounted for the difference in sample sizes by measuring the outcomes in ratios. For example, when the results were combined into a single blinded list, the over-representation of analogical results over more purpose-aligned keyword results may have shifted the users’ overall perceived value of the list to be more or less positive. Users’ perception of diverse results may have been further affected by their relative over-representation. For example, increased cognitive load for processing analogical mapping [51, 52, 102] may suggest that results that fully match on the purpose search query may have been perceived even more favorably than analogical results, due to a negative spill over effect from the rest of the articles in the list, which were less likely matched on the purpose. Investigating whether such factors led to compounding effects beyond our ratio-based measures of usefulness remains an open question for future work.

7.2.3 Controlling the Diversity of Search Results. Our work is also limited by the lack of controllability in sampling the search results beyond purpose similarity. As described in Section 2.2.1, from pilot tests in our corpus we discovered that even close purpose matches of scientific articles already had high variance in terms of the mechanisms they proposed which allowed us to focus our approach to sampling based solely on purpose similarity. The simplicity of this approach also means fewer hyper parameters in the sampling mechanism compared to other approaches [61, 62]. However, all the approaches including this work thus far lacked a mechanism for explicitly controlling the diversity in retrieved search results which remains a fruitful avenue for future work. For example, prior research has uncovered the nuanced effects of distance (e.g., near vs. far sources of inspiration [24, 97]), suggesting the benefit of targeting analogs at different distance from the source problem for the right context. Future research may also uncover further complexities in the relationship between novelty and purpose-match. The result of our mediation analysis (Table 5) showed that the novelty of content among the search results in Study 1 was not a significant factor to the same extent that the three levels of purpose match was. However, the relationship between novelty and purpose match may be more complex than the levels of manipulation presented in this work. For example, [30] suggested a greater importance of novelty than usefulness for predicting creativity scores. Future work may design mechanisms to manipulate the variance in content novelty and alignment in the purpose-mechanism schema to uncover dynamics between the two that go beyond the results from mediation analyses presented here (Section 3.3.3). Furthermore,

challenges with the abstraction of purposes remain open, for example, how core versus peripheral attributes of research purposes may be identified, and how they may be selectively matched at a specific level of the conceptual hierarchy. Finally, not all query formulations are created equal in terms of their suitability for analogical search. We observed in the case studies that participants wanted to express different query intent via reformulation (Section 5.2.3). While participants could reformulate their search queries and examine the returned results from our analogical search engine in real-time, it was unclear whether and how specific query formulations may lead to more or less diverse results, and how subsequent queries may be updated after reviewing them. As such, systems that assist users in the potentially tedious process of query reformulation [113] (for example, by way of automatic query expansion [18]) in the context of analogical search will be important.

7.2.4 Studying the Effect of Larger Context of Scientific Innovation on Analogical Innovation. Due to our focus on ideation outcomes, our results do not explain how these ideas may be integrated, developed, and shared across the research communities. Studying the lifetime of ideas that goes beyond their inception will deepen our understanding of the factors that currently make analogical innovation such a rare event in sciences (for example, Hofstra et al. suggested that more semantically distant conceptual combinations receive far less uptake [58]). Through interviewing our study participants and other colleagues in academia we found emerging structures related to this challenge. Our interviews informed us that in general the context in which a scientist exists—such as the scientist's role in a project, the maturity of a project, and the broader academic culture—can ultimately change how they interact with and seek analogical inspirations. For example, a third-year PhD student studying chemical engineering commented “In the current stage of my project it's more about parameter-tuning—running multiple experiments at once and comparing which configuration works the best$\ldots$ If I were a first year PhD student maybe I would be in a broader field and exploration.” In contrast, a PhD in biology who recently defended noted that “analogical inspirations would perhaps be more useful if you're looking for a postdoc or a faculty position.”

In addition, the underlying career incentive structures in academia may also affect researchers’ perception of and openness to analogical inspirations. One of the study participants commented “since I'm already a third year PhD student and my project is further along and more firmed up, I'm not really looking for really far inspirations$\ldots$ first we push the specific way we have in mind with many iterations on the experiments until, say, publication.” In addition to the career-wise incentives there are other types of competitive resourcefulness (e.g., social resources such as the advisors’ and colleagues’ expertise that participants can easily tap into; physical and other forms of resources such as tangible artifacts like previously developed code packages or experimental processes and setups). These factors can influence scientists’ perception of their advantage and lead them to interpret analogical inspirations as more or less useful, feasible, and directly applicable to their research. This observation is further suggested by survey results that asked our participants: “Could this article be useful to you?,” their ratings were significantly higher for keyword articles than analogy articles despite them having come up with creative adaptation ideas more often with analogy articles. Therefore, future work that studies incentive structures, the quality of ideation outcome, their feasibility, the differences in research context e.g., frames of research contribution such as discovery-oriented vs. novel system development-oriented, and taking a longitudinal observation of the variation in such factors will add a significant depth to our understanding.