Perspective: Large Language Models in Applied Mechanics

In natural language processing (NLP), a language model is a probabilistic model of sequences of words. When conditioned on a partial text sequence, a language model learns from its training corpus to predict the most likely missing text. By learning to do this task well, such models implicitly absorb linguistic rules, norms, idioms, and even world knowledge from the text on which they are trained. The era of “large” language models (LLMs) began with the 2017/2018 publication of the transformer architecture [1] and 110M- and 345M-parameter BERT models [2] (Fig. 1).

When a model like BERT is pretrained on a large text corpus with an unsupervised language modeling objective, it serves as a rich starting point for further training (fine-tuning) on specific supervised learning tasks, such as information extraction or text classification; analogous to learning a language before learning to perform a specific task in that language. Until recently, the paradigm of “general-purpose language model pretraining followed by task-specific fine-tuning” has dominated NLP. However, more recently, larger LLMs have begun to obviate the need for fine-tuning altogether.

Recently, very large language models (VLLMs) such as 540B-parameter PaLM [3] and the 175B+ GPT 3/4 series [4,5] have demonstrated the ability to perform zero- and few-shot learning (Fig. 2) on the basis of pretraining alone. In consumer-facing models such as ChatGPT, this consists of both text-completion-based language model pretraining and additional, yet still general-purpose, human-in-the-loop optimization for instruction-taking [6]. During zero-shot learning (applied to Ref. [7] in Fig. 2), the model is prompted with the textual description of a task, and it generates answer text based solely on the linguistic and real-world knowledge it absorbed during pretraining. In few-shot learning, this prompt is accompanied by a small number of exemplars (shown using Refs. [7,8] in Fig. 2) which demonstrate for the model how to perform the task [4]. Increasingly, these learning paradigms outperform traditionally fine-tuned models (and humans) on a wide variety of linguistic tasks [5]. These emergent capabilities are exciting for science because they vastly lower the barrier to effective scientific data mining. They require no large datasets of expert annotations, and eliminate the time, computational power, and expertise associated with traditional model training. Instead, they at most need a small set of expert exemplars and no additional fine-tuning.

These very large models have flaws. Because they are optimized for linguistic plausibility, they have a tendency to hallucinate, or fabricate, fictitious information, and they can make reasoning errors when faced with tasks involving multi-step logical inference [5]. They also tend to produce “boring” output that represents a middle-of-the-road consensus of information absorbed from their training data, which limits their ability to, for example, produce novel scientific hypotheses [9]. The emerging field of prompt engineering [10] explores prompting strategies for mitigating these issues, such as “chain of thought prompting” [11] which explicitly demonstrates step-by-step reasoning to avoid logical errors or “retrieval-augmented generation” which incorporates retrieved documents into the input prompt [12] in order to discourage hallucination. This latter approach is an example of “augmented generation,” where a LLM is optimized to operate in concert with additional functions such as web search, calculation, or a calendar [13,14].

At the time of writing, there is a distinction between “medium-large” language models (MLLMs) such as BERT and its variants, and “VLLMs” such as PaLM and GPT-3/4 (see Fig. 3). MLLMs generally are not capable of zero- or few-shot learning, and must be fine-tuned on training data. However, they can be purpose-pretrained on specific corpora (e.g., on a corpus of mechanics literature) and can be fine-tuned in innovative ways, such as for integration with scientific knowledge bases (e.g., Ref. [15]). VLLMs generally are capable of zero- and few-shot learning, but are too large to purpose-pretrain or easily fine-tune on commercially available compute (with the partial exception of GPT-3, whose application programming interface (API) allows for opaque, proprietary, and limited fine-tuning). In this perspective, we use the term MLLM to refer to LLMs small enough to allow (and require) traditional fine-tuning, where VLLMs are more suited for zero- or few-shot learning, with at-best limited fine-tuning ability. Another research direction in LLMs studies the interplay between large general-purpose models and smaller, more focused models, to leverage the advantages of both LLM types [16].

The line between MLLM and VLLM is ill-defined and liable to change in the next few years, as models and computational resources continue to advance. For most teams, this line lies between the 11B-parameter T5 [17], which is often fine-tuned, and the 120B-parameter Galactica [18], which is designed for inference only (no fine-tuning) on the highest-end commercially available graphical processing units (GPUs).

A collection of recent LLMs have been pretrained specifically on corpora of scientific literature and found to perform better on scientific NLP tasks than ones trained on a general-purpose text corpus. Examples include MLLMs like SciBERT [19] and ScholarBERT [20], which outperform the original BERT on such tasks, and the VLLM Galactica [18], which outperforms GPT-3. This idea of domain-specific pretraining has been applied even more narrowly to materials science. Examples include MatBERT [21], which is pretrained on a corpus of materials papers and, when fine-tuned, outperforms SciBERT on materials-specific NLP tasks. Similar models include MatSciBERT [22] and MaterialBERT [23], as well as even more focused models such as BatteryBERT [24], OpticalBERT [25], and TransPolymer [26].

Parallel work has explored how to use LLMs as “soft knowledge bases,” focusing on what factual information they absorb through their pretraining and how to extract it. This idea is key to their use in scientific applications, which are heavily dependent on factual knowledge. Petroni et al. [27] use “fill-in-the-blank” prompts (“clozures”) to demonstrate that even the original BERT model encodes meaningful factual information (e.g., the birthplace of Dante Alighieri). This approach has become a template for zero-shot prompting of larger models as well. In the evaluation of Galactica, the model achieved 43% accuracy at predicting chemical reaction products from reagents [18]. Kandpal et al. [28] use similar prompts to show that models struggle to learn facts that are not well-supported in their training data.

When considering LLMs as “soft” knowledge bases, a related line of work asks how to combine them with traditional “hard” knowledge bases such as material property datasets. Safavi et al. combined vector representations from traditional knowledge bases with those produced by language models for improved link-prediction within these knowledge bases [15]. Similar work has primarily been pursued in the biomedical domain, aligning LLM-derived representations with data from biomedical databases [29–31].

The idea that a language model can encode useful world knowledge precedes contemporary models. In materials, the use of word vector models [32,33] by Tshitoyan et al. [34] on thermoelectric compounds attracted significant attention because it showed that representations extracted from even such a simple model can predict the thermoelectric properties of compounds. This approach has been applied to exploration of the properties of polymers [35], multi-principal element alloys [36], and fuel cells [37].

Domain-specific pretraining and fine-tuning for alignment between LLMs and structured knowledge bases lend themselves to use in MLLMs, as VLLMs require more time, computational infrastructure, and data than is available to anyone except a small handful of companies. Galactica is an exception here, as it was released by Meta for general-purpose scientific applications. However, the explosion of interest in VLLMs has also touched on their use in science, with scientists in varying fields expressing both interest in the potential and concern for the pitfalls of general-purpose large generative models [38,39]. This excitement has spread to material mechanics, where developments such as Refs. [40–42] take advantage of OpenAI’s proprietary tuning API to demonstrate that minimal supervised fine-tuning enables VLLMs like GPT-3 to make high-fidelity property predictions. For VLLMs that cannot be tuned, Polak and Morgan [43] demonstrate a zero-shot prompting strategy for material information extraction from the published literature.

The use of LLMs in mechanics has been sparse. Independent of its use in LLMs, the Transformer architecture has attracted attention as a means for modeling of material properties [44–46]. There is also excitement about the use of LLMs for large-scale text mining in mechanics and materials science [47]. Some pre-LLM, traditional NLP approaches have been applied to mechanics problems [48–52], but to the best of our knowledge, the only instance of LLMs explicitly used for mechanics is Ref. [53], which applies a supervised BERT model to describe textual mechanics problems to other solver methods.

The use of LLMs in applied mechanics is challenging due to the focus in mechanics on structural design and phenomenological description across a wide range of topics. Traditionally, MLLMs (which have dominated the recent study of NLP in science) have succeeded in fields where the knowledge represented in the literature can be efficiently extracted and organized into structured schemas (e.g., ontologies), which can then be used as training datasets. For example, in materials science, descriptors such as the stoichiometric formulas of alloys can be readily extracted and linked to physical properties in the form of structured databases, such as those used for CALPHAD (e.g. Thermo-Calc, Pandat) and density functional theory (DFT) (e.g., OQMD: Open Quantum Materials Database, Materials Project). This formulaic nature that enables database construction also allows for efficient annotation of training and evaluation data for NLP models. For example, Tshitoyan et al. [34] validate vector representations of material formulas against a database of thermoelectric figures-of-merit.

One example of how schema creation can be difficult in mechanics is illustrated by the most downloaded Nature Materials paper of 2021, Ref. [7], which describes a non-cuttable composite composed of ceramic spheres embedded in an aluminum foam matrix capped with steel face plates. If one wanted to create a large-scale database linking composites to their properties, it would be challenging to design a schema flexible enough to represent all possible composites structures and connect those structures to emergent mechanical phenomena, such as the strain rate and local resonance effects that drive the non-cuttability seen in Ref. [7].

Beyond characterizing materials, many papers in applied mechanics are standalone investigations of context-dependent phenomena. An example is Ref. [54], which is a systematic study of elastically bistable ring origami, and one of the top five “most read” papers in the Journal of Applied Mechanics in May 2023. Elastic bistability is a useful phenomenon, but it occurs in many contexts beyond ring origami, such as clamped beams [55] and metamaterials [56]. Similar to representing composite behavior, it would be challenging to design a database schema capable of unifying the observed behaviors and these three disparate examinations of elastic bistability.

The diverse, difficult-to-categorize nature of applied mechanics research represents a key challenge for traditional NLP methods, including MLLMs. However, by reducing the need for traditional fine-tuning (and the large-scale databases it requires), VLLMs represent a possible and powerful means of overcoming these challenges.

We discuss several key emergent capabilities of LLMs that represent promising directions for their use in applied mechanics.

The first and perhaps most immediate such capability is information extraction. With little or no supervision, VLLMs can perform sophisticated reading comprehension tasks over a large number of documents (e.g., the examples shown in Fig. 2). This capability allows for quick iteration on extraction task, allowing scientists to efficiently experiment with what information it might be helpful to extract at scale from the literature. It skirts mechanics’ traditional resistance to schematization by allowing the use of ad hoc schemas designed for specific purposes, such as identifying a corpus of papers studying one particular phenomenon (e.g., elastic bistability and non-cuttability) across different contexts, and extracting parallel information from papers to study in a systematic way.

In applying machine learning to mechanics, language models can supplement or replace the need for traditional feature engineering by projecting text sequences to fixed-width vector representations which encode useful meaning. LLMs can just as readily encode the phrase “hard ceramic spheres in a highly compressible aluminum foam matrix” [7] as they can any other structural composite design description, regardless of the number of the words in the description or their meaning. Combined with zero- or few-shot information extraction (discussed above) for annotating training data, these representations can then be used in downstream machine learning models, such as predicting the properties of composites from descriptions of their structure. This capability is powerful in allowing the exploitation of the kinds of ad hoc schemas described above, for concrete tasks such as property or application prediction.

VLLMs such as PaLM and GPT-3/4 are pretrained on large corpora of code in addition to natural language text, allowing them to generate code in response to prompts. GitHub CoPilot [57] is perhaps the most salient example of this ability. This capability is likely to have major impact in mechanics, as both the computational and experimental mechanics communities rely on algorithms in their research pipelines. Potential examples include finite element subroutines [58–60], voxel-based mechanics models [61], and algorithms for experimental characterization [62,63]. For these tasks and others, LLMs can readily develop new code and translate between programming languages.

VLLMs such as GPT-4 and PaLM encode a large amount of memorized knowledge in their parameters and can be used as “soft knowledge bases.” Furthermore, they can synthesize information across their memorized training data, allowing quick answers to complex questions, where a traditional search engine could only return relevant documents that would have to be further parsed to get the desired answer. For example, a query asking to describe elastic bistability succinctly (and correctly) synthesizes information across multiple sources (Fig. 4). In theory, this allows quick access to answers which are otherwise diffuse or difficult to locate in the mechanics literature. However, as we discuss in the next sections, there are limits to LLMs’ potential for factuality.

VLLMs’ generative abilities can extend beyond simple question-answering, into more interpretive questions such as interesting hypotheses to explore. By synthesizing and summarizing information across large swathes of their training corpora, LLMs are able to coherently answer questions such as “what direction should I explore for…” or “what experiment should I perform to find out…” This capability can potentially allow LLMs to serve as engines for creativity in addition to knowledge and comprehension, both in mechanics and in science generally [64].

There are key limitations of large language models which need to be overcome before they can achieve their full potential as engines for speeding progress in mechanics and physical sciences.

Hallucination occurs when a language model generates information that is linguistically probable rather than factually accurate. This tendency threatens any use of LLMs as knowledge bases, and cuts into their advantage over more traditional information retrieval engines. One approach to mitigating this issue is retrieval-augmented generation [12,65], which first retrieves documents related to a query, and then prompts the LLM to generate text based on those documents. This approach is one opportunity for hybrid approaches using both MLLMs and VLLMs, as shown in Fig. 5, where database entries could be populated such as those in Refs. [66,67]. A mechanics-specific MLLM can be trained, applied across its entire training corpus, and used to retrieve relevant documents in a neural retrieval arrangement [68], and then a VLLM used to synthesize these results into a coherent response (e.g., WebGPT [69]).

Reasoning and comprehension errors threaten the use of LLMs as information extraction agents, hindering their ability to perform complex and useful comprehension across large corpora of mechanics literature. While the source of such errors is the same as hallucination (linguistic plausibility over factual/logical accuracy), the solution space is somewhat different. Extensions of few-shot learning such as chain-of-thought prompting [11] and faithful chain-of-thought prompting [70] encourage the model to generate intermediate reasoning steps, and have been found to improve the overall accuracy in performing complex reasoning tasks. Within mechanics specifically, the open questions include what comprehension and extraction tasks are useful to perform, and what strategies are needed for models to perform them reliably.

A key factor in the application of LLMs is interpretability. When a researcher queries a LLM for a scientific fact, uses it to suggest a hypothesis, or uses it to extract knowledge from a text, it is important to understand the source(s) of the model’s response. This is important not just from a scientific ethics perspective (properly citing the sources of used information) but from an even-more-basic verification perspective. No fact or suggestion generated by a LLM can be verified as true without some understanding of the underlying source for that information. Given the propensity of LLMs to hallucinate, this lack of verifiability brings with it serious risks of propagating error through work that uses LLMs as knowledge sources.

Both retrieval-augmented generation and chain-of-thought prompting represent means for improving model interpretability (and thus verifiability). Retrieval-augmented generation produces the textual sources underlying the model’s generated information, while chain-of-thought prompting exposes a facsimile of the model’s intermediate reasoning, allowing potentially easier resolution of logical mistakes.

When asked interpretive or speculative questions, VLLMs tend to produce coherent-but-uninteresting output [9], a result yet again of these models’ focus on producing the most likely rather than the most interesting possible text. This compromises their ability to serve as engines for scientific creativity by suggesting hypotheses or promising research directions. The increasing power of LLMs hints at their potential for helping to automate hypothesis discovery, but generating detailed scientific ideas that are both novel and useful remains an open problem [64]. While it is a less-studied problem than those previously discussed, there has been work on using scientific NLP to inculcate creativity in science, often focused on analogy as a useful construct [71–73].

In the previous section, we discuss several key capabilities of LLMs and speculate about how they might apply to the field of mechanics. However, one of the biggest open questions in applying LLMs to mechanics is which of these capabilities are likely to produce the most effective results in terms of the overall goals of the field. Although the information extraction capabilities of VLLMs offer unprecedented ability to perform complex reading comprehension tasks at scale, higher impact results may arise from using VLLMs for hypothesis generation, or otherwise. The answers to this big question will decide the role of LLMs in applied mechanics going forward, and these answers will remain unknown until mechanics-based LLMs are actually built and applied.

NRB and SHD acknowledge support from NSWC Grant #N00174-22-1-0020. CM acknowledges support from the NASA Space Technology Graduate Research Opportunities Fellowship (Grant No. 80NSSC19K1164). DD acknowledges support from NSF Grant 2033558. TMP and MPE acknowledge the support of a Department of Defense Vannevar Bush Fellowship, Grant ONR N00014-18-1-3031.

There are no conflicts of interest.

No data, models, or code were generated or used for this paper.