An Experimental Study of Prompt Engineering Techniques for Optimizing Large Language Model Inference

Prompt-based learning emerged as a major paradigm shift in natural language processing by enabling pretrained language models to perform downstream tasks through input reformulation rather than full task-specific retraining. This review synthesizes influential studies published up to 2021 and evaluates how prompt-based methods improved inference effectiveness, few-shot adaptation, and parameter efficiency. The review focuses on five major developments: cloze-style knowledge probing, few-shot in-context learning, automated prompt discovery, prompt-based fine-tuning in low-resource settings, and continuous prompt optimization. Evidence from the pre-2022 literature shows that prompting substantially improved task adaptation when aligned with pretraining objectives. GPT-3 demonstrated that large language models could perform a wide range of tasks through zero-shot, one-shot, and few-shot prompting without gradient updates. In smaller-model settings, PET/iPET and LM-BFF showed that prompt-based fine-tuning substantially outperformed conventional fine-tuning in low-resource classification tasks. By 2021, prefix-tuning and prompt tuning further established that competitive downstream performance could be achieved while updating only a very small fraction of model parameters. The review concludes that, up to 2021, prompt engineering should be understood primarily as a framework for task reformulation and efficient adaptation rather than as the reasoning-and compression-centered paradigm that developed later. (Brown et al., 2020; Gao et al., 2021; Lester et al., 2021; Li & Liang, 2021; Liu et al., 2021; Schick & Schütze, 2021).