Back

A Comprehensive Overview of Large Language Models

Humza NaveedAsad KhanShi QiuMuhammad SaqibSaeed AnwarMuhammad UsmanNick BarnesAjmal Saeed Mian

article2023ACM Transactions on Intelligent Systems and Technology1,976 citations

Synthesizes the rapid progress of large language models across architectural designs, training strategies, context extensions, multimodal integrations, and evaluation benchmarks into a unified reference for researchers.

Listen

The rapid emergence and widespread adoption of Large Language Models have transformed artificial intelligence by demonstrating near-human performance across complex language, reasoning, and multimodal tasks. However, the sheer volume and velocity of recent advancements make it difficult for decision-makers and practitioners to maintain a clear, unified understanding of the technological landscape. The article provides a comprehensive overview of the fundamental design choices, training methodologies, adaptation techniques, and operational efficiency mechanisms across modern language models containing ten billion or more parameters.

The article conducts a broad review of leading foundational models and instruction-tuned systems. It examines core structural components—such as attention mechanisms, normalization techniques, and mixture-of-experts designs—alongside pre-training objectives, scaling dynamics, distributed optimization strategies, and alignment protocols like reinforcement learning from human feedback. Additionally, the review analyzes parameter-efficient adaptation, post-training compression, multimodal extensions, tool-augmented systems, and standard evaluation benchmarks.

The analysis reveals several critical findings for developing and deploying language systems. First, performance depends heavily on data quality and balanced scaling; expanding training token volume in equal proportion to model size yields far better results than merely increasing parameter counts. Second, base pre-trained models struggle with zero-shot generalization and safety unless refined through instruction tuning and human preference alignment, where small, highly curated datasets can match the gains of massive, uncurated corpora. Third, architectural adaptations such as parallel attention layers, multi-query attention, and mixture-of-experts routing substantially reduce computational overhead, enabling multi-fold improvements in throughput and energy efficiency without degrading output quality. Finally, augmenting models with external retrieval mechanisms and tool interfaces allows smaller, cost-effective architectures to match or surpass the factual accuracy of systems several times their size.

These findings indicate that achieving peak performance does not require brute-force infrastructure investment alone, but rather strategic data curation, efficient architectures, and targeted downstream adaptation. Organizations can significantly lower computational expenses and operational risks by adopting parameter-efficient fine-tuning, model quantization, and retrieval-augmented generation instead of full model training. Stakeholders planning deployment should prioritize robust data filtering pipelines, evaluate modular mixture-of-experts architectures, and implement strict alignment safeguards to mitigate hallucination and safety vulnerabilities in production.

While the findings provide high confidence regarding mainstream architectural and training paradigms, the article highlights ongoing uncertainties, such as conflicting evidence on optimal positional encodings and the instability of ultra-large models during extended training runs. Decision-makers should exercise caution and conduct controlled, application-specific pilot testing when applying aggressive compression methods, such as low-bit quantization or unstructured pruning, to ensure task performance remains uncompromised.

Australian National UniversityCommonwealth Scientific and Industrial Research OrganisationKing Fahd University of Petroleum and MineralsSDAIA-KFUPM Joint Research Center for Artificial Intelligence (JRCAI)The Chinese University of Hong KongUniversity of Engineering and TechnologyUniversity of MelbourneUniversity of SydneyUniversity of Technology SydneyUniversity of Western Australialanguage modellarge language modelsroboticslarge language modelspre-trained language modelslanguage agents
arXiv: 2307.06435humza909/LLM_Survey
Cover for A Comprehensive Overview of Large Language Models

Abstract

Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the existing literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to not only provide a systematic survey but also a quick comprehensive reference for the researchers and practitioners to draw insights from extensive informative summaries of the existing works to advance the LLM research.

Topics

language modellarge language modelsroboticsnatural language understanding

Keywords

large language modelspre-trained language modelslanguage agentsmachine translationLarge-scale Chinese Question Matching CorpusMMLUSentencePiece tokenizerlarge-scale training

Table of Contents

  • 1 Introduction
  • 2 Background
  • 2.1 Tokenization
  • 2.2 Encoding Positions
  • 2.3 Attention in LLMs
  • 2.4 Activation Functions
  • 2.5 Layer Normalization
  • 2.6 Distributed LLM Training
  • 2.7 Libraries
  • 2.8 Data PreProcessing
  • 2.9 Architectures
  • 2.10 Pre-Training Objectives
  • 2.11 LLMs Scaling Laws
  • 2.12 LLMs Adaptation Stages
  • 2.12.1 Pre-Training
  • 2.12.2 Fine-Tuning
  • 2.12.3 Prompting/Utilization
  • 3 Large Language Models
  • 3.1 Pre-Trained LLMs
  • 3.1.1 General Purpose
  • 3.1.2 Coding
  • 3.1.3 Scientific Knowledge
  • 3.1.4 Dialog
  • 3.1.5 Finance
  • 3.2 Fine-Tuned LLMs
  • 3.2.1 Instruction-Tuning with Manually Created Datasets
  • 3.2.2 Instruction-Tuning with LLMs Generated Datasets
  • 3.2.3 Aligning with Human Preferences
  • 3.2.4 Continue Pre-Training
  • 3.2.5 Sample Efficiency
  • 3.3 Increasing Context Window
  • 3.4 Augmented LLMs
  • 3.4.1 Retrieval Augmented LLMs
  • 3.4.2 Tool Augmented LLMs
  • 3.5 LLMs-Powered Agents
  • 3.6 Efficient LLMs
  • 3.6.1 Parameter Efficient Fine-Tuning
  • 3.6.2 Quantization
  • 3.6.3 Pruning
  • 3.7 Multimodal LLMs
  • 3.8 Summary and Discussion
  • 3.8.1 Architecture
  • 3.8.2 Training Strategies
  • 3.8.3 Supervised Models vs Generalized Models
  • 3.8.4 Zero-Shot vs Few-Shot
  • 3.8.5 Encoder vs Decoder vs Encoder-Decoder
  • 4 Model Configurations
  • 5 Datasets and Evaluation
  • 5.1 Training Datasets
  • 5.2 Evaluation Datasets and Tasks
  • 5.2.1 Multi-task
  • 5.2.2 Language Understanding
  • 5.2.3 Story Cloze and Sentence Completion
  • 5.2.4 Physical Knowledge and World Understanding
  • 5.2.5 Contextual Language Understanding
  • 5.2.6 Commonsense Reasoning
  • 5.2.7 Reading Comprehension
  • 5.2.8 Mathematical Reasoning
  • 5.2.9 Problem Solving and Logical Reasoning
  • 5.2.10 Cross-Lingual Understanding
  • 5.2.11 Truthfulness
  • 5.2.12 Biases and Ethics in AI
  • 6 Applications
  • 7 Challenges and Future Directions
  • 8 Conclusion
  • References

Knowls

  1. Knowl 1 — Taxonomy of Large Language Model Lifecycle and Research Domains

    model/method

    Large Language Model (LLM) research and engineering encompasses seven primary interdependent branches across model lifecycle, optimization, and deployment:

    1. Pre-Training: Foundational self-supervised training on massive web and specialized corpora (tens of billions to trillions of tokens). It includes the selection of transformer architectures (causal decoder, non-causal/prefix decoder, encoder-decoder, mixture-of-experts), pre-training objectives (full language modeling, prefix language modeling, masked language modeling, mixture-of-denoisers), scaling law analyses, and distributed training systems (data, tensor, pipeline, 3D/5D, and optimizer parallelism).

    2. Fine-Tuning: Post-pretraining adaptation to improve downstream task execution, intent adherence, and safety. This spans transfer learning on task-specific data, instruction-tuning using curated human datasets (e.g., P3, Super-NaturalInstructions, Flan) or synthetically generated multi-turn/complex instructions (e.g., Self-Instruct, Evol-Instruct), and human preference alignment via Reinforcement Learning with Human Feedback (RLHF), Direct Preference Optimization (DPO), and Reinforcement Learning from AI Feedback (RLAIF).

    3. Efficient LLMs: Methods designed to mitigate computational, memory, and latency bottlenecks during training, adaptation, and deployment. This includes Parameter-Efficient Fine-Tuning (PEFT; e.g., Adapters, LoRA, Prompt/Prefix Tuning, BitFit), model compression via Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT), model pruning (structured and unstructured), and context window interpolation.

    4. Inference and Prompting: Prompt engineering and execution strategies requiring no parameter modification, including zero-shot prompting, few-shot in-context learning (ICL), Chain-of-Thought (CoT), Self-Consistency, and Tree-of-Thought (ToT) reasoning.

    5. Evaluation and Benchmarking: Multi-dimensional assessment across Natural Language Understanding (NLU; classification, NLI, question answering), Natural Language Generation (NLG; summarization, translation), multi-task reasoning (MMLU, SuperGLUE, BIG-bench), coding, mathematics, factual truthfulness, toxicity, and bias.

    6. Applications and Augmentation: Extending LLM capabilities to specialized domains (medicine, finance, law, code synthesis, education) and extending foundational models with external memory, Retrieval-Augmented Generation (RAG), external API/tool manipulation (e.g., ToolLLM, Gorilla), multimodal perception (vision, audio, video), and autonomous multi-agent systems.

    7. Challenges and Limitations: Fundamental challenges including hallucinations (input-conflicting, context-conflicting, fact-conflicting), catastrophic forgetting in continual learning, safety and adversarial jailbreaking vulnerabilities, memorization/privacy leakage, and hardware inference latency.

  2. Knowl 2 — Pre-Trained Large Language Models Architecture and Specifications

    data/table

    Pre-trained LLMs (ge10textB\\ge 10\\text{B} parameters) demonstrate diverse architectural choices in normalization, positional embeddings, activation functions, and parallelization frameworks.

    Model Params Architecture Objective Tokens/Data Norm Positional Activation Parallelism
    T5 11B Enc-Dec Span Corruption 1T tokens Pre-RMS Relative ReLU D + M
    GPT-3 175B Causal-Dec Next Token 300B tokens LayerNorm Learned GeLU Model
    mT5 13B Enc-Dec Span Corruption 1T tokens Pre-RMS Relative ReLU -
    PanGu-α\alpha 200B Causal-Dec Next Token 1.1TB LayerNorm Query layer - 5D (D+OP+P+O+R)
    CPM-2 198B Enc-Dec MoE Span Corruption 2.6TB Pre-RMS Relative ReLU D + M
    BLOOM 176B Causal-Dec Next Token 366B tokens LayerNorm ALiBi GeLU D + T + P
    PaLM 540B Causal-Dec Next Token 780B tokens LayerNorm RoPE SwiGLU D + M
    Chinchilla 70B Causal-Dec Next Token 1.4T tokens Pre-RMS Relative GeLU -
    UL2 20B Enc-Dec MoD 1T tokens - - - Model
    GLM-130B 130B Non-Causal-Dec Blank Infill 400B tokens DeepNorm RoPE GeGLU Model
    LLaMA-2 70B Causal-Dec Next Token 2T tokens Pre-RMS RoPE SwiGLU -
    LLaMA-3.1 405B Causal-Dec Next Token 15T tokens Pre-RMS RoPE SwiGLU D + T + P + C
    DeepSeek-v2 67B MoE-Dec Next Token 8.1T tokens Pre-RMS RoPE SwiGLU D + P

    In this comparison, Enc-Dec denotes Encoder-Decoder, Causal-Dec denotes Causal Decoder, Non-Causal-Dec denotes Non-Causal Decoder, and MoE denotes Mixture-of-Experts. Parallelism modes include Data Parallelism (D), Tensor Parallelism (T), Pipeline Parallelism (P), Model Parallelism (M), Context Parallelism (C), Optimizer Parallelism (OP), and Rematerialization (R). MoD denotes Mixture-of-Denoisers.

  3. Knowl 3 — DeepSeek Empirical Compute-Optimal Scaling Laws for Non-Embedding LLM Parameters

    equation

    Empirical scaling law experiments across compute budgets CC ranging from 101710^{17} to 3×10203 \times 10^{20} training FLOPs (tested over varying model sizes, dataset sizes, batch sizes, and learning rates) define optimal non-embedding model parameters MoptM_{opt}, dataset size DoptD_{opt} (in tokens), batch size BoptB_{opt}, and learning rate ηopt\eta_{opt} as power-law functions of total compute CC:

    Bopt=0.2920⋅C0.3271B_{opt} = 0.2920 \cdot C^{0.3271}

    ηopt=0.3118⋅C−0.1250\eta_{opt} = 0.3118 \cdot C^{-0.1250}

    Mopt=Mbase⋅CaM_{opt} = M_{base} \cdot C^{a}

    Dopt=Dbase⋅CbD_{opt} = D_{base} \cdot C^{b}

    where the baseline constants and scaling exponents are:

    Mbase=0.1715,Dbase=5.8316,a=0.5243,b=0.4757M_{base} = 0.1715, \quad D_{base} = 5.8316, \quad a = 0.5243, \quad b = 0.4757

    These scaling relationships indicate that as the compute budget CC increases, the optimal batch size BoptB_{opt} increases while the optimal learning rate ηopt\eta_{opt} decreases, with compute allocated in near-equal proportions between non-embedding model capacity and training token volume (a≈ba \approx b).

  4. Knowl 4 — Architectural and Training Optimizations for Large Language Model Pre-Training

    empirical result

    Large-scale empirical investigations across pre-trained models identify key structural and optimization mechanisms that govern training stability, throughput, and downstream performance:

    • Parallel Attention and Feed-Forward: Structuring the transformer block such that multi-head attention and feed-forward operations execute in parallel on the normalized inputs: xl+1=xl+Attn(LN1(xl))+FF(LN2(xl))x_{l+1} = x_l + \text{Attn}(\text{LN}_1(x_l)) + \text{FF}(\text{LN}_2(x_l)) reduces training time and increases computational throughput by approximately 15%15\% without degrading performance compared to conventional sequential stacking (validated in PaLM, GPT-NeoX-20B, and CodeGen).

    • Weight Initialization & Residual Scaling: Initializing feed-forward output projection layers before residual addition with a scaling factor of 2Ld\frac{2}{L\sqrt{d}} (where LL is the layer depth and dd is the model hidden dimension) prevents activation values from expanding exponentially as model depth and width grow (GPT-NeoX-20B).

    • Training Stability & Divergence Recovery: Pre-layer normalization (Pre-Norm) and DeepNorm stabilize training for large networks, though Pre-Norm can degrade fine-tuned downstream performance compared to post-layer normalization beyond 100B parameters. Gradient norm spikes at the embedding layer can destabilize training; embedding layer gradient shrinking and the removal of additive biases in dense and normalization layers significantly enhance numerical stability. When loss spikes occur, optimal mitigation consists of rolling back to a checkpoint 100 steps prior and skipping 200--500 subsequent training batches.

    • Precision Tradeoffs: FP16 mixed precision frequently induces loss spikes due to limited dynamic range, whereas BF16 provides superior numerical stability across large-scale distributed runs by maintaining a dynamic range equivalent to FP32 while executing precision-sensitive steps (such as softmax and gradient accumulation) in full precision.

  5. Knowl 5 — Alignment Frameworks for LLMs: Reinforcement Learning from Human Feedback and Direct Supervised Fine-Tuning

    model/method

    To prevent toxic, untruthful, or unhelpful outputs, LLMs are aligned to satisfy Helpfulness, Honesty, and Harmlessness (HHH criteria) via three distinct technical paradigms:

    1. Reinforcement Learning from Human Feedback (RLHF):

      • Supervised Fine-Tuning (SFT): The base model is trained on curated human demonstration pairs.
      • Reward Modeling (RM): A classification model is trained on human preference rankings over multiple model-generated responses to output a scalar reward score.
      • Policy Optimization: The SFT model is optimized against the scalar reward using Proximal Policy Optimization (PPO) alongside a Kullback-Leibler (KL) divergence penalty to prevent the policy from diverging excessively from the initial model.
      • Rule/Separated Rewards: Frameworks such as Sparrow and LLaMA-2-Chat partition reward modeling into distinct helpfulness and safety reward models, utilizing rejection sampling combined with PPO.

    2. Direct Supervised Alignment (SFT-based Alignment):

      • Direct Preference Optimization (DPO): Bypasses explicit reward model training and reinforcement learning loops by parameterizing the reward function directly using the language model policy, optimizing a binary cross-entropy loss directly on preferred versus dispreferred completion pairs.
      • Reward Ranked Fine-Tuning (RAFT), Preference Ranking Optimization (PRO), and RRHF: Rank candidate responses via an external scoring function and update the model directly using standard supervised cross-entropy objectives on top-ranked outputs.
      • Chain-of-Hindsight (CoH): Presents both preferred and dispreferred sequences to the model with natural language feedback tokens, teaching the model to condition directly on positive evaluative language.

    3. Synthetic and Principle-Driven Feedback (RLAIF):

      • Constitutional AI / RLAIF: Replaces human annotators with an automated model guided by a constitution of behavioral principles to self-critique, revise, and generate preference data for alignment.
      • Self-Correction Prompting: Introduces verification and moral safety reasoning steps into Chain-of-Thought prompts prior to final answer generation, reducing unsafe outputs without model weight modification.

  6. Knowl 6 — Parameter-Efficient Fine-Tuning (PEFT) Strategies for Large Language Models

    model/method

    Parameter-Efficient Fine-Tuning (PEFT) techniques adapt pre-trained LLMs to downstream tasks by modifying or inserting a small fraction (0.001%−3%0.001\% - 3\%) of parameters while freezing the base backbone:

    • Adapter Tuning: Inserts lightweight bottleneck layers comprising a down-projection matrix Wdown∈Rd×mW_{down} \in \mathbb{R}^{d \times m}, a non-linear activation σ(⋅)\sigma(\cdot), and an up-projection matrix Wup∈Rm×dW_{up} \in \mathbb{R}^{m \times d} (with bottleneck dimension m≪dm \ll d) either sequentially within attention and feed-forward blocks or in parallel. Mixture-of-Adapters (AdaMix) routes inputs dynamically across multiple adapter sub-modules during training and averages their weights during inference to eliminate runtime overhead.

    • Low-Rank Adaptation (LoRA): Freezes the pre-trained weight matrix W0∈Rd×kW_0 \in \mathbb{R}^{d \times k} and models the parameter update via low-rank decomposition: W=W0+ΔW=W0+αrBAW = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A where B∈Rd×rB \in \mathbb{R}^{d \times r} and A∈Rr×kA \in \mathbb{R}^{r \times k} are trainable low-rank matrices with rank r≪min⁡(d,k)r \ll \min(d, k), and α\alpha is a constant scaling hyperparameter. The learned matrices are fused into W0W_0 prior to inference, introducing zero additional latency.

    • Prompt Tuning and Prefix Tuning: Prompt Tuning prepends trainable continuous virtual token embeddings to the input sequence (P-Tuning prepends them to every transformer layer via a prompt encoder MLP). Prefix Tuning prepends trainable key and value vectors directly to the transformer attention mechanisms at all layers, allowing subsequent tokens to attend to virtual task-specific prefixes.

    • BitFit (Bias Tuning): Freezes all transformer weight matrices and optimizes exclusively the additive bias terms throughout the network, matching full fine-tuning performance on small-to-medium dataset regimes.

  7. Knowl 7 — LLM Compression Methods: Quantization, Quantization-Aware Training, and Pruning

    model/method

    To reduce the massive GPU memory and compute footprints required for LLM inference (e.g., ≥350GB\ge 350\text{GB} in FP16 for 175B models), three primary model compression classes are utilized:

    1. Post-Training Quantization (PTQ): Operates on pre-trained weights without extensive retraining:

      • LLM.int8(): Identifies activation outliers (emergent large-magnitude features appearing in a small subset of channels above 6B scale) and separates them into a high-precision FP16 matrix multiplication branch, while quantizing the remaining 99.9%99.9\% of weights and activations to INT8.
      • SmoothQuant: Applies per-channel scaling transformations that divide activations by a smoothing factor ss and multiply weights by ss, migrating quantization difficulty from activations to weights to allow uniform INT8 matrix multiplications for both.
      • OPTQ / GPTQ: Employs second-order error compensation based on Optimal Brain Compression (OBC) to quantize weights layer-by-layer in arbitrary order using fast Cholesky updates.
      • Outlier-Aware Weight Quantization (OWQ): Assigns higher precision to weight columns corresponding to high-magnitude activation channels.

    2. Quantization-Aware Training (QAT): Adjusts weights during training or fine-tuning under quantized constraints:

      • QLoRA: Quantizes the frozen base model to 4-bit NormalFloat (NF4; an information-theoretically optimal quantile distribution for normally distributed weights), applies Double Quantization (quantizing the quantization constants), uses Paged Optimizers, and trains LoRA adapter weights in 16-bit precision.
      • PEQA: Compresses fully connected layers to low bit-widths and fine-tunes only the module-level scalar quantization parameters.

    3. Model Pruning:

      • Unstructured Pruning (Wanda): Prunes weights independently per output feature by evaluating importance as ∣Wij∣⋅∥Xj∥2|W_{ij}| \cdot \|X_j\|_2 (the product of weight magnitude and the L2L_2-norm of corresponding input activations) without requiring fine-tuning.
      • Structured Pruning (LLM-Pruner): Identifies structural dependencies across coupled transformer layers and groups of hidden states, prunes non-critical groups (rows, columns, attention heads), and recovers lost task capacity via post-pruning LoRA fine-tuning.

  8. Knowl 8 — Augmented LLM Paradigms: Retrieval-Augmentation, External Tools, and Autonomous Agents

    model/method

    Augmented LLMs integrate language models with external data stores, execution environments, and memory architectures:

    • Retrieval-Augmented Generation (RAG): Addresses knowledge staleness and hallucinations by dynamically retrieving relevant documents from an external corpus via dense (e.g., BERT-based) or sparse (BM25) indexers.

      • Zero-Shot & Active Retrieval: RAG passes retrieved context directly into the prompt. Active retrieval systems (e.g., Forward-Looking Active REtrieval / FLARE) monitor token generation confidence and trigger external retrieval only when low-confidence tokens are generated.
      • Pre-Trained / Integrated RAG: RETRO incorporates frozen chunks from an external 2-trillion token database into LLM layers using chunked cross-attention during pre-training, enabling a 7.5B model to rival a 280B standard model.
      • Fusion-in-Decoder (FiD): Processes retrieved passages independently in the encoder and fuses their representations across decoder cross-attention layers to scale context without quadratic self-attention cost.

    • Tool-Augmented LLMs: Enhances LLMs with external software APIs, interpreters, and symbolic tools.

      • Zero-Shot Tool Use: Models interpret API documentation directly in-context or represent available tools as special virtual tokens (e.g., ToolkenGPT).
      • Trained Tool Manipulation: ToolLLM and Gorilla fine-tune models on extensive API datasets (e.g., RapidAPI) using Depth-First Search Decision Trees (DFSDT) to generate multi-step planning ground truths.

    • LLM-Powered Autonomous Agents: LLMs serve as central cognitive controllers executing an action loop: Plan →\rightarrow Tool Selection →\rightarrow Execute →\rightarrow Inspect →\rightarrow Generate.

      • Planning: Elicited via Chain-of-Thought, Tree-of-Thought, Reasoning via Planning (RAP; repurposing an LLM as a world model to simulate future outcomes), or retrospective reflection (Retroformer).
      • Memory Systems: Maintain episodic short-term memory (recent interactions) and reflective long-term memory (summarized failure evaluations and feedback, as in Reflexion).
      • Physical Grounding: Systems like SayCan combine high-level LLM task planning ('Say') with learned robotic affordance value functions ('Can') to ensure real-world executable trajectories.

  9. Knowl 9 — Multimodal Large Language Model Architectures and Cross-Modal Alignment

    model/method

    Multimodal Large Language Models (MLLMs) adapt text-based LLMs to perceive and reason over visual, auditory, and video inputs via three structural paradigms:

    1. Cross-Attention Pre-Training: Incorporates cross-attention layers into the LLM architecture (e.g., Flamingo) to condition language generation on intermediate visual features extracted from a frozen vision encoder.

    2. Learnable Alignment Interfaces (Bottleneck / Projection):

      • Querying Transformer (Q-Former; BLIP-2): Uses a two-stage pre-training paradigm: Stage 1 bootstraps vision-language representation learning using learnable queries interacting with a frozen visual encoder; Stage 2 connects the Q-Former output to a frozen LLM for generative text synthesis.
      • Linear Projection & Adapters: MiniGPT-4 trains only a single linear projection layer between a frozen visual backbone and LLM. LLaMA-Adapter and LaVIN insert parameter-efficient visual adapters (such as mixture-of-modality adapters) with zero-initialized attention into frozen LLM blocks.

    3. Multimodal Prompting & Rationale Generation:

      • Multimodal Chain-of-Thought (Multimodal-CoT): Divides multimodal reasoning into two explicit stages: first generating text rationales from image and text inputs, then conditioning the answer inference step on the original inputs concatenated with the generated rationale.

  10. Knowl 10 — Comparative Benchmark Performance of Top-Performing LLMs across NLP and Reasoning Tasks

    data/table

    Evaluation of leading LLMs across standardized Natural Language Understanding (NLU), Natural Language Generation (NLG), multi-task reasoning, and problem-solving benchmarks highlights relative capabilities under zero-shot and few-shot evaluation settings.

    Task Benchmark Top-1 Model (Score) Top-2 Model (Score) Top-3 Model (Score)
    Multi-Task MMLU GPT-4 (86.4; 5-shot) Gemini Ultra (83.7; 5-shot) Flan-PaLM-2 Large (81.2; 5-shot)
    Multi-Task BIG-bench Chinchilla 70B (65.1; 5-shot) Gopher 280B (53.97; 5-shot) PaLM 540B (53.7; 5-shot)
    NLU SuperGLUE ERNIE 3.0 12B (90.6; -) PaLM 540B finetuned (90.4; -) T5 11B (88.9; -)
    Story Reasoning HellaSwag GPT-4 (95.3; 10-shot) Gemini Ultra (87.8; 10-shot) PaLM-2 Large (86.8; 1-shot)
    Story Reasoning StoryCloze GPT-3 175B (87.7; few-shot) PaLM-2 Large (87.4; 1-shot) OPT 175B (79.82; -)
    Physical Commonsense PIQA PaLM-2 Large (85.0; 1-shot) LLaMA 65B (82.8; 0-shot) MT-NLG 530B (81.99; 0-shot)
    Open-Domain QA TriviaQA PaLM-2 Large (86.1; 1-shot) LLaMA-2 70B (85.0; 1-shot) PaLM 540B (81.4; 1-shot)
    Reading Comprehension BoolQ PaLM 540B finetuned (92.2; -) T5 11B (91.2; -) PaLM-2 Large (90.9; 1-shot)
    Commonsense Reasoning WinoGrande GPT-4 (87.5; 5-shot) PaLM-2 Large (83.0; 1-shot) PaLM 540B (81.1; 0-shot)
    Mathematical Reasoning GSM8K GPT-4 (92.0; 5-shot) PaLM-2 Large (80.7; 8-shot) U-PaLM 540B (58.5; -)
    Mathematical Reasoning MATH Gemini Ultra (53.2; 4-shot) PaLM-2 Large (34.3; 4-shot) LLaMA-2 65B (13.5; 4-shot)
    Coding Synthesis HumanEval Gemini Ultra finetuned (74.4; 0-shot) GPT-4 (67.0; 0-shot) Code Llama 34B (48.8; 0-shot)
    Truthfulness TruthfulQA LLaMA 65B (57.0; -) - -

    Across multi-task benchmarks (MMLU) and complex mathematical/coding tasks (GSM8K, MATH, HumanEval), closed frontier models (GPT-4, Gemini Ultra) consistently achieve top performance, while instruction-tuned and compute-scaled open models (PaLM-2, LLaMA-2, Chinchilla) exhibit competitive zero-shot and few-shot scores across commonsense reasoning, QA, and reading comprehension.

  11. Knowl 11 — Context Length Extension and Positional Extrapolation in LLMs

    model/method

    LLMs trained on fixed context windows fail to generalize to longer sequence lengths at inference without degradation. Three primary methodological classes enable context extension:

    1. Position Interpolation (PI): Directly interpolates position indices within the pre-trained context window rather than extrapolating beyond it. For an extension factor s>1s > 1, Position Interpolation scales the position index mm to m′=m/sm' = m / s, mapping input sequence positions into the range [0,Ltrain][0, L_{train}]. This requires as few as 1,000 fine-tuning steps to preserve short-context performance while extending attention up to 32k32\text{k} tokens. Variants include Power Scaling (Giraffe) and Neural Tangent Kernel (NTK)-aware interpolation (YaRN), which adjusts high-frequency and low-frequency components of Rotary Positional Embeddings (RoPE) differently.

    2. Efficient and Sparse Attention Mechanisms:

      • Transient Global Attention (LongT5 / TGlobal): Blends local token sliding-window attention with windowed token averaging to create global attention states, enabling sequence extrapolation to 16k tokens.
      • Dilated Attention (LongNet): Employs dilated attention patterns with exponentially increasing segment lengths, scaling sequence length capacity up to 1 billion tokens.
      • Shift-Short Attention (LongLoRA): Enforces non-overlapping group attention and shifted group attention during parameter-efficient fine-tuning to approximate full attention while retaining dense global attention at inference.

    3. Training-Free Context Extrapolation:

      • LM-Infinite: Implements Λ\Lambda-shaped attention masks (restricting attention to initial prompt tokens and a local sliding window) within the original pre-trained positional bounds.
      • Parallel Context Windows (PCW): Partitions long text inputs into multiple parallel chunks within the pre-trained window size and assigns identical positional encodings to each chunk, aggregating attention across context windows without fine-tuning.

  12. Knowl 12 — Structural Taxonomy of LLM Hallucinations and Core Vulnerabilities

    limitation

    Despite large-scale pre-training and alignment, LLMs remain subject to systemic failures across hallucination categories, security vulnerabilities, and data retention:

    1. Hallucination Taxonomy:

      • Input-Conflicting Hallucination: Generated outputs directly contradict or diverge from the context, constraints, or factual premises supplied in the user's prompt.
      • Context-Conflicting Hallucination: The model produces statements that internally contradict previously generated content within the same conversational or document thread.
      • Fact-Conflicting Hallucination: The generated text violates established real-world facts and commonsense knowledge despite appearing coherent and plausible.

    2. Catastrophic Forgetting: Fine-tuning LLMs on specialized downstream tasks causes rapid decay and loss of generalized pre-trained capabilities. Mitigations require concatenating a subset of general pre-training data during every fine-tuning iteration or employing modular architectures (such as Randomly Routed Experts in PanGu-Σ\Sigma) that isolate domain updates to specific subnetworks.

    3. Security and Jailbreak Susceptibility: Even aligned models remain vulnerable to prompt injection, jailbreaking attacks, and adversarial prompting that bypass HHH safeguards. Adversarial fine-tuning can increase safety robustness but often induces an 'alignment tax' by degrading standard downstream generalization performance.

    4. Training Data Memorization: Large models exhibit significant verbatim data memorization; for example, PaLM at the 540B parameter scale memorizes approximately 2.4%2.4\% of its training data, enabling privacy extraction attacks where private identifiers, source code, or personal data can be reconstructed.

Coverage note — The extracted knowls systematically cover the paper's core contributions: the overarching research taxonomy, pre-trained model specifications and architectures, empirical pre-training optimizations, compute-optimal scaling laws, alignment frameworks (RLHF vs SFT-based), PEFT techniques, model compression/quantization methods, augmented LLMs (RAG, tools, agents), multimodal architectures, benchmark performance synthesis, context extension methods, and core failure modes/challenges. Individual descriptive domain summaries (such as the high-level application narratives for education, law, and medicine) were subsumed into the general taxonomy and application knowls to maintain technical density.

References

  1. 1.A. Chernyavskiy, D. Ilvovsky, P. Nakov, Transformers:“the end of history” for natural language processing?, in: Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part III 21, Springer, 2021, pp. 677–693.
  2. 2.A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. Bowman, Superglue: A stickier benchmark for general-purpose language understanding systems, Advances in neural information processing systems 32 (2019).
  3. 3.D. Adiwardana, M.-T. Luong, D. R. So, J. Hall, N. Fiedel, R. Thoppilan, Z. Yang, A. Kulshreshtha, G. Nemade, Y. Lu, et al., Towards a human-like open-domain chatbot, arXiv preprint arXiv:2001.09977 (2020).
  4. 4.B. A. y Arcas, Do large language models understand us?, Daedalus 151 (2) (2022) 183–197.
  5. 5.A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (8) (2019) 9.
  6. 6.T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901.
  7. 7.J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
  8. 8.M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, in: NAACL-HLT, Association for Computational Linguistics, 2018, pp. 2227–2237.
  9. 9.M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019).
  10. 10.C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Research 21 (1) (2020) 5485–5551.
  11. 11.L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Raffel, mt5: A massively multilingual pre-trained text-to-text transformer, arXiv preprint arXiv:2010.11934 (2020).
  12. 12.Z. Zhang, Y. Gu, X. Han, S. Chen, C. Xiao, Z. Sun, Y. Yao, F. Qi, J. Guan, P. Ke, et al., Cpm-2: Large-scale cost-effective pre-trained language models, AI Open 2 (2021) 216–224.
  13. 13.T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, ´ R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al., Bloom: A 176b-parameter open-access multilingual language model, arXiv preprint arXiv:2211.05100 (2022).
  14. 14.S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al., Opt: Open pre-trained transformer language models, arXiv preprint arXiv:2205.01068 (2022).
  15. 15.A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., Palm: Scaling language modeling with pathways, arXiv preprint arXiv:2204.02311 (2022).
  16. 16.H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, X. Dehghani, S. Brahma, et al., Scaling instruction-finetuned language models, arXiv preprint arXiv:2210.11416 (2022).
  17. 17.V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al., Multitask prompted training enables zero-shot task generalization, arXiv preprint arXiv:2110.08207 (2021).
  18. 18.Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al., Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 5085–5109.
  19. 19.Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, H. Hajishirzi, Self-instruct: Aligning language model with self generated instructions, arXiv preprint arXiv:2212.10560 (2022).
  20. 20.L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, Advances in Neural Information Processing Systems 35 (2022) 27730–27744.

Citation

MLA
Naveed, H., et al. “A Comprehensive Overview of Large Language Models”. arXiv, 2023, http://arxiv.org/abs/2307.06435v10.
APA
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023). A Comprehensive Overview of Large Language Models. arXiv. http://arxiv.org/abs/2307.06435v10
Chicago
Naveed, H., A. U. Khan, S. Qiu, et al. 2023. “A Comprehensive Overview of Large Language Models”. arXiv. http://arxiv.org/abs/2307.06435v10.
Harvard
Naveed, H. et al. (2023) “A Comprehensive Overview of Large Language Models”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2307.06435v10.
Vancouver
1. Naveed H, Khan AU, Qiu S, Saqib M, Anwar S, Usman M, Akhtar N, Barnes N, Mian A (2023) A Comprehensive Overview of Large Language Models. arXiv

BibTeX

@article{naveed2023comprehensive,
  title = {A Comprehensive Overview of Large Language Models},
  author = {Naveed, Humza and Khan, Asad Ullah and Qiu, Shi and Saqib, Muhammad and Anwar, Saeed and Usman, Muhammad and Akhtar, Naveed and Barnes, Nick and Mian, Ajmal},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2307.06435v10},
  eprint = {2307.06435}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/