Back

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

SHIYANG LIXiaoyong JinYao XuanXiyou ZhouWenhu ChenYu-Xiang WangXifeng Yan

article2019NeurIPS2,040 citations

Proposes convolutional self-attention and a LogSparse Transformer architecture that reduces memory complexity to O(L(log L)^2), solving the key challenges of local context insensitivity and quadratic memory bottlenecks in long-sequence time series forecasting.

Listen

Accurate time series forecasting is critical for operational planning, energy management, and resource allocation across industries like retail, utilities, and transportation. While deep neural networks have largely replaced traditional statistical methods by learning patterns across large collections of related time series, standard recurrent neural networks struggle to capture extended long-term dependencies. The standard Transformer model resolves this by allowing direct access to past historical context, but it introduces two severe limitations in time series forecasting: it evaluates individual points without recognizing local context and shapes, making it sensitive to anomalies, and its memory footprint grows quadratically with sequence length, making long-sequence forecasting computationally prohibitive.

The article evaluates whether adapting the Transformer architecture with context-aware mechanisms and sparse attention can significantly improve forecasting accuracy while overcoming these memory and locality bottlenecks.

The authors proposed and tested two enhancements: convolutional self-attention, which extracts local shape patterns via causal convolution, and a LogSparse Transformer, which selectively samples historical context to reduce memory requirements. The approach was validated through synthetic experiments designed to isolate long-term dependency retention, alongside empirical evaluations across standard real-world benchmarks covering electricity consumption, freeway traffic, solar generation, wind power, and the M4 hourly competition dataset across multiple time horizons.

The experimental findings show significant performance and computational improvements. First, the proposed Transformer architecture consistently outperformed traditional statistical techniques, matrix factorization, and recurrent neural network baselines across all datasets; for example, on the hourly traffic dataset, it reduced the median quantile loss from 0.161 (recurrent baseline) to 0.122. Second, synthetic testing confirmed that while recurrent networks experienced severe performance degradation as history lengths exceeded 96 steps, the Transformer maintained steady accuracy. Third, incorporating causal convolutions yielded up to a 9% relative error reduction on complex traffic data and accelerated model training convergence. Finally, the LogSparse attention mechanism matched or exceeded the accuracy of full-attention models on fine-grained series while consuming significantly less memory, allowing the model to process up to 768 time steps under constraints where standard models were capped at under 300 steps.

These results demonstrate that organizations can deploy high-capacity Transformer architectures for long-horizon and fine-grained forecasting without experiencing prohibitive hardware or memory costs. Modeling local shape patterns mitigates the risk of misinterpreting isolated spikes or anomalies, translating into more reliable forecasts for capacity planning and resource distribution.

Organizations handling large-scale, high-frequency time series data should consider adopting context-aware, sparse Transformer models over recurrent architectures for production forecasting. Prior to broad deployment, teams should conduct internal pilot tests to tune convolutional kernel sizes and determine appropriate sparsity levels tailored to their specific data granularity. Further work is recommended to optimize sparsity strategies for smaller datasets and implement dedicated sparse matrix GPU kernels to maximize runtime efficiency.

The primary limitation of the study is that the sparse attention implementation was evaluated via masking rather than low-level custom sparse computation kernels, meaning observed memory reductions reflect algorithmic bounds rather than optimized execution runtimes. Additionally, performance gains were less pronounced on simple datasets where basic seasonal covariates already provided sufficient context. Nonetheless, there is high confidence in the core finding that context-aware sparse attention improves forecasting accuracy and scalability on complex, long-horizon data.

University of California, Santa Barbaraattention mechanismelectricity-consumptionencoder-decoderTime SeriesAttentionTransformer
arXiv: 1907.00235
Cover for Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

Abstract

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length LL, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only O(L(log⁡L)2)O(L(\log L)^{2}) memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.

Topics

attention mechanismelectricity-consumptionencoder-decoderrandom access memorysynthetic dataforecasting problems

Keywords

Time SeriesAttentionTransformerForecastingTime Series ForecastingMemorySelf-attentionModelarXivarXiv Preprint arXiv

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Background
  • 4 Methodology
  • 4.1 Enhancing the locality of Transformer
  • 4.2 Breaking the memory bottleneck of Transformer
  • 5 Experiments
  • 5.1 Synthetic datasets
  • 5.2 Real-world datasets
  • 6 Conclusion
  • References
  • A Supplementary Materials
  • A.1 Proof of Theorem 1
  • A.2 Training
  • A.3 Evaluation
  • A.4 Implementation of sparse attention and its memory cost
  • A.5 Visualization of attention matrix

Knowls

  1. Knowl 1 — Probabilistic Autoregressive Time Series Forecasting Formulation

    definition

    Given a collection of NN univariate time series {zi,1:t0}i=1N\{z_{i,1:t_0}\}_{i=1}^N, where zi,teextnullextandzi,tinRz_{i,t} e ext{null} ext{ and } z_{i,t} in \mathbb{R} denotes the observation of time series ii at relative time step tt, and associated time-based covariate vectors {xi,1:t0+τ}i=1N\{x_{i,1:t_0+\tau}\}_{i=1}^N with dimension dd known across both the conditioning history 1:t01:t_0 and the forecast horizon t0+1:t0+τt_0+1:t_0+\tau (such as calendar features), the goal is to model the joint conditional distribution of the future trajectory:

    p(zi,t0+1:t0+τ∣zi,1:t0,xi,1:t0+τ;Φ)=∏t=t0+1t0+τp(zi,t∣zi,1:t−1,xi,1:t;Φ)p(z_{i,t_0+1:t_0+\tau} \mid z_{i,1:t_0}, x_{i,1:t_0+\tau}; \Phi) = \prod_{t=t_0+1}^{t_0+\tau} p(z_{i,t} \mid z_{i,1:t-1}, x_{i,1:t}; \Phi)

    where Φ\Phi denotes the learnable parameters shared across all NN time series.

    The autoregressive step model p(zt∣z1:t−1,x1:t;Φ)p(z_t \mid z_{1:t-1}, x_{1:t}; \Phi) operates on an augmented step vector yty_t and historical matrix YtY_t:

    yt=[zt−1∘xt]∈Rd+1,Yt=[y1,…,yt]T∈Rt×(d+1)y_t = [z_{t-1} \circ x_t] \in \mathbb{R}^{d+1}, \quad Y_t = [y_1, \dots, y_t]^T \in \mathbb{R}^{t \times (d+1)}

    where [⋅∘⋅][\cdot \circ \cdot] denotes vector concatenation. A neural sequence model f(Yt)f(Y_t) is trained to output the parameters of the predictive distribution of ztz_t conditioned on YtY_t.

  2. Knowl 2 — Convolutional Self-Attention Mechanism

    model/method

    Canonical self-attention matches queries and keys using point-wise linear projections (kernel size 1), making it insensitive to local context and prone to misinterpreting anomalies or local shapes. Convolutional self-attention replaces point-wise projections with 1D causal convolutions of kernel size k≥1k \ge 1 with stride 1 (and left-padding of k−1k-1 to prevent future information leakage) when producing query and key matrices for each attention head h∈{1,…,H}h \in \{1, \dots, H\} from input sequence Y∈RL×(d+1)Y \in \mathbb{R}^{L \times (d+1)}:

    Qh=CausalConv1Dk(Y,WhQ)∈RL×dkQ_h = \text{CausalConv1D}_k(Y, W_h^Q) \in \mathbb{R}^{L \times d_k}

    Kh=CausalConv1Dk(Y,WhK)∈RL×dkK_h = \text{CausalConv1D}_k(Y, W_h^K) \in \mathbb{R}^{L \times d_k}

    Vh=YWhV∈RL×dvV_h = Y W_h^V \in \mathbb{R}^{L \times d_v}

    where WhQ,WhK∈Rk×(d+1)×dkW_h^Q, W_h^K \in \mathbb{R}^{k \times (d+1) \times d_k} are learnable convolutional kernel weights, and WhV∈R(d+1)×dvW_h^V \in \mathbb{R}^{(d+1) \times d_v} is a linear projection weight matrix.

    The head output OhO_h is computed via scaled dot-product attention with causal mask matrix M∈RL×LM \in \mathbb{R}^{L \times L} (Mi,j=0M_{i,j} = 0 for j≤ij \le i and Mi,j=−∞M_{i,j} = -\infty for j>ij > i):

    Oh=Attention(Qh,Kh,Vh)=softmax(QhKhTdk+M)VhO_h = \text{Attention}(Q_h, K_h, V_h) = \text{softmax}\left(\frac{Q_h K_h^T}{\sqrt{d_k}} + M\right) V_h

    When k=1k=1, convolutional self-attention degrades to canonical point-wise self-attention.

  3. Knowl 3 — LogSparse Self-Attention Mechanism

    model/method

    In standard full self-attention over sequence length LL, every position ll attends to all past positions Il={j:j≤l}I_l = \{j : j \le l\}, requiring O(L2)O(L^2) memory and computation.

    LogSparse self-attention restricts each position ll at layer kk to only attend to past positions at exponentially increasing distance intervals (powers of 2) as well as to itself:

    Ilk={l−2⌊log⁡2l⌋,l−2⌊log⁡2l⌋−1,…,l−20,l}I_l^k = \left\{l - 2^{\lfloor \log_2 l \rfloor}, l - 2^{\lfloor \log_2 l \rfloor - 1}, \dots, l - 2^0, l\right\}

    where ⌊⋅⌋\lfloor \cdot \rfloor denotes the floor function.

    Under this indexing scheme:

    1. The number of dot products computed per cell per layer is ∣Ilk∣=⌊log⁡2l⌋+2=O(log⁡L)|I_l^k| = \lfloor \log_2 l \rfloor + 2 = O(\log L).
    2. The per-layer memory footprint drops from O(L2)O(L^2) to O(Llog⁡L)O(L \log L).
    3. Stacking O(log⁡L)O(\log L) layers guarantees that information from any cell j≤lj \le l can reach cell ll, yielding an overall space complexity of O(L(log⁡L)2)O(L (\log L)^2).

  4. Knowl 4 — Information Flow and Path Multiplicity in LogSparse Self-Attention

    theoretical result

    Let Ilk={l−2⌊log⁡2l⌋,l−2⌊log⁡2l⌋−1,…,l−20,l}I_l^k = \{l - 2^{\lfloor \log_2 l \rfloor}, l - 2^{\lfloor \log_2 l \rfloor - 1}, \dots, l - 2^0, l\} be the receptive index set of cell ll at layer kk, and let SlKS_l^K be the set of cell indices whose information reaches cell ll after stacking KK LogSparse layers through directed paths Pjl=(j,p1,p2,…,pK−1,l)P_{jl} = (j, p_1, p_2, \dots, p_{K-1}, l) with j∈Ip11,p1∈Ip22,…,pK−1∈IlKj \in I_{p_1}^1, p_1 \in I_{p_2}^2, \dots, p_{K-1} \in I_l^K.

    1. Path Existence: For any sequence length LL, any position l∈{1,…,L}l \in \{1, \dots, L\}, and any predecessor j≤lj \le l, there exists at least one valid directed information path from cell jj to cell ll when stacking ⌊log⁡2l⌋+1\lfloor \log_2 l \rfloor + 1 layers. Therefore, stacking ⌊log⁡2L⌋+1\lfloor \log_2 L \rfloor + 1 layers ensures full receptive coverage Sl⌊log⁡2L⌋+1={j:j≤l}S_l^{\lfloor \log_2 L \rfloor + 1} = \{j : j \le l\} for all ll.

    2. Path Multiplicity: For any j<lj < l, the number of distinct valid directed paths from cell jj to cell ll grows super-exponentially with temporal distance at a rate of O(⌊log⁡2(l−j)⌋!)O(\lfloor \log_2(l - j) \rfloor !), ensuring dense multi-path information flow between distant temporal steps.

  5. Knowl 5 — Local and Restart Extensions for Sparse Attention

    model/method

    Two structural extensions can be combined with LogSparse self-attention without increasing the asymptotic O(L(log⁡L)2)O(L(\log L)^2) space complexity:

    1. Local Attention: Each cell ll densely attends to all cells within an immediate left neighborhood window of size O(log⁡2L)O(\log_2 L) (i.e., {l−O(log⁡2L),…,l}\{l - O(\log_2 L), \dots, l\}) to capture fine-grained local patterns and trends. Beyond this local window, the LogSparse exponential stride index set is applied.

    2. Restart Attention: The input sequence of length LL is partitioned into consecutive subsequences of length Lsub∝LL_{\text{sub}} \propto L (e.g., Lsub=L/8L_{\text{sub}} = L/8). The LogSparse attention index pattern restarts from zero at the beginning of each subsequence.

    Both local and restart attention reduce the shortest path length between cells and increase the number of information routing paths while maintaining an O(Llog⁡L)O(L \log L) per-layer memory budget.

  6. Knowl 6 — Decoder-Only Transformer Architecture for Probabilistic Time Series Forecasting

    model/method

    The forecasting model uses an autoregressive Transformer decoder architecture:

    1. Input Layer: Continuous covariates (e.g., age, hour-of-the-day, day-of-the-week) are normalized to zero mean and unit variance. Categorical time series IDs are passed through a learned embedding matrix of dimension 20. Learnable position embeddings of dimension 20 are added to the time series ID embeddings, and the sum is concatenated with other covariates and the past observation zt−1z_{t-1} to form the input representation.

    2. Decoder Stacks: The input passes through KK stacked layers (K=3K=3 for coarse datasets, K=6K=6 or 1010 for fine datasets), each containing multi-head convolutional/LogSparse self-attention with H=8H=8 heads followed by a position-wise feedforward network with two linear layers and a ReLU activation.

    3. Probabilistic Output: A linear output layer maps the top hidden states to the parameters of the chosen likelihood distribution (e.g., mean μt\mu_t and scale σt\sigma_t of a Gaussian distribution, using a softplus activation for positive parameters), combined with affine scale handling.

    4. Training: Parameters are optimized to maximize the full Gaussian log-likelihood across time steps using BERTAdam (Adam with linear warmup and learning rate annealing) to stabilize convergence.

  7. Knowl 7 — Quantile Loss Evaluation Metric for Probabilistic Forecasting

    equation

    Given observed values xt(i)x_t^{(i)} and predicted empirical ρ\rho-quantiles x^t(i)\hat{x}_t^{(i)} for quantile parameter ρ∈(0,1)\rho \in (0, 1) across time series ii and time steps tt, the normalized ρ\rho-quantile loss Rρ(x,x^)R_\rho(x, \hat{x}) is defined as:

    Rρ(x,x^)=2∑i,tDρ(xt(i),x^t(i))∑i,t∣xt(i)∣R_\rho(x, \hat{x}) = \frac{2 \sum_{i, t} D_\rho(x_t^{(i)}, \hat{x}_t^{(i)})}{\sum_{i, t} |x_t^{(i)}|}

    where the point-wise asymmetric quantile loss Dρ(x,x^)D_\rho(x, \hat{x}) is:

    Dρ(x,x^)=(ρ−I{x≤x^})(x−x^)D_\rho(x, \hat{x}) = \left(\rho - \mathbb{I}_{\{x \le \hat{x}\}}\right)(x - \hat{x})

    and I{x≤x^}\mathbb{I}_{\{x \le \hat{x}\}} is an indicator function equal to 1 if x≤x^x \le \hat{x} and 0 otherwise. Typical evaluation points include the median forecast (rho=0.5\\rho=0.5, denoted R0.5R_{0.5}) and the tail risk forecast (rho=0.9\\rho=0.9, denoted R0.9R_{0.9}).

  8. Knowl 8 — Robustness to Long-Term Dependencies on Synthetic Piecewise Sinusoidal Data

    empirical result

    To test the capability of capturing long-term dependencies, models were evaluated on synthetic piecewise sinusoidal signals defined by:

    f(x)={A1sin⁡(πx/6)+72+Nx,x∈[0,12)A2sin⁡(πx/6)+72+Nx,x∈[12,24)A3sin⁡(πx/6)+72+Nx,x∈[24,t0)A4sin⁡(πx/12)+72+Nx,x∈[t0,t0+24)f(x) = \begin{cases} A_1 \sin(\pi x / 6) + 72 + N_x, & x \in [0, 12) \\ A_2 \sin(\pi x / 6) + 72 + N_x, & x \in [12, 24) \\ A_3 \sin(\pi x / 6) + 72 + N_x, & x \in [24, t_0) \\ A_4 \sin(\pi x / 12) + 72 + N_x, & x \in [t_0, t_0 + 24) \end{cases}

    where x∈Zx \in \mathbb{Z}, A1,A2,A3∼Uniform(0,60)A_1, A_2, A_3 \sim \text{Uniform}(0, 60), A4=max⁡(A1,A2)A_4 = \max(A_1, A_2), and Nx∼N(0,1)N_x \sim \mathcal{N}(0, 1). The task requires predicting the final 24 steps ([t0,t0+24)[t_0, t_0+24)) given the preceding t0t_0 history points, requiring the model to remember A1A_1 and A2A_2 across t0−24t_0-24 steps of irrelevant signal A3A_3.

    Evaluating across history lengths t0∈{24,48,72,96,120,144,168,192}t_0 \in \{24, 48, 72, 96, 120, 144, 168, 192\}:

    • At t0=24t_0 = 24, both DeepAR (3-layer LSTM with hidden size h∈{20,40,80,140,200}h \in \{20, 40, 80, 140, 200\}) and a 3-layer canonical Transformer achieve low R0.5R_{0.5} loss (≈0.02\approx 0.02).
    • As t0t_0 increases beyond 9696 up to 192192, DeepAR's R0.5R_{0.5} loss degrades sharply to >0.10> 0.10 across all hidden sizes.
    • The Transformer maintains a constant low R0.5≈0.02R_{0.5} \approx 0.02 across all t0∈[24,192]t_0 \in [24, 192].

  9. Knowl 9 — Benchmark Comparison of Forecasting Methods across Granularities

    data/table

    Probabilistic forecasting performance (R0.5/R0.9R_{0.5} / R_{0.9} loss) evaluated on real-world datasets across rolling 1-day (1d), 7-day ahead (7d), 2-day ahead (2d), and 30-day ahead (30d) horizons. Baseline models include ARIMA, ETS, TRMF (matrix factorization), DeepAR (autoregressive LSTM), and DeepState (state-space RNN).

    Dataset ARIMA ETS TRMF DeepAR DeepState Ours
    electricity-c (1d) 0.154/0.102 0.101/0.077 0.084/- 0.075/0.040 0.083/0.056 0.059/0.034
    electricity-c (7d) 0.283/0.109 0.121/0.101 0.087/- 0.082/0.053 0.085/0.052 0.070/0.044
    traffic-c (1d) 0.223/0.137 0.236/0.148 0.186/- 0.161/0.099 0.167/0.113 0.122/0.081
    traffic-c (7d) 0.492/0.280 0.509/0.529 0.202/- 0.179/0.105 0.168/0.114 0.139/0.094
    electricity-f (1d) - - 0.094/- 0.082/0.063 - 0.074/0.042
    traffic-f (1d) - - 0.213/- 0.230/0.150 - 0.139/0.090
    solar (1d) - - 0.241/- 0.222/0.093 - 0.210/0.082
    M4-Hourly (2d) - - - 0.090/0.030 - 0.067/0.025
    wind (30d) - - 0.311/- 0.286/0.116 - 0.284/0.108

    The convolutional Transformer achieves superior accuracy across all datasets. The performance advantage over RNN models (DeepAR and DeepState) is most significant on datasets with strong multi-scale seasonality and sharp weekday-versus-weekend shifts, such as traffic-c (R0.5=0.122R_{0.5} = 0.122 vs 0.1610.161) and traffic-f (R0.5=0.139R_{0.5} = 0.139 vs 0.2300.230).

  10. Knowl 10 — Effect of Convolutional Kernel Size on Transformer Attention and Training Dynamics

    empirical result

    Evaluating full attention Transformers with varying 1D causal convolution kernel sizes k∈{1,2,3,6,9}k \in \{1, 2, 3, 6, 9\} for 7-day rolling-day prediction yields the following R0.5/R0.9R_{0.5} / R_{0.9} losses:

    1. electricity-c: k=1k=1: 0.060/0.0300.060/0.030; k=2k=2: 0.058/0.0300.058/0.030; k=3k=3: 0.057/0.0310.057/0.031; k=6k=6: 0.057/0.0310.057/0.031; k=9k=9: 0.059/0.0340.059/0.034.
    2. traffic-c: k=1k=1: 0.134/0.0890.134/0.089; k=2k=2: 0.124/0.0850.124/0.085; k=3k=3: 0.123/0.0830.123/0.083; k=6k=6: 0.123/0.0830.123/0.083; k=9k=9: 0.122/0.0810.122/0.081.

    On the challenging traffic-c dataset, increasing kk from 1 to 9 produces an approximate 9%9\% relative improvement in R0.5R_{0.5} error. Furthermore, training loss curves show that models with larger context kernels (k=3,9k=3, 9) converge faster and achieve consistently lower negative log-likelihood (NLL) training error than canonical point-wise attention (k=1k=1), demonstrating that locality awareness stabilizes query-key matching.

  11. Knowl 11 — Performance of LogSparse Attention under Memory and Sequence Length Constraints

    data/table

    Comparison of full attention versus LogSparse attention (with and without causal convolution with kernel size k=6k=6) on fine-grained time series (electricity-f and traffic-f) under two evaluation regimes:

    1. Equal Memory Budget: LogSparse uses full context (Le1=768L_{e1}=768, Lt1=576L_{t1}=576), while full attention context length is reduced (Le2=293L_{e2}=293, Lt2=254L_{t2}=254) to equalize per-layer memory footprint (L×attended cells≈2932≈768×112L \times \text{attended cells} \approx 293^2 \approx 768 \times 112).
    2. Equal Length Constraint: Full and sparse models receive identical context lengths (Le=768L_{e}=768, Lt=576L_{t}=576).
    Constraint Dataset Full Sparse Full + Conv Sparse + Conv
    Memory electricity-f (1d) 0.083/0.051 0.084/0.047 0.078/0.048 0.079/0.049
    Memory traffic-f (1d) 0.161/0.109 0.150/0.098 0.149/0.102 0.138/0.092
    Length electricity-f (1d) 0.082/0.047 0.084/0.047 0.074/0.042 0.079/0.049
    Length traffic-f (1d) 0.147/0.096 0.150/0.098 0.139/0.090 0.138/0.092

    Under identical memory budgets on traffic-f, Sparse + Conv substantially outperforms Full + Conv (0.138/0.0920.138/0.092 vs 0.149/0.1020.149/0.102) by enabling longer history modeling (L=576L=576 vs 254254). Under equal sequence length, Sparse + Conv matches or slightly exceeds Full + Conv (R0.5=0.138R_{0.5}=0.138 vs 0.1390.139), confirming that O(L(log⁡L)2)O(L(\log L)^2) sparsity incurs no accuracy loss while alleviating the memory bottleneck.

Coverage note — No substantial contributed material was omitted; all core models (convolutional attention, LogSparse attention), theoretical proofs of path existence, training techniques, evaluation metrics, and empirical experiments across synthetic and real-world datasets are covered.

References

  1. 1.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
  2. 2.James Durbin and Siem Jan Koopman. Time series analysis by state space methods. Oxford university press, 2012.
  3. 3.Valentin Flunkert, David Salinas, and Jan Gasthaus. Deepar: Probabilistic forecasting with autoregressive recurrent networks. arXiv preprint arXiv:1704.04110, 2017.
  4. 4.Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
  5. 5.Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014.
  6. 6.Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems, pages 7785–7794, 2018.
  7. 7.Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95–104. ACM, 2018.
  8. 8.Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor-train rnns. arXiv preprint arXiv:1711.00073, 2017.
  9. 9.Danielle C Maddix, Yuyang Wang, and Alex Smola. Deep factors with gaussian processes for forecasting. arXiv preprint arXiv:1812.00098, 2018.
  10. 10.Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318, 2013.
  11. 11.Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  12. 12.Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
  13. 13.Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. arXiv preprint arXiv:1805.04623, 2018.
  14. 14.Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933, 2016.
  15. 15.George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):91–109, 1968.
  16. 16.George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
  17. 17.Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. Temporal regularized matrix factorization for high-dimensional time series prediction. In Advances in neural information processing systems, pages 847–855, 2016.
  18. 18.Nicolas Chapados. Effective bayesian modeling of groups of related count time series. arXiv preprint arXiv:1405.3738, 2014.
  19. 19.Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053, 2017.
  20. 20.Xiaoyong Jin, Shiyang Li, Yunkai Zhang, and Xifeng Yan. Multi-step deep autoregressive forecasting with latent states. URL http://roseyu.com/time-series-workshop/submissions/2019/timeseries-ICML19_paper_19.pdf, 2019.
  21. 21.Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, and Douglas Eck. An improved relative self-attention mechanism for transformer with application to music generation. arXiv preprint arXiv:1809.04281, 2018.
  22. 22.Daniel Povey, Hossein Hadian, Pegah Ghahremani, Ke Li, and Sanjeev Khudanpur. A time-restricted self-attention layer for asr. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5874–5878. IEEE, 2018.
  23. 23.Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv:1802.05751, 2018.
  24. 24.Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802–808, 2018.
  25. 25.Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao. Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review. International Journal of Automation and Computing, 14(5):503–519, 2017.
  26. 26.Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193–202, 1980.
  27. 27.Hrushikesh N Mhaskar and Tomaso Poggio. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14(06):829–848, 2016.
  28. 28.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  29. 29.Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW, 125, 2016.
  30. 30.Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Conditional time series forecasting with convolutional neural networks. arXiv preprint arXiv:1703.04691, 2017.
  31. 31.Scott Gray, Alec Radford, and Diederik P. Kingma. Gpu kernels for block-sparse weights. arXiv preprint arXiv:1711.09224, 2017.
  32. 32.Tim Cooijmans, Nicolas Ballas, César Laurent, Çaglar Gülçehre, and Aaron Courville. Recurrent batch ˘ normalization. arXiv preprint arXiv:1603.09025, 2016.
  33. 33.Rose Yu, Yaguang Li, Cyrus Shahabi, Ugur Demiryurek, and Yan Liu. Deep learning: A generic approach for extreme condition traffic forecasting. In Proceedings of the 2017 SIAM International Conference on Data Mining, pages 777–785. SIAM, 2017.
  34. 34.Guoqiang Zhang, B Eddy Patuwo, and Michael Y Hu. Forecasting with artificial neural networks:: The state of the art. International journal of forecasting, 14(1):35–62, 1998.
  35. 35.Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  36. 36.Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018.
  37. 37.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf, 2018.
  38. 38.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  39. 39.Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  40. 40.Nikolay Laptev, Jason Yosinski, Li Erran Li, and Slawek Smyl. Time-series extreme event forecasting with neural networks at uber. In International Conference on Machine Learning, number 34, pages 1–5, 2017.

Citation

MLA
Li, S., et al. “Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting”. arXiv, 2019, http://arxiv.org/abs/1907.00235v3.
APA
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., & Yan, X. (2019). Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. arXiv. http://arxiv.org/abs/1907.00235v3
Chicago
Li, S., X. Jin, Y. Xuan, et al. 2019. “Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting”. arXiv. http://arxiv.org/abs/1907.00235v3.
Harvard
Li, S. et al. (2019) “Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1907.00235v3.
Vancouver
1. Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang Y-X, Yan X (2019) Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. arXiv

BibTeX

@article{li2019enhancing,
  title = {Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting},
  author = {Li, Shiyang and Jin, Xiaoyong and Xuan, Yao and Zhou, Xiyou and Chen, Wenhu and Wang, Yu-Xiang and Yan, Xifeng},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1907.00235v3},
  eprint = {1907.00235}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors