Back

Scalable Adaptive Computation for Iterative Generation

Allan JabriDavid J. FleetTing Chen

article2023ICML208 citations

Proposes Recurrent Interface Networks, an architecture that routes information between high-dimensional data and a compact set of latent tokens to achieve state-of-the-art pixel diffusion generation up to ten times more efficiently than standard U-Nets.

Listen

Generating high-dimensional content such as high-resolution images and videos requires massive computing power. Standard deep learning architectures allocate computation uniformly across all input areas, spending equal effort on complex visual details and simple, redundant regions like empty backgrounds. This rigid allocation creates a significant computational bottleneck that limits scalability, especially on modern accelerator hardware that prefers fixed computation structures.

The article demonstrates and evaluates the Recurrent Interface Network, a neural network architecture designed to decouple heavy computation from input dimensions. The network dynamically routes compute capacity to information-dense regions, enabling efficient, adaptive generation of high-resolution images and video.

To evaluate the system, the authors conducted benchmark experiments applying the architecture to pixel-level denoising diffusion models across standard datasets, including ImageNet at resolutions from 64x64 up to 1024x1024, Kinetics-600 video prediction, and CIFAR-10. The architecture splits hidden units into a lightweight data interface and a compact set of latent tokens that handle the core computation. Stacked blocks alternate between reading data into latents, computing via self-attention, and writing updates back to the interface. To eliminate the high warm-up overhead of routing, the method introduces latent self-conditioning, which reuses latent context from prior iterative generation steps without requiring expensive backpropagation through time.

The experiments show that the proposed architecture achieves superior generation quality compared to standard convolutional U-Net models while dramatically improving efficiency. For ImageNet class-conditional generation, the architecture reduced computation per inference step by up to tenfold compared to leading diffusion baselines, scaling directly to 1024x1024 pixel images without relying on cascaded models or guidance techniques. For Kinetics-600 video prediction, the model improved generation quality scores while reducing computation per step tenfold. Ablation analyses confirmed that latent self-conditioning and multi-block iterative routing are critical, as attention visualizations demonstrated that the model learns to selectively focus computation on complex regions and dynamic object motions rather than static backgrounds.

These findings indicate that generative models do not need complex, multi-stage pipelines or hand-crafted spatial architectures to achieve state-of-the-art results. By relying on domain-agnostic attention operations and dynamic resource allocation, organizations can substantially lower inference computation costs and hardware resource requirements for visual generation workloads.

Teams developing high-dimensional generative pipelines should explore adopting decoupled attention architectures and latent self-conditioning to reduce computational overhead. Prior to full production deployment, further work should evaluate combining this architecture with orthogonal techniques, such as classifier guidance and latent diffusion, to assess maximum performance limits. Readers should note that while the empirical results demonstrate robust gains across images and video, evaluating broader data modalities and refining the latent conditioning dynamics remain ongoing areas for further research.

GoogleUniversity of California Berkeleydiffusion probabilistic modelshigh dimensional datalatent factorScalable adaptive computationIterative generationLatent computation
arXiv: 2212.11972
  • Paper: Flow Reasoning Models: Turning Flows Into Efficient Recurrent Reasoners, Alec Helbling et al. (2026). Flow Reasoning Models carry iterative self-conditioning into recurrent generation, extending the use of evolving internal state across repeated denoising-like updates.
  • Paper: Thinking with Looped Flows, Ayhan Suleymanzade et al. (2026). Thinking with Looped Flows extends recurrent state updates to iterative flow generation, exploring how repeated computation can improve generated solutions.
Cover for Scalable Adaptive Computation for Iterative Generation

Abstract

Natural data is redundant yet predominant architectures tile computation uniformly across their input and output space. We propose the Recurrent Interface Network (RIN), an attention-based architecture that decouples its core computation from the dimensionality of the data, enabling adaptive computation for more scalable generation of high-dimensional data. RINs focus the bulk of computation (i.e. global self-attention) on a set of latent tokens, using cross-attention to read and write (i.e. route) information between latent and data tokens. Stacking RIN blocks allows bottom-up (data to latent) and top-down (latent to data) feedback, leading to deeper and more expressive routing. While this routing introduces challenges, this is less problematic in recurrent computation settings where the task (and routing problem) changes gradually, such as iterative generation with diffusion models. We show how to leverage recurrence by conditioning the latent tokens at each forward pass of the reverse diffusion process with those from prior computation, i.e. latent self-conditioning. RINs yield state-of-the-art pixel diffusion models for image and video generation, scaling to 1024×1024 images without cascades or guidance, while being domain-agnostic and up to 10× more efficient than 2D and 3D U-Nets.

Topics

diffusion probabilistic modelshigh dimensional datalatent factorlatent variablerouting problemsrouting schemeefficient computationparallel computation

Keywords

Scalable adaptive computationIterative generationLatent computationIterative routing

Table of Contents

  • 1. Introduction
  • 2. Method
  • 2.1. Background: Iterative Generation with Diffusion
  • 2.2. Elements of Recurrent Interface Networks
  • 2.3. Latent Self-Conditioning
  • 3. Experiments
  • 3.1. Implementation Details
  • 3.2. Experimental Setup
  • 3.3. Comparison to SOTA
  • 3.4. Ablations
  • 3.5. Visualizing Adaptive Computation
  • 4. Related Work
  • 5. Conclusion
  • Acknowledgements
  • References
  • A. Architecture Implementation Pseudo-code
  • B. More Details of Training / Sampling Algorithms, and Noise schedules
  • C. Hyper-parameters and Other Training Details
  • D. Sample Visualizations

Knowls

  1. Knowl 1 — RINs route computation through a compact latent set

    model/method

    A Recurrent Interface Network (RIN) separates hidden units into an interface X∈Rn×dxX\in\mathbb{R}^{n\times d_x}, whose nn tokens correspond to the input, and a latent set Z∈Rm×dzZ\in\mathbb{R}^{m\times d_z}, where mm can be much smaller than nn. The interface is formed by tokenizing the input (for example, into image or video patches) and adding positional encodings; the latents begin as learned embeddings, with conditioning embeddings such as diffusion time or class appended to them.

    Each RIN block reads interface information into latents with cross-attention, performs repeated latent self-attention and token-wise MLP computation, then writes latent information back to the interface with cross-attention and an interface MLP. In schematic form, the block applies Z←Z+MHA(Z,X)Z\leftarrow Z+\mathrm{MHA}(Z,X), computes on ZZ using KK latent self-attention/MLP layers, and applies X←X+MHA(X,Z)X\leftarrow X+\mathrm{MHA}(X,Z) followed by an MLP on XX. Here MHA(Q,KV)\mathrm{MHA}(Q,KV) is multi-head attention with queries QQ and keys and values from KVKV; MLPs act on each token’s channels with shared weights. Layer normalization is applied to attention queries. Stacking blocks lets the updated interface and latents affect later routing. A linear readout maps final interface tokens to local output patches, which are assembled into the output.

    The interface-linked operations grow with input token count, while the more expensive global self-attention is concentrated on the latent set. Thus the model can process many input tokens without applying pairwise self-attention across all of them.

  2. Knowl 2 — Latent self-conditioning carries routing context between diffusion steps

    model/method

    For a denoising call at time tt, RIN latent initialization can incorporate latent activations from a previous reverse-diffusion step t′t'. If ZinitZ_{\mathrm{init}} is the learned latent initialization, Zt′Z_{t'} is the previous call’s latent output, and MLP\mathrm{MLP} and LayerNorm\mathrm{LayerNorm} are learned transformations, the initialization is

    Zt=Zinit+LayerNorm(Zt′+MLP(Zt′)).Z_t=Z_{\mathrm{init}}+\mathrm{LayerNorm}\bigl(Z_{t'}+\mathrm{MLP}(Z_{t'})\bigr).

    The layer-normalization scale and bias are initialized to zero, so the initialization begins as ZinitZ_{\mathrm{init}} early in training. During training, with the chosen self-conditioning probability, the network first runs on the current noisy input with zero context latents, uses the resulting latents as an estimate of prior context, stops gradients through that estimate, and runs again with the estimate as context. When self-conditioning is not selected, the prediction uses zero context. During sampling, the first call starts with zero latents and each subsequent call uses the preceding call’s latent output. This carries routing context across iterations without backpropagation through the diffusion trajectory; the paper reports less than 25% additional training time and negligible inference cost.

  3. Knowl 3 — ImageNet pixel-diffusion results across resolutions

    empirical result

    The paper reports class-conditional ImageNet pixel-diffusion results without guidance for its benchmark models. RIN FID / Inception Score (IS) / reported GFLOPs are: at 64×6464\times64, 1.23/66.5/1061.23 / 66.5 / 106; at 128×128128\times128, 2.75/144.1/1942.75 / 144.1 / 194; at 256×256256\times256, 4.51/161.0/3344.51 / 161.0 / 334; at 256×256256\times256 with input scaling, 3.42/182.0/3343.42 / 182.0 / 334; at 512×512512\times512 with input scaling, 3.95/216.0/4153.95 / 216.0 / 415; and at 1024×10241024\times1024 with input scaling, 8.72/163.9/11208.72 / 163.9 / 1120.

    The comparisons show both strong quality and substantially lower reported compute than the listed U-Net diffusion baselines, but not superiority on every metric against every method. For example, at 128×128128\times128, RIN’s FID is 2.75, compared with 2.43 for classifier-free guidance, while RIN uses 194 GFLOPs versus 538 for ADM. At 256×256256\times256, input-scaled RIN obtains FID 3.42 and IS 182.0, compared with FID 4.59 and IS 186.7 for guided ADM. At 512×512512\times512, input-scaled RIN obtains FID 3.95 and IS 216.0, compared with 7.72 and 172.7 for guided ADM; the RIN reports 415 GFLOPs versus 4122 for unguided ADM. The 1024×10241024\times1024 result demonstrates pixel-space scaling to that resolution without a cascade.

  4. Knowl 4 — Video prediction improves quality while reducing per-step compute

    empirical result

    On Kinetics-600 video prediction at 16×64×6416\times64\times64, using five conditioning frames, RIN obtains FVD 11.5 and IS 17.7 with 400 DDPM sampling steps, and FVD 10.8 and IS 17.7 with 1000 steps. Both settings report 386 GFLOPs and 411 million parameters. The compared Video Diffusion model, which uses reconstruction guidance, reports FVD 16.6, IS 15.64, 4136 GFLOPs, and 1100 million parameters. Thus the reported RIN results improve both video metrics while using about one-tenth the per-step GFLOPs. The RIN video model uses 3-D patches but otherwise follows the same architecture as the image models.

  5. Knowl 5 — RIN configurations retain small latent sets at high input resolution

    experimental setup

    The reported configurations use substantially fewer latent tokens than interface tokens, including for high-resolution inputs. In the detailed configurations, ImageNet models use: 64×6464\times64—128 latents and 256 interface tokens; 128×128128\times128—128 and 1024; 256×256256\times256—256 and 1024; 512×512512\times512—256 and 4096; and 1024×10241024\times1024—256 and 16384. The corresponding latent dimensions are 1024, 1024, 1024, 768, and 768; interface dimension is 256 for the 64×6464\times64 model and 512 for the others. These models use 4, 6, 6, 6, and 6 blocks, respectively, with latent processing depths per block of 4, 4, 4, 6, and 8. Their reported GFLOPs are 106, 194, 334, 415, and 1120.

    For Kinetics-600, the model uses 256 latents for 2048 interface tokens, latent dimension 1024, interface dimension 512, six blocks, and depth four. In an additional ImageNet tokenization ablation, the interface ranged from 64 tokens with 8×88\times8 patches to 4096 tokens with 1×11\times1 patches; the reported results indicate that RINs can handle this range, with performance remaining reasonable even when tokens are larger patches.

  6. Knowl 6 — Training self-conditioning rate affects sample quality

    empirical result

    An ablation on ImageNet at 64×6464\times64 and 128×128128\times128 varied the fraction of training examples using latent self-conditioning. A rate of zero disables self-conditioning during both training and inference; a positive rate selects self-conditioning for that fraction of training examples, while inference always uses it. The plotted FID and IS results show a clear association between increasing training self-conditioning rate and improved sample quality. The paper used a rate of 0.9 for its best reported results.

  7. Knowl 7 — Repeated read-write routing improves over a single interface update

    empirical result

    The block-stacking ablation varied the number of read-write operations while comparing models across parameter counts. A single read-write operation, without latent self-conditioning, behaved similarly to a single-read-write latent architecture and reached a quality plateau earlier as model size increased. More frequent read-writes delayed saturation and produced substantially better sample quality, supporting the use of stacked blocks to let interface and latent context inform subsequent routing. These ablations used smaller ImageNet models with latent dimension 768, a learning rate of 2×10−32\times10^{-3}, and 150,000 updates for 64×6464\times64 or 220,000 updates for 128×128128\times128.

  8. Knowl 8 — Read attention becomes sample-specific and emphasizes changing content

    empirical result

    Visualizations of RIN read-attention maps, averaged over latents, show how interface information is routed into the latent computation during iterative generation. For ImageNet images, the maps are relatively similar across samples early in reverse diffusion, then become sparser and more specific to each sample as generation proceeds, emphasizing visually complex regions. For Kinetics-600 prediction, the maps emphasize motion and other regions that cannot be produced simply by copying the conditioning frames. These observations are consistent with adaptive allocation of latent processing, although the visualizations are evidence from attention patterns rather than a direct measurement of per-region compute.

  9. Knowl 9 — ImageNet evaluation and training conditions

    experimental setup

    For ImageNet generation, the paper uses center crops and random left-right flips as its only data augmentations. FID and Inception Score are evaluated on 50,000 generated samples using 1000 DDPM sampling steps. The models are pixel-space denoising diffusion models; the benchmark comparisons do not use guidance or cascades. The implementation uses a continuous-time noise schedule, with sigmoid temperature 0.9 as the default because sigmoid schedules were found more stable than cosine schedules during training, especially at higher image resolutions. Input scaling is reported for the larger-resolution models. For Kinetics-600, evaluation uses 50,000 samples and reports FVD and IS with 400 or 1000 DDPM steps.

  10. Knowl 10 — RIN performance on CIFAR-10 is close to the reported state of the art

    empirical result

    On CIFAR-10, the paper reports FID 1.81 for RIN, compared with the cited EDM result of 1.79. The RIN result was obtained without EDM’s improved sampling procedure. The CIFAR-10 RIN had 31 million parameters, described as about half the model size, and trained in three hours—about one-tenth the training time—using comparable compute. This result indicates that the architecture also works on a small image dataset, despite lacking the specialized convolutional structure of U-Nets.

Coverage note — The detailed cosine-versus-sigmoid schedule ablation and the full optimizer, hardware, and training-hyperparameter inventory are omitted because they are supporting implementation details rather than central contributions.

References

  1. 1.Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. {TensorFlow}: a system for {Large-Scale} machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp. 265–283, 2016.
  2. 2.Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  3. 3.Burtsev, M. S., Kuratov, Y., Peganov, A., and Sapunov, G. V. Memory transformer. arXiv preprint arXiv:2006.11527, 2020.
  4. 4.Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., and Zisserman, A. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018.
  5. 5.Chen, T. On the importance of noise schedules for diffusion models. arXiv preprint arXiv:2301.10972, 2023.
  6. 6.Chen, T., Saxena, S., Li, L., Fleet, D. J., and Hinton, G. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852, 2021.
  7. 7.Chen, T., Li, L., Saxena, S., Hinton, G., and Fleet, D. J. A generalist framework for panoptic segmentation of images and videos. arXiv preprint arXiv:2210.06366, 2022a.
  8. 8.Chen, T., Saxena, S., Li, L., Lin, T.-Y., Fleet, D. J., and Hinton, G. A unified sequence interface for vision tasks. arXiv preprint arXiv:2206.07669, 2022b.
  9. 9.Chen, T., Zhang, R., and Hinton, G. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022c.
  10. 10.Clark, A., Donahue, J., and Simonyan, K. Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571, 2019.
  11. 11.Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
  12. 12.Dhariwal, P. and Nichol, A. Diffusion models beat GANs on image synthesis. In NeurIPS, 2022.
  13. 13.Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022.
  14. 14.Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  15. 15.Figurnov, M., Collins, M. D., Zhu, Y., Zhang, L., Huang, J., Vetrov, D., and Salakhutdinov, R. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1039–1048, 2017.
  16. 16.Fukushima, K. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks, 1(2): 119–130, 1988.
  17. 17.Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., and Bengio, Y. Coordination among neural modules through a shared global workspace. arXiv preprint arXiv:2103.01197, 2021.
  18. 18.Graves, A. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016.
  19. 19.Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
  20. 20.Gregor, K., Danihelka, I., Graves, A., Rezende, D., and Wierstra, D. Draw: A recurrent neural network for image generation. In International conference on machine learning, pp. 1462–1471. PMLR, 2015.
  21. 21.He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  22. 22.Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  23. 23.Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  24. 24.Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
  25. 25.Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.
  26. 26.Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. JMLR, 2022a.
  27. 27.Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video Diffusion Models. In NeurIPS, 2022b.
  28. 28.Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021a.
  29. 29.Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651–4664. PMLR, 2021b.
  30. 30.Joulin, A. and Mikolov, T. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in neural information processing systems, 28, 2015.
  31. 31.Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.
  32. 32.Kingma, D., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021.
  33. 33.Krizhevsky, A., Nair, V., and Hinton, G. Cifar-10 (canadian institute for advanced research). URL http://www.cs.toronto.edu/~kriz/cifar.html.
  34. 34.Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 2012.
  35. 35.LeCun, Y., Boser, D., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4): 541–551, 1989.
  36. 36.Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., and Teh, Y. W. Set transformer. CoRR, abs/1810.00825, 2018. URL http://arxiv.org/abs/1810.00825.
  37. 37.Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning, pp. 3744–3753. PMLR, 2019.
  38. 38.Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525–11538, 2020.
  39. 39.Luc, P., Clark, A., Dieleman, S., Casas, D. d. L., Doron, Y., Cassirer, A., and Simonyan, K. Transformation-based adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.04035, 2020.
  40. 40.Luhman, T. and Luhman, E. Improving diffusion model efficiency through patching. arXiv preprint arXiv:2207.04316, 2022.
  41. 41.Nash, C., Carreira, J., Walker, J., Barr, I., Jaegle, A., Malinowski, M., and Battaglia, P. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.09494, 2022.
  42. 42.Nichol, A. and Dhariwal, P. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
  43. 43.Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  44. 44.Peebles, W. and Xie, S. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
  45. 45.Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019.
  46. 46.Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.
  47. 47.Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer, 2015.
  48. 48.Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  49. 49.Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
  50. 50.Savinov, N., Chung, J., Binkowski, M., Elsen, E., and Oord, A. v. d. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.06749, 2021.
  51. 51.Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
  52. 52.Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  53. 53.Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  54. 54.Strudel, R., Tallec, C., Altché, F., Du, Y., Ganin, Y., Mensch, A., Grathwohl, W., Savinov, N., Dieleman, S., Sifre, L., et al. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:2211.04236, 2022.
  55. 55.Sukhbaatar, S., Weston, J., Fergus, R., et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
  56. 56.Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018.
  57. 57.Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  58. 58.Walker, J., Razavi, A., and Oord, A. v. d. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021.
  59. 59.Weston, J., Chopra, S., and Bordes, A. Memory networks. arXiv preprint arXiv:1410.3916, 2014.
  60. 60.Yin, H., Vahdat, A., Alvarez, J., Mallya, A., Kautz, J., and Molchanov, P. Adavit: Adaptive tokens for efficient vision transformer. arXiv preprint arXiv:2112.07658, 2021.
  61. 61.You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.
  62. 62.Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283–17297, 2020.

Citation

MLA
Jabri, A., et al. “Scalable Adaptive Computation for Iterative Generation”. International Conference on Machine Learning, vol. 202, 2023, pp. 14569–89, https://proceedings.mlr.press/v202/jabri23a.html.
APA
Jabri, A., Fleet, D. J., & Chen, T. (2023). Scalable Adaptive Computation for Iterative Generation. International Conference on Machine Learning, 202, 14569–14589. https://proceedings.mlr.press/v202/jabri23a.html
Chicago
Jabri, A., D. J. Fleet, and T. Chen. 2023. “Scalable Adaptive Computation for Iterative Generation”. International Conference on Machine Learning 202: 14569–89. https://proceedings.mlr.press/v202/jabri23a.html.
Harvard
Jabri, A., Fleet, D.J. and Chen, T. (2023) “Scalable Adaptive Computation for Iterative Generation”, International Conference on Machine Learning. PMLR, pp. 14569–14589. Available at: https://proceedings.mlr.press/v202/jabri23a.html.
Vancouver
1. Jabri A, Fleet DJ, Chen T (2023) Scalable Adaptive Computation for Iterative Generation. In: International Conference on Machine Learning. PMLR, pp 14569–14589

BibTeX

@InProceedings{pmlr-v202-jabri23a,
  title = 	 {Scalable Adaptive Computation for Iterative Generation},
  author =       {Jabri, Allan and Fleet, David J. and Chen, Ting},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {14569--14589},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/jabri23a/jabri23a.pdf},
  url = 	 {https://proceedings.mlr.press/v202/jabri23a.html},
  abstract = 	 {Natural data is redundant yet predominant architectures tile computation uniformly across their input and output space. We propose the Recurrent Interface Network (RIN), an attention-based architecture that decouples its core computation from the dimensionality of the data, enabling adaptive computation for more scalable generation of high-dimensional data. RINs focus the bulk of computation (i.e. global self-attention) on a set of latent tokens, using cross-attention to read and write (i.e. route) information between latent and data tokens. Stacking RIN blocks allows bottom-up (data to latent) and top-down (latent to data) feedback, leading to deeper and more expressive routing. While this routing introduces challenges, this is less problematic in recurrent computation settings where the task (and routing problem) changes gradually, such as iterative generation with diffusion models. We show how to leverage recurrence by conditioning the latent tokens at each forward pass of the reverse diffusion process with those from prior computation, i.e. latent self-conditioning. RINs yield state-of-the-art pixel diffusion models for image and video generation, scaling to1024×1024 images without cascades or guidance, while being domain-agnostic and up to 10× more efficient than 2D and 3D U-Nets.}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/