Multimodal Token Fusion for Vision Transformers

Yikai WangXinghao ChenLele CaoWenbing HuangFuchun SunYunhe Wang

article2022CVPR315 citations

encoder-decoder

arXiv: 2204.08721

Topics

encoder-decoder

1. Introduction
2. Related Work
3. Methodology
3.1. Basic Fusion for Vision Transformers
3.2. Multimodal Token Fusion
3.3. Residual Positional Alignment
3.4. Homogeneous Modalities
3.5. Heterogeneous Modalities
4. Experiments
4.1. Multimodal Image-to-Image Translation
4.2. RGB-Depth Semantic Segmentation
4.3. Vision and Point Cloud 3D Object Detection
5. Ablation Study
6. Conclusion
Acknowledgement
References

Knowls

Knowl 1 — TokenFusion: adaptive replacement of uninformative tokens
model/method
TokenFusion is a multimodal fusion method for vision transformers with $M$ input modalities. Each modality is represented as a sequence of spatial or semantically corresponding tokens and is processed through a transformer stream. At every transformer layer, a learned token-importance predictor identifies tokens that contribute little to the modality-specific computation. Important tokens continue through their original transformer stream, whereas uninformative tokens are replaced by features projected from aligned tokens in another modality. The method therefore introduces cross-modal interaction while preserving most of the original single-modal transformer data flow and parameterization. For homogeneous modalities, transformer blocks can be shared; for heterogeneous modalities, separate transformer streams can be connected through learned inter-modal projections.
Knowl 2 — Alignment-aware token projection
definition
For modality $m$ , let $x_m[n]$ denote its $n$ -th input token, and let $x_{m'}[n']$ denote the token in modality $m'$ that is aligned with it. TokenFusion defines a token projection from modality $m$ to modality $m'$ as
$\operatorname{Proj}^{T}_{m'}(x_m[n])=h\!\left(x_{m'}[n']\right),$
where $h$ is the identity map for homogeneous modalities and a shallow multilayer perceptron for heterogeneous modalities. Applying the projection to every token gives
$\operatorname{Proj}^{M}_{m'}(x_m)=\left[\operatorname{Proj}^{T}_{m'}(x_m[1]);\ldots;\operatorname{Proj}^{T}_{m'}(x_m[N])\right].$
Here $N$ is the number of tokens being projected, and the alignment determines $n'$ for each source token. Unlike alignment-agnostic fusion, which concatenates modality tokens and leaves correspondences to be learned implicitly, this operation explicitly injects known pixel-, patch-, or 3D-coordinate correspondences. The projection can be applied to input features or to intermediate transformer features.
Knowl 3 — Differentiable token pruning and cross-modal substitution
equation
Let $e_m^l\in\mathbb{R}^{N\times C_l}$ be the feature tokens of modality $m$ at layer $l$ , where $N$ is the token count and $C_l$ is the layer's channel dimension. TokenFusion predicts one importance score per token with
$s^l(e_m^l)=\operatorname{MLP}_{\mathrm{score}}(e_m^l)\in[0,1]^N,$
where the score vector is broadcast across channels. The score gates the input to multi-head self-attention:
$\widehat e_m^l=\operatorname{MSA}\!\left(\operatorname{LN}(e_m^l)\odot s^l(e_m^l)\right),$
where $\operatorname{LN}$ is layer normalization, $\operatorname{MSA}$ is multi-head self-attention, and $\odot$ denotes element-wise multiplication with channel-wise broadcasting. If $\mathcal{L}_m$ is the task loss for modality $m$ , the training objective is
$\mathcal{L}=\sum_{m=1}^{M}\left(\mathcal{L}_m+\lambda\sum_{l=1}^{L}\left\|s^l(e_m^l)\right\|_1\right),$
where $L$ is the number of transformer layers and $\lambda\geq 0$ controls the sparsity penalty. Before each transformer layer, a token whose score is below a threshold $\theta$ is replaced by the aligned feature from another modality:
$e_m^l\leftarrow e_m^l\odot\mathbf{1}_{s^l(e_m^l)\geq\theta}+\operatorname{Proj}^{M}_{m'}(e_m^l)\odot\mathbf{1}_{s^l(e_m^l)<\theta}.$
Here $\mathbf{1}_{condition}$ is a binary token mask, $m'\neq m$ is the source modality, and $\operatorname{Proj}^{M}_{m'}(e_m^l)$ has the same token and channel dimensions as $e_m^l$ . The $\ell_1$ term encourages low scores for replaceable tokens, while the differentiable gating allows the model to learn which tokens should be retained or fused.
Knowl 4 — Residual Positional Alignment
model/method
TokenFusion introduces Residual Positional Alignment (RPA) to prevent token substitution from discarding the destination token's positional identity. A transformer positional embedding associated with the destination token remains attached to that token even when its content feature is replaced by a projected feature from another modality. Thus, the substituted representation combines the source modality's content with the destination location's positional embedding. Positional embeddings are shared between modalities in homogeneous fusion and are modality-specific in heterogeneous fusion. During training, positional-embedding back-propagation is stopped after the first layer: gradients are retained for the first-layer positional embeddings, while positional embeddings used at later layers are frozen. This preserves explicit spatial or token-index alignment after fusion.
Knowl 5 — Homogeneous-modality TokenFusion
model/method
For aligned homogeneous modalities such as RGB and depth or multiple image representations, TokenFusion uses the same multi-head self-attention and feed-forward parameters across modalities, but separate layer-normalization parameters so that each modality retains its own feature statistics. Patch projections, attention projections, MLPs, and positional embeddings are shared. The projection function is the identity and a pruned token is replaced by the token at the same spatial position in another modality. With two modalities, replacement is bidirectional. With $M>2$ modalities, each modality's $N$ token positions are randomly partitioned before training into $M-1$ equal-sized groups, one group for each possible replacement modality; these assignments remain fixed throughout training. If $a_{m'}(m)\in\{0,1\}^N$ is the fixed mask assigning positions in modality $m$ to source modality $m'$ , the replacement rule is
$e_m^l\leftarrow e_m^l\odot\mathbf{1}_{s^l(e_m^l)\geq\theta}+\sum_{\substack{m'=1\\m'\neq m}}^{M}a_{m'}(m)\odot\operatorname{Proj}^{M}_{m'}(e_m^l)\odot\mathbf{1}_{s^l(e_m^l)<\theta}.$
The fixed allocation ensures that every pruned token has a predetermined alternative modality while the learned scores determine whether replacement occurs.
Knowl 6 — Heterogeneous fusion of point-cloud and image transformers
model/method
For structurally different modalities, TokenFusion keeps separate transformer architectures and unshared parameters, then connects them through shallow MLP projections. In the paper's 3D detection application, sampled point-cloud seed tokens are processed by a Group-Free-style transformer, while image patches and object-query tokens are processed by a YOLOS-style transformer. Token pruning is applied to point-cloud seed tokens. A pruned 3D point is projected to its corresponding image patch using camera calibration, and the image-patch feature is transformed to the point-stream embedding dimension before substitution.

For a point with 3D coordinates $(x,y,z)$ , camera intrinsic matrix $K\in\mathbb{R}^{4\times4}$ , and camera extrinsic matrix $R_t\in\mathbb{R}^{4\times4}$ , the homogeneous image projection is
$\begin{bmatrix}u\\v\\z'\end{bmatrix}=K R_t\begin{bmatrix}x\\y\\z\\1\end{bmatrix},$
where $(u/z',v/z')$ is the projected pixel coordinate. If $P$ is the image-transformer patch size and $W$ is the image width, the corresponding image-patch index is
$n_{\mathrm{img}}=\left\lfloor\frac{\lfloor v/z'\rfloor}{P}\right\rfloor\left\lfloor\frac{W}{P}\right\rfloor+\left\lfloor\frac{\lfloor u/z'\rfloor}{P}\right\rfloor.$
The same geometric correspondence is used to project 3D object labels to the image branch. This design enables simultaneous 3D and 2D detection without requiring the two modalities to have the same token count, embedding dimension, or transformer depth.
Knowl 7 — Image-to-image translation results on Taskonomy
empirical result
TokenFusion was evaluated on Taskonomy using 1,000 training and 500 validation multimodal images sampled from the dataset. Inputs and predictions were resized to $256\times256$ pixels. The generator and discriminator were transformers with either 10 layers (tiny) or 20 layers (small), learning rate $2\times10^{-4}$ , shared transformer parameters across input modalities, modality-specific layer normalization, shared positional embeddings, sparsity weight $\lambda=10^{-4}$ , and threshold $\theta=2\times10^{-2}$ . Lower values are better; RGB predictions use FID/KID with the reported values scaled by $10^{-2}$ , while other predictions use MAE/MSE with values scaled by $10^{-1}$ .

The strongest CNN baseline, CEN, obtained scores of $62.63/1.65$ , $84.33/2.70$ , $1.12/2.51$ , $1.10/1.72$ , and $0.28/0.66$ for Shade+Texture $\rightarrow$ RGB, Depth+Normal $\rightarrow$ RGB, RGB+Shade $\rightarrow$ Normal, RGB+Normal $\rightarrow$ Shade, and RGB+Edge $\rightarrow$ Depth, respectively. Transformer feature concatenation obtained $76.13/2.85$ , $102.70/4.74$ , $1.52/3.15$ , $1.33/2.20$ , and $0.40/0.83$ with the tiny backbone, and $72.55/2.39$ , $96.04/4.09$ , $1.18/2.73$ , $1.30/2.07$ , and $0.35/0.68$ with the small backbone. TokenFusion obtained $50.40/1.03$ , $76.35/2.19$ , $0.73/1.83$ , $0.95/1.54$ , and $0.21/0.57$ with the tiny backbone, and $43.92/0.94$ , $70.13/1.92$ , $0.58/1.51$ , $0.79/1.33$ , and $0.16/0.47$ with the small backbone. Thus, for Shade+Texture $\rightarrow$ RGB, the small TokenFusion model reduced FID from the CEN value of $62.63$ to $43.92$ , a reported relative decrease of $29.8\%$ . Qualitative outputs showed richer colors and details; learned fusion patterns also indicated that texture tokens tended to retain detailed boundaries while borrowing facial information from shade features.
Knowl 8 — RGB-depth semantic segmentation results
empirical result
TokenFusion was evaluated for RGB-depth semantic segmentation on NYUDv2 and SUN RGB-D. NYUDv2 used 795 training and 654 test images with 40 classes; SUN RGB-D used 5,285 training and 5,050 test images with 37 classes. The tiny and small models followed the B1 and B2 SegFormer backbones and were initialized from ImageNet-1k pretraining. RGB and depth used shared transformer blocks and positional embeddings with separate layer normalization, $\lambda=10^{-3}$ , and $\theta=2\times10^{-2}$ . Metrics are pixel accuracy, mean accuracy, and mean intersection-over-union, all reported as percentages.

On NYUDv2, RGB-only transformer baselines achieved $(75.2,62.5,49.7)$ for the tiny backbone and $(76.0,63.0,50.6)$ for the small backbone. Simple RGB-depth token concatenation achieved $(76.5,63.4,50.8)$ and $(77.1,63.8,51.4)$ , whereas TokenFusion achieved $(78.6,66.2,53.3)$ and $(79.0,66.9,54.2)$ for the tiny and small backbones, respectively. On SUN RGB-D, the corresponding RGB-only results were $(82.3,60.6,47.0)$ and $(82.9,61.3,48.1)$ ; concatenation achieved $(82.8,61.4,47.9)$ and $(83.5,62.0,49.0)$ ; and TokenFusion achieved $(84.0,63.3,51.4)$ and $(84.7,64.1,53.0)$ . The TokenFusion transformer models therefore exceeded the reported CNN-based state of the art while using relatively light backbones.
Knowl 9 — Point-cloud and image 3D detection results
empirical result
TokenFusion jointly learned 3D detection from point clouds and 2D detection from images on SUN RGB-D and ScanNetV2. SUN RGB-D used the standard split and 10 object classes; ScanNetV2 used 1,201 training and 312 test scans with 18 classes, sampling 10 image frames per scene. The 3D branch followed Group-Free and the 2D branch followed YOLOS. Results use [email protected] and [email protected]; values in parentheses are averages across five runs, while unparenthesized values are the best run.

On SUN RGB-D, Group-Free with point clouds alone obtained $63.0\,(62.6)$ and $45.2\,(44.4)$ , and appending RGB vectors directly to point features reduced performance to $62.1\,(61.0)$ and $42.7\,(41.9)$ . TokenFusion with the Group-Free $L6,O256$ configuration and YOLOS-tiny obtained $64.5\,(64.2)$ and $47.8\,(47.3)$ ; using YOLOS-small obtained $64.9\,(64.4)$ and $48.3\,(47.7)$ . The best listed CNN result, imVoteNet, obtained $63.4$ [email protected].

On ScanNetV2, Group-Free with $L6,O256$ obtained $67.3\,(66.3)$ and $48.9\,(48.5)$ , while direct RGB concatenation obtained $66.3\,(65.7)$ and $47.5\,(47.0)$ . The larger $L12,O512$ Group-Free model obtained $69.1\,(68.6)$ and $52.8\,(51.8)$ , dropping to $68.2\,(67.6)$ and $50.3\,(49.4)$ with direct RGB concatenation. TokenFusion achieved $68.8\,(68.0)$ and $51.9\,(51.2)$ with YOLOS-tiny and $70.8\,(69.8)$ and $54.2\,(53.6)$ with YOLOS-small. The qualitative detections showed that image features helped recover objects with sparse or missing points and disambiguate heavily overlapping point sets.
Knowl 10 — Ablation of sparsity, token fusion, and positional alignment
empirical result
Ablations were conducted on NYUDv2 RGB-depth segmentation and SUN RGB-D point-cloud/image 3D detection. Without the sparsity penalty, no fusion gave $(75.2,62.5,49.7)$ segmentation scores and $(62.8,45.1)$ detection scores. Randomly replacing 10% of tokens gave $(75.6,63.0,50.1)$ and $(62.3,44.5)$ , while replacing 30% gave $(74.2,61.0,48.2)$ and $(59.5,42.4)$ . Adding the $\ell_1$ sparsity penalty without token fusion changed the results only slightly to $(75.0,62.5,49.5)$ and $(62.6,44.9)$ .

The combination of learned token fusion and RPA produced $(78.6,66.2,53.3)$ for segmentation and $(64.9,48.3)$ for detection. A separate RPA ablation with the sparsity penalty showed that no token fusion gave $(75.2,62.5,49.7)$ and $(62.8,45.1)$ , RPA without token fusion gave $(75.7,62.9,50.3)$ and $(63.0,45.3)$ , learned token fusion without RPA gave $(78.3,65.8,52.9)$ and $(63.6,46.2)$ , and the complete method gave $(78.6,66.2,53.3)$ and $(64.9,48.3)$ . These results indicate that the sparsity penalty is primarily useful for exposing tokens to fuse, learned replacement is substantially better than random replacement, and RPA is especially important when token fusion is used.

Coverage note — No substantial contributed material was omitted; qualitative visualizations and the paper's implementation details are incorporated into the method and empirical-result knowls rather than treated as separate knowls.

References

1.Aljaz Bozic, Pablo R. Palafox, Justus Thies, Angela Dai, and Matthias Nießner. Transformerfusion: Monocular RGB scene reconstruction using transformers. In NeurIPS, 2021.
2.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
3.Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying, Danny Z Chen, and Jian Wu. A hierarchical graph network for 3d object detection on point clouds. In CVPR, 2020.
4.Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
5.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
6.Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner. 3d-mpa: Multi-proposal aggregation for 3d semantic instance segmentation. In CVPR, 2020.
7.Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. arXiv preprint arXiv:2106.00666, 2021.
8.Yu Fu, TianYang Xu, XiaoJun Wu, and Josef Kittler. Ppt fusion: Pyramid patch transformerfor a case study in image fusion. arXiv preprint arXiv:2107.13967, 2021.
9.Saurabh Gupta, Pablo Arbelaez, and Jitendra Malik. Perceptual organization and recognition of indoor scenes from RGB-D images. In CVPR, 2013.
10.JunYoung Gwak, Christopher Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. arXiv preprint arXiv:2006.12356, 2020.
11.Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. Fusenet: Incorporating depth into semantic segmentation via fusion-based CNN architecture. In ACCV, 2016.
12.Jin-Fan Hu, Ting-Zhu Huang, and Liang-Jian Deng. Fusformer: A transformer-based fusion approach for hyperspectral image super-resolution. arXiv preprint arXiv:2109.02079, 2021.
13.Huawei. Mindspore. https://www.mindspore.cn/, 2020.
14.Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In NeurIPS, 2021.
15.Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. arXiv preprint arXiv:2102.03334, 2021.
16.Seungyong Lee, Seong-Jin Park, and Ki-Sang Hong. Rdfnet: RGB-D multi-level residual feature fusion for indoor semantic segmentation. In ICCV, 2017.
17.Guosheng Lin, Fayao Liu, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for dense prediction. In IEEE Trans. PAMI, 2019.
18.Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fuseformer: Fusing fine-grained information in transformers for video inpainting. In ICCV, 2021.
19.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
20.Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV, 2021.
21.Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
22.Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. arXiv preprint arXiv:2107.00135, 2021.
23.Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Greg Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
24.Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multimodal fusion transformer for end-to-end autonomous driving. In CVPR, 2021.
25.Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Imvotenet: Boosting 3d object detection in point clouds with image votes. In CVPR, 2020.
26.Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277–9286, 2019.
27.Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.
28.Charles R Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS, 2017.
29.Xie Qian, Lai Yu-kun, Wu Jing, Wang Zhoutao, Zhang Yiming, Xu Kai, and Wang Jun. Mlcvnet: Multi-level context votenet for 3d object detection. In CVPR, 2020.
30.Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NeurIPS, 2021.
31.Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, 2012.
32.Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D scene understanding benchmark suite. In CVPR, 2015.
33.Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.
34.Sijie Song, Jiaying Liu, Yanghao Li, and Zongming Guo. Modality compensation network: Cross-modal adaptation for action recognition. In IEEE Trans. Image Process., 2020.
35.Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2019.
36.Abhinav Valada, Rohit Mohan, and Wolfram Burgard. Self-supervised model adaptation for multimodal semantic segmentation. In IJCV, 2020.
37.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
38.Vibashan VS, Jeya Maria Jose Valanarasu, Poojan Oza, and Vishal M Patel. Image fusion transformer. arXiv preprint arXiv:2107.09011, 2021.
39.Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. In NeurIPS, 2020.
40.Yikai Wang, Fuchun Sun, Ming Lu, and Anbang Yao. Learning deep multimodal feature representation with asymmetric multi-layer fusion. In ACM MM, 2020.
41.Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
42.Amir Roshan Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
43.Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. arXiv preprint arXiv:2006.05682, 2020.
44.Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021.
45.Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.

Citation

MLA

Wang, Y., et al. “Multimodal Token Fusion for Vision Transformers”. arXiv, 2022, http://arxiv.org/abs/2204.08721v2.

APA

Wang, Y., Chen, X., Cao, L., Huang, W., Sun, F., & Wang, Y. (2022). Multimodal Token Fusion for Vision Transformers. arXiv. http://arxiv.org/abs/2204.08721v2

Chicago

Wang, Y., X. Chen, L. Cao, W. Huang, F. Sun, and Y. Wang. 2022. “Multimodal Token Fusion for Vision Transformers”. arXiv. http://arxiv.org/abs/2204.08721v2.

Harvard

Wang, Y. et al. (2022) “Multimodal Token Fusion for Vision Transformers”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2204.08721v2.

Vancouver

1. Wang Y, Chen X, Cao L, Huang W, Sun F, Wang Y (2022) Multimodal Token Fusion for Vision Transformers. arXiv

BibTeX

@article{wang2022multimodal,
  title = {Multimodal Token Fusion for Vision Transformers},
  author = {Wang, Yikai and Chen, Xinghao and Cao, Lele and Huang, Wenbing and Sun, Fuchun and Wang, Yunhe},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2204.08721v2},
  eprint = {2204.08721}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Multimodal Token Fusion for Vision Transformers

Topics

Table of Contents

Knowls

Knowl 1 — TokenFusion: adaptive replacement of uninformative tokens

Knowl 2 — Alignment-aware token projection

Knowl 3 — Differentiable token pruning and cross-modal substitution

Knowl 4 — Residual Positional Alignment

Knowl 5 — Homogeneous-modality TokenFusion

Knowl 6 — Heterogeneous fusion of point-cloud and image transformers

Knowl 7 — Image-to-image translation results on Taskonomy

Knowl 8 — RGB-depth semantic segmentation results

Knowl 9 — Point-cloud and image 3D detection results

Knowl 10 — Ablation of sparsity, token fusion, and positional alignment

References

Citation

BibTeX

Access the Paper