论文标题
嵌套：用于脑肿瘤分割的嵌套模态感知变压器
NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation
论文作者
论文摘要
多模式MR成像通常用于临床实践中，以通过提供丰富的互补信息来诊断和研究脑肿瘤。以前的多模式MRI分割方法通常通过在网络的早期/中阶段连接多模式MRIS来执行模态融合，这几乎无法探索模态之间的非线性依赖性。在这项工作中，我们提出了一种新颖的嵌套模态感知变压器（嵌套形式），以明确探索多模式MRIS在脑肿瘤分段中的模式内和模式间关系。我们建立在基于变压器的多模型和单一码头结构的基础上，我们对不同模态的高级表示进行嵌套的多模式融合，并在较低的尺度上应用对模态敏感的门控（MSG），以进行更有效的跳过连接。具体而言，多模式融合是在我们提出的嵌套模态感知特征聚集（NMAFA）模块中进行的，该模块通过三个方向的空间意见变压器增强了单个模态内的长期依赖性，并通过交叉模态注意力变压器在模态之间进一步补充了模态信息之间的关键上下文信息。关于BRATS2020基准和私人脑膜瘤细分（Maniseg）数据集的广泛实验表明，嵌套形式显然比最先进的表现明显优于最先进的。该代码可从https://github.com/920232796/nestedformer获得。
Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion by concatenating multi-modal MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based multi-encoder and single-decoder structure, we perform nested multi-modal fusion for high-level representations of different modalities and apply modality-sensitive gating (MSG) at lower scales for more effective skip connections. Specifically, the multi-modal fusion is conducted in our proposed Nested Modality-aware Feature Aggregation (NMaFA) module, which enhances long-term dependencies within individual modalities via a tri-orientated spatial-attention transformer, and further complements key contextual information among modalities via a cross-modality attention transformer. Extensive experiments on BraTS2020 benchmark and a private meningiomas segmentation (MeniSeg) dataset show that the NestedFormer clearly outperforms the state-of-the-arts. The code is available at https://github.com/920232796/NestedFormer.
