Transformers: A Powerful Tool for Image Analysis and Generation

Level: Intermediate Length: 4 hours Format: In-Person Lecture Intended Audience: Students, researchers, and engineers from academia and industry with a good background of deep learning architectures like CNNs and LSTMs seeking to broaden and deepen their knowledge on state-of-the-art architectures. Description: After the tremendous success transformers had in the field of text analysis, machine translation, and de-novo text generation, it took time for transformers to enter the scene of image analysis and image generation. As high demand once limited data availability and compute resource, we now see more successful approaches in training transformers begin to mark a change that is potentially as impactful as the introduction of CNNs for image classification. In this course, we explain the thought model and elementary mathematics of the attention mechanism underlying transformers. You will learn the reason for their modeling capacity and understand why this creates the need of larger training datasets. The course traces the development of transformers for image analysis tasks and shows ways to pre-train transformers on weak or unlabeled data. The course concludes with examples of applications used in medical image analysis task. Learning Outcomes: This course will enable you to: - describe the benefits transformer architectures offer over CNNs and their limitations - reproduce the elementary building blocks (self- and cross-attention) of transformer architectures - sketch approaches to apply transformers to image analysis tasks as well as generative modeling - describe approaches to improve memory and time complexity - give examples of transformers applied to medical image analysis Instructor(s): Markus Thorsten Wenzel has worked on machine learning methods for medical applications since 2005 and has published more than 30 conference and journal papers on the subject. He received his PhD for his work on decision support systems for breast care. As key scientist for cognitive computing at Fraunhofer MEVIS, he is an experienced teacher and lecturer for academia and continuing education for participants from industry and research. He has acquired and led several international research projects as well as industry cooperations. Felix Thielke has been researching machine learning applications for medical image analysis at Fraunhofer MEVIS since 2019. He studied computer science with a focus on AI, cognition, and robotics at the University of Bremen, where he also gained years of experience in image analysis and deep learning through successful participation in the international RoboCup competitions. He is currently conducting his PhD research on deep learning based solutions to support the diagnosis and treatment of liver diseases. Event: SPIE Medical Imaging 2023 Course Held: 21 February 2023