ISD Research Team Develops the First Foundation Model for Marine Image Analysis with Instance Visual Description

Congratulations to Prof. Sai-Kit YEUNG (Professor, Division of Integrative Systems and Design, Department of Computer Science and Engineering, and Department of Ocean Science) and Mr. Ziqiang ZHENG (PhD Candidate, Department of Computer Science and Engineering) for getting their groundbreaking research paper accepted as an oral presentation at one of the top-tier conferences - European Conference on Computer Vision (ECCV) 2024. The research team develops the first foundation model for image analysis for marine realms with instance visual description, together with the largest marine image dataset.

The oceans, covering 70% of the Earth’s surface, are teeming with life and play a pivotal role in global climate regulation, yet remain largely unexplored and poorly understood due to their vastness and inaccessibility. Therefore, analyzing and understanding marine imagery has gained increasing attention within both computer vision and marine communities. However, building a foundation model for marine visual analysis is very challenging. The scarcity of labeled data is the most hindering issue, and marine photographs illustrate significantly different appearances and contents from general in-air images. Based on their previous work MarineGPT, the first vision-language model specifically on the marine domain with extensive marine knowledge, Prof. YEUNG and his team created MarineInst20M, the largest marine image dataset to date, with 2.42 million images and 19.2 million masks in total and introduced MarineInst, a foundation model for marine visual analysis which can segment and describe the marine object instances. The dataset and model support a wide range of marine visual analysis tasks, from image-level scene understanding to regional mask-level instance understanding. What’s more, the model exhibits strong generalization ability and flexibility to support various downstream tasks with state-of-the-art performance.

European Conference on Computer Vision (ECCV) ranks top 2 on Research.com’s 2023 Best Computer Science Conference List. Getting the paper accepted as an oral presentation with only 2.33% (200/8585) of total submissions in ECCV 2024 recognizes the innovative capacity of ISD and Prof. Yeung’s team. The research contributes to progress in AI for Science at large and indicates HKUST’s advanced ability in frontier research on Marine AI.

For more information about MarineInst, please visit the website: https://marineinst.hkustvgd.com.

(Article reposted from HKUST Academy of Interdisciplinary Studies)