Visual perception including localization, object recognition, and semantic segmentation are important abilities for nowadays’ intelligent agents, for example, robots in various applications.

For years, our group studies 2D images, videos, 3D point clouds, texts, audios to enable the robots to see and understand the surroundings as human beings.

In this talk, I will show key common challenges in computer vision topics, that is, scale changes, data imbalance, lighting changes. Normally, the majority class may dominate the training process, making it difficult for the model to accurately classify the minority class. How to utilize limited context information to accurately locate “hard” keypoints is still unexplored. Specifically, I will present our image-based coarse-to-fine human keypoint estimation work, which solves the scale and hard sample mining for efficient training.

Humans easily capture and understand the scene structure and contents, but not for robots even with developed deep learning. While standard image models jointly train an image feature extractor and a linear classifier to predict some label, CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples. At test time the learned text encoder synthesizes a zero-shot linear classifier by embedding the names or descriptions of the target dataset’s classes. Based on the power of CLIP and CoOp, we discuss the prompt engineering and show our work in improving performance of specific classes for large-scale pre-training models.

Xiaoyan Jiang is currently an associate professor in the School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, China. She received the Ph.D. degree in computer science from Friedrich-Schiller University of Jena, Germany, in 2015. 

Her research interests are computer vision and artificial intelligence topics including semantic segmentation, pose estimation, person re-identification, multi-modal fusion, and SLAM. She has multiple publications as the first/corresponding author in top journals/conferences, for example, Trans.: SMC, Pattern Recognition, TITS, SPIC, Knowledge-Based Systems, ICIP, CAIP, etc. She holds multiple funds both from research organizations and enterprises. She also cooperates with enterprises aiming to design AI solutions for practical scenarios.