Generating 3D Animals with YouDream

Generative AI can struggle to create realistic animals and 2D representations often have mistakes like extra limbs and tails. If 2D wasn’t hard enough, there are researchers working on generative 3D models. 3D models present an extra challenge because there is paucity of training datasets.In this episode, PhD students Sandeep and Oindrila walked us through their work on creating 3D animals using 2D data. Join us to learn about their pipelines, quality control, tie in with iNaturalist, and how this tech could streamline FX pipelines.

Guests

Sandeep Mishra: About me I am an ECE PhD student at the University of Texas at Austin, guided by Prof. Alan Bovik. I am a member of Laboratory for Image and Video Engineering (LIVE), where we work towards the development and optimization of blind/non-blind image and video quality assessment/enhancement techniques. I am also highly interested in generative modeling for image/video and 3D content, focusing on making generative methods more accessible to the creator community. During my PhD, I have had the opportunity to work at ByteDance and Google Research (ongoing) as a research intern. Before pursuing a PhD, I worked as a Lead Research Engineer at Samsung Research Institute Bangalore in the Vision Research Group, which primarily contributes to the latest vision-based solutions for Samsung’s flagship devices. While at Samsung, we developed an Image Super-Resolution solution for the device Gallery, which has been deployed on various Samsung devices across the globe since 2020. We also developed a Video Super-Resolution solution for the device capture pipeline. I graduated with a dual degree from the Department of Electronics and Electrical Communication Engineering, Indian Institute of Technology Kharagpur. My specialization was Visual Image Processing and Embedded Systems. I also completed a minor in Computer Science and Engineering. Research Interests Applications of generative modeling, image/video quality assessment and enhancement, and self-supervised learning Core technical skills Experience with large-scale distributed training, million-scale dataset generation and handling, and proficient in Python.

Oindrila Saha: I am a Research Scientist at Adobe Research. Previously, I completed my PhD at the UMass Amherst working with Prof. Subhransu Maji. Before that, I was a Research Fellow at Microsoft Research, working with Dr. Prateek Jain and Dr. Harsha Simhadri. I earned my Bachelors and Masters from Indian Institute of Technology Kharagpur. I am interested in generating visual content with control and consistency. I am also interested in exploring effective methods for controlling generation. Internships and Academic Collaborations. I am always looking for motivated students and collaborators. If you would like to explore these research directions through an internship at Adobe Research or an academic collaboration, feel free to reach out!