Taichi Nishimura, Ph.D.

Softwares

• Shutoh, A fast scene detector implemented in C++20. Inspired by PySceneDetect, Shutoh aims to provide a powerful and flexible alternative with enhanced performance. Shutoh achieves 2x~4x faster than PySceneDetect and supports not only rule-based (PySceneDetect), but also ML-based and neural-based scene detectors.
• line/lighthouse, A user-friendly library for reproducible video moment retrieval and highlight detection (MR-HD). It supports six models, three features, and five datasets. In addition, we prepare an inference-only API and demo for developers to use MR-HD methods easily.

Publication

International Journal

• Text-driven Affordance Learning from Egocentric Vision, Advanced Robotics (minor revision), Tomoya Yoshida, Shuhei Kurita, Taichi Nishimura, Shinsuke Mori
• Vision-Language Models Learn Super Images for Efficient Partially Relevant Video Retrieval, ACM Transactions on Multimedia Computing, Communications, and Applications (ACM TOMM), Taichi Nishimura, Shota Nakada, Masayoshi Kondo
• Recipe Generation from Unsegmented Cooking Videos, ACM Transactions on Multimedia Computing, Communications, and Applications (ACM TOMM), [code] Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, and Shinsuke Mori
• State-aware Video Procedural Captioning, Multimedia Tools and Applications (MTAP) Vol 82 (24), Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, and Shinsuke Mori
• Structure-Aware Procedural Text Generation from an Image Sequence, IEEE Access Vol. 9, [slide] Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, Yoko Yamakata, and Shinsuke Mori

International Conference

• Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision,

  IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR25), [slide] Tomoya Yoshida, Shuhei Kurita, Taichi Nishimura, Shinsuke Mori (Highlight)
• Language-based Audio Moment Retrieval, IEEE International Conference on Acoustic, Speech, and Signal Processing (ICASSP25), [code] [slide] Hokuto Munakata, Taichi Nishimura, Shota Nakada, Tatsuya Komatsu
• DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information, IEEE International Conference on Acoustic, Speech, and Signal Processing (ICASSP25), Shota Nakada, Taichi Nishimura, Hokuto Munakata, Tatsuya Komatsu
• Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos, IEEE/CVF Winter Conference on Applications of Computer Vision (WACV25), Takehiko Ohkawa, Takuma Yagi, Taichi Nishimura, Ryosuke Furuta, Atsushi Hashimoto, Yoshitaka Ushiku, Yoichi Sato
• Lighthouse: A User-Friendly Library for Reproducible Video Moment Retrieval and Highlight Detection, The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP24) - System Demonstration Track, [code] Taichi Nishimura, Shota Nakada, Hokuto Munakata, Tatsuya Komatsu
• Pre-trained models, Datasets, Data Augmentation, and Inference Time Augmentation for Language-based Audio Retrieval, IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events Workshop (DCASE24 Workshop), Hokuto Munakata, Taichi Nishimura, Shota Nakada, Tatsuya Komatsu
• Training Strategy of Massive Text-to-audio Models and GPT-based Query-Augmentation, IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events Challenge Task 8: Language-Based Audio Retrieval (DCASE24 Challenge), Hokuto Munakata, Taichi Nishimura, Shota Nakada, Tatsuya Komatsu
• Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos, The 1st Workshop on Learning from Procedural Videos and Language in conjunction with CVPR2024 (LPVL24), Takehiko Ohkawa, Takuma Yagi, Taichi Nishimura, Ryosuke Furuta, Atsushi Hashimoto, Yoshitaka Ushiku, Yoichi Sato
• Automatic Construction of a Large-Scale Corpus for Geoparsing Using Wikipedia Hyperlinks, The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING2024), Keyaki Ohno, Hirotaka Kameko, Keisuke Shirai, Taichi Nishimura and Shinsuke Mori
• Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows, The 29th International Conference on Computational Linguistics (COLING2022), [code] Keisuke Shirai, Atsushi Hashimoto, Taichi Nishimura, Hirotaka Kameko, Shuhei Kurita, Yoshitaka Ushiku and Shinsuke Mori
• Multimodal Dish Pairing: Predicting Side Dishes to Serve with a Main Dish, The 1st International Workshop on Multimedia for Cooking, Eating, and related APPlications 2022 in conjunction with ACMMM2022 (CEA++2022), Taichi Nishimura, Katsuhiko Ishiguro, Keita Higuchi, and Masaaki Kotera (Best Paper Award)
• Recipe Recommendation for Balancing Ingredient Preference and Daily Nutrients, The 1st International Workshop on Multimedia for Cooking, Eating, and related APPlications 2022 in conjunction with ACMMM2022 (CEA++2022), Sara Ozeki, Masaaki Kotera, Katsuhiko Ishiguro, Taichi Nishimura, Keita Higuchi
• Multimedia Retrieval of Historical Materials, Digital Humanities 2022 (DH2022), Jieyong Zhu, Taichi Nishimura, Makoto Goto, and Shinsuke Mori
• Image Description Dataset for Language Learners, The 13th International Conference on Language Resources and Evaluation (LREC2022), Kento Tanaka, Taichi Nishimura, Hiroaki Nanjo, Keisuke Shirai, Hirotaka Kameko, and Masatake Dantsuji
• Cross-modal Representation Learning for Understanding Manufacturing Procedure, The 23th International Conference on Human-Computer Interaction (HCII2022), Atsushi Hashimoto, Taichi Nishimura, Yoshitaka Ushiku, Hirotaka Kameko, and Shinsuke Mori
• State-aware Video Procedural Captioning, The 29th ACM International Conference on Multimedia (ACMMM2021), [code] Taichi Nishimura, Atsushi Hashimoto, Yoshitaka Ushiku, Hirotaka Kameko, and Shinsuke Mori
• Egocentric Biochemical Video-and-Language Dataset, The 4th Workshop on Closing the Loop Between Vision and Language in conjunction with ICCV2021 (CLVL2021), Taichi Nishimura, Kojiro Sakoda, Atsushi Hashimoto, Yoshitaka Ushiku, Natsuko Tanaka, Fumihito Ono, Hirotaka Kameko, and Shinsuke Mori
• Visual Grounding Annotation of Recipe Flow Graph, The 12th International Conference on Language Resources and Evaluation (LREC2020), Taichi Nishimura, Suzushi Tomori, Hayato Hashimoto, Atsushi Hashimoto, Yoko Yamakata, Jun Harashima, Yoshitaka Ushiku and Shinsuke Mori
• Procedural Text Generation from a Photo Sequence, The 12th International Conference on Natural Language Generation (INLG2019), [slide] Taichi Nishimura, Atsushi Hashimoto, and Shinsuke Mori
• Frame Selection for Producing Recipe with Pictures from an Execution Video of a Recipe, The 11th Workshop on Multimedia for Cooking and Eating Activities in conjunction with ICMR2019 (CEA2019), [slide] Taichi Nishimura, Atsushi Hashimoto, Yoko Yamakata, and Shinsuke Mori (Best Paper Award)

Preprint

• EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos with Procedural Texts, arXiv:2410.05343, [code] [slide] Yuto Haneji, Taichi Nishimura, Hirotaka Kameko, Keisuke Shirai, Tomoya Yoshida, Keiya Kajimura, Koki Yamamoto, Taiyu Cui, Tomohiro Nishimoto, Shinsuke Mori
• BioVL-QR: Egocentric Biochemical Video-and-Language Dataset Using Micro QR Codes, arXiv:2404.03161, [code] [slide] Tomohiro Nishimoto, Taichi Nishimura, Koki Yamamoto, Keisuke Shirai, Hirotaka Kameko, Yuto Haneji, Tomoya Yoshida, Keiya Kajimura, Taiyu Cui, Chihiro Nishiwaki, Eriko Daikoku, Natsuko Okuda, Fumihito Ono, Shinsuke Mori
• On the Audio Hallucinations in Large Audio-Video Language Models, arXiv:2401.09774, Taichi Nishimura, Shota Nakada, Masayoshi Kondo

Domestic Journal and Conference (Japanese)