Generation of vignette illustrations from video

Summary

A variety of summarization techniques have recently been published to manage the growing volume of media data, but most of them are oriented toward homogeneous media conversion and are limited in terms of compression ratio. In this study, we focus on vignette illustration. An Example of vignette illustration is shown below.

 

Vignette illustration expresses the story content of a video work or game in a compact way and
allows the viewer to understand the world perspective with a single glance. If video can be converted to vignette illustration, it is expected to provide a more highly compressed summary of the media information. In addition, visualization of the world view is also realized at the same time. In this study, we propose VigNet (video image generative network) for semiautomatic generation of vignette illustration.

First, input a video into VigNet. The following is an example of an input video.

video Alike．Licensed under CC BY 4.0

Second, VigNet selects some frames for the character and some frames for the stage. Users can choose their favorite frame images from the group of character frames and the group of stage frames.

Character frame examples. Each character detected is enclosed in a red frame.

Stage frame examples

Finally, VigNet automatically generates a vignette illustration from the two frame images selected by the user. This allows the system to generate the following vignette illustrations while reflecting viewer preferences

Members

NameAffiliationWeb site
Mayu NamaiKeio University

Publications

International Journals

  1. Mayu Namai, Issei Fujishiro：”VigNet: Semiautomatic generation of vignette illustrations from video,” to appear in Transactions on Image Electronics and Visual Computing, Vol, 12, No. 1 (special issue on IEVC 2024), June 2024

Presentations (Refereed)

  1. Mayu Namai, Issei Fujishiro：”Prototyping of semiautomatic generation of vignette illustrations from video,” in proceedings of Visual Computing 2023, pp. 2:1―2:4, Shibaura Institute of Technology, September 18, 2023 (poster)

Presentations (Unrefereed)

  1. Mayu Namai, Issei Fujishiro：”Semiautomatic generation of vignette illustrations from video: Reflecting viewer preferences,” in Proceedings of Expressive Japan 2023 (The Institute of Image Information and Television Engineers Technical Report, Vol. 47, No. 9, pp.301-304, AIT2023-120), Tokyo Polytechnic University, March 6, 2023
  2. Mayu Namai, Issei Fujishiro：”Semiautomatic generation of vignette illustrations from video,” in Proceedings of the 85th National Convention of International Processing Society of Japan, Vol. 2, pp. 255―256 (6X-03), The University of Electro-Communications, online & onsite hybrid, March 4, 2023
  3. Mayu Namai, Issei Fujishiro：”Prototyping toward automatic generation of vignette illustrations,” in Proceedings of Visual Computing Workshop 2022 in Lake Suwa, Lecture No. 16, Nagano, November 25–26, 2022, [Abstract: The Journal of the Institute of Image Electronics Engineers of Japan, Vol. 52, No. 1, pp. 252–253, January 2023]

Others

  1. Issei Fujishiro, Mayu Namai：”Visual Intelligence: The World of AI-Driven Graphics (2) VigNet: Semiautomatic generation of vignette illustrations from video,” KEIO TECHNO-MALL 2023, Tokyo International Forum, December 15, 2023 (poster)
  2. Mayu Namai，Issei Fujishiro：”Prototyping toward automatic generation of vignette illustrations,” Information Visualization Camp, Poster Presentation, Tokyo Institute of Technology, September 24-25, 2022

Awards

Domestic awards

  1. Mayu Namai：the 2023 Masakazu Nakanishi Encouragement Awards，Graduation Research, Department of Information and Computer Science, Faculty of Science and Technology, Keio University, Semiautomatic generation of vignette illustrations from video，March 23, 2023
  2. Mayu Namai，Issei Fujishiro：Excellent poster awards・studioVOXEL award，in Proceedings of Expressive Japan 2023, Semiautomatic generation of vignette illustrations from video: Reflecting viewer preferences，March 6, 2023
  3. Mayu Namai，Issei Fujishiro：Student encouragement awards，in Proceedings of the 85th National Convention of International Processing Society of Japan，Semiautomatic generation of vignette illustrations from video, March 4, 2023

Grants

  1. Grant-in-Aid for Challenging Research (Pioneering)：20K20481 (2022―2023)
  2. Grant-in-Aid for Challenging Research ((Exploratory)：23K18468 (2023―)

Back to SAM team page