Open source · 2020
Deepstory
Turn any text into a video of a character telling it, in their own voice
My New Media graduation artwork (SM4701): a Flask web app that chains GPT-2, a custom DCTTS voice model, Speech-Driven Animation, and the First Order Motion Model, so a character writes, speaks, and acts out your story.
- PyTorch
- GPT-2
- DCTTS
- Flask
- First Order Motion
- GitHub stars
- 103
- From the open-source community.
- Forks
- 23
- Other people building on it.
- Dev log
- 30+
- Posts written while building it.
- Models
- 4
- Chained into one pipeline.
The idea
Deepstory is an artwork that combines natural language generation, text-to-speech, and facial animation in one media application. You give it a text, or let GPT-2 write one, and it turns that into a video of a character speaking your story in their own voice.
The pipeline
Deepstory is a Flask web app with several PyTorch models behind it:
- Text: a GPT-2 model fine-tuned on The Witcher books (and one on Waiting for Godot) continues or writes the story.
- Speech: custom-trained DCTTS models for Geralt and Yennefer synthesize each sentence. I scraped and cleaned the voice lines and transcripts from The Witcher 3 myself.
- Animation: Speech-Driven Animation generates the talking face, and the First Order Motion Model transfers it onto a still character image.
The hard parts
Chaining models multiplies their failure modes. A lot of the work went into the joins between stages:
- Text normalization happens at synthesis time, so punctuation survives and can drive variable pauses in the audio.
- Sentence-to-speaker mapping, so not every line gets voiced by the same character.
- Trimmed, temp-file audio so the animation lines up and RAM and VRAM don't overflow.
- GPT-2 token limits: inference is capped at the nearest 1024 minus the predicted length.
Dev log
I documented the whole build, from scraping audio files to the memory leaks, in a 30+ post dev log. You can also try the live demo on Google Colab.