Deepstory: GPT-2, DCTTS, and talking Witcher characters

Before "multimodal" was a buzzword, I wanted to see whether I could make a Witcher character tell me a brand-new story, in their own voice, with their face moving. That became Deepstory, my New Media graduation project. It now has 103 stars and 23 forks on GitHub.

While building it I kept a dev log of 30 posts. This post looks back at the parts that mattered most.

Three models, one pipeline

1. Write. A GPT-2 model fine-tuned on The Witcher novels continues a prompt (training GPT-2). I later added a second model trained on Waiting for Godot, just because.
2. Speak. A DCTTS model that I trained myself turns the text into speech. The voice data was scraped by hand from The Witcher 3 game, along with its transcripts (audio pre-processing).
3. Move. Speech-Driven Animation and the First Order Motion Model animate a still image of the character so it matches the audio (from SDA to first-order model).

A small Flask app ties it all together.

The unglamorous part

Each model works fine on its own. Chaining them is where things break:

• GPT-2 happily writes 80-word sentences, and the TTS model falls apart on them, so the text needs sentence splitting before synthesis.
• The animation models expect clean, normalized audio, so loudness has to be normalized between stages.
• Holding three models in VRAM on one GPU means loading and unloading them carefully (the memory issue).

The interface mattered more than I expected

The first versions were scripts. They worked, but only I could use them. The project only became an artwork once it had an interface: a page to load or generate text, map each sentence to a speaker, synthesize, review, and animate. I rewrote the Flask interface more than once. In the end, the GPT-2 page could continue a story from the original book and let you pick between several generated continuations, instead of accepting whatever came out first.

That's the moment I understood that putting a model in front of people is a design problem, not just an engineering one.

Looking back

If I built Deepstory today, almost every model would be different:

• Text: a modern LLM writes far more coherent stories than a fine-tuned GPT-2, and it can follow style instructions without fine-tuning.
• Speech: zero-shot voice models can imitate a voice from seconds of audio. Training a separate DCTTS model per character, with all the scraping and cleaning that took, would be unnecessary.
• Video: talking-head models now go from one image plus audio to video in a single step, instead of chaining two animation models.

But the parts that took the most time wouldn't change much. Data work was most of the project: extracting audio, segmenting it, normalizing silences, and checking files by hand. The joins between models still need the most care. And writing as I went made it possible to write this post six years later.

I'd also think harder about ethics. Cloning a voice was a technical stunt in 2020. Today it's easy, and it's a real risk. A project like this should now come with clear consent for any voice it uses, and visible labelling that the output is synthetic.

Deepstory is still on GitHub, dependencies updated, with a Colab demo if you want to try the ancient technology yourself.