Scriptable Cinematics with Coroutines

I've been working on and off on a small 2D ARPG for a year or so. There's very little to show right now (and it's almost entirely using placeholder assets) but I'm incredibly proud of some of the underlying tech.

The custom engine is built on top of MonoGame (C#) and includes a fairly comprehensive Lua scripting system (via MoonSharp). One of the cool features of Lua is support for coroutines - a form of cooperative multitasking. You can start a coroutine and that coroutine can pause itself to be resumed later.

Building Cinematics

One area that's particularly suited for using coroutines is scripted cinematics. A cinematic script can be a coroutine that yields whenever some action needs to take multiple frames or whenever the script needs to wait for something to complete. Using a coroutine means I can write these scripts completely linearly. Here's a short example of something that would actually work in my engine:

-- These are just aliases of existing functions to make things a bit faster to
-- write.
local yield = coroutine.yield
local CA = CinemaActions

return Cinematic("MyFirstCinematic", function (data)
	local scene = data.Parameters["Scene"]
    
    -- Get our "actors" for the cinematic
    local person1 = scene:FindEntityByName("Person1")
    local person2 = scene:FindEntityByName("Person2")
    
    -- We need to register the fact that we're using these entities as actors
    -- in our cinematic. Other systems (like player input) can disable or
    -- cancel themselves if they see that an entity is taking part in a
    -- cinematic.
    yield (CA.ActorRegistration{person1, person2})
    
    -- Have person1 walk to person2
    yield (CA.NavigateToEntity(person2, person1))
    
    -- Wait 2 seconds
    yield (2)
    
    -- Have person2 walk to (100, 200)
    yield (CA.NavigateToPosition({100, 200}, person2)
    
    -- Have both people walk to each other in parallel
    yield (CA.Parallel(
    	CA.NavigateToEntity(person2, person1)
        CA.NavigateToEntity(person1, person2)
    ))
    
    -- All done!
    -- Actors automatically get released at the end of the cinematic
end)

Of course, there's a lot going on behind the scenes here. There's a navigation system behind NavigateToEntity and NavigateToPosition, and there's some scene management going on at the top of the script.

But the focus here is the use of yield() - every time we call that we return a value to the cinematic system. In some cases that value is an action - like NavigateToEntity - in which case the action will be executed to completion before our coroutine continues. In other cases the action is a number in which case the cinematic waits that long (in seconds) before continuing our coroutine.

Cinematic Actions

What's really cool about this system is that not only can you write cinematics linearly in Lua, but it cleanly integrates with C# as well. C# has support for defining iterators as functions with similar syntax to Lua's yield:

public IEnumerator<int> MyIterator()
{
    yield return 1;
    yield return 2;
    yield return 5;
}

foreach (int value in MyIterator())
{
	Console.WriteLine(value); // Prints 1, then 2, then 5
}

Sort of similar to Lua coroutines, right? Which means we can implement anything from individual actions in a cinematic to entire scripts in a similar way on the C# side. For example, here's a trivial implementation of an action that just waits a few seconds before letting further actions continue:

public class WaitAction
{
    private float timeLeft;
    
    private WaitAction(float timeToWait)
    {
    	this.timeLeft = timeToWait;
    }
    
    // The private constructor and static function here are just to keep
    // the API as clean as possible - only one way to create this action from
    // both C# and Lua. The [ScriptFunction] attribute is how we register this
    // into the CinemaActions module for Lua.
    [ScriptFunction("Wait", Module = CinemaActionsModule.Name)]
    public static CinemaAction Create(float timeToWait)
    {
    	return new WaitAction(timeToWait).Run;
    }
    
    private IEnumerator<object> Run(ICinematicData data)
    {
    	while (this.timeLeft > 0)
        {
            this.timeLeft -= (float)data.GameTime.ElapsedGameTime.TotalSeconds;
            
            // yielding null ends the current frame but continues
            // execution next frame.
            yield return null;
        }
    }
}

A bit more boilerplate - mostly because this action should be exposed to both Lua and C# - but still pretty simple. Here's another simple example showing how actions can be nested:

private IEnumerator<object> Run(ICinematicData data)
{
    // Wait 10 seconds
    yield return WaitAction.Create(10);
    
    // do something, idk
    
    // Wait 5 seconds
    yield return WaitAction.Create(5);
}

Of course, nesting actions can be done over and over again, though usually you don't need to nest things too much.

Backend Implementation

The implementation of the cinematic system is done on the C# side. The basic idea is that the root cinematic object maintains a "stack" - the active action is always on the top of the stack. Actions can be from C# as IEnumerator<object> or Lua coroutines. At the start of a cinematic the initial action (usually the Lua script itself) is added as the only item on the stack.

Every frame, the cinematic gets the next result from the active action, running whatever logic is inside it. When the action returns, the return value is examined:

• If the action didn't return any more values (i.e. we reached the end of the enumerator) then we remove the action from the stack and immediately continue the new topmost action (or end the cinematic if nothing is left)
• If it's null (or nil, if you're using Lua) the return value is ignored and the game proceeds to the next frame.
• If the value implements IEnumerator<object> it is placed at the top of the stack to be executed immediately
• If the value is a CinemaAction delegate (delegate IEnumerator<object> CinemaAction(ICinematicData data)) then it is immediately executed and the resulting IEnumerator<object> is added to the stack and executed.
• If the value is a Lua function then the function is added to the stack and immediately executed as a coroutine.
• If the value is a number then a WaitAction is added to the stack and immediately executed. Other "shortcuts" work similarly.

There are some other minor complications thrown in as well - parallel actions are a bit harder to deal with and don't work with the stack. Currently the cinematic spins up "sub-cinematics" for each action running in parallel - that way each parallel action gets its own stack.

Closing Words

I'll leave off with one of the more interesting bits of cinematic scripting I've done. Recently I've been writing scripts for the combat tutorial - and part of the flexible nature of this cinematic system is that I can have the player perform actions while the cinematic continues running in the background, waiting for them to complete a task. So to end this here's a section of that script that starts the tutorial combat encounter:

local encounter = scene:FindEntityByName("Enc_Dummies"):GetComponent(Components.Encounter)

local lowHealth = false

-- This will be called every time the player gets hurt during
-- this part of the tutorial. If the player is about to die,
-- we set a flag and reset their health back to max.
local modifyHit = function (_, e)
    local h = player:GetComponent(Components.Health)
    if e.Damage >= h.Health then
        lowHealth = true
        e.Damage = 0
        h.Health = h.MaxHealth
    end
end

player:GetComponent(Components.DamageReceiver).ModifyDamageEvent.add(modifyHit)

-- Remove the player from the cinematic so player input starts working again
data.Cinematic:DeregisterActor(player)

-- Enables AI for the encounter
encounter:BeginEncounter()
while encounter.Running do
    if lowHealth then
        -- Let the player know they ran out of health (though we reset it in
        -- modifyHit) and then continue the encounter.
        encounter:PauseEncounter()
        yield (CA.ActorRegistration{player})
        yield (CA.Dialogue(dialogue, "swordtut_fail"))
        
        -- Continue the encounter - we're not resetting anything but the player
        -- health so it should get a bit easier if they've gotten any hits on
        -- enemies.
        data.Cinematic:DeregisterActor(player)
        encounter:BeginEncounter()
        lowHealth = false
    else
        -- do nothing until the encounter is over or the player has low health
        yield (nil)
    end
end

-- Remove our modify damage function
player:GetComponent(Components.DamageReceiver).ModifyDamageEvent.remove(modifyHit)