[WIP] Unreal Source Explained

Unreal Source Explained (USE) is an Unreal source code analysis, based on profilers.
For more infomation, see the repo in github.

Thread Management

RunnableThread and Runnable

FRunnableThread(link) is a cross platfrom abstract “native thread” interface in Unreal. It has different implementaion in different OS, for example, FRunnableThreadWin(link) in Windows, FRunnableThreadPThread(link) in POSIX-compliant OS, i.e., iOS and Android.

FRunnable(link)’s inherited class represets the actual running workload, e.g., FRenderingThread(link).

Each FRunnableThread runs one FRunnable.

Take the POSIX FRunnableThreadPThread for example, static function _ThreadProc()(link) is the thread entry point and it calls every FRunnableThreadPThread::Run(), as follows.

/**
* The thread entry point. Simply forwards the call on to the right
* thread main function
*/
static void *STDCALL _ThreadProc(void *pThis) {
	FRunnableThreadPThread* ThisThread = (FRunnableThreadPThread*)pThis;
	...
	// run the thread!
	ThisThread->PreRun();
	ThisThread->Run();
	ThisThread->PostRun();

	pthread_exit(NULL);
	return NULL;
}

then, FRunnableThreadPThread::Run() calls FRunnable::Run()(link).

RunnableThread and Runnable is the low-level thread management in Unreal. The following Async Task and Task Graph depend on Runnable.

Queued Thread and Async Task

Queued Thread and Async Task is Unreal’s thread pool implementation. All kinds of tasks are scheduled among a pool of threads.

FQueuedThreadPoolBase(link) manages this task queue and thread queue, as follow,

/**
 * Implementation of a queued thread pool.
 */
class FQueuedThreadPoolBase : public FQueuedThreadPool {
protected:
	/** The work queue to pull from. */
	TArray<IQueuedWork*> QueuedWork;
	/** The thread pool to dole work out to. */
	TArray<FQueuedThread*> QueuedThreads;
	/** All threads in the pool. */
	TArray<FQueuedThread*> AllThreads;
	...
};

There are 4 threads pools(link) in maximum, but the most important is the GThreadPool, which is the default thread pool for most tasks.

FQueuedThread(link) inherits from FRunnable, it represents the task worker thread. Its Run() waits for its DoWorkEvent, if signaled, it runs its current QueuedWork by calling IQueuedWork::DoThreadedWork(), as follow,

class FQueuedThread : public FRunnable {
protected:
	/** The event that tells the thread there is work to do. */
	FEvent* DoWorkEvent;
	/** The work this thread is doing. */
	IQueuedWork* volatile QueuedWork;
	/** The pool this thread belongs to. */
	class FQueuedThreadPool* OwningThreadPool;
	/** My Thread  */
	FRunnableThread* Thread;

	virtual uint32 Run() override {
		while (!TimeToDie.Load(EMemoryOrder::Relaxed)) {
			// We need to wait for shorter amount of time
			bool bContinueWaiting = true;
			while( bContinueWaiting ) {
				// Wait for some work to do
				bContinueWaiting = !DoWorkEvent->Wait( 10 );
			}

			IQueuedWork* LocalQueuedWork = QueuedWork;
			QueuedWork = nullptr;
			FPlatformMisc::MemoryBarrier();
			while (LocalQueuedWork) {
				// Tell the object to do the work
				LocalQueuedWork->DoThreadedWork();
				// Let the object cleanup before we remove our ref to it
				LocalQueuedWork = OwningThreadPool->ReturnToPoolOrGetNextJob(this);
			} 
		}
		return 0;
	}
	...
	void DoWork(IQueuedWork* InQueuedWork) {
		// Tell the thread the work to be done
		QueuedWork = InQueuedWork;
		FPlatformMisc::MemoryBarrier();
		// Tell the thread to wake up and do its job
		DoWorkEvent->Trigger();
	}
};

FEngineLoop::PreInit()(link) calls FQueuedThreadPoolBase::Create()(link) to creates FQueuedThreads.

IQueuedWork(link) is the task interface, it can’t be any simpler:

class IQueuedWork {
public:
	virtual void DoThreadedWork() = 0;
	virtual void Abandon() = 0;

public:
	virtual ~IQueuedWork() { }
};

IQueuedWork’s most important implementation is FAsyncTask(link).
FAsyncTask is a template class, you can create your own async task class as the sample code in FAsyncTask’s comment(link). Various specific tasks start their work as the following.

Task Graph

Parallel programming is hard, multithreaded programming in fine granularity is even harder, because you may spend lots of time to take care of locking, waiting, race condition in every detailed level.

Task Graph is a parallel programming with coarse granularity, and it handles dependency among async tasks. A simplified task graph of a game may be depicted as follow:

We can divide the whole game into several big tasks, tasks in parallel can shared read some data but never shared write, tasks with dependency must finish in order.

FTaskGraphImplementation(link) is the most important manager of Task Graph, it manages an array of FWorkerThreads,

/**
*	FTaskGraphImplementation
*	Implementation of the centralized part of the task graph system.
*	These parts of the system have no knowledge of the dependency graph, they exclusively work on tasks.
**/
class FTaskGraphImplementation : public FTaskGraphInterface {
	...
	/** Per thread data. **/
	FWorkerThread		WorkerThreads[MAX_THREADS];
	/** Number of threads actually in use. **/
	int32				NumThreads;
	/** Number of named threads actually in use. **/
	int32				NumNamedThreads;
	/** Number of tasks thread sets for priority **/
	int32				NumTaskThreadSets;
	/** Number of tasks threads per priority set **/
	int32				NumTaskThreadsPerSet;
	...
}

FWorkerThread(link) is a wrapper of the running thread and the task.

struct FWorkerThread {
	/** The actual FTaskThread that manager this task **/
	FTaskThreadBase*	TaskGraphWorker;
	/** For internal threads, the is non-NULL and holds the information about the runable thread that was created. **/
	FRunnableThread*	RunnableThread;
	/** For external threads, this determines if they have been "attached" yet. Attachment is mostly setting up TLS for this individual thread. **/
	bool				bAttached;
	...
};

FTaskThreadBase(link) implements FRunnable, and it has only two inherited class:

FNamedTaskThread(link), named task thread runs the engine built-in tasks, there are 5 named thread(link): StatsThread, RHIThread, AudioThread, GameThread and ActualRenderingThread
FTaskThreadAnyThread(link), unnamed task thread runs the user defined tasks.

Named task thread are created in various places. But the unnamed task threads are created inside the constructor(link) of FTaskGraphImplementation, we can observe this by the thread creation(link):

You can create your own task as the demo code in TGraphTask’s comment(link). Keep in mind that the task argument can not be references, but pointer is OK.
The following image is the call stacks filtered by “TGraphTask”, you may notice the both the render thread and the game thread use the task graph to accomplish many important tasks, even include the world ticking.

[WIP] Unreal Source Explained

Contents

Thread Management

RunnableThread and Runnable

Queued Thread and Async Task

Task Graph