Conversational AI Avatar Study: LLM and Audio2Face Comparison

TTFT stands for “Time To First Token” or, in the audio context, the time to the first audio byte. It is the interval from when the user stops speaking until the first audio chunk of the language model response arrives. It does not include Audio2Face or the playback buffer: it is purely the LLM “thinking” time.

The lower the TTFT, the sooner the audio can start being processed. And the more stable it is (lower standard deviation), the more predictable the user experience.

Another common metric in model evaluation is TPOT (Time Per Output Token), but this study did not measure it because in this scenario it does not provide relevant information for the user experience. Once the first audio chunk is received, the bottleneck shifts to Audio2Face, which has enough time to process subsequent chunks as they continue to arrive. Therefore, TTFT is the metric that best represents the perceived responsiveness of the system.

All providers were measured under the same conditions to ensure results are comparable:

• Voice Activity Detector (VAD): the same acoustic detector in all cases, based on signal level (RMS), with a threshold of 0.01 and a 150 ms confirmation window. It activates when the user stops speaking.
• Server silence: 200 ms for all providers, the extra time the server waits before sending the request to the LLM.
• Test questions: 20 questions organized into four difficulty levels (5 easy, 5 medium, 5 complex, 5 very complex), with 1 warm-up question excluded from the analysis.
• Deployment region: GPT Realtime is served from the Azure Sweden Central region. Gemini Live does not expose a fixed region; Google automatically routes requests.

Each of the 20 questions was executed once per provider, recording the TTFT obtained. Aggregate metrics (mean, standard deviation, minimum and maximum) were calculated from the set of 20 measurements per model.

Questions were classified based on the expected reasoning effort from the model. Easy questions correspond to greetings or simple general knowledge; medium questions require brief explanations; complex questions involve reasoning or recommendation elaboration; and very complex questions demand extensive responses with planning, comparison or detailed concept explanation. The full list can be found in the appendix.

The most significant finding is each model behavior as complexity increases. Gemini 3.1 Live maintains a nearly constant latency (~1,400 ms) regardless of difficulty. GPT models, on the other hand, scale significantly: GPT-Realtime goes from 1,201 ms on easy questions to 2,308 ms on very complex ones.

Note: GPT-Realtime was deployed in the Azure Sweden Central region, while Gemini 3.1 Live uses Google automatic infrastructure routing. Consequently, some of the observed differences in TTFT may be influenced by network latency between the client and each provider infrastructure.

Gemini is the fastest and most consistent. With a mean of 1,407 ms and a standard deviation of only ±118 ms, Gemini 3.1 Live is the most predictable provider. It responds consistently within the same range, regardless of the question asked.

Gemini latency does not scale with complexity. Easy question: 1,372 ms. Very complex question: 1,416 ms. The difference is just 44 ms, essentially statistical noise. This suggests the model is optimized for constant latency, not for spending more time reasoning based on difficulty. During testing, no significant differences in response quality were observed that would justify this constant latency. The most interesting finding is not who is faster, but that Gemini has constant latency regardless of complexity while GPT scales its reasoning time with difficulty.

GPT-Realtime is the fastest on easy questions. On simple questions it reaches 1,201 ms, faster than Gemini. The problem arises with complexity: on very complex questions it scales to 2,308 ms, nearly double.

GPT-Realtime-Mini has severe outliers. Two very complex questions spike its latency to 4,446 ms and 3,781 ms. With a standard deviation of ±872 ms it is the most unpredictable of all, not recommended for production with complex questions.

Response quality. Beyond latency, qualitative differences were observed between models:

• Gemini: the only provider that responds with a plausible time when asked. It maintains consistent voice and tone throughout the conversation.
• GPT-Realtime, 1.5 and 2: they refuse to give the time, likely due to lack of access to the system clock. Unexpected voice changes were also detected during conversation: at certain moments the assistant would noticeably shift its voice timbre or intonation, occasionally sounding like a voice of the opposite gender. These changes disrupted the sense of continuity.
• GPT-Realtime-Mini: fails at practical reasoning. When asked whether it is better to walk or drive to a car wash 20 meters from home, it suggested driving. It was the only model that answered incorrectly.

These observations are qualitative, based on experience during testing. They are not part of a systematic quality study, but they are relevant to the production decision.

NVIDIA Audio2Face does not process the full audio at once. Instead, it receives it divided into small fragments (chunks) that are sent continuously via gRPC. Each chunk contains a few hundred milliseconds of audio and, as soon as Audio2Face receives one, it begins generating the corresponding facial animations without waiting for the rest of the sentence.

Chunk size represents a trade-off between latency and efficiency: smaller chunks allow avatar animation to start sooner, but increase the number of requests and processing overhead. Larger chunks reduce that overhead but require waiting longer before the animation can begin.

• Test environments: RTX 4070 (laptop, local Docker), RTX 4080 Super (desktop, local Docker), RTX 5090 (desktop, local Docker) and NVIDIA A100 (Azure Container App). A detailed hardware list can be found in the appendix.
• Study design: sweep of 25 chunk sizes (from 300 ms to 1,500 ms in 50 ms steps), with 5 chunks per step (125 chunks total per environment). The gRPC channel is reused to eliminate connection overhead.

The RTX 5090 achieves the lowest latency across all chunk sizes. However, the difference compared to the RTX 4080 Super is relatively small (24-63 ms in the tests performed), so the impact on total system latency is limited. From a latency standpoint, both GPUs offer very similar behavior.

Regarding the NVIDIA A100 deployed on Azure Container Apps, its performance is also very close to the RTX 5090, with a small additional latency attributable to network communication. This type of deployment is an alternative for architectures where Audio2Face runs centrally. However, this study is limited to latency analysis and does not evaluate aspects such as operational cost or performance under concurrent load.

The results and recommendations of this study should be interpreted with the following constraints in mind:

• Single-user environment. All tests were conducted with a single concurrent user. System behavior under multiple concurrent load, especially in the A100 cloud configuration, has not been evaluated and may differ significantly from the values presented here.
• Uncontrolled network latency. The TTFT of each provider includes network latency between the client and the model infrastructure. GPT-Realtime was served from Azure Sweden Central, while Gemini 3.1 Live uses Google automatic routing based on client location. Some of the differences observed between providers may be due to this variable, not to the model intrinsic performance.
• Non-systematic quality evaluation. Observations about response quality (coherence, voice tone, reasoning capability) are qualitative and were collected during latency tests. They are not part of a structured quality study and should not be taken as definitive conclusions about each model capabilities.
• Synchronization buffer and render excluded. The total reported latency does not include the synchronization buffer between audio and blendshapes, nor the Evergine render time. Both components are constant across configurations and do not affect the comparison, but they do affect the actual latency perceived by the user.

Appendix: Question Bank Used

Appendix: Test Environments