Building OpenAI Realtime voice agents in React Native

Putting that same agent into a mobile app is a little different.

Once you get beyond the demo, you start running into problems that don't have much to do with the model itself:

• The agent hears its own voice through the phone speaker.

• Background noise gets mistaken for speech.

• A short pause gets mistaken for the end of a sentence.

• The user interrupts, but the agent keeps talking.

• Your React Native app needs to manage a native audio session across iOS and Android.

• Tool calls need to get routed back into the app and its current state.

We built @synervoz/openai-realtime-toolkit to handle that layer.

It's an MIT-licensed React Native package for building OpenAI Realtime voice agents on iOS and Android. It manages audio capture and playback, the Realtime session, echo cancellation, tool calls, and barge-in. It can also run voice activity detection and semantic turn detection locally on the device when you want more control over when the agent should listen, respond, or stop talking.

The API is intentionally small: one provider and two hooks.

OpenAI handles the intelligence of the agent and the realtime protocol. But if you're connecting directly from a mobile app, there is still a fair amount of plumbing around it.

Mobile audio

You need to capture the microphone, play the model's response, maintain the correct audio route, and deal with things like headphones being connected or disconnected.

Then there's echo cancellation.

If you're using WebRTC, echo cancellation is generally part of the audio stack. If you're connecting directly to Realtime over WebSocket, you need to configure the native platform audio correctly yourself.

On iOS that means using Apple's voice-processing audio path. On Android it means configuring the appropriate communication audio mode and source.

Get it wrong and the model can hear its own output coming back through the microphone.

Turn-taking

OpenAI already provides server-side turn detection, including semantic turn detection, and for many applications it works well.

But mobile environments vary enormously.

Someone using your app in a quiet bedroom is a very different case from someone using it in a car, gym, warehouse, kitchen or café.

Sometimes you want tighter control over questions like:

How loud does something need to be before it counts as speech?

How long should we tolerate a pause?

Does what the user said actually sound like a completed thought?

How quickly should playback stop when the user starts talking?

Those decisions can also be useful to make locally.

For example, if the agent is already playing audio from a buffer on the phone and the user starts speaking, you don't need to wait for a server round trip before turning that playback down.

That immediate response makes interruption feel much more natural.

What the toolkit handles

Without a library, a direct React Native integration typically needs some combination of:

• native microphone capture and audio playback

• audio-session and route management

• acoustic echo cancellation

• the OpenAI Realtime WebSocket session

• reconnect and lifecycle handling

• voice activity detection

• turn detection

• interruption and barge-in behavior

• tool-call dispatch

• iOS and Android native integration

None of these problems is particularly exotic. There's just a lot of them.

And they're mostly infrastructure rather than whatever makes your application unique.

That's the main reason we built the toolkit: we'd rather have developers spend their time on what the voice agent actually does than on getting Android audio routing or interruption timing right.

A voice agent in one screen

A basic agent looks like this:

import {
  OpenAIRealtimeToolkitProvider,
  useOpenAIRealtimeToolkit,
  useTool,
} from '@synervoz/openai-realtime-toolkit'

export default function App() {
  return (
    <OpenAIRealtimeToolkitProvider
      instructions="You are a terse, friendly voice assistant."
    >
      <Screen />
    </OpenAIRealtimeToolkitProvider>
  )
}

function Screen() {
  const { isRunning, start, stop } = useOpenAIRealtimeToolkit()

  useTool({
    name: 'get_time',
    description: 'Get the current time.',
    handler: async () => ({
      time: new Date().toLocaleTimeString(),
    }),
  })

  return (
    <TouchableOpacity onPress={isRunning ? stop : start}>
      <Text>{isRunning ? 'Stop' : 'Start talking'}</Text>
    </TouchableOpacity>
  )
}

That gives you a working OpenAI Realtime voice agent with a tool call.

Underneath, the toolkit creates the audio graph, connects the microphone to OpenAI Realtime, plays the response, enables echo cancellation, and dispatches tool calls back into your application.

Because the tools are just JavaScript handlers, they can interact directly with your app.

For example:

"Show me the cheaper options."

could call a tool that changes a filter in your UI.

Or:

"Book the 3:30 appointment."

could call the same application logic your booking screen already uses.

This is one of the reasons we like running the agent in the client. Voice becomes another interface to the application rather than a separate service sitting beside it.

Local turn detection

Local turn handling is optional.

const { localTurnHandling } = useOpenAIRealtimeToolkit()

localTurnHandling.setEnabled(true)

When enabled, the toolkit uses two stages.

First, on-device voice activity detection decides whether someone is actually speaking.

Then a semantic model estimates whether what they said sounds complete.

Those are different questions.

Consider:

"What's the weather in..."

versus:

"What's the weather in Berlin?"

Both can be followed by exactly the same amount of silence. But only one sounds like a finished thought.

Separating speech detection from semantic completion makes it possible to tune the interaction more intelligently than simply waiting for a fixed silence timeout.

There's also a separate fast path for interruption. When the user starts speaking while the agent is talking, playback can duck immediately, then pause or cancel according to the configured timing.

For most applications you don't need to tune all of this manually. The toolkit includes three presets:

localTurnHandling.setConfig(QUIET_CONFIG)
localTurnHandling.setConfig(BALANCED_CONFIG)
localTurnHandling.setConfig(NOISY_CONFIG)

You can switch them while a session is running.

That makes it practical to use different behavior for, say, a quiet indoor assistant versus a hands-free app intended for a gym or car.

When this architecture makes sense

This toolkit is mainly for developers who:

• are building a React Native app

• want to use OpenAI Realtime directly

• want the voice agent integrated tightly with their existing application

• care about mobile audio behavior and interruption

• don't want to build and maintain a separate agent runtime

It's particularly useful when voice is intended to control the application itself.

Your tools execute directly in the React Native application, so they can work with application state, local storage, device APIs and your existing backend calls.

There isn't another server-side tool protocol required just to get an agent action back into the UI.

When LiveKit is probably a better choice

LiveKit solves a broader problem.

Its agent architecture runs server-side, with the mobile client connecting over WebRTC. That gives you capabilities this toolkit isn't trying to reproduce: telephony and SIP, multi-party rooms, server-side agent workers, observability, provider flexibility, and the ability to build STT → LLM → TTS pipelines using different vendors.

If you're building an agent that needs to work across phone calls, browsers and mobile apps, or you want the agent runtime centralized on your infrastructure, LiveKit is probably the better architecture.

The tradeoff is that your agent is no longer running inside the application.

For mobile applications where voice is primarily another way of controlling the app, we think the client-side architecture is considerably simpler.

A tool handler can just be:

useTool({
  name: 'set_background_color',
  handler: ({ color }) => setBackgroundColor(color),
})

There's no data channel or application protocol required to get that action from a server-side worker back into the component tree.

The other difference is interruption.

Because the toolkit owns playback locally, it can react to detected speech immediately rather than waiting for an interruption signal to make a round trip through the agent infrastructure.

And there is no agent runtime to deploy or scale.

You should still use a small backend endpoint to mint ephemeral OpenAI credentials before shipping. But that's very different from operating the agent itself on your servers.

Why WebSocket instead of WebRTC?

OpenAI recommends WebRTC for client applications, while this toolkit currently connects to Realtime over WebSocket.

That's a tradeoff.

WebRTC is better suited to unreliable networks. Its media transport handles packet loss and network changes more gracefully, which matters when someone moves between Wi-Fi and cellular or has a poor connection.

WebSocket gives us a simpler direct audio path and avoids adding a WebRTC stack to the application.

For applications expected to operate frequently on unreliable mobile networks, WebRTC is worth considering and may be the better choice.

For applications primarily running on good Wi-Fi or cellular connections, we've found the direct architecture attractive enough to make that trade.

Current limitations

The toolkit is deliberately fairly narrow today.

It's for React Native on iOS and Android, and it currently targets OpenAI Realtime.

It doesn't give you a provider-independent STT → LLM → TTS pipeline, telephony, multi-party rooms, or a server-side agent runtime.

The package also contains native code, so Expo users need a development build rather than Expo Go. React Native's New Architecture is required.

And while the wrapper is MIT licensed, the underlying audio engine uses prebuilt Switchboard SDK binaries.

Those constraints are worth understanding before choosing the architecture.

Try it

npm install @synervoz/openai-realtime-toolkit

The READ ME contains the quickest path to a working agent.

There's also documentation for:

Getting started

Turn detection

Tools

Example app

Shared test credentials are included so you can run the example before setting up your own. They're intended for evaluation only; use your own ephemeral credentials before shipping.

The project is built on the Switchboard SDK.

If you're building something adjacent — another model provider, another platform, or a more complicated realtime audio pipeline — open an issue. We're interested in seeing where people take it.