EdgeSpeech

You want to develop an app that your users can talk to; what are your options? You could rely on high level cloud AI to process your audio or you could rely on low level audio processing on the device; each has its tradeoffs.

The Hidden Costs of Cloud AI

To keep things simple, you might consider using a cloud AI provider that can already handle audio. However, this simplicity comes with hidden costs. Chief among them are privacy, latency, and service fees that only grow when your user base increases.

Privacy

By its very nature, cloud-based voice AI requires sending data to a third-party server for processing. This can raise privacy concerns, especially in applications that handle sensitive information.

On-device voice AI processes data locally, ensuring that user data remains private and secure.

Latency

Audio data requires significant bandwidth to transmit to the cloud. This can result in latency issues, especially in real-time applications where quick responses are critical.

On-device voice AI has a much shorter round trip meaning your app can respond sooner.

Service Fees

Audio data sent to voice AI cloud services requires processing at the provider which incurs high token usage.

On-device voice AI can reduce this by processing data locally, allowing much smaller payloads to be sent to the cloud requiring fewer tokens to interpret.

The Hidden Complexity of On-Device Audio

If the tradeoffs of cloud AI are too severe you may consider on-device audio processing. However, those cloud AI providers are hiding a lot of complexity in their service; complexity you’ll need to implement locally all by yourself. Setting up a low level real-time audio pipeline comes with a number of challenges.

Voice Activity Detection (VAD)

First, you need to know when someone is speaking. Processing audio consumes resources so it’s important to avoid processing silence. This involves integrating a voice activity detector (VAD) and tuning it for potentially noisy environments.

Speech-to-Text (STT)

Next, you need to convert the speech to text (STT). This involves integrating an STT model that’s both small enough to run on-device and efficient enough to run in real-time.

Text-to-Speech (TTS)

Finally, you need to convert text back into speech (TTS). This requires integrating a TTS model that can run efficiently on the device but also sounds natural; robotic voices aren’t going to cut it.

The Best of Both Worlds

EdgeSpeech is the best of both worlds. It dramatically simplifies on-device audio so you can focus on building your app, while at the same time saving you money by avoiding sending large payloads to cloud providers.

The 96% Savings of On-Device AI

On-device voice AI can save about 96% of the costs associated with cloud AI services.

Consider a voice AI assistant handling 1,000 conversations per day, each lasting 5 minutes. If you used OpenAI Realtime API to do speech-to-speech, this would cost roughly $7,200 per month. Here’s the breakdown:

Meanwhile, using EdgeSpeech to do speech-to-speech on the device means you can send lightweight text-only messages to the ChatGPT API instead. This would cost roughly $281 per month; a savings of more than 25x.

Integrate EdgeSpeech

EdgeSpeech uses a provider and hook pattern. All you need to do is wrap your app in the EdgeSpeechProvider and then call the useEdgeSpeech hook. The hook returns actions to listen and speak as well as reactive values of the speech-to-text (STT) transcripts.

import { EdgeSpeechProvider, useEdgeSpeech } from '@synervoz/edgespeech'

// use the hook to communicate with the local model
function VoiceChat() {
  const { listen, speak, onTranscriptComplete } = useEdgeSpeech()

  onTranscriptComplete(async (text) => {
    const response = await chat(text)
    await speak(response)
  })

  return <Button onPress={listen} title="Start Listening" />
}

// wrap your app in the EdgeSpeechProvider
export default function App() {
  return (
    <EdgeSpeechProvider appId="YOUR_APP_ID" appSecret="YOUR_APP_SECRET">
      <VoiceChat />
    </EdgeSpeechProvider>
  )
}

Conclusion

EdgeSpeech provides a powerful and cost-effective solution for integrating voice AI capabilities into your React Native applications. By processing audio data locally, it addresses privacy concerns, reduces latency, and significantly lowers service fees associated with cloud-based AI services. With its simple integration and easy-to-use hooks, developers can quickly add speech-to-text and text-to-speech functionalities to their apps, enhancing user experience while keeping costs manageable.