Sxian/clt 2919/integrate platform audio by xianshijing-lk · Pull Request #669 · livekit/python-sdks

xianshijing-lk · 2026-05-15T23:31:13Z

Summary

Adds PlatformAudio support to the Python SDK, enabling microphone capture via WebRTC's Audio Device Module (ADM) with built-in voice processing.

Changes

New: livekit-rtc/livekit/rtc/platform_audio.py

PlatformAudio - Main class for ADM access
- recording_devices() / playout_devices() - Device enumeration
- set_recording_device() / set_playout_device() - Device selection by GUID
- create_audio_source() - Create audio source with processing options
PlatformAudioSource - Audio source backed by ADM (no capture_frame() needed)
PlatformAudioOptions - Configure AEC, NS, AGC, prefer_hardware
AudioDeviceInfo - Device info (index, name, id/GUID)

Modified: livekit-rtc/livekit/rtc/track.py

LocalAudioTrack.create_audio_track() now accepts both AudioSource and PlatformAudioSource

Modified: livekit-rtc/livekit/rtc/init.py

Export: PlatformAudio, PlatformAudioSource, PlatformAudioOptions, PlatformAudioError, AudioDeviceInfo

Modified: livekit-rtc/livekit/rtc/room.py

Commented out ready_for_room_event (not yet in released livekit-ffi)

Modified: examples/basic_room.py

Complete rewrite demonstrating both audio modes:
- --platform-audio - Use ADM with voice processing (recommended)
- --file WAV_PATH - Publish WAV file via synthetic mode
- --list-devices - List available audio devices
- --mic-id / --speaker-id - Select specific devices
- --room - Room name
Demonstrates mixing both modes (microphone + file simultaneously)

New: examples/README.md

Documentation for all examples
Detailed explanation of PlatformAudio vs Synthetic mode

PlatformAudio vs Synthetic Mode
┌───────────────────────────────┬───────────────┬───────────────────────────────────────┐
│ Feature │ PlatformAudio │ Synthetic │
├───────────────────────────────┼───────────────┼───────────────────────────────────────┤
│ Voice processing (AEC/NS/AGC) │ Built-in │ Manual │
├───────────────────────────────┼───────────────┼───────────────────────────────────────┤
│ Raw frame access │ No │ Yes │
├───────────────────────────────┼───────────────┼───────────────────────────────────────┤
│ External audio libs needed │ No │ Yes │
├───────────────────────────────┼───────────────┼───────────────────────────────────────┤
│ Use case │ Voice calls │ Custom processing, TTS, file playback │
└───────────────────────────────┴───────────────┴───────────────────────────────────────┘
Both modes can run simultaneously (e.g., mic + background music).

Test Procedure

List audio devices

cd examples
python basic_room.py --list-devices
Expected: Lists available microphones and speakers with device IDs.

Connect with PlatformAudio

Start LiveKit server

livekit-server --dev

In another terminal

export LIVEKIT_URL=ws://localhost:7880
export LIVEKIT_API_KEY=devkey
export LIVEKIT_API_SECRET=secret

python basic_room.py --platform-audio --room test-room
Expected: Connects to room, publishes microphone track with voice processing.

Test with specific device

python basic_room.py --platform-audio --mic-id "" --room test-room
Expected: Uses specified microphone.

Test WAV file playback (synthetic mode)

python basic_room.py --file test.wav --room test-room
Expected: Publishes audio from WAV file.

Test mixed mode (PlatformAudio + file)

python basic_room.py --platform-audio --file test.wav --room test-room
Expected: Publishes two audio tracks - microphone and file.

Verify with second participant

Open https://meet.livekit.io and join the same room to verify audio is received.

- Update rust-sdks submodule to latest main (497527d4) - Regenerate protobuf files - Includes: Fix LocalTrackPublished handle leak, WebRTC ADM integration, and other improvements Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

cloudwebrtc · 2026-05-19T13:26:38Z

@copilot, please check the ruff format CI, and try to fix the format issue.

Copilot · 2026-05-19T13:41:30Z

@copilot, please check the ruff format CI, and try to fix the format issue.

Addressed in 356d5c0. I reproduced the Ruff workflow locally, removed the unused import blocking Ruff, applied ruff format to the files flagged by CI, and make check now passes.

CLAassistant · 2026-05-19T13:41:31Z

All committers have signed the CLA.

devin-ai-integration

Devin Review found 1 new potential issue.

View 11 additional findings in Devin Review.

devin-ai-integration · 2026-06-10T04:33:15Z

+        prefer_hardware: Prefer hardware audio processing when available.
+            - iOS: Uses Voice Processing I/O (VPIO) for low-latency processing
+            - Android: Hardware support varies by device
+            - Desktop: Generally not available
+            Default: True on iOS, False on other platforms.
+    """
+
+    echo_cancellation: bool = True
+    noise_suppression: bool = True
+    auto_gain_control: bool = True
+    prefer_hardware: bool = False


🚩 PlatformAudioOptions.prefer_hardware default vs proto documentation discrepancy

The proto comment for prefer_hardware in audio_frame_pb2.pyi:975-977 says Default: true and the PlatformAudioOptions docstring says Default: True on iOS, False on other platforms, but the Python dataclass default is prefer_hardware: bool = False. Since _to_proto() always explicitly sets this field, the FFI backend will always receive False unless the user overrides it — the backend never gets the chance to apply platform-specific defaults. This may be intentional (letting the Python SDK control the value explicitly), but the docstring claiming 'Default: True on iOS' is misleading since the Python default is unconditionally False.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration

Devin Review found 1 new potential issue.

View 11 additional findings in Devin Review.

devin-ai-integration · 2026-06-10T04:46:14Z

+    prefer_hardware: bool = False
+
+    def _to_proto(self) -> proto_audio_frame.AudioSourceOptions:
+        return proto_audio_frame.AudioSourceOptions(
+            echo_cancellation=self.echo_cancellation,
+            noise_suppression=self.noise_suppression,
+            auto_gain_control=self.auto_gain_control,
+            prefer_hardware=self.prefer_hardware,
+        )


🟡 _to_proto() always explicitly sets prefer_hardware=False, overriding server-side platform-specific defaults

The PlatformAudioOptions dataclass defaults prefer_hardware to False and _to_proto() always sets it explicitly in the proto message. Because the proto field uses optional semantics (verified: HasField('prefer_hardware') returns True when explicitly set to False vs False when unset), the server can distinguish "not set" from "explicitly false". The proto comment at livekit-rtc/livekit/rtc/_proto/audio_frame_pb2.pyi:984 says the server default is true (hardware-accelerated VPIO on iOS). By always sending an explicit False, the Python SDK prevents the server from ever applying its recommended platform-specific defaults. The docstring at line 121 compounds this by claiming "Default: True on iOS, False on other platforms" — behavior the code doesn't implement.

Fix approach

Use Optional[bool] for prefer_hardware with a default of None, and only set the proto field when the user explicitly specifies a value. This lets the server apply its own platform-specific defaults when the user hasn't expressed a preference.

Prompt for agents

The prefer_hardware field in PlatformAudioOptions defaults to False and is always sent explicitly to the server via _to_proto(). Since the proto field uses optional semantics (proto3 optional), the server can distinguish 'not set' from 'explicitly set to false'. The proto comment says the server's default is true (for iOS VPIO hardware acceleration). By always sending an explicit False, the SDK overrides the server's platform-specific defaults. The fix involves: 1. In PlatformAudioOptions (platform_audio.py line 127), change prefer_hardware from bool = False to Optional[bool] = None 2. In _to_proto() (platform_audio.py lines 129-135), only set prefer_hardware in the proto message when self.prefer_hardware is not None 3. Update the docstring (line 121) to clarify that None means 'let the server decide based on platform' This allows the server to apply its recommended default (True on iOS, False elsewhere) when the user doesn't explicitly specify a preference.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration

Devin Review found 0 new potential issues.

View 11 additional findings in Devin Review.

xianshijing-lk requested review from cloudwebrtc and lukasIO as code owners May 15, 2026 23:31

This comment was marked as resolved.

Sign in to view

xianshijing-lk force-pushed the sxian/CLT-2919/integrate_platformAudio branch from f519dde to 52bed9f Compare May 15, 2026 23:39

xianshijing-lk changed the base branch from main to sxian/CLT-2919/update-livekit-ffi-0.12.57 May 15, 2026 23:46

Base automatically changed from sxian/CLT-2919/update-livekit-ffi-0.12.57 to main May 18, 2026 10:26

xianshijing-lk and others added 5 commits May 18, 2026 11:53

Update livekit-ffi to 0.12.57

a4d3a74

- Update rust-sdks submodule to latest main (497527d4) - Regenerate protobuf files - Includes: Fix LocalTrackPublished handle leak, WebRTC ADM integration, and other improvements Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

generated protobuf

30da99a

implement platform audio on python

5f3eca3

added unit tests for platformAudio

40e1e65

addressed the comment and fixed publish_mic.py and local_audio.py

2ef05f0

xianshijing-lk force-pushed the sxian/CLT-2919/integrate_platformAudio branch from 52bed9f to 2ef05f0 Compare May 18, 2026 18:53

updated the readme

f2d2da4

Copilot started work on behalf of cloudwebrtc May 19, 2026 13:37 View session

Copilot finished work on behalf of cloudwebrtc May 19, 2026 13:48

xianshijing-lk added 2 commits June 10, 2026 11:46

Merge branch 'main' into sxian/CLT-2919/integrate_platformAudio

34d5eb3

rebased and fixed the tests

e118dc5

xianshijing-lk force-pushed the sxian/CLT-2919/integrate_platformAudio branch from 356d5c0 to e118dc5 Compare June 10, 2026 04:26

devin-ai-integration Bot reviewed Jun 10, 2026

View reviewed changes

fix the CIs

03b1e05

devin-ai-integration Bot reviewed Jun 10, 2026

View reviewed changes

fix the windows bot

718346c

devin-ai-integration Bot reviewed Jun 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sxian/clt 2919/integrate platform audio#669

Sxian/clt 2919/integrate platform audio#669
xianshijing-lk wants to merge 10 commits into
mainfrom
sxian/CLT-2919/integrate_platformAudio

xianshijing-lk commented May 15, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

cloudwebrtc commented May 19, 2026

Uh oh!

Copilot AI commented May 19, 2026

Uh oh!

CLAassistant commented May 19, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jun 10, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jun 10, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

xianshijing-lk commented May 15, 2026

Start LiveKit server

In another terminal

Uh oh!

This comment was marked as resolved.

Uh oh!

cloudwebrtc commented May 19, 2026

Uh oh!

Copilot AI commented May 19, 2026

Uh oh!

CLAassistant commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented May 19, 2026 •

edited

Loading