Smooth playback of real-time AMR audio packets in web client

I’m developing a real-time PTT (Push-to-Talk) system. The server receives compressed AMR packets, decompresses them to WAV format, and sends them to a web client. However, the audio playback on the client isn’t smooth. I suspect it’s because each packet sent to the client includes the entire WAV header, causing interruptions.
How can I optimize this process for smoother real-time audio playback? What’s the best practice for handling streaming audio in this scenario?

Here’s a snippet of what I’m currently working with on the client side:

class AudioPlayer {
    constructor(websocketUrl) {
        this.socket = new WebSocket(websocketUrl);
        this.audioContext = new (window.AudioContext || window.webkitAudioContext)();
        this.socket.onmessage = (event) => this.handleAudioData(event.data);
    }

    async handleAudioData(data) {
        const arrayBuffer = await data.arrayBuffer();
        const audioBuffer = await this.audioContext.decodeAudioData(arrayBuffer);
        this.playAudioBuffer(audioBuffer);
    }

    playAudioBuffer(audioBuffer) {
        const source = this.audioContext.createBufferSource();
        source.buffer = audioBuffer;
        source.connect(this.audioContext.destination);
        source.start();
    }
}

Thanks in advance for any help or insights!