I’m developing an AI voice conversation application for the client side, utilizing the Play.ai WebSocket API. The documentation is available here.
I completed the initial setup following the instructions provided, and the WebSocket connection was established successfully.
The system transmits audio input in the form of base64Data,
Which I have successfully decoded, as evidenced by the playback of the initial welcome message.
However, when I attempt to send my audio, I encounter an issue. According to their specifications, the audio must be sent as a single-channel µ-law (mu-law) encoded at 16000Hz and converted into a base64 encoded string.
This is precisely what I am attempting to accomplish with the following code:
function startRecording() {
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm;codecs=pcm' });
mediaRecorder.ondataavailable = handleAudioData;
mediaRecorder.start(1000);
})
.catch(error => console.error('Error accessing microphone:', error));
}
function handleAudioData(event) {
if (event.data.size > 0) {
event.data.arrayBuffer().then(buffer => {
const byteLength = buffer.byteLength - (buffer.byteLength % 2);
const pcmDataBuffer = new ArrayBuffer(byteLength);
const view = new Uint8Array(buffer);
const pcmDataView = new Uint8Array(pcmDataBuffer);
pcmDataView.set(view.subarray(0, byteLength));
const pcmData = new Int16Array(pcmDataBuffer);
const muLawData = encodeToMuLaw(pcmData);
const base64Data = btoa(String.fromCharCode.apply(null, muLawData));
socket.send(base64Data);
});
}
}
function encodeToMuLaw(pcmData) {
const mu = 255;
const muLawData = new Uint8Array(pcmData.length / 2);
for (let i = 0; i < pcmData.length; i++) {
const s = Math.min(Math.max(-32768, pcmData[i]), 32767);
const sign = s < 0 ? 0x80 : 0x00;
const abs = Math.abs(s);
const exponent = Math.floor(Math.log(abs / 32635 + 1) / Math.log(1 + 1 / 255));
const mantissa = (abs >> (exponent + 1)) & 0x0f;
muLawData[i] = ~(sign | (exponent << 4) | mantissa);
}
return muLawData;
}
On the proxy side, I am forwarding the request to the API as specified in the documentation:
function sendAudioData(base64Data) {
const audioMessage = {
type: 'audioIn',
data: base64Data
};
playAiSocket.send(JSON.stringify(audioMessage));
console.log('Sent audio data');
}
ws.on('message', function incoming(message) {
baseToString = message.toString('base64')
sendAudioData(baseToString);
});
The console logs appear to be correct although the chunks that I’m sending appear to be much larger than the chunks I’m reviewing in the welcome message, but I am not receiving any response from the API after the welcome message, not even an error message. The file seems to be transmitting endlessly. Could anyone please assist me in identifying the issue?
I’m expecting assistance in case there are any inaccuracies in my method or if I’m overlooking something.
