Hello I’m stuck in making an API to return chunked data to Nuxt3 frontend in realtime. I’m actually trying to implement streaming of ChatGPT API responses that this git codes achieve for Next.js in my Nuxt3 app.
https://github.com/shivanshubisht/chatgpt
The following is the data flow which I’m trying to make happen.
(a) Nuxt3 frontend <-> (b) Wrapper API running on nitro for Nuxt3 <-> (c) ChatGPT API Handler on nitro <-> (d) ChatGPT API
Environment: “nuxt”: “^3.2.3”, Windows 11, node.js 16
I have a problem with data transfer between (b) wrapper and (c) handler, while (c) handler itself perfectly works to pull stream data from ChatGPT API in realtime. There seems an issue with either/both (c) to return the same chunked data that it received from ChatGPT or/and (b) to receive the data from (c) in stream. I guess that both have issue, though.
Here are the codes that I think have some problems:
(b) Wrapper API: /server/api/response.post.ts
import { OpenAIStream, OpenAIStreamPayload } from '@/server/api/openaistream'
type RequestData = {
currentModel: string
message: string
}
export default defineEventHandler(async (event) => {
const { message } = await readBody(event);
const payload: OpenAIStreamPayload = {
model: 'gpt-3.5-turbo',
messages: [{ role: 'user', content: message }],
temperature: 0.7,
top_p: 1,
frequency_penalty: 0,
presence_penalty: 0,
max_tokens: 500,
stream: true,
n: 1,
}
const stream = await OpenAIStream(payload);
return new Response(stream);
})
(c) ChatGPT handler: /server/api/openaistream.ts
import {
createParser,
ParsedEvent,
ReconnectInterval,
} from 'eventsource-parser';
export type ChatGPTAgent = 'user' | 'system';
export interface ChatGPTMessage {
//abbreviated
}
export interface OpenAIStreamPayload {
//abbreviated
}
export async function OpenAIStream(payload: OpenAIStreamPayload) {
const encoder = new TextEncoder();
const decoder = new TextDecoder();
let counter = 0;
const res = await fetch('https://api.openai.com/v1/chat/completions', {
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${process.env.OPENAI_API_KEY ?? ''}`,
},
method: 'POST',
body: JSON.stringify(payload),
});
const stream = new ReadableStream({
async start(controller) {
// callback
function onParse(event: ParsedEvent | ReconnectInterval) {
if (event.type === 'event') {
const data = event.data;
// https://beta.openai.com/docs/api-reference/completions/create#completions/create-stream
if (data === '[DONE]') {
console.log("END!");
controller.close();
return;
}
try {
const json = JSON.parse(data);
const text = json.choices[0].delta?.content || '';
// check if it receives chunked data from chatgpt API
console.log(text);
if (counter < 2 && (text.match(/n/) || []).length) {
// this is a prefix character (i.e., "nn"), do nothing
return;
}
const queue = encoder.encode(text);
controller.enqueue(queue);
counter++;
} catch (e) {
// maybe parse error
controller.error(e);
}
}
}
for await (const chunk of res.body as any) {
//check if it can decode the chunk data to send as stream data
console.log(decoder.decode(chunk))
parser.feed(decoder.decode(chunk));
}
},
});
return stream;
}
I guess that h3/nitro for Nuxt3 might work differently in handling stream data from Next.js/Vercel but I can’t figure out how it should work because of very few resources. Could anyone please share any thoughts or suggestion? Thank you!
The API handler properly sends chunked stream data to another API running on nitro in realtime.
