How to get image metadata into useChat within Svelte?

I am writing a ChatBot using Svelte and am using the useChat function to interface with my backend to talk to an LLM.

<script lang="ts">
    import ChatList from '$lib/components/ChatList.svelte';
    import PromptForm from '$lib/components/PromptForm.svelte';
    import { useChat, type Message } from 'ai/svelte';
    import { API_BASE_URL } from '$lib/config';

    export let id: string | undefined;
    export let initialMessages: Message[] | undefined;

    const { messages, append, isLoading, input } = useChat({
        initialMessages,
        id,
        api: `${API_BASE_URL}/api/chat`,
    });

    let tempUserInput = '';

    async function handleFinished() {
        await append({ id, content: tempUserInput, role: 'user' });
        tempUserInput = '';
    }
</script>

<div class="flex flex-col min-h-screen bg-background">
    <div class="flex-1 overflow-y-auto pt-6">
        <ChatList {messages} />
    </div>

    <div class="fixed inset-x-0 bottom-0 bg-metallic-gradient">
        <div class="mx-auto sm:max-w-2xl sm:px-4 flex gap-2">
            <div
                class="space-y-4 border-t bg-background px-4 py-2 shadow-subtle sm:rounded-t-xl sm:border md:py-4 flex gap-2 items-center grow"
            >
                <PromptForm on:submit={handleFinished} {input} {isLoading} />
            </div>
        </div>
    </div>
</div>

This is working fine for just chats, but sometimes I want to be able to get back images also from the LLM. According to the documentation, I think I can use experimental_attachments for this.

But I am not sure what to return from my backend to get the information about additional attachments into this experimental_attachments field.

Right now, I am returning a string, and that automatically seem to go inside the message.content field.

I tried returning a json instead that looks like this –

{
    'content': self.settings['dev_test_string'],
    'role': 'assistant',
    'experimental_attachments': {
        'content_type': 'img',
        'url': 'some_string'
    }
}

But this parses the whole of the return json into the message.content field. What should I return from the backend so that I have more control over what goes into the message object?