Symfony deserialize ManyToMany relation into existing objects

I have been trying to deserialize a JSON object with Doctrine ManyToMany relation.

The Categories are always new instances instead of existing objects. My hope is that someone here can show me how to recursively find the objects and replace them with existing objects (and merge the changes if needed ).

I want to use Symfony components and not a 3rd party library.

{
    id: 12,
    name: "360 Wallmount",
    categories: [
        {id: 15},
        {id: 12}
    ]
}

When deserialize the JSON above with Symfony deserializer

$entity = $this->serializerManager->deserialize($product, $request->getContent());

It does not replace the product categories with existing doctrine objects

Here is the Deserialize function

public function deserialize($entity, mixed $data, $groups = null)
{
    $context = [
        AbstractObjectNormalizer::DEEP_OBJECT_TO_POPULATE => true,
        AbstractNormalizer::OBJECT_TO_POPULATE => $entity,
        AbstractNormalizer::GROUPS => $groups
    ];

    return $this->serializer->deserialize($data, $entity::class, 'json', $context);
}