How to properly scale user’s input on a scaled map in MaplibreGL?

I’ve been using MaplibreGL (which is pretty much MapboxGL) to create a map and place a draggable marker on it. I need the map to be scaled (not resized), so it shows the same area despite the size of the map. However, when downscaling the map, the user’s input seems to become inaccurate.

this is the HTML for the map:

<div class="map-container maplibregl-map mapboxgl-map" style="transform: scale(0.657) translate(40px, 40px); width: 720px; height: 800px;">
<div class="maplibregl-canvas-container mapboxgl-canvas-container maplibregl-interactive mapboxgl-interactive maplibregl-touch-drag-pan mapboxgl-touch-drag-pan maplibregl-touch-zoom-rotate mapboxgl-touch-zoom-rotate">
    <canvas class="maplibregl-canvas mapboxgl-canvas" tabindex="0" aria-label="Map" role="region" width="647" height="719" style="width: 720px; height: 800px;"></canvas>
    <!-- marker -->
    <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="45px" height="45px" viewBox="0 0 200 200" style="transform: translate(-50%, -50%) translate(406px, 335px) rotateX(0deg) rotateZ(0deg); pointer-events: auto;" xml:space="preserve" class="maplibregl-marker mapboxgl-marker maplibregl-marker-anchor-center mapboxgl-marker-anchor-center"><path class="mapmarker" d="M100.3,2.7c-32.5,0-58.9,26.4-58.9,58.9s59.1,133.1,59.1,133.1s58.7-100.5,58.7-133.1 S132.8,2.7,100.3,2.7z M100.5,85.3c-13.3,0-24.2-10.8-24.2-24.2c0-13.3,10.8-24.2,24.2-24.2s24.2,10.8,24.2,24.2 C124.7,74.5,113.8,85.3,100.5,85.3z"></path></svg>
</div>
<div class="maplibregl-control-container mapboxgl-control-container">
    <div class="maplibregl-ctrl-top-left mapboxgl-ctrl-top-left"></div>
    <div class="maplibregl-ctrl-top-right mapboxgl-ctrl-top-right"></div>
    <div class="maplibregl-ctrl-bottom-left mapboxgl-ctrl-bottom-left">
        <div class="maplibregl-ctrl mapboxgl-ctrl" style="display: none;"></div>
    </div>
    <div class="maplibregl-ctrl-bottom-right mapboxgl-ctrl-bottom-right"></div>
</div>

.map-container {
    transition: all .3s;
    transform-origin: 0 0;
}

Whenever dragging the map or the marker, it does not end up where the mouse pointer is. I suppose the mouse coordinates get downscaled aswell but I have no idea how to fix this issue.

I have created a demo fiddle with a downscaled demo map here: https://jsfiddle.net/bjwtcqo6/