CSS, JS Mousedown Getting Stuck During Drag Event

I’m working on the Etch-a-Sketch project from The Odin Project and I’m trying to make it so that my grid element (‘div’ in this case) changes color when my cursor is over it and while my mouse is down.

The code I have now works, however, there’s an issue with the mousedown portion where it will get intermittently stuck down. It seems to occur during an accidental drag event. I’ll notice a hand drag icon appear on my cursor and then all the other grid elements I mouseover will change color regardless of the mousedown condition.

Is there a way to turn off just the ‘draggable’ portion of a ‘div’ or is there another workaround I should be considering?

const grid = document.querySelector('.grid');
grid.addEventListener('mouseover', colorGrid);

let mousedown = false;
grid.addEventListener('mousedown', function(){
    mousedown = true;
});

grid.addEventListener('mouseup', function(){
    mousedown = false;
});


function colorGrid(e){
    if(mousedown) e.target.style.backgroundColor = markerColor.value;

}

.grid {
    display: grid;
    grid-template: repeat(var(--columns-row), 1fr) / repeat(var(--columns-row), 1fr);
    border: solid black 1px;
    width: 100%;
}

.grid div {
    border: 1px dotted lightgrey;
    background-color: var(--bgColor);
 }

 .grid div:hover {
    background-color: var(--color);
 }