Prevent useState from multiplying eventListeners

I’m trying to create some sort of typing test, making letters be removed from a string if they have been typed correctly. My initial approach was to use a state to save the words in, and if the correct key has been pressed, remove the first letter from the string. However, this results in a lot of bugs, such as that after around 20 correct keys, the function starts to remove multiple characters from the string and when pressed on some keys, the string even starts building itself back, pressing on ‘e’ and ‘r’ in my case. It is not an infinite render, so I’m not sure why. I think what is happening is that the eventlistener is multiplying, so it starts to remove multiple letters, if I’d console log the key pressed this confirms this as well. Here is the code.

    const [words, setWords] = useState("a lot of words are put here")
    const removeLetter = (e) => {
        if (e.key === words[0]) {
            setWords((letters) => letters.substring(1));
    }
    document.addEventListener("keypress", function(e) {removeLetter(e)})