How do I append an image after clicking a button and remove it on every other click in a container within JS?

I’m trying to create a mushroom log where a container populates with a unique group of images after pressing different buttons.

An image appears after clicking the first button, but it doesn’t go away after clicking again. What do I need to change in order for the image to disappear on every other click or on other button clicks?

HTML:

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Mushroom Log</title>
    <link rel="stylesheet" href="style.css">
</head>

<body>
    <h1>Mushroom Log</h1>
    <br>
    <section id="container"></section>
    <h2>Select a Mushroom Type</h2>
    <br>
    <button type="button" class="button1">Light-Spored Gilled Mushrooms</button>
    <button type="button" class="button2">Brown-Spored Gilled Mushrooms</button>
    <button type="button" class="button3">Dark-Spored Gilled Mushrooms</button>
    <button type="button" class="button4">Polypores and Crust-Fungi</button>
    <button type="button" class="button5">Puffballs, Earthballs, and Earthstars</button>
    <button type="button" class="button6">Jelly-like Fungi</button>
</body>
<script src="app.js"></script>

</html>

Javascript:

const btn1 = document.querySelector('.button1');

function lightSpored() {
    let beenCalled = false;
    if (!beenCalled) {
        beenCalled = true;
        btn1.addEventListener('click', () => { container.append(fungi1) });
    } else {
        container.innerHTML = ' ';
        beenCalled = false;
    }
}

lightSpored();