Nested HTML element not assigning to slot

In the below code I am trying to assign <span slot='test-slot'>b</span> to <slot name='test-slot'>a</slot> but the assignment does not work. If I bring <span slot='test-slot'>b</span> outside of its parent <div> container the assignment does take place as expected.

Why is this? Is there anyway you can assign from nested elements with the slot element? If not, any alternatives? This is obviously a reduced test case but in my real web component, it is much more intuitive for a user to add an element with the slot tag within other containers.

<!DOCTYPE html>
<html>
    <head>
    </head>
    <body>
        <test-element>
            <div>
                <span slot='test-slot'>b</span>
            </div>
        </test-element>

        <template id='template-test-element'>
            <slot name='test-slot'>a</slot>
        </template>

        <script>
            class TestElement extends HTMLElement {
                constructor() {
                    super();

                    // Initialise shadow root and attach table template
                    let shadowRoot = this.attachShadow({ mode: "open" });
                    shadowRoot.appendChild(document.getElementById("template-test-element").content.cloneNode(true));
                }
            }

            customElements.define('test-element', TestElement);
        </script>
    </body>
</html>