Adding fixed position on scroll causes flickering element

I have a use case, where I need to make en element positioned fixed whenever you scroll below it.
I can’t use position: sticky in this case as the html is not right for it.

I’m using IntersectionObserver to check scroll position, and adding a class with position: fixed.

However, adding position fixed causes the fixed element to flicker/blink like crazy. How can I prevent this?

Js:

const watcher = document.querySelector('.sticky-nav');
const createStickyTOC = () => {
    const toc_config = {
        root: null,
        rootMargin: '0px 0px 0px 0px',
        threshold: [.9]
    }

    const toc = (entries) => {
        entries.forEach((entry) => {
            // Sticky when scrolling below target
            entry.target.nextElementSibling.classList.toggle('sticky', !entry.isIntersecting && window.scrollY >= entry.boundingClientRect.top);
        })
    }

    if ('IntersectionObserver' in window) {
        const toc_observer = new IntersectionObserver(toc, toc_config);
        toc_observer.observe(watcher);
    }
};

css:

.sticky {
    position: fixed;
    top: 0;
    left: 0;
    z-index: 10;
}

See jsFiddle here.