panzoom set specific bounds to not show space in container

Currently using timmywils panzoom library: https://github.com/timmywil/panzoom

Im currently trying to build a “map” where you can pan around and click on specific elements. Im currently having some issues with this though.

When panning I dont want to show space thats outside of the bounds if that makes sense.

In the image below, I would imagine the “div.adventure-map” to have a background image.
This would okay to show because the container element behind it is completely covered.
enter image description here

However, this image would NOT be okay because the container element behind it isnt covered completely and would just be black.
enter image description here

I feel like im approaching this problem incorrectly but am having a bit of a brain fart.

Im starting to think maybe just implementing my own drag/pan solution would be better. Would appreciate some guidance/help!