How to maintain scroll position when moving element

I have an element (container) with horizontal scroll. For example:

<div class="rows">
  <div class="row"></div>
  <div class="row"></div>
  <div class="row"></div>
  <div class="row">
    <div id="container"></div>
  </div>
  <div class="row"></div>
  <div class="row"></div>
  <div class="row"></div>
  <div class="row"></div>
  <div class="row"></div>
</div>

If you move the element to another row, it will reset the horizontal scroll in the container. Why is that? How can I maintain the scroll position? Is it possible to fix with only CSS?

Here’s a demo:
https://jsbin.com/gugotewane/2/edit?html,css,js,output