Encoding that Works on Localhost Messing Up On Heroku (Apostrophes, quotes, etc.)

I have (very circuitously…I know) figured out a way that (at least on my localhost) I can get an external .txt file from a url saved in my Rails database.

I know it’s not the best Rails-y way (TRUST ME I KNOW) but I have tried so many better ways and everything ends in encoding errors. This is the only way I’ve gotten it to work.

Here’s the way the code looks so far:

<main class="container py-5">
  <header class="text-center">
    <h1 class="header-font">A Place for Testing Stuff...</h1>
  </header>


  <% empty_chapters = Chapter.where(text: nil) %>

  <section>
    <h2>Some Info</h2>
    <ul>
      <li><strong>Total Chapters:</strong> <%= number_with_delimiter(Chapter.all.count ) %></li>
      <li><strong>Empty Chapters:</strong> <%= number_with_delimiter(empty_chapters.count ) %></li>
    </ul>
  </section>

  <section>
    <h2>Empty Chapters For Testing</h2>
    <% empty_chapters.limit(10).each do |chapter| %>

      <% aws_url = "https://the-petulant-poetess.s3.us-east-2.amazonaws.com/stories/#{ chapter.story.author.id }/#{ chapter.id }.txt" %>

      <div class="my-4">
        <h5 class="bg-accent-main color-always-white p-3"><%= link_to chapter.title, chapter_path(chapter) %> (<%= link_to chapter.story.title, story_path(chapter.story) %>)</h5>
        
        <%= simple_form_for [chapter] do |f| %>
            <%= f.hidden_field :text, id: "chapter#{ chapter.id }field" %>
            <%= f.submit "Update Chapter Text", class: "btn btn-subtle" %>
        <% end %>


        <p class="strong mb-0 pb-0">AWS Link</p>
        <a href="<%= aws_url %>" target="_blank"><%= aws_url %></a>
        
        <p class="strong mt-3">Database Chapter</p>
        <%= truncate(chapter.text, length: 1000) %>
        
        <p class="strong mt-3">JS Chapter</p>
        <div id="js<%= chapter.id %>"></div>

      </div>
    <% end %>
  </section>

</main>



<script type="text/javascript">
  window.onload = function () {
    <% empty_chapters.limit(10).each do |chapter| %>
      <% aws_url = "https://the-petulant-poetess.s3.us-east-2.amazonaws.com/stories/#{ chapter.story.author.id }/#{ chapter.id }.txt" %>

      var chapter = new Headers();
      chapter.append('Content-Type','text/html; charset=UTF-8');

      fetch('<%= aws_url %>', chapter)
        .then(function (response) {
            return response.arrayBuffer();
        })
        .then(function (buffer) {
            const decoder = new TextDecoder('iso-8859-1');
            const text = decoder.decode(buffer);
            var array_of_lines = text.split("n");
            array_of_lines = array_of_lines.map(i => '<p>' + i + '</p>');
            var string_of_html = array_of_lines.join("").replaceAll('<p></p>','');
            const element = document.getElementById("js<%= chapter.id %>");
            element.innerHTML = string_of_html
            document.getElementById("chapter<%= chapter.id %>field").value = string_of_html
        });

    <% end %>
  }
</script>

Despite the absolutely ridiculous way this code is Macguyvered together, it does actually work on localhost.

Unfortunately, on my production environment on Heroku the encryption errors are back. Now ' and " turn into â, and I wouldn’t be surprised if there were other things getting mis-encoded too. From the lots of research I’ve done, it looks like a UTF-8 encoding issue, but I can’t see why it happens on Heroku not on localhost.

Can anyone help me get this working? It’s getting ridiculous how many different ways I’ve tried to get a simple .txt url uploaded into my rails database only to get shanghaied by encoding errors.