How to correctly use UseEffect with Nested And Paginated Fetches

Please roast my code here.

I am using UseEffect to trigger a request to fetch some data that my React Component will render. Technically, what I have is working. However, I feel like I am not using hooks correctly in this scenario. There’s multiple pieces of state I am working with and I am also hanging on to some data and checking it against state before updating it. It feels very hacky but I have had a hard time coming up with an improved solution.

For describing this I will use more of an analogous example rather than the actual data we are working with. Who doesn’t love authors and books?

The fetch queries are somewhat complex because we need to:

1. Fetch a list of authors. The list we get back will be used to nest the secondary queries. Each record looks like: { id: integer, name: string, books: Book[]}, where Book looks like {id: integer}. This happens with an existing custom hook that handles things like pagination for us.
2. Because the original list only contains the id of the books for each author we need to perform a fetch for a list of books based on the author’s book ids. The books we get back from the original request look like: {id: integer, author_id: integer, name: string}.
3. The API we are using has a pagination limit of 12. When we fetch the needed books we will have some missing if we do not automatically re-fetch for each additional page. For example, given 5 authors we may have 2 authors with 11 books total, leaving only 1 book for the remaining 3 authors in the same paginated request. So, we need to loop through the pages and fetch each book from the authors we have.

Here is the existing code:

const customHook = () => { /* returns books for us */ }

const BooksByAuthorPage = () => {
  const { authors } = customHook();
  const [booksByAuthor, setBooksByAuthor] = UseState({});

  UseEffect(() => {  
    const fetchPage = async (page, bookIds) => {
      const bookUrl = `/api/books?ids=${bookIds}&page=${page}`;
      const response = await fetch(bookUrl);
    
      // The first value is the list of books for that page
      // The second value indicates if there is a next page to fetch from
      return [response.body.data, response.body.links.next];
    }

    const fetchAllPages = async (bookIds) => {
      let page = 1;
      let next = true;
      const booksByAuthorId = {};

      while (next) {
        let [books, next] = fetchPage(page, bookIds)
        next = next;
        books.forEach(book => {
          if (booksByAuthorId[book.author_id]) {
            booksByAuthorId[book.author_id].push(book);
          } else {
            booksByAuthorId[book.author_id] = [book];
          }
        });

        setBooksByAuthor((prevBooks) => {
          const newBooksByAuthor = { ...prevBooks };

          Object.keys(booksByAuthorId).forEach((authorId) => {
            const newBooks = booksByAuthorId[authorId];
            // If the author id is already added as a key
            if (newBooksByAuthor[authorId]) {
              // If the book does not yet exist on the author then add it
              newBooks.forEach((newBook) => {
                if (
                  !newBooksByAuthor[authorId].find(
                    (existingBook) => existingBook.id === newBook.id
                  )
                ) {
                  newBooksByAuthor[authorId].push(newBook);
                }
              });
            // Otherwise add the new key with the list
            } else {
              newBooksByAuthor[authorId] = [newBook];
            }
          });

          return {
            ...newBooksByAuthor,
          };
        });
      }
      
      // trigger the next paginated request if needed
      if (next) { 
        page += 1 
      }
    }

    const bookIds = authors.flat_map(a => a.books.map(b => b.id));
    fetchAllPages(bookIds);
  }, [authors]);
};

Woof that was a lot to type.

Okay so here are the reasons I think this code needs to be improved.

1. On each paginated request I am tracking data in booksByAuthorId and comparing it to the state stored in booksByAuthor before updating it via setBooksByAuthor. (If I don’t do this I will overwrite the existing values from previous pages, which I do not want).
2. I’m also tracking a mutable next and page value inside the UseEffect. My functional friends are gonna come after this one.
3. My spidey senses tell me this code is going to break so easily.

So, help me out. How can I do this better. Am I not ‘thinking in hooks’ correctly? Is this actually the best way to do this? Is the real problem the structure of the API responses?

Thanks for reading and for any insight.