Why can’t I download media from Telegram API (GramJS) passing only the mandatory attributes?

I am using GramJS, it works fine when I pass the whole media object to downloadMedia, where I guess GramJS itself picks the necessary values and formats the actual query to Telegram’s API.

But if I extract the four values that I believe are mandatory (id, accesHash, fileReference and dcId) and pass just those the response is empty:

Downloading file with the following properties: {
  id: Integer { value: 5935164553170105629n },
  accessHash: Integer { value: 2808544...7639434n },
  fileReference: <Buffer 02 79 47 88 7b 00 00 12 ca 66 cc 95 68 49 1f 19 33 5f b3 a5 29 e6 15 8a 24 f9 8f 17 4e>,
  dcId: 4
}
Error fetching messages: Error: Invalid image response from Telegram

I am trying to do it this way because first I fetch all the messages, each with these four attributes, then I download the images of only some of them (and I don’t want to store the whole media object).

Is this impossible in GramJS?

I tried using directly the MTProto API instead, but oh boy it is hard, I don’t even manage to find that server salt value…