I am selecting rows from a Table in Apache Arrow JS, and than trying to convert those rows into a Table.
I load the data as such:
import { tableFromIPC } from "apache-arrow";
import { fetchData } from "../../services/dataStreamer";
import { readFileSync } from "fs";
const arrow = readFileSync("./src/__tests__/services/example_data.ipc");
const table = tableFromIPC(arrow);
const row = table.get(0) as any;
The row looks like this {"annotation": ["1","2","3"], "id": 1234}
, where annotation is of Vector
type and id of int
type.
Once I have the StructRowProxy
I try convert it into a table.
I have this code to convert into a table,
import { makeTable, RecordBatch, StructRowProxy, Table, tableToIPC } from "apache-arrow";
export const fetchData = (data: { [key: number]: StructRowProxy }) => {
const table = Object.values(data)
.map((row) => {
const rowData = Array.from(row)
.map((rowData, _rowIndex) => {
const fieldName = rowData[0];
const fieldValue = rowData[1] as any;
const a = {
[fieldName]: new Array(fieldValue),
} as any;
const t = makeTable(a);
console.log(t.data);
return {
[fieldName]: fieldValue,
};
})
.reduce((acc, curr) => {
return { ...acc, ...curr };
});
makeTable(rowData);
let batch = new RecordBatch(rowData);
const table = new Table([batch]);
console.log(table);
return table;
})
.reduce((acc, table) => acc.concat(table));
console.log(table);
const ipc = tableToIPC(table as any as Table);
console.log(ipc);
return new Blob([ipc]);
};
The issue appears that the int value gets converted into 1234n
, which isn’t accepted.
Any other ideas how to go from StructRowProxy
to Table
?