I am using this crate with reqwest and async_compression::GzipDecoder to decode CSV files while loading them from the web.
For some reason, the process seems to allocate a lot of memory, causing OOMs in production. In some cases, allocations are almost as high as the size of the source file loaded, which can be tens of GiBs.
Do you have a spontaneous idea why this could be the case? Not pointing the finger to csv_async, I still need to run some heap profiling once I get some time.
Cheers, Chris
I am using this crate with reqwest and async_compression::GzipDecoder to decode CSV files while loading them from the web.
For some reason, the process seems to allocate a lot of memory, causing OOMs in production. In some cases, allocations are almost as high as the size of the source file loaded, which can be tens of GiBs.
Do you have a spontaneous idea why this could be the case? Not pointing the finger to csv_async, I still need to run some heap profiling once I get some time.
Cheers, Chris