Apex Code Development You need to sign in to do that
Don't have an account?
Clint Majors getBody decompression results in heap size error
I'm working with a third party API that provides a large amount of data to by synced with my org. Unfortunately, the API does not support paging or limiting my results. I simply get all of my data at one time. After implementing req.setHeader('Accept-Encoding', 'gzip') and req.setCompressed(true) I am now able to get a response, but when I call getBody on the response it is decompressed and causes my Heap Size to jump to about 30 MB. This is preventing me from saving the response and processing it in a seperate transaction. Any advice on how to deal with large amounts of data with third party APIs?
A solution (such as the one your are using) that converts the entire JSON string into objects requires enough heap space for both the JSON string and the resulting object graph. So this is one of the few occasions where using the JSONParser class instead could help, especially if you are only interested in a few of the pieces of data. Using that class you can examine the tokens in the JSON and only keep the ones you are interested in and so (potentially) greatly cut down on the space used for the resulting object graph.
An alternative way to perhaps achieve the same result would be to use JSON.deserialize(jsonString, apexType) and leave out most of the fields on the Apex class whose type you supply but whether that works depends on how the method is implemented internally which is not documented.
Obviously if the JSON string consumes the heap you are still stuck.
(A last resort would be to build a proxy service on a platform that doesn't have such tight constraints such as Heroku and call that proxy service using some "chunk" identifier of your own design. That proxy service can then make the request per chunk to the current end pint and only return the relevant part.)
Please mark this post as solved if it helps.
Best Regards,
Nagendra.P