Hi All,
I’m using lookerAPI to try and circumvent looker’s 5000 row limit, but when I change the limit to over 1,000,000, I run into an error:
“[{‘looker_error’: ‘Refusing to run query with large limit 999223 that Looker cannot stream incrementally. Please choose a smaller limit or remove table calculations.’}]”
Is there a way to be able to pull all the data I need incrementally (for example, first do elements 1-50,000, then 50,001 to 100,000, etc)?
There isn’t a way to page through the data with the API, but you could probably figure out a way to break up the data without explicitly specifying elements. Filter for date ranges, for example, or for ID ranges, and you could probably accomplish the same thing. You could also hit the /queries/{query_id}/run/sql endpoint to get the SQL generated for the query, add an OFFSET or WHERE row_id > x to that, and then feed that back into the API to create + run the query.
Or, you could look into converting the table calculations into LookML or SQL if possible— Totally depends on what the functions are though… What kind of calculation are you doing?
