Note: This information is accurate as of Looker 4.22. One cannot and should not assume that this information applies to releases prior to Looker 3.46.
What can stream?
Looker currently has the ability to stream results from the following SQL dialects as of Looker 4.0. If you're a dialect not on this list, you're not streaming (or if you are, Looker doesn't know it). Streaming isn't guaranteed for these dialects, but it is a possibility. Notes indicate when streaming support was introduced for a particular dialect after Looker 3.46 (when this post was originally written), but before Looker 4.0.
- amazon aurora
- google cloud postgres
- vertica v5
spark1.4 (removed in Looker 4.0)
- spark2.0 (Looker 4.0+)
- bigquery (Looker 3.48+)
- bigquery standard sql (Looker 3.56+)
- denodo (Looker 3.50+)
When will it stream?
We want to be streaming any time the user runs a query in the SQL Runner or requests to download data, but we are subject to the limitations below. In short, if the request goes through the data are displayed in the browser Looker won't stream, but if the data are exported from the browser Looker will stream.
What are some examples of exported data? CSVs pulled from an explore and JSON downloads from the SQL Runner. What are some examples of data rendered in the browser? Explorations run in Looker, queries, and public URL visits in the browser.
When won't it stream?
Looker can't stream when the download requires multi-row post-query processing. What the heck does that mean? Any time Looker needs to process the results 1) after they return from the database and 2) needs to look at multiple rows of data to do that processing.
Want examples? Table calculations can require Looker to use several rows to produce a single cell's data point. They don't stream. Value formatting doesn't need more information; all it needs is the value and the format. It streams just fine.
Special Pivot Notice
In Looker 4.2, we implemented a new technique for in-database pivoting that allows Looker to stream results from many database technologies. This means that these dialects will no longer be subject to the 30,000 cell download limit as of Looker 4.2. Which SQL dialects are these? Of the above "streamable dialects"the following do not support streamable pivots:
Special Dimension Fill notice
In Looker 4.0, we introduced a slick feature to fill in missing values for many dimension types. Dimension Fill happens after the query, so Looker can't stream if you're using it (queries will be filled by default). Large downloads won't stream, but fear not—Looker will refuse to fill the results. Simply add a limit or turn off dimension filling and download again.
Special API notices
Calls using the legacy API—the one with dedicated users—will not stream. Calls using API 3.0—the one with credentials tied to a user—are a bit more nuanced. Looker will stream results to the API endpoint, but the current Ruby SDK does not handle the streamed results in chunks and will process them in one go.
If you know that row data results will never be larger than your system's memory (due to size or specified limits), then you probably won't need to worry about streaming too much. If your results can be larger than what your system can handle, you should consider processing the row data results chunk by chunk as they are received from the stream.
Download unpivoted results from one of the above dialects and you should be fine as long as you're not using table calcs. If you're using dimensional filling, you shouldn't damage your instance, but you may not get everything you want. The SQL Runner will stream if the query is going against a database using one of the above SQL dialects. Be aware of your system's constraints if you're using the API.