Continuing the discussion from ETL Tool Recommendations:
Hi @lloydtabb, thanks for the info!
I saw this post as well:
and the TABLE_DATE_RANGE feature makes me wonder what the best approach is.
My first thought was just to append a current date column each day to the snapshot and keep appending to the same table, but your post here seems to suggest it is better to create a new table each day with the snapshot date in the table name?
Being a BigQuery novice still, is one clearly better than the other?
For context, I am currently cross joining a simple calendar date column against my table of interest and deriving my historical numbers using date logic. This works, but as you can guess, is not fast, as I am effectively multiplying my table row count once for each date that I care to process…