Ro's LookML Autogeneration System

Ro uses Looker as its BI tool. Looker is based on the idea of analysts building Looker “models” written in its LookML language; these models describe a graphical point-and-click interface that a business-side individual can use to query the specified data sets without needing to write code.

Operations, data models additions/modifications, product offerings, and medical condition offerings are scaling far too quickly at Ro for the standard Looker development model to be sustainable without having a small army of a data team to provide the labor required. Applying our general analytic tendencies to the situation, we found that much of the creation and maintenance labor required for LookML models was repetitive and error prone. Separately, LookML lacked certain features that we were having trouble doing without, particularly around documentation and the idea of view blocks. So, now we automatically generate most of our LookML code instead.

Basic Architecture

Completely independently of Looker, we code Looker views, explores, and models in our own Ro Looker templating language. A Python engine that we’ve built turns these templates into LookML and writes the LookML files to our Looker GitHub repo. A web hook pushes the LookML from GitHub to our Looker instance. The number of lines written in our Ro Looker templating language is less than 10% of the number of lines in the LookML that is generated .

Key Features

Key features of the Ro LookML code generation system include:

Automatically add dimensions to a Looker view when a new column is added to the underlying table

We couple this with our automatic pivot table generator — when Ro rolls out a new supported health condition, for example, this means that in most cases all of the fields for the new conditions appear first in our derived tables then in Looker automatically.

Reducing our custom permissions system from 15 lines per Looker field to a single word

We do permissions on a field-by-field basis instead of model-by-model.

Explore-level documentation via Sphinx

We provide documentation specific to each explore in Looker, complete with examples and technical notes. Our documentation (written in Markdown) files are wrapped in with our Looker templating, and our documentation-generation engine uses our Looker templating files to autogenerate portions of the documentation site.

New looker concept: view blocks

Groups of Looker views can appear together in several different Looker explores. For example, our “member” block contains many views with various types of member-level information; this block appears in the majority of our Looker explores. Looker does not have any existing structure that encapsulates this idea of multiple views being grouped.

Automate organization of explore views

We use some tricks to organize views in a large explore:

Remove the need to specify dimension type

We automatically deduce this from the underlying table column’s type. We also have automated levels for timestamp dimensions with different levels of time breakouts (date, day of week, month, hour, minute, etc).

Make group labels easier/automatic

In Looker currently, each dimension/measure needs to have its group label (if any) specified explicitly. With the Ro system, we can assign group labels to field name prefixes, and any field with one of these prefixes will be added to the corresponding group automatically.

Automatic parameter lists

Currently, Looker requires that you manually specify parameters. Our system can auto-generate parameter lists from unique values in a column or unique dimensions in a dimension group.

A variety of other tricks

  • Ex: all measures use the first defined drillset by default
  • Ex: removing the need for things like “${TABLE}”

Further reading

This article originally appeared on our blog. You can find further articles detailing of some of the above features here and here.

15 Likes

Sami, this is SO cool. You all at Ro keep impressing me with the awesome things you create on top of Looker.

Thanks for sharing it here!

2 Likes

Thanks, Izzy!

This looks great. Could you share some screenshots/examples of how you implement the Explore-level documentation? We are looking into a solution for better documentation (basic tooltip definition, full definition, examples).

This is purely an off-platform documentation solution right? Did you find any issues with your users needing to go to a separate site to see this documentation? Have you looked in to using Looker Documents for this?

Hey! We use Sphinx, which is a standard documentation system for developers.

2 Likes

@romain.ducarrouge I think they stole your name for their model.

2 Likes

Or, your parents named you after their model.

Which is actually more plausible than it should be, given the company! :joy: (sorry @samiyabroudi)

1 Like

Finally, I am now a model! :star:

3 Likes

Would you say… you are a ro’ model? >_>

3 Likes

pretty good

I might be being stupid but I cant find any link to github etc. Have you opened sourced this?

1 Like

We have unfortunately not open-sourced our LookML generation system - generalizing it for open source use is not something we’ve had time for yet…

This is quite interesting and I would personally consider implementing the LookML generation system if it saves us that 90% of LookML.