New LookML — Why?


(Nate Pickens) #1

Overview

“New LookML” is best described as a refresh of the LookML syntax. “YAML LookML”'s syntax was, well, YAML, a generic data-serialization language created long before Looker came into existence. New LookML borrows a lot from the YAML syntax, making it easy for existing LookML developers to learn and transition to it. At the same time, it does away with a lot of the pitfalls and inconsistencies of YAML that make LookML more difficult to learn for new users and more error-prone even for experienced LookML developers.

But even more importantly than the changes to syntax, transitioning off of YAML to a native language designed specifically for LookML’s needs has allowed the Looker IDE experience to be expanded into something much more dynamic, interactive, and helpful for LookML developers.

The two primary goals of new LookML can therefore be summarized as follows:

  1. Provide a syntax that—while familiar to existing LookML developers—resolves the various aspects of YAML that often trip people up.
  2. Power a rich and assistive IDE to make LookML development an easy and delightful experience.

Let’s take a closer look at why these goals were set and how new LookML reaches them.

Syntax

YAML was designed with human-readability in mind, which is one of the reasons it was used for LookML originally. It provides all of the constructs LookML needs: key-value pairs, lists, nested structures, etc. But it’s not perfect. There are three primary aspects of YAML that lead to confusion which the alternative syntax of new LookML resolves.

1. Hierarchy

YAML uses indentation — Mess up indentation by a level or two, and now all of a sudden your code means something completely different. It’s often hard to tell at a glance what the issue is, especially when the YAML is still valid but doesn’t make sense as LookML.

New LookML uses curly braces — New LookML solves the indentation-based hierarchy confusion by doing away with it entirely. Blocks of LookML are clearly designated with a simple set of curly braces.

2. Consistency

YAML has multiple syntaxes to say the same thing — Strings can be quoted or not, there are many word options for booleans, and arrays look very different depending on whether you want each item on a new line or not. This makes the syntax more complex and therefore more difficult for new LookML developers to learn.

New LookML is consistent — Strings are always quoted, booleans are always yes or no, and lists are always defined with square brackets and comma-separated items. There is one and only one right way to define these things.

3. Meaning

YAML doesn’t know about LookML things — YAML provides syntax, but nothing more. It has no concept of defining a thing and referencing it somewhere else, as with fields, views, and joins. This means it can’t help you as you write LookML.

New LookML is LookML — New LookML is an actual language—not just a syntax. So there are inherent rules and meaning behind the things you define with it. New LookML understands all the constructs of LookML, like field definitions and what the valid parameters are when defining a dimension. This allows it to power a full-featured IDE experience that both enables learning for newcomers to LookML and enhances the productivity of experienced developers.

IDE

With YAML LookML, you basically get a plain text editor to develop in. It can’t offer any assistance, because as mentioned above, it doesn’t understand the LookML meaning behind any of the things you’re writing.

The enhanced IDE built atop new LookML, on the other hand, offers a much more guided and helpful development experience. This is where the real magic of new LookML shines! With new LookML, the IDE auto suggests LookML parameters and field names, checks for errors, and provides documentation—all on the fly as you type!

Get started!

Check out the following posts to learn more about new LookML and the new IDE features and how to get started converting your YAML LookML projects to the new syntax so you can take advantage of it all:


The New LookML IDE Shows New and Pre-Existing Parameters
Looker 4.0 Release Notes
Data Actions
Legacy Feature Notice: Old Style YAML LookML (Looker 4.4+)
BigQuery Resources Limit (max_billing_tier)
(Maximilian Roos) #2

The IDE experience looks awesome, congrats on that.

It looks like the language decision is made. But I’m curious (and a bit surprised). Why couldn’t you do those IDE changes with YAML? It looks like the new LookML language represents the same data structures as YAML (and then the additional meaning comes from a layer above those structures). If YAML’s liberal syntax was a problem for users / your compiler, there are other standard alternatives to creating your own language.

For us, this seems to be a step away from what we love about Looker - a flat, explicit way of modeling and representing our data with the minimum of interference. A new proprietary language adds to that interference.


(john) #3

Definitely understand some of the motivation mentioned here. We are generating (a large number of) LookML programmatically and also a bit curious about using non-standard markup language going forward.
Does looker have plan to supply tool/scripts for YAML-to-new-LookML transformation, and how long would the YAML-based syntax be supported for?

Thanks!


(Dirty Looker) #4

Hey @Maximilian_Roos,

We originally picked YAML because it was our favorite standard, there are no alternative standards because we had already picked the one we felt was the best. Unfortunately, we’ve grown beyond what it offers and need to provide something better. LookML has to evolve with the demand of what data modelers need.

Here’s some huge wins that we get with just this first release:

YAML has a lot of sharp edges, if you don’t know YAML the learning cliff is painful and brutal. We didn’t want to adopt a standard with more guard rails that sacrifice readability for error proofing. New LookML doesn’t suffer this issue.

Building a parser with YAML wasn’t sustainable. There were many ways to do the same thing in YAML(for example instantiating a collection or referencing object/parameter inheritance) We couldn’t build a rich enough IDE under these constraints, we could detect for wrong things, but are unable to suggest the right thing. We didn’t feel this was good enough and wanted the IDE to guide and help the developer as much as possible.

Finally, YAML is not a standard designed nor intended for data modeling. YAML is amazing and awesome serialization language that is super readable. YAML is fantastic for configurations, but not great for multi-file objects that build up a data model. With New LookML we’re going to keep all of the great ideas they had AND add all the ones we want.

We want LookML to be richer, more expressive, and more powerful for developers. This doesn’t exist so we simply have to build it.


(Dirty Looker) #5

@john_handshake,

I believe we have efforts to expose the YAML -> LookML generator as part of our API or a script. @natepickens can chime in more on the details for support timeline. I believe everything will always still work, but it may not contain the new features we will add to LookML post 4.0


(Michael Erasmus) #6

@mikhailxu How about open sourcing the just the LookML parser (or at least a specification for the parser)?

We’re hoping to be able to enforce some of our own company naming conventions and best practices with a githook. Writing scripts to read a standard yaml file and walk the abstract syntax tree could make this pretty easy, which is much harder now.


(Gaurav Shah) #7

Hi @mikhailxu any updates on exposing the convertor as an api ?

generating new lookml is hard, json/yaml is easier to generate.

We have scripts that generates view based on a large schema (400 fields) and gets updated every other week.


(Gaurav Shah) #8

@john_handshake were you able to move ahead of it ? facing similar problem while generating the new lookml


(john) #9

Hey Gaurav great to meet someone in the same boat.
we’ve been punting for now as we haven’t found time to deal with automating the conversion.
with the new releases and official deprecation perhaps it’s time for us to allocate resources though :frowning:
what language is your generation script written in? perhaps there are opportunities to collaborate.


(Dirty Looker) #10

Hey @Gaurav_Shah and @john_handshake,

We don’t have any updates on this yet, we will send your feedback to our product team!


(Gaurav Shah) #11

we are scripting in ruby, we have just one view. Sure we can try collaborating. One way I was thinking is to write out JSON with “key_$_name”, and then perform string manipulation.

if we had to write the following:

view: raw_s3_events {
 sql_table_name: default.raw_events ;;
 suggestions: no
dimension: actor_ab_segment {
 view_label: "actor"
 label: "ab_segment"
 type: string
 sql: ${TABLE}.`actor`["ab_segment"] ;;
 description: "actor=>ab_segment"
}

we could start with

view_$_raw_s3_events: {
 sql_table_name: default.raw_events ;;
 suggestions: no
dimension_$_actor_ab_segment: {
 view_label: "actor"
 label: "ab_segment"
 type: string
 sql: ${TABLE}.`actor`["ab_segment"] ;;
 description: "actor=>ab_segment"
}

And then perform a string replace.


(fabio) #12

Michael, I have just shared an unofficial LookML parser here: Parser for new LookML?


(KALLAAR KATHIR) #13

Does this mean we need not to do “validate lookml” once after saving the view, since the syntax and other validation are realtime/on the fly?


(Chris Seymour) #14

Hey @kallaar,

On-the-fly error checking will catch most LookML errors, but not all of them, so it is still a good idea to run the validator with the “Validate LookML” button after saving your changes. For example, an invalid field reference due to a missing join would not be surfaced in the on-the-fly checking, since it depends on interaction among LookML objects across several files. You can read more about this here: New LookML — Deep dive into the new syntax and the new IDE