Looker Community

Is LookML a Subset of any other format or standard?

I am looking to create a simple script or maybe even CLI tool outside of Looker that accepts JSON documents, and recursively parses their content into boiler plate LookML.

Our use case derives from storing JSON in Snowflake DB Variant data type column. Looker power users are finding it error prone (case-sensitivity, sampling at relevant scale to catch rare fields) to create LookML that exposes the key value pairs within the JSON documents as LookML dimensions, by hand. As more of our data sources evolve in the direction of NoSQL/JSON, this need will become increasingly urgent.

Planning the tool, I’d want to know if LookML is a subset of some other notation standard. For example, LookML looks similar to YANG format (rfc6020), but not quite the same. Knowing the language spec would be vital to implementing the proper marshaling.

Clearly, the standard exists within the Looker ecosystem, as LookML is readily implemented in and validated by the platform. If LookML is not directly compatible with an existing standard, I’m wondering if the definition could be made public so others might develop tooling around it’s syntax, specifically for transforming in-memory representations of LookML structures to valid storable LookML.

Thanks!

Hi Greg, replying to your post, we may be able to help. We are a new Looker partner specializing in JSON transformation. We provide a tool (free trial available) that uses some advanced mathematics to make transforming JSON of any complexity into tables very easily. We are also a Snowflake partner.

Let me explain what we have now, and also some things coming. Today you can use our solution called REFORM to connect a wide variety of JSON sources, including any API, MongoDB, S3, Azure, Postgres JSON and more. You can build virtual tables directly over the JSON and stream them to a variety of targets automatically, including Snowflake, Redshift, ThoughtSpot and lots of others.

For your current use case, you would need to connect to the source JSON, create the tables or LookML dimensions you want then push those to Snowflake. REFORM is very accurate and specifically addresses several of the issues you mention such as rare fields, case sensitivity ect. It’s much simpler and more powerful than variant queries.

Potentially more interesting is we are working on supporting Snowflake variant column as a source in the near future. In this case you will be able to connect to JSON stored in variant and build the dimensions you need without dealing with the errors or challenges of variant queries.

In summary, I agree, JSON is a growing and more important part of many data pipelines, so we are building technology that focuses on this. Feel free to give the trial a spin, I hope it helps.

These open-source tools for reading and writing LookML might be of interest: https://medium.com/@leapingllamas/2019-lookml-open-source-state-of-the-union-d0470012fed0