Introducing lkml, a speedy LookML parser in pure Python

I’m excited to debut lkml, a pure Python parser for LookML.

You can run lkml from the command line (it will output a JSON string) or import it as a Python package (it will output a nested dictionary).

@fabio at Looker already built a fantastic, open-source parser in Node, but I find that the Node dependency sometimes makes it difficult to adopt for Python-based workflows that data people are more familiar with.

I decided to embrace the challenge of hand-writing a parser in Python without any external dependencies or libraries.

I didn’t have access to the actual grammar used for LookML, so I reverse engineered it myself. Don’t worry, I’ve tested lkml on over 160K lines of public LookML I downloaded through the GitHub API and it works like a charm!

lkml is fast too. Excluding file I/O, it parses a typical file in a handful of milliseconds. Based on my tests, it’s at parity with the Node parser.

Lastly, lkml has a full unit test suite with CI.

Installation

lkml is up on pip, so the following command will install it

pip install lkml

Please also check out the project on GitHub for a more detailed description and instructions. Contributions and issues are welcome! If you build anything interesting with this parser, please send me a link!

11 Likes

Huge, Josh! That’s commitment.

Can’t wait to see what gets built out now that the Python-speaking masses (like me :snake: ) have easy access to this.

1 Like

Hi Josh. That’s a neat work. How are you using it? I remember some of my users were interested in doing somewhat reverse: generate/update LookML if the table structure changes.

Thanks! I released it so that others could use it to build or improve linters, syntax highlighting, CI tools, and support more Python-based “built on Looker” projects. There are a couple other open-source projects that I know of that are considering implementing it behind the scenes.

There’s no functionality to do the reverse at the moment (“dump” instead of “load”), but the project is definitely open for contributions!

@joshtemple This is great. I’ve tested this with lookml-tools and I will be swapping out the node LookML parser with this Python lkml parser in the next release – hopefully in the next week
Carl

2 Likes

lookml-tools 2.0.0 is released and it now uses lkml as its LookML parser.

1 Like

amazing work @joshtemple I have been wanting this for a while ! works like a charm for me so far.

1 Like

@yuriy, with version 0.2.0, lkml now supports doing the reverse: generating LookML from Python objects. Hope this is useful for your users!

1 Like

I am familiar with Looker and with Python. Why would I want to use this parser? What are some of the primary use cases that this enables? I am trying to figure out how I would use/employ this. Thanks!

Jeff, I gave an example of how I could envision using it here: Generating LookML from Python with lkml

2 Likes

@JeffH, a parser is a foundational building block for building open-source developer tools in Python for LookML. Having a Python representation of LookML structure allows you to do things like:

  • Write programmatic tests of your LookML (linting)
  • Map out the structure of your LookML with hierarchies and dependencies
  • Modify LookML automatically when changes are made elsewhere
  • Generate LookML from other schemas or data representations (see @LStanevich’s example)

Granted, these are all fairly “advanced” use cases, but I’m excited to see how people will build off of this parser and expand the ecosystem of open-source developer tools for Looker. Let me know if you think of other ideas!

2 Likes

Thanks!
I think it would be useful to have a Database vs. LookML View complete compare, to identify missing LookML dimension fields. When you add fields to your database, they don’t automatically show up in your LookML views.
Also, it would be nice to have a function to alphabetize your dimension field and measure field sections.
I think these things would be possible with this, right?

3 Likes

Those are fantastic ideas, and yes, both should be possible with this tool!

1 Like