New LookML — How to convert a "YAML" LookML project


(Nate Pickens) #1

NOTE: Upgrading to Looker 4.0 or later will not automatically alter existing LookML projects, and they will all continue to work as before. Conversion to new LookML requires human action and is performed on each LookML project individually. This guide explains how to convert a project.

Overview

As discussed the “New LookML — Why?” post, new LookML provides a familiar but easier syntax, plus a rich and assistive IDE to make LookML development an easy and delightful experience.

Looker provides an automated way to convert an “old” or “YAML” LookML project to new LookML. There is some preparation and coordination between LookML developers that needs to happen, however, so be sure to read all the way through this post before converting a project.

Preparation

First and foremost, the task of converting a YAML LookML project to new LookML must be performed by one and only one LookML developer. The conversion takes place within that developer’s Development Mode and the changes are then committed and deployed like any ordinary LookML change.

That being said, the process requires the cooperation of all LookML developers. Because the conversion alters essentially every line of LookML, it’s important to make sure other LookML developers have any changes they’ve been working on committed and deployed. Otherwise they’re going to find themselves dealing with some very nasty Git merge conflicts!

While the process of converting is automated, note that new LookML is much more thorough about error checking than YAML LookML. This means that after the conversion, there may be errors showing up that weren’t previously surfaced. Don’t worry! These errors are real and resolving them will improve the quality of the project. The number will vary from project to project, so while fixing them could take only a few minutes, be prepared for the process to take a few hours, during which development on the project will need to remain paused by all LookML developers.

So, the four steps to prepare are:

  1. Choose one and only one LookML developer to run the conversion process.
  2. Make sure the project is error-free by running the LookML Validator (if there are errors, fix, commit, and deploy before moving forward).
  3. Designate a time to perform the conversion.
  4. Communicate to all other LookML developers that they need to have any changes they’re working on committed and deployed by the designated time. That is, their Development Mode copy of the project should be in a clean Git state (“Up-to-date with Production”).
  5. Communicate to all other LookML developers that they should not make, commit, or deploy any changes until the conversion is complete.

Run the converter

  1. Enter Development Mode.
  2. Open the project that is to be converted.
  3. Make sure the Git status in the upper left corner shows “Up-to-Date with Production”.
  4. Run the LookML Validator to ensure there are no errors in the project—the result should be “No LookML Issues”.
  5. In the drop-down menu to the right of the Git status, choose “Convert Project to New LookML”.
  6. Press “OK” in the confirmation dialog that opens.
  7. Sit back while the converter runs!

Validate

  1. When the converter has finished running, the left pane should look like this:
  2. Press the “Validate Again” button to run the LookML Validator.
  3. If there are no errors, you’re good to go! Skip to the “Deploy” section below.
  4. As noted above, new LookML is more thorough about checking for errors than YAML LookML, so don’t panic if you had no errors before and now you do! If you get stuck resolving them, refer to the “Resolving Errors” appendix below.

Commit and deploy

At this point, simply commit and deploy as you would with any ordinary LookML change. Be aware, though, that this commit will trigger a rebuild of your persistent derived tables (PDTs). Once the commit is deployed, let all the other LookML developers know that the conversion is done and that queries relying on PDTs may take a little longer than usual to run, at least until everything’s rebuilt. They can pull from production in their Development Modes to bring in the changes and then continue developing—only now with an improved and more intuitive LookML syntax, and a much more helpful and powerful IDE!


Appendix: Resolving Errors

New LookML introduces more thorough model validation. As a result, you may see errors after you convert that weren’t surfaced before. Additionally new LookML’s on-the-fly error checking requires a bit more information in some cases; the converter does its best to figure this out for you, but in some cases you may need to add an extra includes line in a view file or two.

Invalid field references

In some contexts, YAML LookML does not ensure that fields or sets you reference are actually defined somewhere in the model. The most likely place this will surface is in drill field lists and sets. For example, if you had drill_fields: [id, name, email] but at some point in the development of your model you removed the name field in favor of a first_name and last_name, YAML LookML will not warn that you are now referencing a field (name) in this drill field list that does not exist.

New LookML, on the other hand, will tell you about these types of errors. So after converting your project and running the LookML Validator, you may see errors about invalid field references that you never saw before. This is a good thing! Fixing these errors will improve the overall quality of your model.

Missing includes in a view file

New LookML’s on-the-fly error checking in some cases requires a bit more context than was necessary in YAML LookML. Specifically, it may be necessary to place an include line in some view files—most commonly when a view extends another.

When there is no ambiguity about which file needs to be included in a view, the converter will automatically add the appropriate include declaration in the new LookML view file and you won’t see any error.

In some cases, however (such as with projects containing multiple models), there may be two or more views with the same name declared in separate view files. The converter won’t know which one is the correct one to include, so it will add a comment at the top of the new LookML file indicating that one of those views should be included. Resolving the resulting error is as simple as going to the view file and uncommenting the correct suggestion.

Stranded extension_required explores

Like the “Invalid field references” issue above, in new LookML the LookML Validator will let you know about explores declared with extension: required, but not actually being extended. These explores should either be reconnected to their extended objects or, if unused, removed.

Version control

Because the LookML conversion process replaces your .lookml view and model files with .lkml ones, if one or more developers commit to your project during conversion, you will end up in the middle of a very nasty merge conflict. As such, only one developer should run the converter and coordinating to ensure that everyone’s development modes are clean and up to date prior to conversion is extremely important. To avoid version control issues, there should be a code freeze during conversion; other developers should not check in changes during the time blocked off for conversion.

If the “converting developer” runs into any issues during the conversion process but prior to committing changes and pushing to production, they may find it helpful to “revert to production”. This option from the drop-down (pictured above in the “Run the converter” step) will undo the conversion, allowing them to start the process over.

Related Info

For related info, check out these articles:


New LookML — Why?
The New LookML IDE Shows New and Pre-Existing Parameters
Why aren't dashboards being upgraded to new LookML using the Upgrade Tool?
Looker 4.0 Release Notes
Error: Could not load model
Deprecation Notice: "decimals" parameter and "int" field type (3.38+)
Legacy Feature Notice: Old Style YAML LookML (Looker 4.4+)
(john) #2

is there away to run the converter via commandline/API? we don’t do any lookML development via UI. everything is auto-generated and having to go to UI to do conversion file-by-file is quite painful.


(Carter Moar) #3

There isn’t a way to do conversion outside of the UI, but it is worth noting that conversion is a project-wide action, so all LookML files in the project get converted. That doesn’t mean that any newly exposed errors won’t require view-by-view attention, but it’s not something that needs to be run for, say, 200 view files.


(john) #4

Thanks @cutler for the reply. What I had in mind was to run the conversion as post-processing after our auto-generation script (for all generated yaml). so any manual step whether 1-click or multiple clicks breaks the automation a bit.
Not huge deal-breaker for now since AFAIK Looker will continue to be backwards compatible for yaml in the near future? but we do have a strong interest in having more programatic ways to access both LookML validator and LookML conversion to better manage workflows (we do auto-generate hundreds of table views and dashboards LookML). currently we just reverse-engineer a large portion of what the Looker UI does.
We could also pursue open-sourcing these but rather not to invest if it’d be redundant with something the Looker team is already working on.


(Nate Pickens) #5

Hey @john_handshake! Our intent with the conversion is that it’s a one-time action. Ideally your auto-generation scripts would be updated to output new LookML directly, avoiding the need for any conversion step afterward.

With regard to backward compatibility, we plan to continue supporting old/YAML LookML for the next six months. So you’ve got plenty of time to transition!


(Segah A. Mir) #6

@john_handshake I am investigating, what sounds like, a similar use-case with one of my clients. It is also very important for them to continue supporting programmatic LookML. We started working through a few possible solutions. Happy to discuss


(john) #7

@segahm yes definitely interested to pick your brain!

@natepickens my main concern is having to build and maintain non-standard serializer and de-serializer going forward (we’ve built our own validator for programatic use) and play catch-up for each version of Looker release. We could open source this stuff as 3rd party, however as mentioned if this is already on Looker’s roadmap we’d rather not duplicate the work.


(Tig Newman) #9

By the way, if you work with LookML dashboards you’ll notice that they still use old LookML. The LookML converter does not affect any LookML dashboards that you have in a project.


Why aren't dashboards being upgraded to new LookML using the Upgrade Tool?
(Nate Pickens) #10

@john_handshake Totally hear you. Providing open source libraries for working with new LookML and/or publishing specification docs for the language are definitely things we’re considering. We’ll keep you posted!


(Maya Simon) #11

As a heads up to others who are converting: We found that the converter had some weird effects that don’t show up as LookML errors in the error checker, but are clearly errors to the user. For example, all of our value formatting that was in single quotes as ‘$#,##0’ was converted to “$” – so users just saw a dollar sign in their sales reports. No big deal to fix, just something to keep an eye out for.


(Morgan Imel) #12

Hey @Maya_Simon,

Thanks for reporting this! I’ll pass along the information and see if we can get this fixed.

Cheers!


(Morgan Imel) #13

Hi @Maya_Simon,

I ran some tests internally and was not able to reproduce this issue. If you (or anyone reading this) comes across this again, feel free to reach out to support@looker.com so we can get to the bottom of this.

Happy Halloween! :ghost:


(Arthur Smith) #14

I should report one issue - the conversion for us ran smoothly except for one thing: we had a measure that was a count using a filter on a yesno field, where the value selected was “true” - this worked in old LookML, but after the conversion it didn’t match any values, until I changed the “true” to “Yes”. Now it works fine again.


(Arthur Smith) #15

Just ran across one additional issue (2 months later!) - a dimension was defined via an SQL regex that included the ‘#’ character. The conversion process seems to have thought the ‘#’ was a comment, and moved it up to a comment entry, so the regex was no longer good and was not even a properly terminated string! We didn’t hit this until somebody tried doing something with that dimension in explore mode and kept getting confusing redshift errors.


(vincent) #16

Hi @apsmith,

Thanks for reporting this, I will let engineering aware of the situation. If you don’t mind, can you email support@looker.com with the definition of the dimension. I would like to reproduce this if I can so I have something to give them.

Cheers,
Vincent