NOTE: Upgrading to Looker 4.0 or later will not automatically alter existing LookML projects, and they will all continue to work as before. Conversion to new LookML requires human action and is performed on each LookML project individually. This guide explains how to convert a project.
As discussed the “New LookML — Why?” post, new LookML provides a familiar but easier syntax, plus a rich and assistive IDE to make LookML development an easy and delightful experience.
Looker provides an automated way to convert an “old” or “YAML” LookML project to new LookML. There is some preparation and coordination between LookML developers that needs to happen, however, so be sure to read all the way through this post before converting a project.
First and foremost, the task of converting a YAML LookML project to new LookML must be performed by one and only one LookML developer. The conversion takes place within that developer’s Development Mode and the changes are then committed and deployed like any ordinary LookML change.
That being said, the process requires the cooperation of all LookML developers. Because the conversion alters essentially every line of LookML, it’s important to make sure other LookML developers have any changes they’ve been working on committed and deployed. Otherwise they’re going to find themselves dealing with some very nasty Git merge conflicts!
While the process of converting is automated, note that new LookML is much more thorough about error checking than YAML LookML. This means that after the conversion, there may be errors showing up that weren’t previously surfaced. Don’t worry! These errors are real and resolving them will improve the quality of the project. The number will vary from project to project, so while fixing them could take only a few minutes, be prepared for the process to take a few hours, during which development on the project will need to remain paused by all LookML developers.
So, the four steps to prepare are:
- Choose one and only one LookML developer to run the conversion process.
- Make sure the project is error-free by running the LookML Validator (if there are errors, fix, commit, and deploy before moving forward).
- Designate a time to perform the conversion.
- Communicate to all other LookML developers that they need to have any changes they’re working on committed and deployed by the designated time. That is, their Development Mode copy of the project should be in a clean Git state (“Up-to-date with Production”).
- Communicate to all other LookML developers that they should not make, commit, or deploy any changes until the conversion is complete.
Run the converter
- Enter Development Mode.
- Open the project that is to be converted.
- Make sure the Git status in the upper left corner shows “Up-to-Date with Production”.
- Run the LookML Validator to ensure there are no errors in the project—the result should be “No LookML Issues”.
- In the drop-down menu to the right of the Git status, choose “Convert Project to New LookML”.
- Press “OK” in the confirmation dialog that opens.
- Sit back while the converter runs!
- When the converter has finished running, the left pane should look like this:
- Press the “Validate Again” button to run the LookML Validator.
- If there are no errors, you’re good to go! Skip to the “Deploy” section below.
- As noted above, new LookML is more thorough about checking for errors than YAML LookML, so don’t panic if you had no errors before and now you do! If you get stuck resolving them, refer to the “Resolving Errors” appendix below.
Commit and deploy
At this point, simply commit and deploy as you would with any ordinary LookML change. Be aware, though, that this commit will trigger a rebuild of your persistent derived tables (PDTs). Once the commit is deployed, let all the other LookML developers know that the conversion is done and that queries relying on PDTs may take a little longer than usual to run, at least until everything’s rebuilt. They can pull from production in their Development Modes to bring in the changes and then continue developing—only now with an improved and more intuitive LookML syntax, and a much more helpful and powerful IDE!
Appendix: Resolving Errors
New LookML introduces more thorough model validation. As a result, you may see errors after you convert that weren’t surfaced before. Additionally new LookML’s on-the-fly error checking requires a bit more information in some cases; the converter does its best to figure this out for you, but in some cases you may need to add an extra
includes line in a view file or two.
Invalid field references
In some contexts, YAML LookML does not ensure that fields or sets you reference are actually defined somewhere in the model. The most likely place this will surface is in drill field lists and sets. For example, if you had
drill_fields: [id, name, email] but at some point in the development of your model you removed the
name field in favor of a
last_name, YAML LookML will not warn that you are now referencing a field (
name) in this drill field list that does not exist.
New LookML, on the other hand, will tell you about these types of errors. So after converting your project and running the LookML Validator, you may see errors about invalid field references that you never saw before. This is a good thing! Fixing these errors will improve the overall quality of your model.
Missing includes in a view file
New LookML’s on-the-fly error checking in some cases requires a bit more context than was necessary in YAML LookML. Specifically, it may be necessary to place an
include line in some view files—most commonly when a view extends another.
When there is no ambiguity about which file needs to be included in a view, the converter will automatically add the appropriate include declaration in the new LookML view file and you won’t see any error.
In some cases, however (such as with projects containing multiple models), there may be two or more views with the same name declared in separate view files. The converter won’t know which one is the correct one to include, so it will add a comment at the top of the new LookML file indicating that one of those views should be included. Resolving the resulting error is as simple as going to the view file and uncommenting the correct suggestion.
Like the “Invalid field references” issue above, in new LookML the LookML Validator will let you know about explores declared with
extension: required, but not actually being extended. These explores should either be reconnected to their extended objects or, if unused, removed.
Because the LookML conversion process replaces your
.lookml view and model files with
.lkml ones, if one or more developers commit to your project during conversion, you will end up in the middle of a very nasty merge conflict. As such, only one developer should run the converter and coordinating to ensure that everyone’s development modes are clean and up to date prior to conversion is extremely important. To avoid version control issues, there should be a code freeze during conversion; other developers should not check in changes during the time blocked off for conversion.
If the “converting developer” runs into any issues during the conversion process but prior to committing changes and pushing to production, they may find it helpful to “revert to production”. This option from the drop-down (pictured above in the “Run the converter” step) will undo the conversion, allowing them to start the process over.
For related info, check out these articles: