How many Projects/Models should I have? 10 Questions to ask

projects
models
organization
develop
lookml

(Eric Feinstein) #1

Setting up a Looker instance from scratch often requires trying to figure out the right number of projects and models. There is no perfect answer, but some considerations to make:

Choosing Number of Projects

How many databases do you plan to connect to? If more than one, do any tables share structure in more than one place?

One connection per project is not necessary, but helpful when views are completely unrelated. If you have the same view in multiple databases (i.e. my archive database and my production database) then you could create two models that use the same views but different connections. This keeps field maintenance low.

  • Keep separate projects for completely unrelated databases.

Do you plan on having everyone develop? Groups of developers?

If you only have a few developers, it is fine to keep everything in one project to make the Git management simpler.
Developers have access to the entire project. They also can access all the data in that project. If you have separate groups, it’s better to have separate projects for that. Also, many developers on one model requires teamwork, if there is a need for separate business rules, it may be better to break up the models.

  • Keep separate projects to separate groups of developers.

Do you have a large number of objects (i.e. 400 tables)?

Break into projects where you can, definitely think about not using the generator if you have no need to bring in every table.

  • Large amounts of views tend to be confusing to develop with, try not to create projects with every view in a database if they do not need to be together.

Do you have multiple datasets that are completely unrelated?

Multiple views in one project cloud up the space and increase your chance of merge conflicts across many developers. If you have unrelated datasets, their own project may be a good idea.

  • Large amounts of views tend to be confusing to develop with, try not to create projects with every view in a database if they do not need to be together.

Do you want to limit the number of git repositories?

Multiple projects require multiple git repositories. This may be a higher effort to maintain.
If rollbacks are likely due to changing business rules, the whole repository is affected, you may want to use more repositories.

  • If git having a single git repository is a necessity, you will need to keep one project.

Do you plan to move parts of existing databases to another database?

It would be easier to build those parts in a separate project, then swap out the connection later and keep all the existing LookML.

  • It is not very easy to take a single project and split it into multiple down the road, consider the future project organization before you get started.

Choosing the number of models

Do you have data in a single database that you normally would restrict some users from seeing (i.e. HR data)?

Separate models allow for securing explores and access to data separately for different users.

  • Keep restricted or secure data in its own model.

Would your users benefit from faster performance due to less metadata being loaded?

If models aren’t shared to many people, the app performs better.

  • Use multiple models to prevent users from being clouded with unnecessary data.

Do you plan on having some role based access automated (LDAP, SAML), or do you intend on adding users manually, with the need to select groups for each one?

You need to build model sets and roles to provide access, if you have less options it is easier to manually add users.

  • When using manual user setup, consider fewer models.

Does your data structure change frequently?

Changes to view names, tables and joins require a bit of attention to detail. Managing changes in a single model is easier than multiple models, plus you can use extensions on explores in the same model.

Also think about extending views to overwrite or add fields, leaving the view essentially stock. Then run the generator (“add view from table”) often to rebuild the base view.

  • Use a single model when you expect to be making many changes for simpler maintenance.

Leave a comment with any project organization tips of your own!