We are self hosting Looker and using the multi-tenancy setup. In our production environment, we have ~500 tenants and in our test environment we have ~5000 tenants. We are seeing our test environment Looker begin to hang in the web UI on GET requests with a lot of objects for a single resource. Even worse, when using the API directly writes have begun to fail for resources with a lot of objects. The write failures start out by being slow, then cascade into 500s and finally bring the entire Looker cluster down.
This seems to be 2 root problems. First, the API doesn’t look to support paging for GET requests on unbounded resources. Is paging on GET requests in the product backlog? Is there a workaround that we can use here until it’s implemented?
Second, Looker seems to start failing on writes after a certain limit. In our case, we have ~5 million content-metadata-access records. This is caused by us having to delete other tenants from spaces that were automatically cascaded from the parent shared node. What are the limits here for Looker in terms of writes?