The content of this article has been updated and moved to Looker’s technical documentation here.
We’re looking to host our Looker Backend on MySQL, but on the same host that is running Looker.
Do you think this is a bad idea?
As a general best practice we recommend hosting the Looker instance and MySQL backend on separate hosts so the two do not need to compete for resources. It is possible to host the two on the same machine but this could reduce the performance increases you might otherwise notice if you make the move to an external MySQL backend. Let us know if you have any further questions here.
Please update the main instructions with the addition from Carter Moar.
Thanks so much for helping us make this article better! We’ve cleaned up a few things and made some additions based on your suggestions, let us know what you think!
It looks much better.
RDS creates a database, so instead of actually creating one, I altered the one created by RDS. [e.g.,
alter database looker_staging DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;).
Do you happen to know if the
max_allowed_packet applies to 5.6.x? I did use that version of MySQL rather than 5.7.x.
In our internal docs, we belabored some of the AWS stuff, e.g.,
- Click Launch DB Instance
- Select MySQL, and then MySQL in the “Production” block
- Take defaults, except
- DB Instance Class: db.m3.medium
- Multi-AZ Deployment: No
- Storage Type: General Purpose (SSD)
- Allocated Storage: 150GB
- DB Instance Identifier: [Your DB Instance Identifier]
- Master Username: [Your Master Username]
- Master Password:
- Confirm Password:
- Advanced Settings: Take defaults except
- VPC: [Your VPC ID]
- Publicly Accessible: No
- Availability Zone: [Your Zone]
- VPC Security Group(s): [Your Security Group]
- Database Name: [Your Database Name]
- Enable Encryption: Yes
- Master Key: (default) aws/rds
- This will give you an endpoint such as
And a typical MySQL connection . . .
mysql -p -P 3306 -u looker_staging -h looker-staging-metadata-db.cy3guwt4hv1r.us-east-1.rds.amazonaws.com looker_staging
(host name changed.)
The max_allowed_packet parameter should also be set to the maximum value for MySQL 5.6.x. Both MySQL 5.6.x and 5.7.x have much smaller default settings for that parameter than Looker requires.
David beat me to it! The max_allowed_packet default has been 4MB since MySQL 5.6.6, when it was 1MB. MySQL 5.5.x, 5.6.x and 5.7.x all support our suggested value - 1073741824.
Thanks you in at least 10 characters.
Hey, I know this page was revised a bit based on suggestions from me and others – did the instructions regarding
touch NORESTART get dropped? I have that in my personal notes.
Hi John -
Yes, that was removed. It’s a component of our internal maintenance process and has no effect outside of our hosted infrastructure.
On a C4.2xlarge:
looker@reports:~/looker$ export LD_LIBRARY_PATH=$HOME/looker/.tmp/:$LD_LIBRARY_PATH looker@reports:~/looker$ java -Xms26000m -Xmx26000m -jar looker.jar migrate_internal_data looker-db Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000167000000, 18175492096, 0) failed; error='Cannot allocate memory' (errno=12)
Had to use a c4.4xlarge
We really don’t have much else running on that instance except some monitoring. I will look at the memory again later.
Thanks for the feedback, John. The AWS C-series (c3, c4) instance types are optimized for additional CPU without scaling up the RAM at the same level as the general purpose (m3, m4) instance types, so the c4.2xlarge only has half the RAM of the recommended m3.2xlarge. I’ll edit the doc to make that a bit more explicit.
I am trying to migrate Backend database to AWS aurora. I am getting following issues:
Starting Looker: Version 4.12.15-02323d…
Error starting Looker. See log file for details.
We do not support AWS aurora for our backend DB as yet.
By default, Looker uses a HyperSQL in-memory database to store its configuration, users, and other data.
On a busy instance this database can grow to be gigabytes in size, which leads to performance issues, Java memory pressure, and extremely long startup times.
In these cases it’s best to replace HyperSQL with a full MySQL backend. Please refer to steps outlined above.
No man. It worked for me. I have fixed it. We can migrate to Aurora. Aurora is Mysql compatable.
Wow, that is good to know. Would you like to share some insights for our Customers, you more then welcome to do it here Srivarun.
Important note: please make sure to use INNODB for all tables!
This has been the default since MySQL 5.5.5 but it is possible to set it to MYISAM which can cause problems.