New Startup options? LOOKERPORT not being honored anymore?

done
watch
normal_priority

(Alex Gray) #1

We have recently updated from 5.18.29 to 6.2.13 and it’s failing to start up:
java -jar ./looker/looker.jar --version
6.2.13

Our startup script hasn’t changed, which means the java command hasn’t change either:
java -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 -XX:MaxMetaspaceSize=800m -Xms2424043k -Xmx2424043k -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/gc.log -jar looker.jar start --no-daemonize -d looker-db.yml --clustered -H 10.55.0.108 --shared-storage-dir /mnt/efs

But with 6.2.13 we get this error in the logs:

WARNING! RUNNING LOOKER AS ROOT IS NOT RECOMMENDED
To run on a privileged port (e.g. 80 or 443), include the --port=<port number> option
If you must run as root on a non-privileged port, include the --root option to bypass this 
warning

But we are NOT running it as root at all. We are running it at the “looker” user. My guess is that looker is trying to run on a privileged port, despite the environment variable LOOKERPORT=9999

Any ideas what could be wrong? Maybe the LOOKERPORT env var change between versions?


(Izzy) #2

I’m not certain about the LOOKERPORT env variable, but AFAIK the recommended way to set the port is by using the -p flag (or --port), per here: https://docs.looker.com/setup-and-management/on-prem-install/looker-startup-options#startup_option_list

I would give that a try, just to make sure. Having LOOKERPORT in your lookerstart.cfg should be okay, but it can’t hurt to put it explicitly in the script to make 100% sure it’s being honored.


(philip.martinelli) #15

Hey @Alex_Gray,

That error raises a bunch of questions:

  • who do they see owning the process via : ps aux | grep java (left most field)

  • what port(s) are looker listening on: netstat -punta | grep -i listen (make sure the pid listed in the output matches that seen in the above). If it’s 9999/19999, both non privileged ports, then the error makes sense (assuming root is indeed the user owning the process).

  • why --no-daemonize: are you using systemd? (just curious)

Thanks,
Philip


(Alex Gray) #16

Thank you for all the responses!!! (and wow, sorry for the formatting of my original question!)

I see that the user “looker” is owning the java process. “root” is definitely not taking in part in anything related to looker :slight_smile:

I am using --no-daemonize because we use supervisord across the board.

You’re gonna hate this, but completely nuked the instances and brought them back up with the same AMI and now it’s working! ARGH! I just checked “ps aux” and it’s the same command and same startup options. It’s using port 9999/19999, verified via netstat, and all is well.

But here is my game plan:

  • Use --port option, not the LOOKERPORT option
  • Use the latest looker-jar(6.4.17), not 6.2.13.

In production we are using 5.18.29, and we want to upgrade to 6.4.17 (last time we tried an upgrade, it didn’t go well, so we’re hoping this latest version does the trick)

Thanks again! (and I’ll try editing my original message to fix up all the nasty formatting issues)


(andy) #17

Are you guys on a ubuntu linux box?


(Alex Gray) #18

@andy, We are on AWS Linux, which is a centos-ish OS :slight_smile:


(Izzy) #19

Did this end up working?


(Alex Gray) #20

Hi Izzy!
In lower environments this is working just fine. I plan to do a dry run in production today, and then make it live tomorrow night. I’m sweating bullets, but I so far I’ve been stress testing the upgrade process all day yesterday with no issues.


(Izzy) #21

Music to my ears! Keep us posted :smile:


(Alex Gray) #22

dang. No love in production again.
We had 3 nodes running 5.18.29.
I stopped one node, upgraded the jar to 6.2.13 and ran the java command.
The migrations seemed to run fine (it look ~3-4m to run), and once that was done, I upgraded the other two nodes, one at a time.

We are now getting a 401:

2019-02-21 21:36:17.618 +0000 [INFO|0000d|http_core] :: Request from 10.10.2.252 |000000| GET /api/3.0/spaces/24, {}

2019-02-21 21:36:17.624 +0000 [INFO|0000d|db:looker] :: (0.000795s) SELECT * FROM `access_token` WHERE (`lookup_id` = 'CsdTw3Gf') LIMIT 1

2019-02-21 21:36:17.628 +0000 [WARN|0000d|http_core] :: (0.010699s) |000000| Request complete: 401 /api/3.0/spaces/24

So maybe our access token is no longer compatible when going from 5.18.29 to 6.2.13?
Maybe we have to regenerate our tokens?


(Izzy) #23

Oh, hm, if this is a 401 just from the API, that would certainly be my first guess, though I have no hard evidence to suggest your tokens would have expired/are incompatible. Shouldn’t be too hard to test that out.

Are you seeing similar behavior in the Looker UI, as well? I’m not familiar with your setup over there, sorry if I’m off the mark.


(Alex Gray) #24

OK. We figured it out. (I replied back to support, but not to this forum).
It was completely our fault (without getting into too many details, we were incorrectly cach’ing looker’s auth token during the upgrade, and thus getting 401’s)
We have successfully updated looker from version 5 to 6 without any issues!