Skip to main content

Gitlab runners faq

updated at: December 20, 2020

What is a Gitlab Runner?​

Gitlab Runner is a process which runs on the host machine and waits for some event to trigger it. Our runners are being triggered when we push to master or alpha/staging.

Our current runners​

Notes:

  • cpu-opt: cpu optimized droplets (has less memory and disk, but more cpu power)
  • general: general purpose (has less cpu powr, but more memory/disk)

Droplet name: Ip prod-runner-sfo-cpu-opt: 138.68.44.9 staging-runner-sfo-general: 165.227.10.88 review-runner-sfo-cpu-opt: 159.65.64.85 alpha-runner-ny-cpu-opt: 157.245.216.50

You can also pause specific runner on the https://gitlab.com/powr/powr/runners/

Why is my deploy pending?​

That's probably because Digital Ocean droplets go to sleep. They wake up when the deploy is triggered, but you can also wake it up by SSH-ing into it or even restarting the runner. Runners are chosen randomly among not busy ones, so you might need to wake all not busy runners up to deploy to happen.

How to manually restart the Runner?​

First, ssh into the droplets:

Ask Puru if you don't have an ssh access. He will need to add your public ssh key to the Digital Ocean settings. Or just ask Sergey or Puru to restart the runners.

And then run: gitlab-runner restart

This will restart runners on that droplet. I don't know what happens if one of the runners would be deploying at this time, so be careful with manual restarting.

How to add a new Gitlab Runner?​

Adding new runner

What to do if Gitlab Runner runs out of disk space?​

Gitlab runner stores docker images and build cache for deploys. In case of prod and staging we have enough disk space since we only deploy one app at a time.

But, for review runner situation is different. There we deploy multiple apps simultaneously, as a result runner can run out of space(we have 100gb there).

Step to clean some space in runner machine:

1) ssh [email protected]

2) docker image prune -f -a (will clear all unused images)

3) cd ../home/gitlab-runner/cache/powr/powr (there we store cache for builds, you can remove unnecessary ones, BUT clearing cache is automated, so ideally you wont need this step, but if needed u can remove cache there)

You can also remove ALL docker images inside runner via:

images=`docker images -q -a`
if [ -n "$images" ]; then
        docker rmi -f $images
fi