Production setup

Server OS

Use latest Ubuntu LTS or latest Debian LTS.

Tip: Ubuntu LTS's have longer lifespans than Debian LTS's.

Requirements

nvm (GitHub repo) installed as non-root user (see next paragraph) or just use pnpm for everything and skip nvm altogether.
node (latest LTS version installed with nvm, see .nvmrc for minimum version) - also non-root.
npm (should be bundled with node) - also non-root.
pnpm (npm i -g pnpm) - also non-root.
an NGINX installation.

Set up user

Don't run the dist-server's process as root. Either create a new user or use an existing user with more limited permissions. All files in the system-files/ directory in this repository assume a non-root user to run the dist-server's process. In a minute, we'll use sudo visudo to allow this user to use sudo without having to specify a password for some commands.

Important: as described in the 'Requirements' paragraph above, all Node.js-related software needs to be installed with this user, not as root!

First, clone this repository. The recommended install location is /opt/dist-server. The /opt directory may be set up quite strict, so you may have to clone as root and then chown recursively as the user we want to run the dist-server process with.

Copy system-files/etc/sudoers.d/dist-server_overrides from the repo to /etc/sudoers.d/ and replace the [USERNAME] placeholder by running sudo visudo -f /etc/sudoers.d/dist-server_overrides (will open safe editor). If you need to rename the file, do not add a . in the filename or the file will be ignored by visudo.

Install dependencies

Go to dist-server's install location. Run pnpm i.

Set up .env

Edit .env, read the comments carefully and fill in. Find an example .env file documented here.

Set up NGINX

First of all, define a default_server server block for port 80 and 443! If NGINX can't find a server block with default_server, it will treat the first server block it finds, that listens on the requested port, as the default server. In NGINX, you can define a server block that contains listen 443 ssl; without specifying a ssl_certificate directive, but with the default server block, sudo nginx -t will throw an error if the ssl_certificate directive is omitted! That will cause trouble with how dist-server works, because we want to support writing listen 443 ssl; directly in your repository's vhost(.*) files, e.g. so you can write listen 443 ssl http2;. That means that if you'd place your include directive above the include directives that were already present, things might break at some point. Our recommendation: create a port 80 and 443 default_server block like this:

NGINX

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  server_name _;
  access_log /var/log/nginx/default.access.log;
  return 403;
}

server {
  listen 443 ssl default_server;
  listen [::]:443 ssl default_server;
  server_name _;
  access_log /var/log/nginx/default.access.log;
  # Create a self-signed certificate that's valid for multiple years using open-ssl, like this:
  # sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt
  ssl_certificate /etc/nginx/ssl/nginx.crt;
  ssl_certificate_key /etc/nginx/ssl/nginx.key;
  return 403;
}

Next, edit /etc/nginx/nginx.conf, find the http {...} block and add include <VHOST_PATH>/*.conf at the bottom (replace <VHOST_PATH> with the value of VHOST_PATH as specified in .env). While you're at it, find the server_names_hash_bucket_size directive in the http {...} block, uncomment it and set it to 128. The reason for this is that some URLs, especially the 'alias' versions, will become really, really long. Later on, when dist-server is up and running and you're trying to deploy a project, you may get errors like nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size [...]. If that happens, simply double the value of server_names_hash_bucket_size, run sudo service nginx reload and redeliver the webhook delivery until the error don't occur anymore. dist-server itself will warn and fail a build whenever at least one of the used hostnames is too long.

Example (with recommended VHOST_PATH):

in .env:

shell

VHOST_PATH="/etc/opt/dist-server/vhosts"

in /etc/nginx/nginx.conf:

NGINX

http {
  # ...stuff
  server_names_hash_bucket_size 64;
  # ...stuff
  include /etc/opt/dist-server/vhosts/*.conf;
}

It's absolutely vital to write /*.conf and not /*, because we need NGINX to ignore .conf.bak files.

Now create a directory at VHOST_PATH manually (mkdir -p <VHOST_PATH>). It will be ensured by starting dist-server's process too, but we may need to add some more files in the same directory.

Set up certbot

Install certbot on the machine. Make sure auto-renewing is turned on. Usually it's turned on automatically using a cron job.

Run certbot at least once, to accept the terms and conditions. We don't want this to get in our way while dist-server is running.

TODO: Check if this is needed: https://blog.arnonerba.com/2019/01/lets-encrypt-how-to-automatically-restart-nginx-with-certbot

Set up Slack

TODO: Describe steps to create Slack app.

Use /invite @[app-name] in a channel to allow the Slack app to post to that channel. From that moment on, you can set SLACK_CHANNEL_NAME in .env to this channel name.

Set up PM2 and systemd service

On production, we want to run the Node.js process with PM2 (link) to restart the server automatically on error, monitor memory usage and process restarts, among other things. We also want to wrap all of this in a Linux systemd service, to make sure the dist-server's process gets started automatically on (re)boot.

As the user dist-server will run with, run pnpm i -g pm2 to install pm2 globally. Test by running pm2 --help.

As root, cd to the dist-server repository and run:

bash

pm2 start ecosystem.config.cjs

This will add dist-server to PM2. This will (probably) start the dist-server process, which you can check by running pm2 status or pm2 monit (if it shows 'online', then the process has started). Remember these two commands, they will be your best friends while debugging.

Now create a PM2 startup script by running pm2 startup. This creates the systemd service. We want to edit this file though, because it pins the current Node.js version, while we want to be able to update to a newer Node.js version without running pm2 unstartup && pm2 startup. The systemd service file should be located at /etc/systemd/system/pm2-<user>.service and look like this:

ini

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=<user>
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=<generated PATH>
Environment=PM2_HOME=/home/<user>/.pm2
PIDFile=/home/<user>/.pm2/pm2.pid
Restart=on-failure

ExecStart=/home/<user>/.local/share/pnpm/global/5/.pnpm/pm2@5.2.0/node_modules/pm2/bin/pm2 resurrect
ExecReload=/home/<user>/.local/share/pnpm/global/5/.pnpm/pm2@5.2.0/node_modules/pm2/bin/pm2 reload all
ExecStop=/home/<user>/.local/share/pnpm/global/5/.pnpm/pm2@5.2.0/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

We could modify the ExecStart, ExecReload and ExecStop lines to use whatever the output of whereis pm2 (as non-root) is for brevity and a PM2 version agnostic path. In our case, this is /home/<user>/.local/share/pnpm/pm2, so we'll change a few lines:

ini

ExecStart=/home/<user>/.local/share/pnpm/pm2 resurrect
ExecReload=/home/<user>/.local/share/pnpm/pm2 reload all
ExecStop=/home/<user>/.local/share/pnpm/pm2 kill

TODO: Is the following still necessary?

Furthermore, change the Environment lines to this (replace <user> with the name of the user that needs to run the dist-server's process):

ini

Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/home/<user>/.pm2
Environment=NODE_VERSION=default

Now, reboot the server, to test if the service works. If it does, pm2 status should show dist-server as 'online' and service pm2-<user> status should say that the service is active.

Note: after updating PM2 or Node.js, the service may no longer successfully restart. In this case, try a server reboot or manually kill all PM2 related processes.

Running the dist-server on a URL

DNS

In the DNS settings of the domain name the builder will be hosted on, create two A records:

One for the BUILDER_HOSTNAME (.env), pointing to this server's IP BUILDER_IP (.env).
One *. variant of the record above.

Example:

in .env:

shell

BUILDER_HOSTNAME=cd.example.io
BUILDER_IP=38.140.12.40

in DNS settings of example.io:

Name	Type	Value
cd	A	38.140.12.40
*.cd	A	38.140.12.40

NGINX

Go to dist-server's install location and run:

bash

# Copy default NGINX config for dist-server itself.
cp ./system-files/etc/nginx/sites-available/dist-server.conf /etc/nginx/sites-available/dist-server.conf

Edit the vhost at /etc/nginx/sites-available/dist-server.conf. Read the comments very carefully and replace all the placeholders (<...>)!

If you want to shield the dist-server dashboard from outsiders, there are multiple options.

Password protect it.
Only allow certain IPs.
Both!

When setting up password protection, read this article. Preferably create a .htpasswd file at /etc/opt/dist-server/.htpasswd, which is the recommended location for all dist-server configuration files and is the recommended location for saving vhosts to as well.

Now run the following commands:

bash

# Create a symlink in /etc/nginx/sites-enabled.
ln -s ../sites-available/dist-server.conf /etc/nginx/sites-enabled/

# Create an SSL certificate for dist-server's public URL (replace placeholder with value in .env).
certbot --nginx --cert-name dist-server -d <BUILDER_HOSTNAME>

# If all went well, apply the NGINX configuration.
sudo service nginx reload

🎉 Congrats!

You should now be able to reach the dist-server dashboard at BUILDER_HOSTNAME (.env) if you've set ENABLE_DASHBOARD=true in .env. Try setting up the webhook for your first repository to see if everything works as expected.

Production setup #

Server OS #

Requirements #

Set up user #

Install dependencies #

Set up .env #

Set up NGINX #

Example (with recommended VHOST_PATH): #

Set up certbot #

Set up Slack #

Set up PM2 and systemd service #

Running the dist-server on a URL #

DNS #

Example: #

NGINX #

🎉 Congrats! #

Production setup

Server OS

Requirements

Set up user

Install dependencies

Set up .env

Set up NGINX

Example (with recommended VHOST_PATH):

Set up certbot

Set up Slack

Set up PM2 and systemd service

Running the dist-server on a URL

DNS

Example:

NGINX

🎉 Congrats!