Flynn

Open-source PaaS software

Available from https://flynn.io/

Perl applications

There's no out of the box support for Perl PSGI applications, but it just takes a single command to setup the environment.

flynn -a yourappname env set \
  BUILDPACK_URL=https://github.com/pnu/heroku-buildpack-perl \
  PERL5LIB=/app/lib:/app/local/lib/perl5

There has to be an app.psgi file (the actual filename is app.psgi, that's not a placeholder) in the top directory of your application. If you use the default Catalyst application structure, you should have a 'yourappname.psgi' which can just be renamed to app.psgi

Useful commands

Dashboard login recovery

To recover the dashboard login token:

flynn -a dashboard env | grep LOGIN_TOKEN

App runs out of space on the overlay partition

For example if your web application generates a lot of temporary files and doesn't clean up or get restarted often enough. Increase the temporary disk space for web service (default 100MB)

flynn limit set web temp_disk=200MB

This also applies to the mariadb app - if /tmp fills up, this will cause problems so you can expand it in a similar way to above. If you have a lot of data, you may need to increase the deployment timeout first. The following commands set the timeout to 1 hour and then increases the disk space for mariadb to 1GB.

flynn -a mariadb deployment timeout 3600
flynn -a mariadb limit set temp_disk=1G

Restart any app

This works simply because setting an environment variable causes a new release of the app to be started. It doesn't matter if the environment variable isn't being changed, it will still cause a restart.

flynn -a router env set RESTART=1

Increase file descriptors

Increase file descriptors for the router in case of heavy load or long-running connections. Default is 10,000

flynn -a router limit set app max_fd=40000

Update a release using JSON

To edit linux capabilities (for example), create a json file containing the things you need to update within the release

{
  "processes": {
    "web": {
      "linux_capabilities": [
        "CAP_NET_RAW",
        "CAP_NET_BIND_SERVICE",
        "CAP_DAC_OVERRIDE",
        "CAP_SETFCAP",
        "CAP_SETPCAP",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_MKNOD",
        "CAP_CHOWN",
        "CAP_FOWNER",
        "CAP_FSETID",
        "CAP_KILL",
        "CAP_SYS_CHROOT",
        "CAP_NET_ADMIN"
      ]
    }
  }
}

And then use the cli to push the changes to the application.

flynn release update release.json

Github repo deployed via webhook

See https://github.com/lmars/flynn-webhook-deploy

Accessing files from a job's filesystem

Primarily for use where there is a job running and you need a file it has generated. ssh to the flynn host and look for the files in

/var/lib/flynn/volumes/zfs/mnt/ext2/$JOBID/overlay-upperdir

Persistent volume for docker apps

Edit the release to indicate the jobs should get a persistent volume

flynn release update /dev/stdin <<< '{"processes": {"app": {"volumes": [{"path": "/data"}]}}}'

Using a custom TLS wildcard cert during setup

This is for using a wildcard cert for the dashboard, controller, git etc.

export TLS_CA="
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
"

export TLS_CERT="
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
"

export TLS_KEY="
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
"

export CONTROLLER_KEY="..."
flynn-host bootstrap

Custom certs for individual applications can be updated using `flynn route` from within the application directory.

flynn route <route-id> update \
    -k your-domain.key -c yur-domain.crt

Or from any folder, using -a

flynn -a yourapp route <route-id> update \
    -c your-domain.crt -k your-domain.key

Updating to a new release

Always perform either a filesystem snapshot or a cluster backup first!

flynn cluster backup --file=backup.tar

Although there's a `flynn-host update` command, the outcome is a little unreliable as of March 2017 so the preferred update route is as follows.

If you can install a new host, do this first and bootstrap from the backup. Then update the DNS to point to the new host IP.

If you have to re-use the same host, do the backup as detailed above, copy your backup.tar.gz to somewhere safe! If you are using mariadb, check inside the .tar and make sure it contains mysql.sql.gz in addition to flynn.json and postgres.sql.gz

curl -fsSL -o /tmp/install-flynn https://dl.flynn.io/install-flynn
/tmp/install-flynn --clean  ## Plus any extra parameters you used originally!
CLUSTER_DOMAIN=flynn.bocks.com flynn-host bootstrap --min-hosts 1 --from-backup=backup.tar

There's a bug relating to mounted filesystems but this worked ok for me:

DATE=`date +%Y%m%d`
flynn cluster backup --file=/root/backup-$DATE.tar

service flynn-host stop
umount /var/lib/flynn/volumes/zfs/mnt/squashfs/*
umount /var/lib/flynn/volumes/zfs/mnt/data/*
umount /flynn-default/*
umount /flynn-default
umount /tmp/tmp.*/mnt

curl -fsSL -o /tmp/install-flynn https://dl.flynn.io/install-flynn
chmod 755 /tmp/install-flynn
/tmp/install-flynn --clean
service flynn-host start
CLUSTER_DOMAIN=flynn.bocks.com flynn-host bootstrap --min-hosts 1 --from-backup=/root/backup-$DATE.tar

Generating a trusted TLS cert for a single app

Use the client at https://github.com/google/acme

acme reg -gen -accept email@example.com
acme cert -dns example.com flynnapp1.example.com flynnapp2.example.com flynapp3.example.com 

.. to generate a cert that would work for 3 web services and then update the flynn route.

To get the route id, just call flynn route

$ flynn -a appname route
ROUTE               SERVICE      ID                                         STICKY  LEADER  PATH
http:www.<domain>   appname-web  http/0bdbc65c-576e-4904-bad9-b0344df721db  false   false   /
http:<domain>       appname-web  http/69a8160c-0ab5-4b71-889c-6a10c675dd1e  false   false   /

Then using the route id:

flynn -a yourapp route update <route-id> -c ~/.config/acme/your-domain.pem -k ~/.config/acme/your-domain.key

to add the certificate to an application.

Git requirements

Requires git 1.8.5 or higher for seamless publishing via git

1.8.3 (CentOS7 default) requires an environment variable to publish app due to the self-signed SSL certificate. Since the CA certificate is stored within ~/.flynn/ when you setup the cluster, the GIT_SSL_CAINFO env can be used to specify the CA used.

$ GIT_SSL_CAINFO=~/.flynn/ca-certs/default.pem git push flynn master

Below 1.7.3 (e.g CentOS6 default of 1.7.1) there's a little more work required because it doesn't support the credential helper. You'll need the key for your flynn cluster which can be found with:

$ grep Key ~/.flynnrc
    Key = "44161646005d26ede2c6687aaaaaaaaa"

$ git remote get-url flynn
https://git.flynn1.bocks.com/myapp.git
$ git remote set-url flynn https://:44161646005d26ede2c6687aaaaaaaaa@git.flynn1.bocks.com/myapp.git

To push the repository with git 1.7.x, you still need the GIT_SSL_CAINFO env as for git 1.8.3

See Updating Git for instructions.

Firewall

Flynn requires a bunch of firewall rules to secure the API from external users.

ufw allow ssh
ufw allow http
ufw allow https
ufw allow 3000:3500/tcp
ufw allow from a.b.c.d   # repeat for each node IP address if in a cluster
ufw enable
ufw allow in on flynnbr0
ufw allow in on flannel.1

This next line is only required if you want to give connection refused instead of silently dropping packets.

ufw default REJECT

Because applications run in Docker, ufw needs to forward some traffic so edit the ufw config /etc/default/ufw and change

DEFAULT_FORWARD_POLICY="DROP"

to

DEFAULT_FORWARD_POLICY="ACCEPT"

Then reload the firewall

ufw reload

The full information about these rules can be found at https://www.philiplb.de/flynn/2016/04/19/flynn-ufw/