Go

Context

Context is a concurrency pattern in Go. It allows request handlers in Go servers to pass cancellation signals to any goroutine that has started from this request.

For example, in a web server, the each request is typically handled by a different goroutine, which in turn may start multiple goroutines to execute database queries or send requests to other services.

Using the context pattern, a struct called the context is passed from one function to the other containing a Done channel, which any function starting goroutines should wait on, typically with select. If the Done channel is closed, any goroutine should do a best effort to stop any operation and free resources used.

Without context, an http request may be cancelled, but gorotoutines pointlessly continue their operations. The resources spared in this way are typically small but in situations of heavy load, requests may become slow and with clients resending their requests when aborted, performance issues may compound.

• You can add values to it like a map using withValue but it's somewhat of an antipattern
• The main use is for canceling goroutines
• Let's say there is an http request coming, and you start an expensive operation
  • In Node.js, you'd execute this operation, which may even take 10 min.
  • But the client has left, not waiting for a response.
  • You would continue processing it, and at the end put it in the request, which will not be sent anywhere.
• Using native promises, there is no way, for the one who started it, to cancel it.
• In Go, your goroutine can listen in the done channel, to know when to cancel.
• Also, you can define a context deadline as a time duration.
• If you switch to channels, it's good practice to always accept a context and always listen in the done channel.
• Also, for every function that does IO operations you can pass context and call the libraries by passing the context, expecting them to accept it. Eg. in database calls.
  • In this way, your can separate between IO and pure functions too.

Docker

Run image with current directory mounted

docker run --rm -it -v $(pwd):/usr/src/project ubuntu:focal

Kubernetes

Don't use it.

GitHub Actions

Example configuration yaml for Django:

name: Django CI

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.8
      uses: actions/setup-python@v1
      with:
        python-version: 3.8
    - name: Install Dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Run Tests
      run: |
        python manage.py test
      env:
        SECRET_KEY: "thisisthesecretkey"
        DATABASE_URL: "postgres://postgres:postgres@localhost:5432/postgres"

  deploy:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/master'
    needs: build
    steps:
      - name: Configure SSH
        run: |
          mkdir -p ~/.ssh/
          echo "$SSH_KEY" > ~/.ssh/heartfort.key
          chmod 600 ~/.ssh/heartfort.key
          cat >> ~/.ssh/config << EOF
          Host heartfort
            HostName heartfort.com
            User root
            IdentityFile ~/.ssh/heartfort.key
            StrictHostKeyChecking no
          EOF
          echo DATABASE_URL=$DATABASE_URL > ~/sshenv
          scp ~/sshenv heartfort:~/.ssh/environment
        env:
          SSH_KEY: ${{ secrets.SSH_KEY }}
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - name: Clone repository
        run: |
          ssh heartfort "rm -rf /opt/apps/srdce"
          ssh heartfort 'cd /opt/apps/ && git clone git@github.com:sirodoht/srdce.git --config core.sshCommand="ssh -i ~/.ssh/id_rsa_github_deploy_key"'
      - name: Install requirements
        run: ssh heartfort 'cd /opt/apps/srdce && python3 -m venv venv && . venv/bin/activate && pip3 install -r requirements.txt'

      - name: Collect static
        run: ssh heartfort "cd /opt/apps/srdce && . venv/bin/activate && DATABASE_URL=$DATABASE_URL python3 manage.py collectstatic --noinput"
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - name: Run migrations
        run: ssh heartfort "cd /opt/apps/srdce && . venv/bin/activate && DATABASE_URL=$DATABASE_URL python3 manage.py migrate"
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - name: Reload server
        run: ssh heartfort 'touch /etc/uwsgi/vassals/srdce.ini'

You have jobs, and each job has steps. It can also depend on other jobs.

A step can be inheritable. Like Docker's FROM. E.g. uses: actions/checkout@v2. The uses key means that it will draw this code to execute: actions/checkout.

If your step has run then it just runs that in the shell.

For using environment variable, one can add them in the repository settings and if they are a secret, to use them like DATABASE_URL: ${{ secrets.DATABASE_URL }}.

To run a job only on master, we use if: github.ref == 'refs/heads/master'.

If you're doing a static website, then it can be like this:

name: Deploy mdBook

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/master'
    steps:
      - uses: actions/checkout@v2

      - name: Configure SSH
        run: |
          mkdir -p ~/.ssh/
          echo "$SSH_KEY" > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa
          cat >> ~/.ssh/config << EOF
          Host venn
            HostName wiki.venn.dev
            User root
            IdentityFile ~/.ssh/id_rsa
            StrictHostKeyChecking no
          EOF
        env:
          SSH_KEY: ${{ secrets.SSH_KEY }}

      - name: Install mdbook
        run: |
          wget https://github.com/rust-lang/mdBook/releases/download/v0.4.7/mdbook-v0.4.7-x86_64-unknown-linux-gnu.tar.gz
          tar -xzvf mdbook-v0.4.7-x86_64-unknown-linux-gnu.tar.gz
      - name: Build site
        run: ./mdbook build

      - name: Cleanup old files
        run: |
          ssh venn "rm -r /var/www/wiki.venn.dev/"
          ssh venn "mkdir -p /var/www/wiki.venn.dev/html"
      - name: Install requirements
        run: rsync -rsP ./book/ root@wiki.venn.dev:/var/www/wiki.venn.dev/html

      - name: Reload nginx
        run: ssh venn "systemctl reload nginx"

Since it's ubuntu (runs-on: ubuntu-latest) we can install stuff and use existing tools like wget and tar.

Dokku

Dokku is a self-hosted heroku-like service. You can git push to start a deployment build with heroku-like buildpacks or Docker, and it will handle updating load balancer (nginx) configuration, zero-downtime deployments, SSL certificates via let's encrypt, etc.

Manual deployment

In case you don't need code to be built, you can manually deploy a pre-built docker image, for example for deploying docker images built by others.

1. Create a new app with dokku apps:create APP_NAME
2. Tag the pre-built docker image as dokku/APP_NAME:VERSION with docker image tag IMAGE dokku/APP_NAME:VERSION. Replace VERSION with 0.0.1,0.0.2 etc.
3. Deploy the docker image using dokku tags:deploy APP_NAME VERSION.

Example: Grafana deployment

In this example we deploy grafana version 7.3.7, using the /var/lib/dokku/data/storage folder for volumes as recommended by dokku.

$ dokku apps:create grafana
$ docker image pull grafana/grafana:7.3.7
$ docker image tag grafana/grafana:7.3.7 dokku/grafana:7.3.7
$ dokku storage:mount grafana /var/lib/dokku/data/storage/grafana:/var/lib/grafana
$ sudo mkdir /var/lib/dokku/data/storage/grafana
$ sudo chown 472:472 /var/lib/dokku/data/storage/grafana/
$ dokku tags:deploy grafana 7.3.7
$ dokku proxy:ports-set grafana http:80:3000

Monads

Probabilistic Programming (List monad)

Let's say we are playing a game, and we have a light switch.

In every turn, if it's on, there's a 50-50% chance to switch it off, but when it's off, there's 2/3 chance to switch it on.

let turn x = if x == "on" then ["off", "on"] else ["on", "on", "off"]

What happens if we start from an "on" position after one turn?

> ["on"] >>= turn
["off","on"]

The values in the list signify all the probable current states (and not a particular state nor the states that we passed from), starting from the on state and after one turn.

So, 1/2 cases it was off, 1/2 it was off. Okay.

What about after two turns?

> ["on"] >>= turn >>= turn
["on","on","off","off","on"]

Okay, 2/3 chance to be on, 1/3 off.

> ["on"] >>= turn >>= turn >>= turn
["off","on","off","on","on","on","off","on","on","off","off","on"]

After three turns it's a bit hard to figure out what's going on so, let's build a function for asking the probability of a state.

count = fromIntegral . length :: [a] -> Float

(??) p x = 100 * (count (filter (== x) p)) / (count p)

?? returns the percentage of elements in p that are equal to x.

count is just a wrapper around length that returns a Float instead of an Int.

> (["on"] >>= turn >>= turn >>= turn) ?? "off"
41.666666666666664
> (["on"] >>= turn >>= turn >>= turn) ?? "on"
58.333333333333336

A more interesting example, let's say that we have a pandemic (cough, cough) and everyday there's:

• 25% chance to get sick when healthy
• 17% chance to get hospitalized, 33% chance to recover when sick,
• and 50-50% chance to die or recover when hospitalized

let vprob x = if x == "healthy" then ["healthy", "healthy", "healthy", "sick"] else if x == "sick" then ["sick", "sick", "sick", "hospitalized", "healthy", "healthy"] else if x == "hospitalized" then ["healthy", "dead"] else ["dead"]

What's the probability one will die after 10 days?

> let p = ["healthy"] >>= gen >>= gen >>= gen >>= gen >>= gen >>= gen >>= gen >>= gen >>= gen >>= gen
> p ?? "dead"
2.2962844

Okay, okay, haskell is full of tricks, what do I care. Well, if you understand how this works, you can do this in python too.

We could call >>= function apply (haskellers: I know apply is a different function, sssh). Apply takes a function that transforms a list and the list and returns the new list.

The transforming function will take one element from the list and return all future states. So, for each previous state, we get a list of new states, therefore we will flatten the result.

def flatten(l): return sum(l, [])
  
def apply(p, f): return flatten([f(x) for x in p])

We can know define the previous vprob probability function:

def vprob(x): return ["healthy", "healthy", "healthy", "sick"] if x == "healthy" else ["sick", "sick", "sick", "hospitalized", "healthy", "healthy"] if x == "sick" else ["healthy", "dead"] if x == "hospitalized" else ["dead"]```

>>> apply(["healthy"], vprob)
['healthy', 'healthy', 'healthy', 'sick']
>>> apply(apply(["healthy"], vprob), vprob)
['healthy', 'healthy', 'healthy', 'sick', 'healthy', 'healthy', 'healthy', 'sick', 'healthy', 'healthy', 'healthy', 'sick', 'sick', 'sick', 'sick', 'hospitalized', 'healthy', 'healthy']

Applying the function multiple times is somewhat ugly in python so we will make a wrapper function apply_times that recursively calls apply:

def apply_times(p, f, c): return p if c == 0 else apply_times(apply(p, f), f, c-1)

>>> apply_times(["healthy"], vprob, 2)
['healthy', 'healthy', 'healthy', 'sick', 'healthy', 'healthy', 'healthy', 'sick', 'healthy', 'healthy', 'healthy', 'sick', 'sick', 'sick', 'sick', 'hospitalized', 'healthy', 'healthy']

And finally the function for determining the probability of a certain state:

def prob(p, s): return len([x for x in p if x == s])/len(p)

>>> prob(apply_times(["healthy"], gen, 2), "hospitalized")
0.05555555555555555

I tried running the apply_times with 10 parameter but python crashed =))

Postgresql

psql output

Sometimes when using psql, lines are really long.

One option is to enable column wrapping. This will wrap lines in columns keeping the max width of the total columns as long as the terminal total width. To enable:

\pset format wrapped

Another options is to enable extended view. This will show each row column in a separate row. To enable:

\x on

Docs link for all psql options: https://www.postgresql.org/docs/13/app-psql.html

nginx

Resources

• nginx documentation
• nginxconfig
• Mozilla SSL Configuration Generator

Sane default configuration examples

/etc/nginx/sites-available/example-static.com.conf for fully static website

• Two server directives, one SSL, one non-SSL.
• non-SSL redirects everything except for Let's Encrypt challenge (at /.well-known/acme-challenge/) to the SSL server.
• SSL cert and key are located in default certbot location: /etc/letsencrypt/live/<website-name>/fullchain.pem.
• HTTP2 enabled.
• Some security headers enabled.
• No dotfiles serving.
• Separate log files for each server.

server {
	listen 443 ssl http2;
	listen [::]:443 ssl http2;
	server_name wiki.venn.dev;
	root /var/www/wiki.venn.dev/html;

	# SSL
	ssl_certificate /etc/letsencrypt/live/wiki.venn.dev/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/wiki.venn.dev/privkey.pem;

	# security
	add_header X-Frame-Options "SAMEORIGIN" always;
	add_header X-XSS-Protection "1; mode=block" always;
	add_header X-Content-Type-Options "nosniff" always;
	add_header Referrer-Policy "no-referrer-when-downgrade" always;
	add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;

	# dotfiles
	location ~ /\.(?!well-known) {
		deny all;
	}

	# restrict methods
	if ($request_method !~ ^(GET|HEAD|CONNECT|OPTIONS|TRACE)$) {
		return '405';
	}

	# logging
	access_log /var/log/nginx/wiki.venn.dev.access.log;
	error_log /var/log/nginx/wiki.venn.dev.error.log warn;

	# favicon.ico
	location = /favicon.ico {
		log_not_found off;
		access_log off;
	}

	# robots.txt
	location = /robots.txt {
		log_not_found off;
		access_log off;
	}

	# assets, media
	location ~* \.(?:css(\.map)?|js(\.map)?|jpe?g|png|gif|ico|heic|webp|tiff?|mp3|m4a|aac|ogg|midi?|wav|mp4|mov|webm)$ {
		expires 7d;
		access_log off;
	}

	# svg, fonts
	location ~* \.(?:svgz?|ttf|ttc|otf|eot|woff2?)$ {
		add_header Access-Control-Allow-Origin "*";
		expires 7d;
		access_log off;
	}
}

server {
	listen 80;
	listen [::]:80;
	server_name wiki.venn.dev;

	location ^~ /.well-known/acme-challenge/ {
		root /var/www/_letsencrypt;
	}

	location / {
		return 301 https://wiki.venn.dev$request_uri;
	}
}

/etc/nginx/sites-available/example-uwsgi-django.com.conf for Django + uWSGI

• Two server directives, one SSL, one non-SSL.
• non-SSL redirects everything except for Let's Encrypt challenge (at /.well-known/acme-challenge/) to the SSL server.
• SSL cert and key are located in default certbot location: /etc/letsencrypt/live/<website-name>/fullchain.pem.
• HTTP2 enabled.
• Some security headers enabled along with HSTS.
• No dotfiles serving.
• Separate log files for each server.
• uWSGI configuration using unix sockets (nginx docs, uwsgi docs).
• Configured for serving Django static files. See here for how to configure Django for static files.

server {
	listen 443 ssl http2;
	listen [::]:443 ssl http2;
	server_name venn.dev;

	# deny requests with invalid host header
	if ( $host !~* ^(venn.dev)$ ) {return 444;}

	# SSL
	ssl_certificate /etc/letsencrypt/live/venn.dev/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/venn.dev/privkey.pem;

	# security headers
	add_header X-Frame-Options "SAMEORIGIN" always;
	add_header X-XSS-Protection "1; mode=block" always;
	add_header X-Content-Type-Options "nosniff" always;
	add_header Referrer-Policy "no-referrer-when-downgrade" always;
	add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
	add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

	# . files
	location ~ /\.(?!well-known) {
		deny all;
	}

	# logging
	access_log /var/log/nginx/venn.dev.access.log;
	error_log /var/log/nginx/venn.dev.error.log warn;

	location / {
		include                       uwsgi_params;
		uwsgi_pass                    unix:/run/uwsgi/venn.sock;
		uwsgi_param Host              $host;
		uwsgi_param X-Real-IP         $remote_addr;
		uwsgi_param X-Forwarded-For   $proxy_add_x_forwarded_for;
		uwsgi_param X-Forwarded-Proto $http_x_forwarded_proto;
	}

	# Django static
	location /static/ {
		alias /opt/apps/venn/static/;
	}

	# favicon.ico
	location = /favicon.ico {
		log_not_found off;
		access_log off;
	}

	# robots.txt
	location = /robots.txt {
		log_not_found off;
		access_log off;
	}
}

server {
	listen 80;
	listen [::]:80;
	server_name venn.dev;

	# ACME-challenge
	location ^~ /.well-known/acme-challenge/ {
		root /var/www/_letsencrypt;
	}

	location / {
		return 301 https://venn.dev$request_uri;
	}
}

/etc/nginx/nginx.conf

user www-data;
pid /run/nginx.pid;
worker_processes auto;
worker_rlimit_nofile 65535;
include /etc/nginx/modules-enabled/*.conf;

events {
	multi_accept on;
	worker_connections 768;
}

http {
	charset uf-8;
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	server_tokens off;
	types_hash_max_size 2048;
	client_max_body_size 16M;
	keepalive_timeout 65;

	server_names_hash_bucket_size 64;
	server_name_in_redirect off;

	# MIME
	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	# Logging
	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	# SSL
	ssl_session_timeout 1d;
	ssl_session_cache shared:SSL:10m;
	ssl_session_tickets off;
	ssl_prefer_server_ciphers on;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;

	# Gzip Settings
	gzip on;
	gzip_vary on;
	gzip_proxied any;
	gzip_comp_level 6;
	gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;

	# Virtual Host Configs
	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

Django

Static files

Django is too cool to serve static files.

We need either something like whitenoise which in the background uses the kernel's sendfile syscall.

Or, we can serve them above Django, from our web server / reverse proxy, if we have one (eg. nginx). See nginx for related configuration examples.

In both cases, one needs to have these two lines in their settings.py:

STATIC_URL = "/static/"
STATIC_ROOT = os.path.join(BASE_DIR, "static")

STATIC_URL is included in the Django project generation. STATIC_ROOT needs to be added manually.

Django docs how-to guide on static files here and here.

Also, enabling manifest static files is usually a good idea for high-quality cache busting. To do this, add this line as well in your settings.py.

STATICFILES_STORAGE = "django.contrib.staticfiles.storage.ManifestStaticFilesStorage"

Git

Clone with non-default SSH key

Maybe you need a specific SSH key to clone from GitHub, and maybe you don’t want to add it as default. One can specify which SSH command Git will use to do git operations.

Given my ssh key at /Users/sirodoht/.ssh/id_ed25519_debug, here is an example:

GIT_SSH_COMMAND='ssh -i /Users/sirodoht/.ssh/id_ed25519_debug -o IdentitiesOnly=yes' git clone git@github.com:debug-org/api.git