Wiki 🌱

Everything I know.

Source

AWS

This is a continually updated page, and very much a work in progress. You can contribute on GitHub.

In this guide I want to go through as many services as possible and (briefly) explain what the service is and how you can benefit from using it.

Compute

  • EC2 (Elastic Compute Cloud) β€” Much like a VPS or a virtual machine. EC2 offers compute power and full administrator access to a server running either Linux, Windows or even macOS. EC2 offers before Intel (X86) and Graviton (ARM) options for its CPU.
  • EC2 Autoscaling β€” Offers services designed to automatically scale the amount of EC2 instances up or down as load/demand fluctuates.
  • Lightsail β€” A service similar to EC2 but with the pricing model of a company like DigitalOcean or Linode as a predetermined amount of data transfer is included in the price.

Storage

  • S3 (Simple Storage Service) β€” A service that offers unlimited storage. You can put files up to 5 Terabytes in so called buckets, which act like folders. You can interface with S3 via the API but there are a lot of third-party clients as well.
  • EFS (Elastic File System) β€” Offers NFS volumes in the cloud, with the added value that you don’t have to provision any storage yourself. EFS will grow the storage automatically for you.
  • EBS (Elastic Block Store) β€” Acts like a USB drive for your compute resources. EBS offers so called volumes which can be up to 16 Terabytes in size and come in many different variants. There’s a variant which offers cheap but slow storage, and a variant which offers very fast storage but that comes at a price.
  • AWS Backup β€” A service to centrally manage backups of EC2 instances, EBS and EFS volumes, DynamoDB tables and much more. AWS Backup allows you to set schedules and even copy backups to other AWS regions.
  • Storage Gateway β€” A solution to connect (on-premises) infrastructure to unlimited cloud storage using standard protocols like NFS, SMB and iCSCI. The gateway connects to storage services like S3, Glacier, and Amazon FSx for Windows File server.

Networking and content delivery

Security

Databases

Serverless

Containers

Analytics

Machine Learning

Other

CDK

Supported languages

  • typescript
  • javascript
  • python
  • java
  • c#

Install

brew install node aws-cdk

Init using typescript

mkdir project
cd project
cdk init app --language typescript

Constructs

npm update
npm install @aws-cdk/aws-lambda

Use construct

import * as lambda from '@aws-cdk/aws-lambda';

Credentials

Using a central IAM account

~/.aws/credentials

[source]
aws_access_key_id = ...
aws_secret_access_key = ...

~/.aws/config

[profile [PROFILE NAME]]
region = eu-west-1
output = json
role_arn = arn:aws:iam::[ACCOUNT ID]:role/[ROLE NAME]
mfa_serial = arn:aws:iam::[ACCOUNT ID]:mfa/[IAM USERNAME]
source_profile = source
export AWS_PROFILE=PROFILE NAME

Script for switching using credentials

#!/usr/bin/env bash

echo "Enter the account for which you want to export credentials and [enter]"
read account

export AWS_ACCESS_KEY_ID=$(aws configure get $account.aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get $account.aws_secret_access_key)
export AWS_REGION=$(aws configure get $account.region)

EC2

Resize root disk

  1. Create and start an EC2 instance with the same OS (+version)
  2. Create a snapshot of the original root volume
  3. Create a new volume from the snapshot (source)
  4. Create a new larger/smaller volume (target)
  5. Connect the source as xvdf
  6. Connect the target as xvdg

Run these commands:

e2fsck -f /dev/xvdf1
resize2fs -M -p /dev/xvdf1

Note: save the amount of blocks this results in to calculate the new blocks: (result) * 4 / 1024 = (new blocks)

dd bs=1M if=/dev/xvdf of=/dev/xvdg count=new blocks
e2fsck -f /dev/xvdg1

Connect the target volume to the original instance as /dev/sda1 and boot from the newly created volume.

EKS

Update kubeconfig

aws eks update-kubeconfig --name <cluster name> --region <region>

Assign OIDC provider

eksctl utils associate-iam-oidc-provider --region=eu-west-1 --cluster=<CLUSTER NAME> --approve

Install Nginx ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/deploy.yaml

Nginx ingress TLS termination on NLB

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.48.1/deploy/static/provider/aws/deploy-tls-termination.yaml
# Change VPC CIDR and ACM ID
k apply -f deploy-tls-termination.yaml

Example service using nginx ingress

apiVersion: v1
kind: Namespace
metadata:
  name: demo-ns
  annotations:
    iam.amazonaws.com/permitted: ".*"
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: demo-ns
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx
--- 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: demo-ns
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
          - containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-ingress
  annotations:
    # use the shared ingress-nginx
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: demo.mijndertstuij.nl
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80

EKSCTL

Basic cluster

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: mijndert-poc
  region: eu-west-1
  version: "1.21"

privateCluster:
  enabled: true

nodeGroups:
  - name: workers
    instanceType: t3.medium
    desiredCapacity: 1
    privateNetworking: true
  
iam:
  withOIDC: true 
eksctl create cluster -f cluster.yaml
eksctl update cluster -f cluster.yaml --aprove
eksctl delete cluster -f cluster.yaml

SSM

Tunnel to a resource in a private subnet using instance connect.

#!/bin/bash

green=`tput setaf 2`
reset=`tput sgr0`

echo "${green}Hostname of the resource you want to connect to:${reset}"
read -e hostname
echo "${green}Port of the thing you want to connect to:${reset}"
read -e port
echo "${green}Connecting (quit with ctrl+c)${reset}"

export ID=$(aws ec2 describe-instances --output=text --query "Reservations[*].Instances[*].InstanceId" --filters Name=tag:Name,Values="BastionStack*")
export AZ=$(aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[Placement.AvailabilityZone]' --filters Name=tag:Name,Values="BastionStack*")

aws ec2-instance-connect \
    send-ssh-public-key \
    --availability-zone $AZ \
    --instance-id $ID \
    --instance-os-user ssm-user \
    --ssh-public-key file://$HOME/.ssh/id_ed25519.pub &>/dev/null & disown;

ssh ssm-user@$ID -NL $port\:$hostname\:$port

~/.ssh/config

host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Cloud

Terraform

AWS provider

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.22.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

State configuration

terraform {
  backend "s3" {
    region         = "<REGION>"
    bucket         = "<BUCKET NAME>"
    key            = "<FILENAME>.tfstate"
    dynamodb_table = "<TABLE NAME>"
    encrypt        = true
  }
}

Get AZs

data "aws_availability_zones" "available" {
}

Latest AMI

data "aws_ami" "latest-ubuntu" {
most_recent = true
owners = ["099720109477"] # Canonical

  filter {
      name   = "name"
      values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-18.04-amd64-server-*"]
  }

  filter {
      name   = "virtualization-type"
      values = ["hvm"]
  }
}

Random string

resource "random_string" "suffix" {
  length  = 8
  special = false
}

Datasources

data "aws_vpc" "selected" {
  tags = {
    environment = var.environment
  }
}

data "aws_subnet_ids" "private" {
  vpc_id = data.aws_vpc.selected.id

  tags = {
    Name = "*private*"
  }
}

data "aws_subnet_ids" "public" {
  vpc_id = data.aws_vpc.selected.id

  tags = {
    Name = "*public*"
  }
}

vpc_zone_identifier = var.public == "true" ? data.aws_subnet_ids.public.ids : data.aws_subnet_ids.private.ids

Development

Hugo

Used for my website

Install

brew install hugo

Init new site

hugo new site <name>

Run server

hugo server -D

Git

Generate key

ssh-keygen -q -a 64 -b 4096 -t ed25519 -C mijndert@mijndertstuij.nl -f ~/.ssh/ed25519

Set default branch name

git config --global init.defaultBranch main

Sync fork

git remote -v
git remove add upstream <url>
git fetch upstream
git checkout main
git merge upstream/main

New repo

git init
touch README.md
git ca
git remote add origin <url>
git push -u origin main

Change branch name to main

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Git send-email on Mac

brew install cpanm
cpanm --sudo Net::SMTP::SSL

.gitconfig

[sendemail]
  from = Your name <your-username@gmail.com>
  smtpserver = smtp.gmail.com
  smtpuser = your-name@gmail.com
  smtppass = your-application-specific-password
  smtpencryption = tls
  chainreplyto = false
  smtpserverport = 587

Create patch

git format-patch --to email@mailinglist.org HEAD^

or

git send-email 0001-name-of-my-patch.patch

or

git format-patch -o outgoing HEAD~5
git send-email outgoing/*.patch
rm outgoing/*.patch
rmdir outgoing

Get branch for all sub-directories

for dir in */
  do (cd $dir && echo "$dir [$(git rev-parse --abbrev-ref HEAD)]")
done

GitHub

Reach GitHub on port 443 using SSH

~/.ssh/config

Host github.com
  Hostname ssh.github.com
  Port 443
  User git
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512

Python

Virtualenv

Create a virtualenv called tfenv:

python3 -m virtualenv tfenv -p python3 --system-site-packages
source tfenv/bin/activate

Tensorflow on M1

Some more packages are still needed but this at least provides a working set. Any errors can be solved by additional pip3 installs.

pip3 install --no-dependencies tensorflow-macos tensorflow-metal absl-py wrapt opt-einsum gast astunparse termcolor flatbuffers google numpy google-api-python-client google-pasta
pip3 install jupyterlab
jupyter-lab

Flask

Hello world

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello world!'

Dockerfile

FROM python:3.8-alpine

WORKDIR /app

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

COPY . .

CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]

Linux

Linux distro’s I used before:

  • Gentoo
  • LFS
  • Debian
  • SuSE
  • CentOS
  • RedHat
  • Fedora
  • Ubuntu
  • Xubuntu
  • Arch
  • NixOS

Acme

An ACME protocol client written purely in Shell (Unix shell) language.

GitHub

Install

git clone https://github.com/Neilpang/acme.sh.git
cd ./acme.sh
./acme.sh --install

Register account

acme.sh --register-account

Configure Nginx

server {
...
  location ~ "^/\.well-known/acme-challenge/([-_a-zA-Z0-9]+)$" {
    default_type text/plain;
    return 200 "$1.YOURTHUMBPRINT";
  }
...
}

Issue a certificate

acme.sh --issue -d example.com -d www.example.com  --stateless

Install the certificate

acme.sh --install-cert -d example.com \
--keypath       /etc/nginx/certificates/example.com.key  \
--fullchainpath /etc/nginx/certificates/example.com.pem \
--reloadcmd     "service nginx force-reload"

Configure vhost

server {
...
  listen 443 ssl;
  listen [::]:443 ssl;

  ssl_certificate /etc/nginx/certificates/example.com.pem;
  ssl_certificate_key /etc/nginx/certificates/example.com.key;

  if ($scheme = http) {
    return 301 https://$server_name$request_uri;
  }
...
}

Cron

Syntax

minute / hour / dom / month / dow

* - any value
, - set of values
- - value range
  • minute

    • 1–59
  • hour

    • 0–23
  • dom

    • 1–31
  • month

    • 1–12
    • JAN–DEC
  • dow

    • 0–6
    • SUN–SAT
0 0 * * * command
0 0 1 * * another command

System cron jobs

Applies to systems fitted with systemd

(crontab -l 2>/dev/null; echo "0 1 * * * ntpdate 0.nl.pool.ntp.org >/dev/null 2>&1") | crontab -
(crontab -l 2>/dev/null; echo "0 2 * * * journalctl --vacuum-size=200M >/dev/null 2>&1") | crontab -

IPtables

OpenVPN server config

server_nic=$(ip route list | grep default | grep -E  'dev (\w+)' -o | awk '{print $2}')
apt-get -qy install iptables-persistent
echo iptables-persistent iptables-persistent/autosave_v4 boolean true | sudo debconf-set-selections
echo iptables-persistent iptables-persistent/autosave_v4 boolean true | sudo debconf-set-selections
iptables -A INPUT -i $server_nic -m state --state NEW -p $openvpn_proto --dport $openvpn_port -j ACCEPT
iptables -A INPUT -i tun+ -j ACCEPT
iptables -A FORWARD -i tun+ -j ACCEPT
iptables -A FORWARD -i tun+ -o $server_nic -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $server_nic -o tun+ -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o $server_nic -j MASQUERADE
iptables -A OUTPUT -o tun+ -j ACCEPT
sed -i 's|#net.ipv4.ip_forward=1|net.ipv4.ip_forward=1|' /etc/sysctl.conf
echo 1 > /proc/sys/net/ipv4/ip_forward

Nginx

Install

curl https://nginx.org/keys/nginx_signing.key | apt-key add -
echo "deb http://nginx.org/packages/ubuntu/ focal nginx" > /etc/apt/sources.list.d/nginx-stable.list
apt update 
apt install nginx

dhparams

openssl dhparam -out /etc/nginx/dhparam.pem 4096

Simple vhost

server {
   listen 80;
   server_name domain.nl www.domain.nl;
   root /home/DOMAIN/www;
   index index.html;
}

Basic auth

Location / {
    auth_basic "Login for <BLABLA>";
    auth_basic_user_file /home/BLABLA.htpasswd
}

Limit by IP

Location / {
    satisfy any;
    allow 1.2.3.4;
    deny all;
}

FastCGI

location ~ \.(hh|php)$ {
   root /home/USERNAME/www;
   include /etc/nginx/fastcgi_params;
   fastcgi_pass 127.0.0.1:9000;
   try_files $uri=404;
   fastcgi_index index.php;
   fastcgi_param SCRIPT_FILENAME /home/USERNAME/www$fastcgi_script_name;
   fastcgi_param PATH_TRANSLATED /home/USERNAME/www$fastcgi_script_name;
}

Redirects

server {
   listen       80;
   server_name  olddomain.nl www.olddomain.nl;
   return       301 http://www.DOMAINNAME.nl$request_uri;
}

Or:

if ($scheme = 'http') {
     rewrite ^ https://$host$request_uri? permanent;
}

Regex example:

# Server-wide redirect
rewrite ^/example/something/?.* http://DOMAIN.nl/example permanent;
# Location-specific redirect
location /example/something/ {
  rewrite /example/something/(.*)$ http://DOMAIN.nl/$1;
}

If your way out of a rewrite:

location ~ ^/feed$ { if ($arg_post_type = "") { rewrite ^/rss2/?$http://feedpress.me/example redirect; } }

Rsync

sync -arpvz dst src
  • a = archive
  • r = recursive
  • p = preserve permissions
  • v = verbose
  • z = compress

Swap

fallocate -l 1G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "/swapfile   none    swap    sw    0   0" >> /etc/fstab

OpenSSH

Daemon settings

sed -z 's/PermitRootLogin yes\|$/PermitRootLogin no/' /etc/ssh/sshd_config
sed -z 's/LoginGraceTime 120\|$/LoginGraceTime 10/' /etc/ssh/sshd_config
sed -z 's/PasswordAuthentication yes\|$/PasswordAuthentication no/' /etc/ssh/sshd_config

Generate SSH key

ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519

Client config

Host *
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519
  ForwardAgent yes
  ConnectTimeout 30
  KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
  Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
  ServerAliveInterval 10
  ServerAliveCountMax 1

Timezone

ln -fs /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime

UFW

ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 22/tcp
ufw --force enable

Unbound

Run

docker run -d --name unbound -p 53:5053/tcp -p 53:5053/udp --restart=unless-stopped mijndert/unbound

Test

docker exec unbound drill -p 5053 mijndertstuij.nl @127.0.0.1

Get statistics

unbound-control -c /var/unbound/unbound.conf stats

Cache

unbound-control -c /var/unbound/unbound.conf dump_cache | grep mijndertstuij.nl

Users

useradd -s /bin/bash -m mijndert -d /home/mijndert
usermod -aG www-data mijndert
usermod -aG sudo mijndert
usermod -aG docker mijndert
mkdir /home/mijndert/www
chown -R mijndert:www-data /home/mijndert/www
chmod -R 0770 /home/mijndert/www
chmod g+s /home/mijndert/www
mkdir /home/mijndert/.ssh
echo "..." > /home/mijndert/.ssh/authorized_keys
chown -R mijndert:mijndert /home/mijndert/.ssh

Software

https://mijndertstuij.nl/uses/

EditorConfig

root = true

[*]
insert_final_newline = true
charset = utf-8
end_of_line = lf
indent_size = 4
indent_style = space
trim_trailing_whitespace = true

[*.{tf,yaml}]
indent_size = 2

[*.md]
trim_trailing_whitespace = false

Firefox

https://www.mozilla.org/firefox/new/

Extensions

  • Behind the overlay revival
  • Bypass firewalls
  • Don’t touch my tabs!
  • Facebook container
  • Firefox Multi-Account Containers
  • HTTPS Everywhere
  • uBlock Origin

iTerm

Restore tmux session

profiles > general > send text at start

tmux ls && read tmux_session && tmux attach -t ${tmux_session:-default} || tmux new -s ${tmux_session:-default}

Podman

brew install podman
podman machine init
podman machine start

Configuration located at /Users/mijndert/.config/containers/podman/machine/qemu

Run container in bridge mode to port forward

podman -r run --network bridge -p 8080:80 docker.io/nginx

M1

brew install simnalamburt/x/podman-apple-silicon
podman machine init
podman machine start

https://github.com/simnalamburt/homebrew-x

Compose

https://github.com/containers/podman-compose

Tor

Tor relay (run on a server with some amount of bandwidth)

docker run -d \
    -v /etc/localtime:/etc/localtime:ro \ 
    -v $PWD/tor-data:/var/lib/tor \ 
    --restart always \ 
    -p 9001:9001 \ 
    --name tor-relay \ 
    mijndert/tor-relay

Vim

editorconfig

mkdir -p ~/.vim/pack/local/start
cd ~/.vim/pack/local/start
git clone https://github.com/editorconfig/editorconfig-vim.git

dracula

mkdir -p ~/.vim/pack/themes/start
cd ~/.vim/pack/themes/start
git clone https://github.com/dracula/vim.git dracula

~/.vimrc

packadd! dracula                                                                                                            
syntax enable                                                                                                              
colorscheme dracula    
set termguicolors