foxhunt/docs/plans/2026-02-22-dev-git-server-implementation.md

# Dev Git Server Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Provision a Scaleway DEV1-S instance in nl-ams-1 running Gitea, accessible only via Tailscale, with `fxhnt.ai` DNS managed by Scaleway.

**Architecture:** Single VM with cloud-init bootstrap. Tailscale provides the network layer — Gitea binds exclusively to the Tailscale interface. No public IP after provisioning. SQLite database (no external dependencies).

**Tech Stack:** Scaleway CLI (`scw`), Tailscale, Gitea, Ubuntu 24.04, cloud-init

---

### Task 1: Create Security Group

Create a temporary security group allowing inbound SSH for initial provisioning.

**Step 1: Create the security group**

```bash
scw instance security-group create \
  name=foxhunt-git-provisioning \
  inbound-default-policy=drop \
  outbound-default-policy=accept \
  zone=nl-ams-1 \
  project-id=c293eb98-228d-427d-9b16-f0941f3f2adb
```

Expected: Returns a security group ID. Save it as `$SG_ID`.

**Step 2: Add SSH inbound rule**

```bash
scw instance security-group-rule create \
  security-group-id=$SG_ID \
  direction=inbound \
  action=accept \
  protocol=TCP \
  dest-port-from=22 \
  zone=nl-ams-1
```

Expected: Rule created successfully.

---

### Task 2: Write Cloud-Init Script

Create the cloud-init user-data script that bootstraps Tailscale and Gitea.

**Step 1: Create the cloud-init file**

Create file: `docs/infra/cloud-init-git-server.yaml`

```yaml
#cloud-config
package_update: true
package_upgrade: true

packages:
  - git
  - sqlite3
  - curl
  - wget

write_files:
  - path: /etc/gitea/app.ini
    permissions: '0640'
    content: |
      [server]
      PROTOCOL         = http
      DOMAIN           = vm-fxhnt-git
      ROOT_URL         = http://vm-fxhnt-git:3000/
      HTTP_PORT        = 3000
      SSH_DOMAIN       = vm-fxhnt-git
      START_SSH_SERVER = true
      SSH_PORT         = 2222
      DISABLE_SSH      = false
      LFS_START_SERVER = true

      [database]
      DB_TYPE  = sqlite3
      PATH     = /var/lib/gitea/data/gitea.db

      [repository]
      ROOT = /var/lib/gitea/repositories

      [lfs]
      PATH = /var/lib/gitea/data/lfs

      [log]
      ROOT_PATH = /var/lib/gitea/log
      MODE      = file
      LEVEL     = Info

      [service]
      DISABLE_REGISTRATION = true

      [security]
      INSTALL_LOCK = false

runcmd:
  # Install Tailscale
  - curl -fsSL https://tailscale.com/install.sh | sh
  - tailscale up --authkey=tskey-api-kkwsqDRmg821CNTRL-d2z3w521Asa1d6Yb1CLatafwK3XJ6gA78 --hostname=vm-fxhnt-git --ssh

  # Wait for Tailscale interface
  - |
    for i in $(seq 1 30); do
      TS_IP=$(tailscale ip -4 2>/dev/null)
      if [ -n "$TS_IP" ]; then
        echo "Tailscale IP: $TS_IP"
        break
      fi
      sleep 2
    done

  # Get Tailscale IP for Gitea binding
  - TS_IP=$(tailscale ip -4)

  # Update Gitea config to bind to Tailscale IP only
  - sed -i "s/^HTTP_PORT.*/HTTP_ADDR = ${TS_IP}\nHTTP_PORT        = 3000/" /etc/gitea/app.ini

  # Create gitea user and directories
  - adduser --system --shell /bin/bash --gecos 'Gitea' --group --disabled-password --home /home/gitea gitea
  - mkdir -p /var/lib/gitea/{custom,data,log,repositories}
  - chown -R gitea:gitea /var/lib/gitea
  - chmod -R 750 /var/lib/gitea
  - mkdir -p /etc/gitea
  - chown root:gitea /etc/gitea
  - chmod 770 /etc/gitea

  # Download and install Gitea
  - |
    GITEA_VERSION=$(curl -s https://api.github.com/repos/go-gitea/gitea/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
    wget -O /usr/local/bin/gitea "https://dl.gitea.com/gitea/${GITEA_VERSION}/gitea-${GITEA_VERSION}-linux-amd64"
    chmod +x /usr/local/bin/gitea

  # Create systemd service
  - |
    cat > /etc/systemd/system/gitea.service << 'UNIT'
    [Unit]
    Description=Gitea
    After=syslog.target network.target tailscaled.service

    [Service]
    RestartSec=2s
    Type=simple
    User=gitea
    Group=gitea
    ExecStart=/usr/local/bin/gitea web --config /etc/gitea/app.ini
    Restart=always
    Environment=USER=gitea HOME=/home/gitea GITEA_WORK_DIR=/var/lib/gitea

    [Install]
    WantedBy=multi-user.target
    UNIT

  - systemctl daemon-reload
  - systemctl enable gitea
  - systemctl start gitea

  # Lock down SSH to Tailscale only (after Tailscale SSH is working)
  - |
    TS_IP=$(tailscale ip -4)
    echo "ListenAddress ${TS_IP}" >> /etc/ssh/sshd_config
    systemctl restart sshd

  # Signal completion
  - echo "PROVISIONING_COMPLETE" > /var/log/cloud-init-done
```

**Step 2: Verify the file is syntactically valid**

```bash
python3 -c "import yaml; yaml.safe_load(open('docs/infra/cloud-init-git-server.yaml'))"
```

Expected: No output (valid YAML).

---

### Task 3: Create the Scaleway Instance

**Step 1: Create the instance with cloud-init**

```bash
scw instance server create \
  name=vm-fxhnt-git \
  type=DEV1-S \
  image=ubuntu_noble \
  zone=nl-ams-1 \
  project-id=c293eb98-228d-427d-9b16-f0941f3f2adb \
  ip=new \
  cloud-init=@docs/infra/cloud-init-git-server.yaml \
  security-group-id=$SG_ID \
  tags.0=env=dev \
  tags.1=service=gitea \
  --wait
```

Expected: Instance created and running. Returns instance ID. Save as `$INSTANCE_ID`.
Note: `ip=new` assigns a temporary public IP needed for cloud-init to download packages and authenticate with Tailscale.

**Step 2: Monitor cloud-init progress**

```bash
# Wait ~2-3 minutes for cloud-init, then check
scw instance server ssh $INSTANCE_ID zone=nl-ams-1 command="tail -20 /var/log/cloud-init-output.log"
```

Expected: Log showing Tailscale and Gitea installation steps. Look for `PROVISIONING_COMPLETE`.

---

### Task 4: Verify Tailscale Connectivity

**Step 1: Check Tailscale from workstation**

```bash
tailscale ping vm-fxhnt-git
```

Expected: Pong from `vm-fxhnt-git` at a `100.x.x.x` address.
Note: If the old Azure `vm-fxhnt-git` is still on the network, there may be a conflict. The new node should take precedence if the old one is removed first — or Tailscale may assign a different name. Check `tailscale status` to verify.

**Step 2: SSH via Tailscale**

```bash
ssh root@vm-fxhnt-git "hostname && tailscale ip -4 && systemctl status gitea --no-pager"
```

Expected: Hostname `vm-fxhnt-git`, Tailscale IP shown, Gitea service active.

**Step 3: Get the Tailscale IP for DNS records**

```bash
TS_IP=$(ssh root@vm-fxhnt-git "tailscale ip -4")
echo "Tailscale IP: $TS_IP"
```

Save this IP for Task 6.

---

### Task 5: Configure Gitea

**Step 1: Create admin user via CLI**

```bash
ssh root@vm-fxhnt-git "sudo -u gitea /usr/local/bin/gitea admin user create \
  --config /etc/gitea/app.ini \
  --username foxhunt-admin \
  --password '<PROMPT_USER_FOR_PASSWORD>' \
  --email admin@fxhnt.ai \
  --admin"
```

Expected: User created successfully. **Ask the user for the admin password before running this.**

**Step 2: Lock the install**

```bash
ssh root@vm-fxhnt-git "sed -i 's/INSTALL_LOCK.*/INSTALL_LOCK = true/' /etc/gitea/app.ini && systemctl restart gitea"
```

Expected: Gitea restarts with install locked.

**Step 3: Verify Gitea web UI is accessible**

```bash
curl -s -o /dev/null -w "%{http_code}" http://vm-fxhnt-git:3000/
```

Expected: `200` (or `302` redirect to login page).

**Step 4: Create foxhunt organization and repo via API**

```bash
# Create org
curl -s -X POST http://vm-fxhnt-git:3000/api/v1/orgs \
  -H "Content-Type: application/json" \
  -u "foxhunt-admin:<PASSWORD>" \
  -d '{"username":"foxhunt","full_name":"Foxhunt","visibility":"private"}'

# Create repo in org
curl -s -X POST http://vm-fxhnt-git:3000/api/v1/orgs/foxhunt/repos \
  -H "Content-Type: application/json" \
  -u "foxhunt-admin:<PASSWORD>" \
  -d '{"name":"foxhunt","description":"Foxhunt HFT Trading System","private":true,"default_branch":"main"}'
```

Expected: Both return `201 Created` with JSON responses.

**Step 5: Push the foxhunt repo**

```bash
cd /home/jgrusewski/Work/foxhunt
git remote add gitea git@vm-fxhnt-git:foxhunt/foxhunt.git
git push gitea --all
git push gitea --tags
```

Expected: All branches and tags pushed. Uses Tailscale SSH via Gitea's built-in SSH server on port 2222 (or system SSH if configured).

---

### Task 6: Detach Public IP

**Step 1: Get the public IP ID**

```bash
scw instance server get $INSTANCE_ID zone=nl-ams-1 | grep -i "public.*ip"
```

Save the IP ID as `$IP_ID`.

**Step 2: Detach the public IP**

```bash
scw instance server detach-ip $INSTANCE_ID zone=nl-ams-1
```

Expected: IP detached.

**Step 3: Delete the temporary public IP**

```bash
scw instance ip delete $IP_ID zone=nl-ams-1
```

Expected: IP released.

**Step 4: Verify still accessible via Tailscale**

```bash
ssh root@vm-fxhnt-git "hostname"
curl -s -o /dev/null -w "%{http_code}" http://vm-fxhnt-git:3000/
```

Expected: SSH works, HTTP returns 200/302. No public internet access.

---

### Task 7: Configure DNS Records

The `fxhnt.ai` zone is now on Scaleway DNS (NS records being migrated from Azure).

**Step 1: Add A record for git.fxhnt.ai (Tailscale IP)**

Note: This is a private Tailscale IP, so `git.fxhnt.ai` will only resolve meaningfully for devices on the Tailscale network.

```bash
scw dns record add fxhnt.ai \
  name=git \
  type=A \
  data=$TS_IP \
  ttl=300
```

Expected: A record created.

**Step 2: Add a CNAME for the apex if desired**

```bash
# Optional: point the root domain somewhere useful
scw dns record add fxhnt.ai \
  name="" \
  type=A \
  data=$TS_IP \
  ttl=300
```

**Step 3: Update Gitea config for git.fxhnt.ai domain**

```bash
ssh root@vm-fxhnt-git "sed -i 's/DOMAIN.*/DOMAIN           = git.fxhnt.ai/' /etc/gitea/app.ini && \
  sed -i 's|ROOT_URL.*|ROOT_URL         = http://git.fxhnt.ai:3000/|' /etc/gitea/app.ini && \
  sed -i 's/SSH_DOMAIN.*/SSH_DOMAIN       = git.fxhnt.ai/' /etc/gitea/app.ini && \
  systemctl restart gitea"
```

Expected: Gitea now responds to `git.fxhnt.ai:3000`.

**Step 4: Verify DNS propagation**

```bash
dig @ns0.dom.scw.cloud git.fxhnt.ai A +short
```

Expected: Returns the Tailscale IP.

---

### Task 8: Remove Old Azure VM from Tailscale

**Step 1: Check old node status**

```bash
tailscale status | grep vm-fxhnt-git
```

Expected: Should show the new Scaleway node. If the old Azure node is still listed, it needs removal.

**Step 2: Remove old node (if still present)**

This must be done from the Tailscale admin console (https://login.tailscale.com/admin/machines) or via `tailscale` CLI with admin privileges. The old Azure node should be removed to avoid hostname conflicts.

**Step 3: Delete the temporary security group**

```bash
scw instance security-group delete $SG_ID zone=nl-ams-1
```

Expected: Security group removed.

---

### Task 9: Final Verification

**Step 1: Verify all access patterns work**

```bash
# Web UI
curl -s -o /dev/null -w "%{http_code}" http://vm-fxhnt-git:3000/
# or
curl -s -o /dev/null -w "%{http_code}" http://git.fxhnt.ai:3000/

# Git clone via HTTP
git clone http://vm-fxhnt-git:3000/foxhunt/foxhunt.git /tmp/foxhunt-test

# Git clone via SSH (port 2222)
git clone ssh://git@vm-fxhnt-git:2222/foxhunt/foxhunt.git /tmp/foxhunt-test-ssh

# Cleanup
rm -rf /tmp/foxhunt-test /tmp/foxhunt-test-ssh
```

Expected: All three access methods work.

**Step 2: Verify no public access**

```bash
# Should fail - no public IP
scw instance server get $INSTANCE_ID zone=nl-ams-1 | grep -i public
```

Expected: No public IP associated.

**Step 3: Commit the cloud-init config**

```bash
cd /home/jgrusewski/Work/foxhunt
git add docs/infra/cloud-init-git-server.yaml docs/plans/2026-02-22-dev-git-server-design.md docs/plans/2026-02-22-dev-git-server-implementation.md
git commit -m "infra: add dev git server cloud-init and design docs"
```