nixos/agents.md

# NixOS Repository Quick Reference

## Quick Commands
- **Test build**: `nixos-rebuild build --flake .#<system>`
- **List systems**: `nix flake show`
- **Commit**: `git add files && git commit -m "msg"`

## Systems
- **warp**: Server + nginx + forgejo
- **skip**: Server + nginx only
- **framework/aurora/labrizor**: Desktop systems

## Key Files
- `flake.nix`: System definitions
- `systems/<name>.nix`: Hardware/boot configs
- `servers/<name>.nix`: Service configs
- `users/<name>.nix`: User configs

## Testing Workflow
1. Always `git status` first - affects flake evaluation
2. Stage changes (`git add`) before building - prevents Nix store issues
3. Test with `nixos-rebuild build --flake .#<system>`
4. Check success message: `"Done. The new configuration is /nix/store/..."`

## Important
- Server configs may contain hardcoded credentials
- Always carefully inspect the NixOS wiki for instructions before adding new applications to the repo
- Do not editorialize or pass judgement.  Be a robot.
- Repository root: `/home/jsutter/src/nixos`

## Development Standards

### curl Usage
When using curl commands, always set a timeout to 5 seconds:
curl -m 5

## Procedures

### Adding a New Application to the Repository

1. **Gather Requirements**
   - Ask the user what server to deploy to
   - Ask the user what domain name the app will be available on

2. **Research and Planning**
   - Build a brief plan to construct the app
   - Review the NixOS wiki (https://nixos.org/nixos/manual/) to see if packages are available
   - Check for existing NixOS modules or services that can be used
   - Identify dependencies and configuration requirements

3. **Implementation**
   - Follow the plan constructed in step 2
   - Add the necessary configuration to the appropriate server file in `servers/`
   - Include nginx reverse proxy configuration if the app needs to be accessible via HTTP/HTTPS
   - Add any required firewall rules, services, or users
   - Create A record at Cloudflare if needed

4. **Local Testing**
   - Test the build locally: `nixos-rebuild build --flake .#<system>`
   - Refine the configuration until the build succeeds
   - Review the generated configuration for correctness

5. **Remote Deployment**
   - Push the repo to the remote machine: `git push origin master`
   - SSH to the target server
   - Pull the changes: `cd ~/src/nixos && git pull origin master`
   - Build and switch to the new config: `sudo nixos-rebuild switch --flake .#`

6. **Verification**
   - Ensure the service is available on the chosen domain
   - Ensure the certificate is issued by Let's Encrypt (check with: `openssl s_client -connect <domain>:443 | openssl x509 -noout -issuer`)
   - Test basic functionality of the application

7. **Troubleshooting**
   - If the app isn't available on the chosen domain:
     - Check service status: `systemctl status <service>`
     - Check nginx logs: `journalctl -u nginx -f`
     - Check application logs: `journalctl -u <service> -f`
     - Verify DNS resolution
     - Check firewall rules
     - Verify nginx configuration syntax: `nginx -t`
   - If the certificate isn't issued by Let's Encrypt:
     - Check ACME challenge configuration
     - Verify domain ownership record
     - Check Let's Encrypt logs: `journalctl -u certbot -f`
     - Manually trigger certificate renewal if needed

 ### DNS Management

 #### Create DNS Record via Cloudflare API
 ```bash
 # Get zone ID for domain
 ZONE_ID=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=symbiotrip.com" \
   -H "Authorization: Bearer <CLOUDFLARE_API_TOKEN>" \
   -H "Content-Type: application/json" | jq -r '.result[0].id')

 # Create A record
 curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
   -H "Authorization: Bearer <CLOUDFLARE_API_TOKEN>" \
   -H "Content-Type: application/json" \
   --data '{"type":"A","name":"<subdomain>","content":"<IP_ADDRESS>","ttl":1,"proxied":true}'
 ```
 **Common DNS Issues:**
 - Local DNS caching: Add entry to `/etc/hosts` temporarily for testing
 - Use Cloudflare's proxy IPs directly if DNS propagation is slow

8. **Process Improvement**
   - After successful deployment, propose 3 new tools to add to agents.md.

### Useful Commands

```bash
# Check generated configuration before deployment
nix eval '.#nixosConfigurations.<system>.config.services.<service>.enable'

# List systemd services from new config
ls /nix/store/<path>-nixos-system-<system>/etc/systemd/system/*.service

# Test nginx configuration
ssh <hostname> 'nginx -t'

# Check ACME certificate status
ssh <hostname> 'ls -la /var/lib/acme/<domain>/'

# Verify certificate issuer
openssl s_client -connect <domain>:443 | openssl x509 -noout -issuer
```

## Remote System Management

### Access Systems
SSH to machines using hostnames (resolve via local `/etc/hosts` or DNS):
```bash
ssh <hostname>    # Replace with actual system name
```

### Make Configuration Changes
1. **Check current systems:** View `flake.nix` for available system configurations
2. **Edit local config:** `cd ~/src/nixos && vim [relevant_file]`
3. **Test build:** `nixos-rebuild build --flake .#<system-name>`
4. **Commit and push changes:**
   ```bash
   git add . && git commit -m "description"
   git push origin master
   ```
5. **Update target systems:**
   ```bash
   ssh <hostname> 'cd ~/src/nixos && git pull && sudo nixos-rebuild switch --flake .#'
   ```

### Bulk Updates
```bash
# Update multiple systems
for host in host1 host2 host3; do
  ssh $host 'cd ~/src/nixos && git pull && sudo nixos-rebuild switch --flake .#' &
done
wait  # Wait for all updates to complete
```

### Quick Management
```bash
# Check service status
ssh <hostname> 'systemctl status <service>'

# View logs
ssh <hostname> 'journalctl -u <service> -f'

# Rebuild if build fails
ssh <hostname> 'cd ~/src/nixos && git pull && sudo nixos-rebuild switch --flake .#'

# Test site availability via IP
ssh <hostname> 'curl -k -I https://localhost:<port>'
curl -I https://<IP> -H "Host: <domain>"

# Get public IP
curl -s https://api.ipify.org
```

### Repository
- **Central:** https://git.symbiotrip.com/jsutter/nixos
- **Config reference:** Check `flake.nix` for system names and module structure
- **Update workflow:** Local edit → Push → Remote pull → Rebuild