Add DNS management and deployment procedures to agents.md

- Add curl timeout requirement (5 seconds)
- Add comprehensive 8-step workflow for new application deployment
- Add troubleshooting procedures for domain availability and SSL certs
- Add infrastructure roadmap for Borg backup server and sops-nix
- Update README with session start protocol and infrastructure tasks

README.md

@@ -1,7 +1,7 @@
 # NixOS Configuration Repository
 
 ## Session Start Protocol
-Always begin by reading agents.md for workflow instructions.
+Always begin by reading agents.md for workflow instructions and development standards.
 
 ## System Installation
 
@@ -37,3 +37,18 @@ nixos-install --flake .#<name> --no-root-password --impure
 nixos-enter --root '/mnt'
 passwd jsutter
 ```
+
+## Infrastructure Roadmap
+
+### Planned Work
+
+#### Borg Backup Server
+- Set up a dedicated Borg backup server for automated backups
+- Configure backup schedules for critical systems
+- Implement retention policies and pruning rules
+
+#### Secrets Management with sops-nix
+- Implement sops-nix for secrets management
+- Move all hardcoded secrets from server configs into sops-nix
+- Set up encryption keys and key rotation policies
+- Document the secrets management workflow

agents.md

@@ -23,11 +23,109 @@
 4. Check success message: `"Done. The new configuration is /nix/store/..."`
 
 ## Important
-- Server configs may contain hardcoded credentials - use agenix or systemd credentials for production
-- **Always carefully inspect the NixOS wiki for instructions before adding new applications to the repo**
-- Both warp and skip build successfully
+- Server configs may contain hardcoded credentials
+- Always carefully inspect the NixOS wiki for instructions before adding new applications to the repo
+- Do not editorialize or pass judgement.  Be a robot.
 - Repository root: `/home/jsutter/src/nixos`
 
+## Development Standards
+
+### curl Usage
+When using curl commands, always set a timeout to 5 seconds:
+curl -m 5
+
+## Procedures
+
+### Adding a New Application to the Repository
+
+1. **Gather Requirements**
+   - Ask the user what server to deploy to
+   - Ask the user what domain name the app will be available on
+
+2. **Research and Planning**
+   - Build a brief plan to construct the app
+   - Review the NixOS wiki (https://nixos.org/nixos/manual/) to see if packages are available
+   - Check for existing NixOS modules or services that can be used
+   - Identify dependencies and configuration requirements
+
+3. **Implementation**
+   - Follow the plan constructed in step 2
+   - Add the necessary configuration to the appropriate server file in `servers/`
+   - Include nginx reverse proxy configuration if the app needs to be accessible via HTTP/HTTPS
+   - Add any required firewall rules, services, or users
+   - Create A record at Cloudflare if needed
+
+4. **Local Testing**
+   - Test the build locally: `nixos-rebuild build --flake .#<system>`
+   - Refine the configuration until the build succeeds
+   - Review the generated configuration for correctness
+
+5. **Remote Deployment**
+   - Push the repo to the remote machine: `git push origin master`
+   - SSH to the target server
+   - Pull the changes: `cd ~/src/nixos && git pull origin master`
+   - Build and switch to the new config: `sudo nixos-rebuild switch --flake .#`
+
+6. **Verification**
+   - Ensure the service is available on the chosen domain
+   - Ensure the certificate is issued by Let's Encrypt (check with: `openssl s_client -connect <domain>:443 | openssl x509 -noout -issuer`)
+   - Test basic functionality of the application
+
+7. **Troubleshooting**
+   - If the app isn't available on the chosen domain:
+     - Check service status: `systemctl status <service>`
+     - Check nginx logs: `journalctl -u nginx -f`
+     - Check application logs: `journalctl -u <service> -f`
+     - Verify DNS resolution
+     - Check firewall rules
+     - Verify nginx configuration syntax: `nginx -t`
+   - If the certificate isn't issued by Let's Encrypt:
+     - Check ACME challenge configuration
+     - Verify domain ownership record
+     - Check Let's Encrypt logs: `journalctl -u certbot -f`
+     - Manually trigger certificate renewal if needed
+
+ ### DNS Management
+
+ #### Create DNS Record via Cloudflare API
+ ```bash
+ # Get zone ID for domain
+ ZONE_ID=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=symbiotrip.com" \
+   -H "Authorization: Bearer <CLOUDFLARE_API_TOKEN>" \
+   -H "Content-Type: application/json" | jq -r '.result[0].id')
+
+ # Create A record
+ curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
+   -H "Authorization: Bearer <CLOUDFLARE_API_TOKEN>" \
+   -H "Content-Type: application/json" \
+   --data '{"type":"A","name":"<subdomain>","content":"<IP_ADDRESS>","ttl":1,"proxied":true}'
+ ```
+ **Common DNS Issues:**
+ - Local DNS caching: Add entry to `/etc/hosts` temporarily for testing
+ - Use Cloudflare's proxy IPs directly if DNS propagation is slow
+
+8. **Process Improvement**
+   - After successful deployment, propose 3 new tools to add to agents.md.
+
+### Useful Commands
+
+```bash
+# Check generated configuration before deployment
+nix eval '.#nixosConfigurations.<system>.config.services.<service>.enable'
+
+# List systemd services from new config
+ls /nix/store/<path>-nixos-system-<system>/etc/systemd/system/*.service
+
+# Test nginx configuration
+ssh <hostname> 'nginx -t'
+
+# Check ACME certificate status
+ssh <hostname> 'ls -la /var/lib/acme/<domain>/'
+
+# Verify certificate issuer
+openssl s_client -connect <domain>:443 | openssl x509 -noout -issuer
+```
+
 ## Remote System Management
 
 ### Access Systems
@@ -69,6 +167,13 @@ ssh <hostname> 'journalctl -u <service> -f'
 
 # Rebuild if build fails
 ssh <hostname> 'cd ~/src/nixos && git pull && sudo nixos-rebuild switch --flake .#'
+
+# Test site availability via IP
+ssh <hostname> 'curl -k -I https://localhost:<port>'
+curl -I https://<IP> -H "Host: <domain>"
+
+# Get public IP
+curl -s https://api.ipify.org
 ```
 
 ### Repository