Systemd on Yarang's Tech Lair

Building a Multi-LLM Distributed Orchestrator with NATS JetStream

Fri, 08 May 2026 21:57:11 +0900

Part 1 discussed the model-specific limitations discovered while running four AIs—Claude, ZAI, Codex, and Gemini—concurrently on the same tasks. This part is about “how we made it possible”—the system design and implementation story.

System Overview

AgentForge consists of three components.

[Task Publisher]
 │ NATS JetStream publish
 ▼
[NATS Broker] ─── af.worker.{id}.inbox
 │ JetStream consume (independent streams per worker)
 ▼
[Worker Pollers] × N (poller.py × 18 instances)
 │ LLM CLI Execution (claude / codex / gemini)
 ▼
[Result Return] af.task.{task_id}.completed

When a publisher posts a task to NATS, each worker, which is independently subscribed, receives the message on its inbox and executes the LLM CLI. The result is then published back to a completion topic.

Why NATS JetStream?

We considered several message broker options: Redis Streams, Kafka, RabbitMQ, and NATS JetStream.

Reasons for choosing NATS JetStream:

Single Binary — Operates with a single nats-server without requiring separate runtimes. It has no dependencies like Kafka’s ZooKeeper or RabbitMQ’s Erlang/OTP.
Built-in Persistence — JetStream is a streaming layer on top of NATS, storing messages to the filesystem. This ensures that unprocessed tasks are not lost even if a worker restarts.
NKey-based Authentication — We can issue independent Ed25519 key pairs for each worker. If one worker is compromised, the credentials of other workers remain valid.
Lightweight — Memory usage is around 30MB on a single server. Even with 18 workers connected, the broker load is minimal.

The Core: Backend Adapter in `poller.py`

The heart of the worker is poller.py. This single file handles NATS subscriptions, LLM CLI execution, and result returns.

Since LLMs have different execution methods, we separated them into a backend adapter dictionary.

_BACKENDS: dict[str, dict] = {
 "claude": {
 "bin": os.environ.get("CLAUDE_BIN", "/usr/local/bin/claude"),
 "tools": os.environ.get("ALLOWED_TOOLS", "Read,Edit,Write,Glob,Grep"),
 "model": os.environ.get("CLAUDE_MODEL", ""),
 },
 "codex": {
 "bin": os.environ.get("CODEX_BIN", "/usr/bin/codex"),
 "model": os.environ.get("CODEX_MODEL", ""),
 "sandbox": os.environ.get("CODEX_SANDBOX", "read-only"),
 },
 "gemini_cli": {
 "bin": os.environ.get("GEMINI_BIN", "/usr/bin/gemini"),
 "model": os.environ.get("GEMINI_MODEL", ""),
 },
}

The MODEL_BACKEND environment variable determines which LLM to use. This allows the same poller.py code to run different LLMs across 18 workers.

Claude Backend

async def run_claude(instructions: str, task_id: str) -> tuple[int, str]:
 cfg = _BACKENDS["claude"]
 cmd = [cfg["bin"], "--print", "--allowedTools", cfg["tools"]]
 if cfg.get("model"):
 cmd += ["--model", cfg["model"]]
 proc = await asyncio.create_subprocess_exec(*cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)

The --print flag is key. It runs Claude Code in non-interactive mode instead of conversational mode, ensuring the results are returned via stdout.

ZAI Backend

ZAI offers an Anthropic API-compatible endpoint, so it doesn’t require a separate backend. Routing is handled by two environment variables.

# /etc/agentforge/cc-zai-high-dev-01.env
ANTHROPIC_BASE_URL=<ZAI endpoint>
ANTHROPIC_AUTH_TOKEN=<ZAI API key>

By injecting this file using systemd’s EnvironmentFile= directive, the claude binary sends requests to the ZAI endpoint. This allows us to connect to a different LLM provider simply by changing environment variables, without altering the code.

Declarative Management: `fleet.yaml` × `servers.yaml`

Manually managing 18 workers is impractical. We declaratively defined the entire infrastructure using two YAML files.

`servers.yaml` — Server Inventory

servers:
 - name: worker-node-1
 role: worker-host
 services: [agentforge-worker, tunnel-arm1]

 - name: broker-host
 role: broker-host
 services: [nats-jetstream, postgres]

 - name: worker-node-2
 role: worker-host
 services: [agentforge-worker, tunnel-arm1]

`fleet.yaml` — Worker Placement

workers:
 - worker_id: cc-go-dev-01
 llm: claude-code
 model: claude-sonnet-4-6
 lang: go
 role: developer
 host: worker-node-1
 enabled: true
 create_pr: true

 - worker_id: codex-py-dev-01
 llm: codex
 model: gpt-5.5
 lang: python
 role: developer
 host: worker-node-1
 enabled: true
 create_pr: false

Changing just the host field moves a worker to a different server. Setting enabled: false stops the deployment script from starting that worker.

Worker Templating System: `provision_worker.py`

Manually writing systemd unit files for each new worker is prone to errors. We automated this using Jinja2 templates and a provisioning script.

Template Structure

templates/
 systemd/
 claude.service.j2 # For claude-code and ZAI alike
 codex.service.j2 # OpenAI Codex
 gemini.service.j2 # Google Gemini CLI

The core part of claude.service.j2:

Environment=MODEL_BACKEND=claude
Environment=CLAUDE_BIN={{ claude_bin }}
{% if claude_model %}
Environment=CLAUDE_MODEL={{ claude_model }}
{% endif %}
{% if env_file %}
EnvironmentFile={{ env_file }}
{% endif %}
Environment=WORK_BASE={{ work_base }}
Environment=WORK_DIR={{ work_base }}/repo
Environment="{{ 'ALLOWED_TOOLS=' + allowed_tools }}"
Environment=CREATE_PR={{ 'true' if create_pr else 'false' }}
{% if create_pr and github_remote %}
Environment=GITHUB_REMOTE={{ github_remote }}
{% endif %}

For ZAI workers, the env_file block is activated, adding the EnvironmentFile. For PR creation workers, github_remote is injected. Other settings use defaults.

`provision_worker.py` Usage

# Preview (no actual deployment)
python3 scripts/provision_worker.py --worker new-worker-id --dry-run

# Actual deployment (including NATS creds issuance)
python3 scripts/provision_worker.py --worker new-worker-id --issue-creds

# Bulk deployment for the entire fleet.yaml
python3 scripts/provision_worker.py --all

Internal operations:

Reads worker entries from fleet.yaml.
Reads target hosts from servers.yaml.
Renders Jinja2 templates.
Deploys /etc/systemd/system/{worker_id}-poller.service via SSH.
Creates the working directory.
Executes systemctl daemon-reload && enable --now.
(Optional) Issues NATS NKey with nsc add user → deploys creds → regenerates auth.conf.

Distributed Hosting: Adding Workers to a Second Server

Running all workers on a single server creates a single point of failure. We added Claude workers to a second host.

The method for workers on the second host to connect to the NATS broker is via an autossh tunnel.

[Unit]
Description=NATS Broker Tunnel
After=network-online.target

[Service]
ExecStart=/usr/bin/autossh -N \
 -L 4222:127.0.0.1:4222 \
 -i /home/ubuntu/.ssh/id_ed25519 \
 broker-host
Restart=always
RestartSec=10

With this configuration active, workers always connect to nats://127.0.0.1:4222. They don’t need to know the broker host’s address. As long as the tunnel is alive, it works the same way from any host.

NATS Credential Operations Experience

NATS NKey management was the most complex part of the implementation.

NATS JetStream’s authentication structure is hierarchical.

Operator (Root Signing Authority)
 └── Account: SYS (System Account)
 └── Account: Services (Worker Account)
 ├── User: cc-dev-01
 ├── User: cc-go-dev-01
 ├── User: codex-py-dev-01
 └── ...

Each worker has an independent User NKey and can publish/subscribe within the permissions scope (af.>, _INBOX.>, $JS.>) of the Services account.

Adding a new worker requires the Operator’s signing key. We initially made the mistake of not backing up this key, leading to its loss. Consequently, we had to regenerate the entire Operator and replace all worker credentials en masse. The service downtime was approximately 60 seconds.

# Regeneration procedure
nsc add operator AgentForge
nsc add account SYS
nsc add account Services
for worker in cc-dev-01 cc-go-dev-01 ...; do
 nsc add user --account Services --name $worker \
 --allow-pub "af.>,_INBOX.>,$JS.>" \
 --allow-sub "af.>,_INBOX.>,$JS.>"
done
nsc generate config --mem-resolver --sys-account SYS > auth.new.conf

Adding a New Worker: The Full Procedure

Since the completion of this system, adding a new worker is straightforward.

Step 1: Add an entry to fleet.yaml

- worker_id: my-new-worker
 llm: claude-code
 model: claude-haiku-4-5
 lang: multi
 role: developer
 host: worker-node-1
 enabled: true
 create_pr: false

Step 2: Preview

python3 scripts/provision_worker.py --worker my-new-worker --dry-run

Step 3: Actual Deployment

python3 scripts/provision_worker.py --worker my-new-worker --issue-creds

That’s it. Template rendering, SSH deployment, NATS credential issuance, and service registration are all handled by a single command.

Next Steps

The current system is structured such that workers process tasks independently. Future plans include:

Routing Policies: Automatically selecting the appropriate worker based on task characteristics (e.g., Go code → claude-go-dev, cost-first → ZAI lightweight tier).
Results Comparison Dashboard: A UI to display fan-out results side-by-side.
Cost Tracking: Aggregating API call costs per worker.

The code is publicly available on GitHub.

Building a Blog AI Auto-Comment System (3/3): Deployment and Troubleshooting

Sun, 03 May 2026 01:20:00 +0900


## Overview

In [Part 1](/ko/post/ai-auto-comment-system-part1-architecture/), we covered the architecture and implementation, and in [Part 2](/ko/post/ai-auto-comment-system-part2-security/), we looked at security enhancements. In this 3rd part, we record the process of deploying to an actual OCI ARM server and the troubleshooting encountered.

In particular, we share in detail the actual debugging process where we tracked and resolved the issue of **GITHUB_TOKEN not loading** over 4 steps. The core of this article is how we narrowed down the cause in a situation where "it's set up, so why isn't it working?"

---

## Infrastructure Configuration

### Server Configuration

| Server | Role | Specs |
|------|------|------|
| ec1 (x86) | Web Server (nginx, Hugo blog) | OCI |
| arm1 (ARM) | Worker Server (Flask, Claude Code) | OCI ARM |

The blog is built and served with Hugo on ec1, while the AI comment worker runs on arm1. The GitHub Webhook is delivered directly to arm1.

### Worker Server Directory Structure

/var/www/auto-comment-worker/ # Application ├── scripts/ │ └── auto-comment-worker.py ├── deploy/ │ └── auto-comment-worker.service ├── venv/ # Python virtual environment └── logs/

/etc/auto-comment-worker/ # Credentials ├── github-token # 640, ubuntu:ubuntu └── credentials/ └── webhook-secret # 600, ubuntu:ubuntu

/home/ubuntu/.local/bin/claude # Claude Code CLI


---

## systemd Service Configuration

### Service File

```ini
[Unit]
Description=Auto Comment Worker for Blog
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/var/www/auto-comment-worker
Environment=PORT=8081
Environment=CLAUDE_CODE_PATH=/home/ubuntu/.local/bin/claude
Environment=BLOG_OWNERS=yarang
Environment=GITHUB_TOKEN_FILE=/etc/auto-comment-worker/github-token
Environment=GITHUB_WEBHOOK_SECRET_FILE=/etc/auto-comment-worker/credentials/webhook-secret
ExecStart=/var/www/auto-comment-worker/venv/bin/python /var/www/auto-comment-worker/scripts/auto-comment-worker.py
Restart=always
RestartSec=10

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=auto-comment-worker

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=false
ReadWritePaths=/var/www/auto-comment-worker /var/log/auto-comment-worker
ReadOnlyPaths=

# Resource Limits
MemoryMax=512M
CPUQuota=50%
TasksMax=100

[Install]
WantedBy=multi-user.target

Key Configuration Explanation

Type=simple: Since the Flask worker runs in the foreground, simple is appropriate. forking is used for processes that daemonize.

User=ubuntu: Although a dedicated service account could be created, it runs as ubuntu because the Claude Code CLI depends on the ubuntu user’s home directory configuration.

ProtectHome=false: Usually set to true, but allows home directory access because Claude Code requires the ~/.agent_forge_for_zai.json configuration file.

ReadOnlyPaths= (Empty value): Initially specified /etc/auto-comment-worker, but left empty due to conflict with ProtectSystem=strict.

Service Management Commands

# Copy service file
sudo cp deploy/auto-comment-worker.service /etc/systemd/system/

# Register and start service
sudo systemctl daemon-reload
sudo systemctl enable auto-comment-worker
sudo systemctl start auto-comment-worker

# Check status
sudo systemctl status auto-comment-worker

# Check logs (real-time)
sudo journalctl -u auto-comment-worker -f

# Check recent logs
sudo journalctl -u auto-comment-worker --since "10 minutes ago"

nginx Reverse Proxy

Webhook Endpoint Configuration

server {
 listen 443 ssl;
 server_name your-domain.com;

 # SSL Configuration
 ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;

 # Webhook Proxy
 location /webhook {
 proxy_pass http://localhost:8081;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;

 # GitHub Webhook signature header forwarding (Required!)
 proxy_set_header X-Hub-Signature-256 $http_x_hub_signature_256;

 # Timeout (Waiting for Claude Code response)
 proxy_read_timeout 120s;
 proxy_connect_timeout 10s;
 }

 # Health check
 location /health {
 proxy_pass http://localhost:8081;
 }
}

proxy_read_timeout 120s is set generously because the Claude Code CLI can take up to 60 seconds to generate an AI response. Since the default timeout for GitHub Webhooks is 10 seconds, asynchronous processing could be considered in practice.

Deployment Process

Manual Deployment (After rsync Failure)

Initially, we attempted deployment with rsync, but it failed because the target directory did not exist on the server:

rsync: [Receiver] mkdir "/var/www/auto-comment-worker/scripts" failed:
No such file or directory

As an alternative, we proceeded with scp-based manual deployment:

# 1. Create directory on server
ssh ubuntu@arm1 "sudo mkdir -p /var/www/auto-comment-worker/scripts"
ssh ubuntu@arm1 "sudo chown -R ubuntu:ubuntu /var/www/auto-comment-worker"

# 2. Transfer files
scp scripts/auto-comment-worker.py ubuntu@arm1:/var/www/auto-comment-worker/scripts/
scp deploy/auto-comment-worker.service ubuntu@arm1:/tmp/

# 3. Install service file
ssh ubuntu@arm1 "sudo cp /tmp/auto-comment-worker.service /etc/systemd/system/"

# 4. Set up Python virtual environment
ssh ubuntu@arm1 "cd /var/www/auto-comment-worker && python3 -m venv venv"
ssh ubuntu@arm1 "cd /var/www/auto-comment-worker && venv/bin/pip install flask flask-limiter marshmallow requests"

# 5. Configure authentication files
ssh ubuntu@arm1 "sudo mkdir -p /etc/auto-comment-worker/credentials"
# Token file is created directly on the server (not transferred via scp)

# 6. Start service
ssh ubuntu@arm1 "sudo systemctl daemon-reload && sudo systemctl enable --now auto-comment-worker"

Heredoc Variable Expansion Pitfall

A common mistake when writing installation scripts with heredoc:

# Single quotes: Variables are NOT expanded!
ssh ubuntu@arm1 << 'ENDSSH'
echo $CREDENTIALS_DIR # Prints empty string
ENDSSH

# No quotes: Variables are expanded locally
ssh ubuntu@arm1 << ENDSSH
echo $CREDENTIALS_DIR # Expanded to local variable value
ENDSSH

To avoid this problem, we switched to executing commands individually instead of using a script.

Troubleshooting: GITHUB_TOKEN Loading Failure

This was the issue that consumed the most time while deploying this system. When the comment Webhook arrived, the following error repeated:

INFO:__main__:GITHUB_TOKEN configured: False
INFO:__main__:GitHub API response status: 401
ERROR:__main__:Failed to get Discussion GraphQL ID

We record the process of tracking down the cause step by step.

Step 1: LoadCredential Path Issue

Initially, we used the LoadCredential directive in systemd:

LoadCredential=github-token:/etc/auto-comment-worker/github-token
Environment=GITHUB_TOKEN_FILE=%d/github-token

%d is a systemd special variable replaced with the credentials directory path. However, this variable was not interpreted as intended, causing the token file path to be set incorrectly.

Solution: Instead of LoadCredential, we specified the absolute path directly.

Environment=GITHUB_TOKEN_FILE=/etc/auto-comment-worker/github-token

Step 2: File Ownership Issue

GITHUB_TOKEN configured: False still appeared. Checking the file revealed:

$ ls -la /etc/auto-comment-worker/github-token
-rw------- 1 root root 93 May 3 01:10 github-token

Since the file owner is root and permissions are 600, the service running as the ubuntu user cannot read this file.

Solution:

sudo chown ubuntu:ubuntu /etc/auto-comment-worker/github-token
sudo chmod 640 /etc/auto-comment-worker/github-token

Step 3: ReadOnlyPaths Conflict

Even after changing ownership, GITHUB_TOKEN configured: False persisted. The cause was the ReadOnlyPaths setting in the systemd service file:

# This setting blocked file reading
ReadOnlyPaths=/etc/auto-comment-worker

ProtectSystem=strict already mounts the entire filesystem read-only. Adding ReadOnlyPaths on top of that can cause mount namespace conflicts in some environments.

Solution: Changed ReadOnlyPaths to an empty value.

ReadOnlyPaths=

Step 4: Python File Permission Validation Code (Root Cause)

Even after resolving all previous 3 steps, the token still did not load. The final cause was the overly strict file permission validation in the Python code:

# File permission 640 → Group read bit (0o040) is set → Denied!
if st.st_mode & (stat.S_IRWXO | stat.S_IRWXG):
 raise PermissionError("Token file must be 600 or 400")

Because we changed to chmod 640 in Step 2, the group read bit was set, triggering this validation. However, the error message did not appear in the logs, delaying discovery — because the PermissionError occurred at the module import time, preventing the service from starting at all.

Solution: As explained in Part 2, we modified it to check only stat.S_IWOTH.

Importance of Debugging Logs

The debugging logs added to track this issue:

logger.info(f"GITHUB_TOKEN configured: {bool(GITHUB_TOKEN)}")
logger.info(f"GitHub API response status: {response.status_code}")
logger.info(f"GitHub API response body: {response.text[:500]}")

Without these logs, it would have taken much longer to identify the cause. Always log token load success/failure and API response status for authentication-related code.

Debugging Flow Summary

[1] LoadCredential %d not interpreted → Changed to absolute path
 ↓ (Still failed)
[2] File owner root:root → Changed to ubuntu:ubuntu
 ↓ (Still failed)
[3] ReadOnlyPaths conflict → Removed
 ↓ (Still failed)
[4] Python permission check S_IRWXG → Relaxed to S_IWOTH
 ↓
 [Resolved!]

Lessons learned from this 4-step debugging:

Change one at a time and verify: If you change multiple settings at once, you won’t know which one is the cause.
Trust logs, but suspect where there are no logs: Exceptions at module load time may not appear in standard logs.
Security validation code can also be a source of bugs: When security code blocks normal operation — balancing security and operations.

Health Check

A health check endpoint to verify the service is running correctly:

@app.route('/health', methods=['GET'])
def health():
 """Health check"""
 return jsonify({'status': 'healthy'})

Monitoring systems periodically call /health to verify the service status:

curl -s http://localhost:8081/health
# {"status": "healthy"}

Future Improvements

Asynchronous Processing: Asynchronize AI response generation using Celery or Redis Queue to respond within the GitHub Webhook timeout (10 seconds).
Retry Logic: Exponential backoff retry on GitHub API call failures.
Monitoring Dashboard: Monitor response time, success rate, and error rate with Prometheus + Grafana.
Automated Deployment: Build an automated deployment pipeline with GitHub Actions on code changes.
Testing: Write integration tests mocking Webhook payloads.

Conclusion

Over three parts, we have recorded the entire build process of the blog AI auto-comment system:

Part 1: Full architecture of giscus → GitHub Webhook → Flask → Claude Code → GraphQL
Part 2: File-based authentication, HMAC verification, input sanitization, systemd security
Part 3: Actual deployment, nginx proxy, 4-step debugging process

The greatest value of this system is that it automates communication with blog readers. While it is difficult for blog operators to respond to every comment immediately, an AI assistant can provide a first response, improving the reader experience.

The full code is available at the GitHub repository.

This article is Part 3 (the final part) of the AgentForge blog auto-comment system series.

Building a Blog AI Auto-Comment System (2/3) — Security Hardening

Sun, 03 May 2026 01:10:00 +0900

Overview

In Part 1, we covered the architecture and implementation of the AI auto-comment system. In this Part 2, we will focus on security aspects.

Systems that receive external Webhooks, manage GitHub API tokens, and process user input require special attention to security. We will explain the process of switching from environment variables to file-based authentication, the reasons behind it, and the design of each security layer.

Security Threat Model

Threats that this system must defend against:

Threat	Attack Vector	Defense
Spoofed Webhook	Attacker sends fake Webhook	HMAC-SHA256 signature verification
Token Leak	Environment variable exposure, log exposure	File-based authentication + permission restrictions
XSS/Injection	Malicious comment content	Input sanitization
Excessive Requests	DDoS, abuse	Flask-Limiter rate limiting
Privilege Escalation	Worker process compromise	systemd security directives
Infinite Loop	AI responding to itself	Marker-based comment detection

File-Based Authentication Management

Problems with Environment Variables

Initially, we managed GitHub tokens and Webhook secrets as environment variables:

# Initial (unsafe)
Environment=GITHUB_TOKEN=ghp_xxxxx
Environment=GITHUB_WEBHOOK_SECRET=my-secret-key

Problems with the environment variable approach:

/proc/PID/environ: Process environment variables are exposed as a file in Linux
Log exposure: Risk of environment variables being logged during debugging
Child process inheritance: When running Claude Code with subprocess.run, all environment variables are inherited
systemd configuration file: If the service file contains plaintext secrets, there is a risk of committing them to git

Switching to File-Based

Store credentials in the file system and specify only file paths in environment variables:

# Improved — only path exposed
Environment=GITHUB_TOKEN_FILE=/etc/auto-comment-worker/github-token
Environment=GITHUB_WEBHOOK_SECRET_FILE=/etc/auto-comment-worker/credentials/webhook-secret

Directory structure of the server:

/etc/auto-comment-worker/
├── github-token # GitHub Personal Access Token (640)
└── credentials/
 └── webhook-secret # GitHub Webhook HMAC Secret (600)

Token File Loading Code

import stat

GITHUB_TOKEN_FILE = os.environ.get('GITHUB_TOKEN_FILE', '')
if GITHUB_TOKEN_FILE and os.path.exists(GITHUB_TOKEN_FILE):
 # Check file permissions
 st = os.stat(GITHUB_TOKEN_FILE)
 if st.st_mode & stat.S_IWOTH:
 raise PermissionError("Token file must not be world-writable")
 with open(GITHUB_TOKEN_FILE, 'r') as f:
 GITHUB_TOKEN = f.read().strip()
else:
 GITHUB_TOKEN = os.environ.get('GITHUB_TOKEN', '')

Core design decisions:

Prefer file if it exists: Environment variables are used only as a fallback
Permission verification: Check permissions before reading the file
strip(): Remove newline characters at the end of the file

File Permission Validation — A Trial and Error Journey

I spent the most time on this part. The initial code was too strict:

# Initial code — too strict
if st.st_mode & (stat.S_IRWXO | stat.S_IRWXG):
 raise PermissionError("Token file must be 600 or 400")

Problem with this code: S_IRWXO | S_IRWXG checks all group permissions (read/write/execute) and all other permissions. That is, if the file permission is 640 (owner read/write, group read), it is rejected.

# Bit mask analysis
S_IRWXG = 0o070 # Group read+write+execute
S_IRWXO = 0o007 # Other read+write+execute

# 640 = 0o640
0o640 & (0o070 | 0o007) = 0o640 & 0o077 = 0o040 # Not 0 → Rejected!

For actual security, what matters is that other users cannot modify the file. Group read permission allows users in the same group to read the file and is not a security issue.

After revision:

# After revision — focus on actual threat
if st.st_mode & stat.S_IWOTH:
 raise PermissionError("Token file must not be world-writable")

You only need to check stat.S_IWOTH (0o002). This confirms only “Does it have write permission for others?”.

Permission	Octal	Initial Code	After Revision
`600`	`0o600`	Allowed	Allowed
`640`	`0o640`	Rejected	Allowed
`644`	`0o644`	Rejected	Allowed
`646`	`0o646`	Rejected	Rejected
`666`	`0o666`	Rejected	Rejected

HMAC-SHA256 Signature Verification

GitHub Webhook sends a signature created by HMAC-SHA256 hashing the request body with the Webhook secret in the X-Hub-Signature-256 header. We verify this to confirm that the request actually came from GitHub.

def verify_webhook_signature(payload: bytes, signature: str) -> bool:
 """GitHub webhook signature verification"""
 if not signature:
 logger.warning("Missing webhook signature")
 return False

 if not WEBHOOK_SECRET:
 logger.warning("WEBHOOK_SECRET not configured - skipping validation")
 return True # Allow for development mode

 try:
 hash_algorithm, github_signature = signature.split('=', 1)
 if hash_algorithm != 'sha256':
 return False

 mac = hmac.new(
 WEBHOOK_SECRET.encode(),
 msg=payload,
 digestmod=hashlib.sha256
 )
 expected_signature = mac.hexdigest()

 # Prevent timing attacks
 if not hmac.compare_digest(expected_signature, github_signature):
 return False

 return True
 except Exception as e:
 logger.error(f"Signature verification error: {e}")
 return False

Key points:

hmac.compare_digest(): Uses constant-time comparison instead of normal == comparison to prevent timing attacks.
raw bytes usage: Uses request.data (original bytes). If parsed with request.json and re-serialized, it may differ from the original.
Development mode: Skips validation if the secret is not set. The secret must be set in production.

nginx Header Forwarding

For signature verification to work properly, nginx must forward the X-Hub-Signature-256 header:

location /webhook {
 proxy_pass http://localhost:8081;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Hub-Signature-256 $http_x_hub_signature_256;
}

Unless X-Hub-Signature-256 is explicitly forwarded, custom headers may not be passed with just the default proxy_pass.

Request Validation (marshmallow Schema)

Validate the structure of the Webhook payload using a marshmallow schema:

class WebhookSchema(Schema):
 action = fields.Str(required=True, validate=validate.Equal('created'))
 comment = fields.Dict(required=True)
 discussion = fields.Dict(required=True)
 repository = fields.Dict(required=True)
 sender = fields.Dict(required=False)

action = 'created' only: Rejects comment edit (edited) or delete (deleted) events.
Required field validation: Returns 400 error if comment, discussion, repository are missing.
ValidationError → Audit log: Invalid requests are logged in the audit log.

Input Sanitization

User comments are external input, so they must be processed:

def sanitize_comment(body: str) -> str:
 """User input sanitization"""
 body = re.sub(r'<[^>]+>', '', body) # Remove HTML tags
 body = html.escape(body) # Escape special characters
 body = body[:1000] # Length limit
 return body

Where this sanitization is applied:

Comment body (comment_body)
Discussion title and body (discussion_title, discussion_body)
Original author name (original_author)
Quoted part when posting AI response

Username Masking

We do not log the full username:

def mask_username(username: str) -> str:
 """Username masking"""
 if not username or len(username) < 4:
 return "***"
 return f"{username[:3]}***"

Logs will only display masked names like yar***. This strikes a balance between privacy protection and debugging convenience.

Rate Limiting

Apply rate limiting per endpoint using Flask-Limiter:

limiter = Limiter(
 app=app,
 key_func=get_remote_address,
 default_limits=["10 per minute"],
 storage_uri="memory://"
)

@app.route('/webhook', methods=['POST'])
@limiter.limit("10 per minute")
def github_webhook():
 ...

10 requests per minute limit: A figure considering normal Webhook call frequency
get_remote_address: Limits based on client IP
memory://: In-memory storage (suitable for single process)

systemd Security Directives

Part 3 will cover systemd deployment in detail, but security-related directives are explained here:

[Service]
# Security hardening
NoNewPrivileges=true # Prevent privilege escalation
PrivateTmp=true # Provide isolated /tmp
ProtectSystem=strict # Read-only file system
ProtectHome=false # Allow home directory access (for Claude Code config)
ReadWritePaths=/var/www/auto-comment-worker /var/log/auto-comment-worker

# Resource limits
MemoryMax=512M # Memory limit
CPUQuota=50% # CPU usage limit
TasksMax=100 # Process count limit

Directive	Effect
`NoNewPrivileges=true`	Privilege escalation via `setuid`, `setgid`, etc. is impossible
`PrivateTmp=true`	Isolated `/tmp` namespace, separated from other processes
`ProtectSystem=strict`	Mounts entire file system as read-only
`ReadWritePaths`	Specifies only paths allowed for writing
`MemoryMax=512M`	Protects the entire system in OOM situations

Conflict between ProtectSystem=strict and ReadOnlyPaths

Initially, adding ReadOnlyPaths=/etc/auto-comment-worker caused an issue where the token file could not be read. Since ProtectSystem=strict already sets the entire file system to read-only, a separate ReadOnlyPaths is unnecessary. It was removed because it could cause conflicts in some environments.

Conclusion

In this Part 2, we covered file-based authentication management, the file permission validation journey, HMAC-SHA256 signature verification, input sanitization, rate limiting, and systemd security directives.

The most important lesson in security: “Too strict validation is as harmful as too loose validation.” The initial code, which only allowed 600 permissions, was secure, but in the actual operating environment, it rejected files with 640 permissions, preventing the service from starting. Focusing only on actual threats (world-writable) is the correct approach.

In the next Part 3, we will cover Deployment and Troubleshooting — systemd service configuration, nginx reverse proxy, and errors actually encountered and the resolution process.

This post is Part 2 of the AgentForge blog automatic comment system series.

Systemd on Yarang's Tech Lair

Building a Multi-LLM Distributed Orchestrator with NATS JetStream

System Overview

Why NATS JetStream?

The Core: Backend Adapter in poller.py

Claude Backend

ZAI Backend

Declarative Management: fleet.yaml × servers.yaml

servers.yaml — Server Inventory

fleet.yaml — Worker Placement

Worker Templating System: provision_worker.py

Template Structure

provision_worker.py Usage

Distributed Hosting: Adding Workers to a Second Server

NATS Credential Operations Experience

Adding a New Worker: The Full Procedure

Next Steps

Building a Blog AI Auto-Comment System (3/3): Deployment and Troubleshooting

Key Configuration Explanation

Service Management Commands

nginx Reverse Proxy

Webhook Endpoint Configuration

Deployment Process

Manual Deployment (After rsync Failure)

Heredoc Variable Expansion Pitfall

Troubleshooting: GITHUB_TOKEN Loading Failure

Step 1: LoadCredential Path Issue

Step 2: File Ownership Issue

Step 3: ReadOnlyPaths Conflict

Step 4: Python File Permission Validation Code (Root Cause)

Importance of Debugging Logs

Debugging Flow Summary

Health Check

Future Improvements

Conclusion

Building a Blog AI Auto-Comment System (2/3) — Security Hardening

Overview

Security Threat Model

File-Based Authentication Management

Problems with Environment Variables

Switching to File-Based

Token File Loading Code

File Permission Validation — A Trial and Error Journey

HMAC-SHA256 Signature Verification

nginx Header Forwarding

Request Validation (marshmallow Schema)

Input Sanitization

Username Masking

Rate Limiting

systemd Security Directives

Conflict between ProtectSystem=strict and ReadOnlyPaths

Conclusion

The Core: Backend Adapter in `poller.py`

Declarative Management: `fleet.yaml` × `servers.yaml`

`servers.yaml` — Server Inventory

`fleet.yaml` — Worker Placement

Worker Templating System: `provision_worker.py`

`provision_worker.py` Usage