The access log that kept every secret

Goal#

Fix an access log that records passwords, session cookies, and security tokens, turning a debugging tool into a credentials store that lives for months and is readable by many.

The idea#

What access logs record

Access logs record every HTTP request: the method, path, headers, query parameters, and sometimes the body. They are used for debugging, compliance (PCI-DSS, HIPAA, SOC 2), analytics, and security monitoring. They answer questions like "Who accessed this endpoint?", "Why did this request fail?", and "Is this user agent a bot?".

A typical access log line in JSON looks like this:

{"method": "POST", "path": "/login", "status": 200, "ms": 45.2}

That line is safe. It says a login succeeded in 45 milliseconds. An unsafe line looks like this:

{"method": "POST", "path": "/login", "query": {}, "headers": {"cookie": "sid=T98z...", "authorization": "Bearer lab_token_..."}, "body": {"email": "ada@example.com", "password": "correct horse battery staple"}, "status": 200, "ms": 45.2}

That line contains the password, the session cookie, and a bearer token. Anyone with access to the logs has access to the credentials.

Why secrets end up in logs

To debug a login problem, someone made the access log write each request in full: the URL with its query string, every header, and the body. The problem was fixed. The logging stayed. Now the log, which is shipped to a logging service, kept for a year, and readable by most of the company, holds every password, session cookie, and password-reset token that passed through.

Once a secret is in the log, it lives as long as the log does. Logs are backed up, replicated, indexed, and shipped to log aggregation systems (Splunk, Datadog, Elasticsearch, CloudWatch Logs). They are stored in S3 or GCS with retention policies measured in months or years. They are accessible to SREs, support engineers, security teams, and anyone who compromises those systems.

How secrets appear in logs

The fix is to sanitize the logs before they are written. Log what explains a request, never what proves identity.

What to log and what to skip

Always log:

• Method and path (but redact secrets in the path, like /reset/{token}).
• Query parameter names only, not values. "query": ["token", "redirect"] is safe. "query": {"token": "abc123"} is not.
• Status code and response time.
• User-Agent header (helps with bot detection and debugging).
• A request ID or trace ID for correlating logs.

Never log:

• Request bodies. If you must log them, use an allowlist: only log fields you explicitly list as safe, and exclude password, secret, api_key, credit_card, ssn, and any other sensitive field.
• The Cookie header. It contains session ids.
• The Authorization header. It contains bearer tokens, basic auth credentials, and API keys.
• The Set-Cookie header in responses. It contains the session id being set.
• Query parameter values. Log the names only.

For debugging, correlation is more useful than content. If you know the request ID, status, and timing, you can find the relevant application logs and reconstruct what happened. You do not need the password or session cookie in the access log.

What people get wrong

Logging everything by default. Many logging libraries serialize the request object to JSON by default, which includes headers and body. The fix is to build the log entry explicitly, field by field, rather than dumping the request object.

Using denylists instead of allowlists. A denylist (password, token, secret) will miss new secret fields (api_key, reset_token, ssn). An allowlist (only log method, path, status, ms, user_agent) is safer. If a field is not on the allowlist, it does not appear in the log.

Redacting after writing. Once the secret is on disk, it is too late. The file has been written, backed up, and shipped to the log aggregator. You cannot safely redact a file that has already left the system. Sanitize before you write.

How production systems handle log sanitization

Framework and library support. Django has LOGGING with custom formatters. Flask and Express have middleware that runs before logging. Rails has config.filter_parameters to redact fields before they reach the logger. Use these hooks to sanitize requests before they are logged.

nginx and Apache. nginx log_format and Apache LogFormat let you choose which fields to log. Exclude $http_cookie, $http_authorization, and query string values. Log only the fields you need.

Structured logging. Use structured logging (JSON) rather than plain text. It is easier to parse, search, and sanitize. But do not serialize the entire request object. Build the log entry explicitly with safe fields only.

Secrets in URLs. Password reset tokens, email confirmation tokens, and API keys sometimes appear in URLs (/reset?token=abc123). The better fix is to not put them in URLs at all. Use POST bodies for tokens. If you must use URLs (for example, a link sent by email), make the tokens single-use and expire them within minutes. Log the path without the query string, or log query parameter names only.

Log retention. Even sanitized logs should not live forever. Set retention policies: 30 days for debug logs, 90 days for access logs, one year for compliance logs. Delete old logs automatically. The shorter the retention, the smaller the window for leaked secrets.

Log access controls. Logs are semi-public. Anyone with access to the log storage (S3 bucket, CloudWatch Logs, Datadog), the log aggregation system, or the backup system can read them. That includes SREs, support engineers, security teams, contractors, and anyone who compromises those systems. Treat logs as if an attacker will read them. Do not log anything you would not want an attacker to have.

The case#

Run the case to see the logged secrets:

uv run python -m wire run 8

case 08: The access log that kept every secret
access log lines: 4
  {"method": "POST", "path": "/login", "query": {}, "headers": {"host": "127.0.0.1:61537", "accept-encoding": "i...
  the password        appears in 1 log line
  the session cookie  appears in 3 log lines
  the reset token     appears in 1 log line

The log records 4 requests. The password appears in the body of the POST to /login. The session cookie appears in the Cookie header of 3 requests. The reset token appears in the query string of GET /reset.

Reasoning towards the fix#

Open lab/cases/case08_logs.py. The log_line function builds one line of the access log as JSON. As written, it logs the method, path, query (the full dict), headers (the full dict), body (the full dict or list), status, and milliseconds.

The handle function implements four endpoints:

• POST /login checks credentials and sets a session cookie.
• GET /me returns the user if the session is valid.
• POST /reset/request generates a password reset token.
• GET /reset?token=... checks if a reset token is valid.

The flow function runs a sign-in, a call with the session, and a password reset, over real HTTP. It returns the access log and the secrets that were used, so they can be looked for in it.

To fix it, sanitize log_line:

def log_line(request: Request, status: int, millis: float) -> str:
    """One line of the access log, as JSON.

    Log what explains the request, never what proves identity: no bodies, no cookies, no
    Authorization header, and only the names of query parameters, since some carry tokens.
    """
    return json.dumps(
        {
            "method": request.method,
            "path": request.path,
            "query": sorted(request.query),
            "user_agent": request.headers.get("user-agent", ""),
            "status": status,
            "ms": round(millis, 1),
        }
    )

This logs:

• The method and path.
• The names of query parameters (sorted), not the values. A line with "query": ["token"] tells you a token was present, but not what it was.
• The User-Agent header, which helps with debugging.
• The status and response time.

It does not log:

• The request body, which contains the password.
• The headers dict, which contains the Cookie and Authorization headers.
• The query parameter values, which contain the reset token.

The checks parse the log and verify that the password, session cookie, and reset token do not appear anywhere in any log line.

Check your fix#

Once you have made the changes, check them:

uv run python -m wire check 8

As shipped, the check output is:

  FAIL  no secret appears in the log (password, session cookie, reset token)
  PASS  every request is still logged (4 of 4 lines)
  PASS  each line still says the method, the path and the status
not yet. Try: uv run python -m wire hint 8

When fixed, all lines show PASS. The password, session cookie, and reset token are not in the log, but every request is still logged with enough information to explain what happened.

Try it on your own machine#

Check your own logs. Search your access logs or application logs for patterns like:

grep -i cookie access.log
grep -i authorization access.log
grep -i password debug.log
grep -E 'token=[^&]+' access.log

If you find any, you are logging secrets.

In production#

Sanitize logs before you write them, not after. Once the secret is on disk, it is too late. You cannot safely redact a file that has already been written, backed up, and shipped to a log aggregator.

What to strip:

• Headers: Cookie, Authorization, X-API-Key, X-Auth-Token, and any other header that carries credentials.
• Body fields: do not log POST bodies at all unless you strip every sensitive field first. Even then, use an allowlist: only log the fields you need.
• Query parameter values: log the names only, not the values. "query": ["token", "redirect"] is safe. "query": {"token": "abc123"} is not.
• Set-Cookie headers: they contain session ids and other secrets. Do not log them.

Most logging libraries and frameworks support sanitization hooks. Use them:

• nginx: use $request_uri instead of $request to avoid logging the query string, or write a custom log format that omits sensitive fields.
• Apache: use LogFormat to exclude headers and query parameters.
• Rails: configure filter_parameters to redact fields before logging.
• Django: use LOGGING config to customize the log formatter and strip sensitive fields.
• Express: write middleware that sanitizes req.headers and req.body before passing the request to the logger.

For API keys and tokens in URLs, the better fix is to not put them in URLs at all. Use Authorization headers for API keys, and POST bodies for tokens. If you must use tokens in URLs (for example, a password reset link sent by email), make them single-use and expire them within minutes.

Treat logs as semi-public. Anyone with access to the log storage, the log aggregation system, or the backup system has access to the logs. That includes SREs, support engineers, security teams, and anyone who compromises those systems. Do not log anything you would not want an attacker to have.

Key points#

• Access logs often contain passwords, session cookies, and tokens. Once logged, they live as long as the log does.
• Sanitize logs before you write them. Log what explains a request, never what proves identity.
• Do not log request bodies, the Cookie header, the Authorization header, or query parameter values.
• Log query parameter names only, not values. Log the User-Agent, method, path, status, and response time.
• In production, treat logs as semi-public. Do not log anything you would not want an attacker to have.

Production checklist#

You have reached the end of the workshop. Here is a checklist for putting these eight cases into production:

1. DNS validation. If you run your own DNS resolver, validate sender, query id, and name before caching.
2. HTTP framing. Reject requests with duplicate Content-Length headers or both Content-Length and Transfer-Encoding.
3. TLS verification. Use ssl.create_default_context() with cafile for private CAs. Never disable verification.
4. Session cookies. Set HttpOnly, Secure, SameSite, and Max-Age on every session cookie. Use the __Host- prefix.
5. Logout. Invalidate the session on the server, not just in the browser.
6. Session fixation. Issue a new session id on login, privilege escalation, and password change.
7. Security headers. Set Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options, Referrer-Policy, and Cache-Control on every response.
8. Log sanitization. Strip bodies, Cookie, Authorization, and query parameter values from logs before you write them.
9. HTTPS everywhere. Use HTTPS in production. Let's Encrypt is free. No exceptions.
10. Defense in depth. Each case is one control. Combine them. Miss one, and an attacker has a foothold. Get them all right, and the cost of compromise rises sharply.

These are not the only controls. They are the ones that system-to-system communication requires, and the ones that teams still get wrong. Build them into your defaults, your frameworks, and your code review checklist. Make the safe thing the easy thing.