SOC 2 as Code: Automating Evidence Collection for a SaaS Startup

In an earlier series I built a fake power grid and automated its NERC CIP evidence (GRC-as-Code: Automating CIP-010 Baselines and Evidence). That’s a niche world. This post takes the exact same idea and points it at the world most companies actually live in: a SaaS startup trying to pass SOC 2.

The problem is the same one every startup hits. A big customer says they won’t sign until they see a SOC 2 report, so suddenly compliance is a revenue blocker, and the usual way to deal with it is a consultant, a spreadsheet, and a scramble. I wanted to see how much of that I could turn into code: a small pipeline that logs into the real systems, collects the evidence, maps it to the SOC 2 controls, and tells me which controls are actually passing. Basically a stripped-down version of what Vanta or Drata do.

I’m not assuming you know any of this. I’ll explain SOC 2 and every tool as it comes up, starting with those two: Vanta and Drata are the two big compliance-automation platforms startups pay for. They connect to your AWS, GitHub, Okta, and so on via API, continuously pull evidence, map it to the controls a framework like SOC 2 requires, and flag what’s failing, instead of someone manually screenshotting settings pages every quarter. That’s the whole category this project is imitating on a much smaller scale.

The honest caveat, up front. I’m not an auditor and this is not a real SOC 2 report. A real report is signed by a licensed CPA firm after they test your controls over months. What I’m building is the evidence-collection layer underneath that — the part that’s actually engineering, and the part a startup spends most of its time on.

What is SOC 2?

SOC 2 is a report a CPA firm produces that says, in effect, “we checked this company’s security controls and here’s how they held up.” It’s the document B2B SaaS companies hand to customers who ask “is it safe to put our data in your product?”

A few things worth knowing before the build:

• Type I vs Type II. Type I says your controls are designed correctly at a single point in time. Type II says they actually operated correctly over a period, usually 3 to 12 months. Customers want Type II. Type I is just the on-ramp.
• Trust Services Criteria (TSC). SOC 2 isn’t a fixed checklist like PCI. It’s built on five criteria: Security (required), Availability, Confidentiality, Processing Integrity, and Privacy. Almost every startup starts with Security only.
• The Common Criteria (CC-series). The Security criterion is broken into numbered controls, CC1 through CC9. The ones that map cleanly to a cloud stack, and the ones I’ll automate, are:

Control area	What it’s really asking	Where the evidence lives
CC6 — Logical access	MFA, least privilege, access reviews, offboarding	AWS IAM, Okta, GitHub
CC7 — System operations	Logging, monitoring, vuln management	CloudTrail, GuardDuty, Dependabot
CC8 — Change management	Code review, no direct pushes to prod	GitHub branch protection, CI
CC9 — Risk / vendors	Vendor list, risk assessment	(mostly policy — noted, not automated here)

The manual version of SOC 2 is someone taking screenshots of AWS console pages the week before the audit. The automated version is a pipeline that collects that same evidence on a schedule, so it’s always current and nobody has to remember to do it.

The scenario

I’m modeling a small B2B SaaS startup going for its first SOC 2 Type II, Security criterion only — the realistic first step. The company runs on a stack you’ll recognize:

• AWS for the application and data.
• GitHub for source code and CI/CD.
• Okta as the identity provider (SSO and MFA).

You can build all of this on free/developer tiers: AWS free tier, a personal GitHub org, and an Okta developer account.

flowchart LR
    subgraph sources["Where the evidence lives"]
        aws["AWS<br/>(IAM, CloudTrail, S3, GuardDuty)"]
        gh["GitHub<br/>(2FA, branch protection, Dependabot)"]
        okta["Okta<br/>(MFA policy, offboarding)"]
    end
    subgraph pipeline["soc2-as-code pipeline"]
        collect["collect.py<br/>(API calls → evidence JSON)"]
        map["map to CC controls"]
        gap["gap check<br/>(pass / fail per control)"]
        evidence["evidence pack<br/>(dated + hashed HTML)"]
    end
    aws --> collect
    gh --> collect
    okta --> collect
    collect --> map --> gap --> evidence

Building the collectors

The whole pipeline is read-only. Each collector logs into one system with a scoped, read-only credential, pulls the facts that matter, and writes them to a structured JSON file. Nothing here changes anything — it just looks and records.

AWS (CC6, CC7)

Using boto3, I check the things a SOC 2 auditor asks about first: is MFA on every IAM user, is the root account locked down, is CloudTrail on, are S3 buckets private and encrypted, is GuardDuty watching.

# collect_aws.py — read-only evidence from IAM, CloudTrail, S3, GuardDuty
import boto3

iam = boto3.client("iam")

def users_without_mfa():
    findings = []
    for user in iam.list_users()["Users"]:
        name = user["UserName"]
        mfa = iam.list_mfa_devices(UserName=name)["MFADevices"]
        if not mfa:
            findings.append(name)          # CC6.1 — every human login needs MFA
    return findings

# ...same idea for: root MFA + no root access keys, password policy,
#    CloudTrail enabled + multi-region, S3 public-access-block, GuardDuty enabled

First real run: AWS and GitHub collect cleanly, Okta skips itself because I hadn’t wired up its API token yet — a missing collector degrades gracefully instead of crashing the whole run.

GitHub (CC8, CC6)

The change-management story lives in GitHub. An auditor wants to see that code can’t reach production without review. So the collector checks org-wide 2FA, and branch protection on the default branch: required reviews, and no force-pushes.

# collect_github.py — org 2FA + branch protection via the GitHub REST API
import requests

H = {"Authorization": f"token {TOKEN}", "Accept": "application/vnd.github+json"}

def branch_protection(owner, repo, branch="main"):
    r = requests.get(
        f"https://api.github.com/repos/{owner}/{repo}/branches/{branch}/protection",
        headers=H,
    )
    if r.status_code == 404:
        return {"protected": False}        # CC8.1 — unprotected default branch is a finding
    p = r.json()
    return {
        "protected": True,
        "required_reviews": p.get("required_pull_request_reviews", {}).get(
            "required_approving_review_count", 0),
        "force_pushes_allowed": p["allow_force_pushes"]["enabled"],
    }

The classic branch-protection screen — GitHub’s newer Rulesets UI looks similar but the collector reads the classic API, so this is the one that actually has to be set for CC8.1 to pass.

Okta (CC6)

Identity is where most of CC6 lives. The Okta collector checks that MFA is actually enforced by policy (not just available), and that deactivated users really lost access — the offboarding control auditors love to test.

# collect_okta.py — MFA policy + recently deactivated users via the Okta API
# GET /api/v1/policies?type=MFA_ENROLL   -> is enrollment required?
# GET /api/v1/users?filter=status eq "DEPROVISIONED"  -> proof offboarding happened

Okta’s “Any two factors” auth policy with the catch-all rule enabled — this is what the collector is actually checking when it says MFA enrollment is required.

Mapping evidence to controls

A pile of API responses isn’t evidence yet. The next stage maps each fact to a specific CC control and marks it pass or fail. This is the part that turns “here’s some AWS data” into “here’s the state of CC6.1.”

# map.py — turn raw collector output into a control matrix
CONTROLS = {
    "CC6.1": lambda e: len(e["aws"]["users_without_mfa"]) == 0
                       and e["okta"]["mfa_enrolled_required"]
                       and e["github"]["org_2fa_required"],
    "CC6.7": lambda e: e["aws"]["s3_all_encrypted"] and e["aws"]["s3_public_blocked"],
    "CC7.2": lambda e: e["aws"]["cloudtrail_multiregion"],
    "CC8.1": lambda e: all(r["required_reviews"] >= 1 for r in e["github"]["repos"]),
    # ...
}

This is a genuine early run against the real account, not a staged one — 5/7 passing. CC6.1 was failing because an IAM user didn’t have MFA yet, and CC7.2 because CloudTrail wasn’t multi-region yet. I fixed both and kept going.

Finding a gap

I was going to plant a few fake failures here to prove the pipeline catches things, but I didn’t need to. While I was working on the lab I opened SSH (port 22) to 0.0.0.0/0 on the default security group, ran the pipeline again out of habit, and CC6.6 immediately flipped red:

The pipeline catches it the moment it happens, not the week before an audit.

And here’s the rule itself in the AWS console, so it’s clear this wasn’t staged output:

The actual change in the console — same security group, same rule the pipeline just flagged.

This is the same move as the CIP-010 post: the value isn’t the checklist, it’s that a machine finds the drift the day it happens instead of the week before the audit. I closed the rule back up right after.

The evidence pack

The last stage turns a run into a dated, self-contained evidence pack — an HTML file per control with the evidence, the source system, a timestamp, who ran it, and a SHA-256 so it can’t be quietly edited later. It’s the SOC 2 equivalent of the RSAW-style output from the CIP-010 post.

CC6.1’s evidence pack: the raw JSON, who ran it, when, and a SHA-256 of the source data so it can’t be quietly edited after the fact.

Making it continuous

A one-time run is a screenshot with extra steps. The point of SOC 2 Type II is that controls hold over time, so I run the collect-map-check on a schedule (GitHub Actions on a cron) and fail the job if any control regresses. That’s the difference between “we passed the audit” and “we stay passing it.”

The same collect → check → evidence pipeline running in GitHub Actions, 7/7 passing — not just on my laptop.

The SOC 2 assessment

Control	Current state	Finding	Risk	Remediation
CC6.1 access + MFA	PASS	Every IAM user, the root account, the Okta org, and the GitHub org all require MFA/2FA.	Low	N/A — enforce MFA for all IAM users and root, require org 2FA in GitHub, set an Okta MFA enrollment policy to required.
CC6.3 offboarding	PASS	No root access keys exist; Okta shows zero stale deprovisioned accounts.	Low	N/A — delete root access keys, ensure terminated users are deprovisioned in the IdP.
CC6.6 network	PASS (fixed)	A security group had SSH open to 0.0.0.0/0 — the pipeline caught it live (see above). Rule removed, control now passes.	Low	Remove the 0.0.0.0/0 rule on port 22; restrict inbound SSH to a specific IP or a bastion.
CC6.7 data at rest	PASS	No public or unencrypted S3 buckets; Block Public Access is on account-wide.	Low	N/A — keep S3 Block Public Access and default encryption enabled on all buckets.
CC7.1 threat detection	PASS	GuardDuty is enabled.	Low	N/A — keep GuardDuty enabled in every active region.
CC7.2 logging	PASS	CloudTrail is multi-region.	Low	N/A — keep a multi-region CloudTrail trail with log file validation enabled.
CC8.1 change mgmt	PASS	The tracked repo requires at least one PR review and force-pushes are disabled on main.	Low	N/A — keep PR reviews required and force-pushes disabled on default branches.

Pros and cons

Pros	Cons
Evidence is always current, not a pre-audit scramble	Read-only API access still has to be set up and scoped carefully
Same pattern scales across every system a startup adds	Policy/people controls (CC1–CC5, CC9) can’t be automated this way
A real auditor trusts a timestamped, hashed pack over a screenshot	You still need the actual audit and the actual policies behind it

Wrap-up

This is the same idea as the CIP-010 post, just pointed at a different stack. Grid or SaaS, the question is the same: do you actually know your control state right now, or only the day before an audit? A script that logs in and checks isn’t a replacement for the auditor, but it’s the difference between finding a gap yourself and having one found for you.

I’m not pretending this makes me a SOC 2 expert. What I can say is I understand, hands-on, what Vanta and Drata are automating under the hood, and I can build the same kind of pipeline for a different framework if a team needed it.

References

• AICPA — SOC 2 / Trust Services Criteria
• boto3 (AWS SDK for Python) · GitHub REST API · Okta API
• GRC-as-Code: Automating CIP-010 Baselines and Evidence — the same idea, OT edition