AI documentation tools: what to check before you give one your repo

Before an AI documentation tool reads your repository, you need seven answers: what access it holds, where the code is processed, what reaches a third-party model, what is retained, what happens if the vendor is breached, who can see the output, and how it is deleted. Vague on any one means no.

I am building a product in this category, so you should read this with that in mind. The checklist is deliberately vendor-neutral. Zipper Blueprint's intended architecture and its unresolved answers are further down.

What are you actually granting?

Start with the credential, because everything else follows from it.

The common pattern is a GitHub App or an OAuth grant. You click install, pick repositories, and the vendor gets a token. Read the permission list on the install screen, not the marketing page. "Read access to contents" means the vendor's systems can pull every file and, usually, the full history. History matters. That is where the API key somebody committed and deleted in 2021 still lives.

Ask three things. What is the exact scope? Is it per-repository or organisation-wide? And who holds the token, you or them? A token the vendor holds is a credential on someone else's infrastructure, and it is only as safe as their key management.

The other pattern is that the vendor holds nothing. The tool runs in your environment, with an IAM role you created, and you can see and revoke every permission. Same work done, very different blast radius.

Where does the code get processed?

There are three real answers, and "in the cloud" is not one of them.

On the vendor's infrastructure. Your code is cloned to their servers. That makes the vendor a subprocessor of your source code, and it belongs in your vendor risk register next to every other one.
On the vendor's infrastructure, in a dedicated tenant. Better isolation, still their boundary.
Inside your own cloud account. The code is read in place and never copied out.

Option three is often called bring your own cloud, or BYOC. It is not automatically safe. A stack running in your account with an over-broad role is still a problem. But it moves the question from "do we trust this vendor's perimeter" to "does this IAM policy do what it says," and the second one is a question your team can answer by reading.

What goes to a third-party model?

This is the question vendors dodge most, because the answer is usually "a lot."

An AI documentation tool sends code, or summaries derived from it, to a language model. You need to know which model, who operates it, and what the terms are. Does the model provider retain prompts? Do they train on them? Is it the vendor's own account at that provider, or yours?

A good answer names the model provider, states in writing that prompts are not used for training, and says whether the call is made from your account or theirs. If the model call happens inside your own AWS account, the data does not leave your boundary to reach it. Pin down what is sent, too: whole files, chunks or summaries, and what the secret stripping misses. Evasion sounds like "we only use industry-leading providers."

What is retained, and for how long?

Ask about four separate things, because vendors will happily answer the easiest.

The source code itself. Cloned and kept? Kept for how long?
Intermediate artifacts. Embeddings, parse trees and summaries are derived from your code and can leak it. If they are retained, the code is retained in a different shape.
Prompts and responses. Logged for debugging? By the vendor, the model provider, or both?
The output. The documentation itself.

"We don't store your code" is a sentence that deserves a follow-up. Do you store embeddings of it? Do you store the prompts? A vendor who can answer all four with a number of days has thought about it. One who says "we delete data upon request" has a policy for when you ask, not for when you don't.

What does a good answer look like?

Good answers share three traits. They are specific, they are checkable, and they include limits.

Specific means a number or a name. "Logs are retained for 30 days" is an answer. "Logs are retained for a limited time" is a way of not answering.

Checkable means you can verify it without trusting the vendor. A CloudFormation template you can read beats a diagram. An IAM policy you can paste into a policy simulator beats a paragraph. A SOC 2 Type II report beats the sentence "we are SOC 2 compliant," and you should check the audit period and the scope, because a report covering only the corporate IT environment tells you nothing about the product that reads your code.

Includes limits means the vendor tells you what they cannot do. Evasion is always smooth. An honest answer has an edge on it: "We don't have that yet." "This depends on how you configure it." "That is covered by the model provider's terms, here is the link."

Compromise belongs here too. Ask what happens when the vendor is breached, and listen for whether the answer is about their controls or about your exposure. If they hold a token and clones of your repositories, a breach is your incident. If they hold nothing, a breach of the vendor does not hand an attacker your source.

What are the seven questions to ask?

Here is the list, with Zipper Blueprint's honest answer under each. The Engineering product is not generally available, so design commitments are separated from details that have not been decided or proven in production.

1. What access is being granted? The intended delivery model is a CloudFormation stack deployed into an AWS account the customer controls. The exact IAM policy and repository integration are not final. A pilot should not begin until the customer can inspect both.

2. Where is the code processed? The design constraint is that analysis runs inside that customer-controlled account and the code is read in place rather than cloned to Zipper Blueprint. That is the central promise of the Engineering product, but it still needs to be demonstrated by the deployable stack.

3. What reaches a third-party model? Not decided yet. The model provider, invocation path, data sent, secret handling and path exclusions all need written answers before a pilot. "Runs in your account" does not answer this question by itself.

4. What is retained, and for how long? Not decided yet. The target is customer-controlled storage, but source, prompts, logs, intermediate artifacts and generated documents each need an explicit retention and deletion rule before production use.

5. What happens on vendor compromise? There is no production service or audit report to point to today. The intended architecture reduces the source-code exposure of a vendor breach, but the control plane, update path and metadata still need a threat model. If your policy requires a SOC 2 or ISO 27001 report, Zipper Blueprint does not meet it today.

6. Who can see the generated output? The customer should control this through IAM and the chosen documentation destination. The exact default policy is not final. The output includes architecture and trust boundaries, so it should be treated as sensitive as the code.

7. How is it deleted? Not defined yet. A credible pilot needs a teardown procedure that covers the stack, roles, logs, model traces, intermediate artifacts and output. The answer should be testable, not just a sentence in a policy.

Frequently asked

Is bring-your-own-cloud automatically secure? No. It changes who runs the compute, not whether the permissions are sensible. A stack in your account with a wildcard IAM role is a bad deal wherever it runs. The advantage is that the policy is something you can read and test yourself.

Does a SOC 2 report settle the question? It settles part of it. Check the audit period and the scope, and confirm it covers the system that touches your code. It also does not tell you what the product does with your repository, which is why the other six questions still apply.

Should we ever give a vendor a clone of the repository? Sometimes, with a narrow scope and a contract that names retention and deletion. For a monorepo that holds the company, the answer is usually to push the analysis to the code instead. The same reasoning applies in technical due diligence, where the target will not hand over a clone at all.