Document AI

The docai app uploads a local PDF to Cloud Storage, runs Google Document AI, and writes compact JSONL records.

Use a Google Document AI Form Parser processor for the default workflow. It can return OCR text, form key-value pairs, and tables. pdf-jsonl-smith reads that rich processor response, but writes only compact records for analysis.

Install the GCP extra before running the app:

uv pip install -e ".[dev,gcp]"

Beginner Setup

There are four Google Cloud pieces involved in a docai run:

project

The Google Cloud container for billing, APIs, IAM permissions, buckets, processors, and later BigQuery datasets. For development, a dedicated project such as pdf-jsonl-smith-dev keeps costs and cleanup easier to understand.

processor

The Document AI processor that reads the PDF. Use Form Parser for the default workflow because it can produce OCR text, form key-value pairs, and tables. The command needs the processor ID, for example 1e1efb128b392292.

bucket

The Cloud Storage bucket where the local PDF is uploaded before Document AI reads it. A bucket is a named container for objects such as PDFs and JSONL files. In gs://pdf-jsonl-smith-dev/input/001058380.pdf, the bucket is pdf-jsonl-smith-dev and the object path is input/001058380.pdf.

auth

The credentials used by the Google client libraries. pdf-jsonl-smith uses Application Default Credentials, so the code does not need an API key.

Recommended development layout:

Project:      pdf-jsonl-smith-dev
Processor:    pdf-jsonl-smith-form-parser-dev
Processor ID: 1e1efb128b392292
Bucket:       gs://pdf-jsonl-smith-dev
Input prefix: gs://pdf-jsonl-smith-dev/input/
Output prefix: gs://pdf-jsonl-smith-dev/output/

Why Form Parser?

Document AI has multiple processor types. Enterprise Document OCR is useful for plain text extraction. Form Parser is a better default for this app because it can extract text, key-value pairs, and tables from the same PDF.

pdf-jsonl-smith still keeps the final output small. It reads the rich Form Parser response, then writes only compact records.

Bucket Basics

The docai command needs a --staging-uri because the app uploads the local PDF before calling Document AI:

--staging-uri gs://pdf-jsonl-smith-dev/input/

If the URI ends in /, it is treated as a prefix and the PDF filename is appended:

sample_pdfs/001058380.pdf
-> gs://pdf-jsonl-smith-dev/input/001058380.pdf

You can also pass a full object path:

--staging-uri gs://pdf-jsonl-smith-dev/input/my-upload-name.pdf

Use --jsonl-gcs-uri only when you also want the compact JSONL output copied back to Cloud Storage.

Auth Basics

For local development, the simplest option is user Application Default Credentials:

gcloud auth application-default login
gcloud config set project pdf-jsonl-smith-dev

For a repeatable runner, create a service account and grant it only the roles needed for this workflow. A good service account description is:

Runs the pdf-jsonl-smith development pipeline: uploads PDFs to Cloud Storage,
calls the Document AI Form Parser, writes compact JSONL outputs, and later
loads/query results in BigQuery.

Practical roles for development:

Document AI API User
Storage Object Admin on the development bucket
BigQuery Job User later, when using the bq app
BigQuery Data Editor on the target dataset later

Avoid API keys for this workflow. The app uploads files, calls Document AI, and will later use BigQuery; service-account or user ADC auth is a better fit.

If you use a downloaded service account key locally, point ADC at it:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json

Treat service account keys as sensitive secrets. For production automation, prefer keyless auth where possible.

Run

pdf-jsonl-smith docai sample_pdfs/001058380.pdf \
  --output 001058380.docai.jsonl \
  --project pdf-jsonl-smith-dev \
  --location us \
  --processor-id 1e1efb128b392292 \
  --staging-uri gs://pdf-jsonl-smith-dev/input/

Use your own project, processor ID, region, and bucket. The --staging-uri value can be a full object URI or a prefix ending in /.

By default, docai emits these compact unit types:

text
key_value
table_cell
list_item

Use --include to narrow what is written:

pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/input/ \
  --include text,key-values

Use --jsonl-gcs-uri to upload the compact JSONL output after writing it locally:

pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/input/ \
  --jsonl-gcs-uri gs://bucket/output/

Inspect Output With jq

Count records by unit type:

jq -r '.unit_type' 001058380.docai.jsonl | sort | uniq -c

Show only main text records:

jq -r 'select(.unit_type=="text") | .text' 001058380.docai.jsonl

Show form key-value pairs:

jq -r 'select(.unit_type=="key_value") | [.key, .value, .confidence] | @tsv' \
  001058380.docai.jsonl

Show table cells with coordinates:

jq -r 'select(.unit_type=="table_cell") |
  [.page_number, .table_index, .row_index, .cell_index, .text] | @tsv' \
  001058380.docai.jsonl

Show list items:

jq -r 'select(.unit_type=="list_item") |
  [.page_number, .row_index, .text] | @tsv' \
  001058380.docai.jsonl

Why Unit Type Matters

The output is one unified JSONL stream. This is convenient for BigQuery, but it means .text contains different kinds of compact records:

  • main document text

  • synthesized key-value text

  • table-cell text

  • list-item text

Filter by unit_type when you want one kind of extracted content.

Policy

docai extracts from rich Document AI payloads, but emits only compact, purpose-built records. The final JSONL does not mirror token boxes, word-level layout, full entity graphs, or other verbose processor internals.