Usage

CLI

Install the GCP extra before using the docai app:

uv pip install -e ".[dev,gcp]"

pdf-jsonl-smith local input.pdf -o text_units.jsonl

Optional metadata:

pdf-jsonl-smith local input.pdf \
  --output text_units.jsonl \
  --source-uri gs://bucket/input.pdf \
  --schema-version 0.1

By default, the CLI prints a short completion summary to stdout:

Wrote 42 records to text_units.jsonl from input.pdf.

Use --quiet to suppress the summary.

The older pdf2jsonl command remains available as a compatibility alias for the local PyMuPDF extractor:

pdf2jsonl input.pdf -o text_units.jsonl

The docai app uploads a local PDF to Cloud Storage, runs Google Document AI, and writes compact JSONL records:

pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/pdf-jsonl-smith/input/

By default, docai emits compact text, key-values, tables, and lists records. Use --include to narrow the output:

pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/pdf-jsonl-smith/input/ \
  --include text,key-values

Use --jsonl-gcs-uri to upload the compact JSONL output after writing it locally.

See Document AI for a friendlier walkthrough and useful jq inspection commands.

The bq app is reserved for the BigQuery workflow:

pdf-jsonl-smith bq

The docai app policy is to extract from rich Google Document AI payloads, but emit only compact, purpose-built records. Full Document AI payloads are not part of the final JSONL output.

Python API

from pdf_jsonl_smith import extract_pdf_to_jsonl

count = extract_pdf_to_jsonl(
    input_path="input.pdf",
    output_path="text_units.jsonl",
    source_uri="gs://bucket/input.pdf",
)

The function returns the number of JSONL records written.

Output Fields

Each line is one standalone JSON object with flat fields:

schema_version STRING
doc_id STRING
source_uri STRING
filename STRING
page_number INT64
unit_type STRING
unit_index INT64
text STRING
bbox_x0 FLOAT64
bbox_y0 FLOAT64
bbox_x1 FLOAT64
bbox_y1 FLOAT64
char_count INT64
word_count INT64
extractor STRING
extracted_at TIMESTAMP
warning STRING