Document AI =========== The ``docai`` app uploads a local PDF to Cloud Storage, runs Google Document AI, and writes compact JSONL records. Use a Google Document AI **Form Parser** processor for the default workflow. It can return OCR text, form key-value pairs, and tables. ``pdf-jsonl-smith`` reads that rich processor response, but writes only compact records for analysis. Install the GCP extra before running the app: .. code-block:: bash uv pip install -e ".[dev,gcp]" Beginner Setup -------------- There are four Google Cloud pieces involved in a ``docai`` run: ``project`` The Google Cloud container for billing, APIs, IAM permissions, buckets, processors, and later BigQuery datasets. For development, a dedicated project such as ``pdf-jsonl-smith-dev`` keeps costs and cleanup easier to understand. ``processor`` The Document AI processor that reads the PDF. Use **Form Parser** for the default workflow because it can produce OCR text, form key-value pairs, and tables. The command needs the processor ID, for example ``1e1efb128b392292``. ``bucket`` The Cloud Storage bucket where the local PDF is uploaded before Document AI reads it. A bucket is a named container for objects such as PDFs and JSONL files. In ``gs://pdf-jsonl-smith-dev/input/001058380.pdf``, the bucket is ``pdf-jsonl-smith-dev`` and the object path is ``input/001058380.pdf``. ``auth`` The credentials used by the Google client libraries. ``pdf-jsonl-smith`` uses Application Default Credentials, so the code does not need an API key. Recommended development layout: .. code-block:: text Project: pdf-jsonl-smith-dev Processor: pdf-jsonl-smith-form-parser-dev Processor ID: 1e1efb128b392292 Bucket: gs://pdf-jsonl-smith-dev Input prefix: gs://pdf-jsonl-smith-dev/input/ Output prefix: gs://pdf-jsonl-smith-dev/output/ Why Form Parser? ~~~~~~~~~~~~~~~~ Document AI has multiple processor types. Enterprise Document OCR is useful for plain text extraction. Form Parser is a better default for this app because it can extract text, key-value pairs, and tables from the same PDF. ``pdf-jsonl-smith`` still keeps the final output small. It reads the rich Form Parser response, then writes only compact records. Bucket Basics ~~~~~~~~~~~~~ The ``docai`` command needs a ``--staging-uri`` because the app uploads the local PDF before calling Document AI: .. code-block:: bash --staging-uri gs://pdf-jsonl-smith-dev/input/ If the URI ends in ``/``, it is treated as a prefix and the PDF filename is appended: .. code-block:: text sample_pdfs/001058380.pdf -> gs://pdf-jsonl-smith-dev/input/001058380.pdf You can also pass a full object path: .. code-block:: bash --staging-uri gs://pdf-jsonl-smith-dev/input/my-upload-name.pdf Use ``--jsonl-gcs-uri`` only when you also want the compact JSONL output copied back to Cloud Storage. Auth Basics ~~~~~~~~~~~ For local development, the simplest option is user Application Default Credentials: .. code-block:: bash gcloud auth application-default login gcloud config set project pdf-jsonl-smith-dev For a repeatable runner, create a service account and grant it only the roles needed for this workflow. A good service account description is: .. code-block:: text Runs the pdf-jsonl-smith development pipeline: uploads PDFs to Cloud Storage, calls the Document AI Form Parser, writes compact JSONL outputs, and later loads/query results in BigQuery. Practical roles for development: .. code-block:: text Document AI API User Storage Object Admin on the development bucket BigQuery Job User later, when using the bq app BigQuery Data Editor on the target dataset later Avoid API keys for this workflow. The app uploads files, calls Document AI, and will later use BigQuery; service-account or user ADC auth is a better fit. If you use a downloaded service account key locally, point ADC at it: .. code-block:: bash export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json Treat service account keys as sensitive secrets. For production automation, prefer keyless auth where possible. Run --- .. code-block:: bash pdf-jsonl-smith docai sample_pdfs/001058380.pdf \ --output 001058380.docai.jsonl \ --project pdf-jsonl-smith-dev \ --location us \ --processor-id 1e1efb128b392292 \ --staging-uri gs://pdf-jsonl-smith-dev/input/ Use your own project, processor ID, region, and bucket. The ``--staging-uri`` value can be a full object URI or a prefix ending in ``/``. By default, ``docai`` emits these compact unit types: .. code-block:: text text key_value table_cell list_item Use ``--include`` to narrow what is written: .. code-block:: bash pdf-jsonl-smith docai input.pdf \ --output compact.jsonl \ --project my-project \ --location us \ --processor-id PROCESSOR_ID \ --staging-uri gs://bucket/input/ \ --include text,key-values Use ``--jsonl-gcs-uri`` to upload the compact JSONL output after writing it locally: .. code-block:: bash pdf-jsonl-smith docai input.pdf \ --output compact.jsonl \ --project my-project \ --location us \ --processor-id PROCESSOR_ID \ --staging-uri gs://bucket/input/ \ --jsonl-gcs-uri gs://bucket/output/ Inspect Output With jq ---------------------- Count records by unit type: .. code-block:: bash jq -r '.unit_type' 001058380.docai.jsonl | sort | uniq -c Show only main text records: .. code-block:: bash jq -r 'select(.unit_type=="text") | .text' 001058380.docai.jsonl Show form key-value pairs: .. code-block:: bash jq -r 'select(.unit_type=="key_value") | [.key, .value, .confidence] | @tsv' \ 001058380.docai.jsonl Show table cells with coordinates: .. code-block:: bash jq -r 'select(.unit_type=="table_cell") | [.page_number, .table_index, .row_index, .cell_index, .text] | @tsv' \ 001058380.docai.jsonl Show list items: .. code-block:: bash jq -r 'select(.unit_type=="list_item") | [.page_number, .row_index, .text] | @tsv' \ 001058380.docai.jsonl Why Unit Type Matters --------------------- The output is one unified JSONL stream. This is convenient for BigQuery, but it means ``.text`` contains different kinds of compact records: * main document text * synthesized key-value text * table-cell text * list-item text Filter by ``unit_type`` when you want one kind of extracted content. Policy ------ ``docai`` extracts from rich Document AI payloads, but emits only compact, purpose-built records. The final JSONL does not mirror token boxes, word-level layout, full entity graphs, or other verbose processor internals.