Qian Yu

Databricks Financial Service customers in the GenAI space have a common use case of ingestion and processing of unstructured documents — PDF/images — then performing downstream GenAI tasks such as entity extraction and RAG based knowledge Q&A. The pain points for the customers for these types of use cases are: - The quality of the PDF/image documents varies since many older physical documents were scanned into electronic form - The complexity of the PDF/image documents varies and many contain tables — images with embedding information — which require slower Tesseract OCR - They would like to streamline postprocess for downstream workloads In this talk we will present an optimized structured streaming workflow for complex PDF ingestion. The key techniques include Apache Spark™ optimization, multi-threading, PDF object extraction, skew handling and auto retry logics