“Recovering Names from the Archives: Vision-Language Models for OCR and Named Entity Recognition ”

Abstract: This talk shares preliminary results from work as part of the Puyallup Family Data Reclamation Project, which in collaboration with the Puyallup Tribal Historic Preservation Department, aims to digitize historical boarding school archival records and make them searchable and accessible to those researching their family history. These archival documents contain a mixture of faded typewriting, cursive handwriting, tables, forms, and handwritten notes, presenting significant challenges for traditional optical character recognition (OCR) techniques. As part of the eScience AI Accelerator, I worked with Curtis Atkisson to test the feasibility of using the vision-language capabilities of large language models to both extract text from these documents and perform named entity recognition (NER) to identify the individuals within the corpus. I will present early results comparing two OCR approaches, using olmOCR and Gemma 4, paired with different NER techniques, including traditional NLP models like Flair and LLM-based extraction with Gemma 4. By evaluating these outputs against human-annotated ground truths, I offer initial insights into the potential of these models to quickly and cost-effectively extract information from large and varied archival collections.

Speaker Bio: Todd Nobles is a PhD student in Sociology at the University of Washington and a CSDE Data Science and Demography T32 trainee. His dissertation examines how environmental hazards shape residential mobility and migration as well as the social and political implications of these shifting populations. In other research, he works with digitizing historical administrative data to study the life-course consequences of internment on German and Austrian Americans during World War I.