Leveraging ChatGPT for Structured Data Extraction

Project Overview#

I developed a production-ready data extraction system that leverages Large Language Models to mine and structure event data from diverse web sources. You can see the implementation in action at bremap.de (German), which aggregates events across Bremen.

Event Aggregation Platform

Technical Architecture#

The project implements a sophisticated data pipeline that combines web crawling, natural language processing, and machine learning to solve the challenge of unstructured data extraction at scale.

Data Collection Infrastructure#

OpenStreetMap Integration#

The system’s data collection begins with programmatic integration with OpenStreetMap’s API, leveraging its semantic tagging system to identify and categorize potential event venues. This provides a structured dataset of locations with associated metadata including:

  • Venue classification (theater, museum, etc.)
  • Geographic coordinates
  • Website URLs
  • Additional venue-specific attributes

Web Crawling Architecture#

I implemented a distributed web crawling system using Scrapy, chosen for its:

  • Built-in concurrency handling
  • Intelligent request throttling
  • Duplicate request detection
  • Robust error handling
  • Extensible middleware system

Machine Learning Approaches Evaluated#

1. Transformer-Based Structure Extraction#

I conducted extensive research into WebFormer’s approach of leveraging HTML structure for information extraction. The architecture’s innovative use of HTML markup as positional encoding showed promise for maintaining structural context during extraction. However, implementation challenges arose due to:

  • Absence of pre-trained models
  • Limited training data availability
  • Complex architecture requirements

2. Fine-Tuned BERT Models#

The investigation into BERT-based approaches revealed both potential and limitations:

Advantages:

  • Strong performance on natural language understanding
  • Ability to capture contextual relationships
  • Extensive pre-training on web text

Challenges:

  • Limited training data for domain-specific fine-tuning
  • Handling dispersed information across documents
  • Resource-intensive training requirements
  • Complex sequence alignment for multiple target fields

3. Traditional NLP Pipeline#

Initial exploration of classical NLP approaches revealed several insights:

WebFormer Architecture WebFormer’s architectural approach to structured information extraction

[Rest of the article continues…]

© 2025