Scraper
Examples
Description
A component that goes through web pages or documents to identify and extract specific information from their content. Its function is to obtain data from unstructured or semi-structured sources through techniques such as pattern matching or parsing, and to hand it over to the system for storage or later processing. Unlike components that consume formal APIs, the scraper accesses the content of the pages or documents directly, searching for, identifying, and extracting the relevant data according to predefined criteria.
Mandatory capabilities
Mandatory capabilities
- Goes through web pages or documents, searching for and extracting specific data through patterns or selectors
- Parses the content to identify the information of interest and separate it from the rest
- Hands the extracted data over to the system for storage or later processing
- Handles unstructured or semi-structured content, without relying on a formal schema exposed by the source
Additional capabilities
Additional capabilities
- Working together with a crawler: receives URLs discovered dynamically by a component that navigates by following links, instead of operating on a predefined list of sources
- Manual trigger: lets an authorized user start the extraction on demand
- Periodic scheduling: runs the scraping at defined intervals
- Extraction logging: records runs, success rates per source, and the data actually extracted
- Tolerance to structural changes: implements multiple selectors or fallback strategies that mitigate the effect of changes to the source pages
Boundaries
What it is not
- Is not a crawler: it does not navigate by following links or discover content on its own; it requires specific URLs or locations as input
- Is not an ETL: the ETL consumes structured data from APIs or databases; the scraper extracts data from unstructured content
- Does not transform or process the extracted data in depth; it only identifies it and hands it over
- It may stop working if the structure of the sources changes and there is no tolerance to those changes
Information flow
Input
URLs or locations of the sources to process; selectors, patterns, or criteria that define which information to extract
→
Output
Extracted and structured data, handed over to the system for storage or later processing
Variants
- Web scraping: extraction of data from HTML pages through CSS selectors, XPath, or other DOM parsing mechanisms
- Document scraping: extraction of data from files with unstructured or semi-structured content (PDF, Word, spreadsheets without a standard format)
Typical dependencies
- External sources to process (web pages, documents accessible by URL)
- Selectors or rules that define what to extract from each source
- Database or another component that receives the extracted data
- A scheduling mechanism or scheduler for periodic runs, when that capability is present
Mapping to the Digital Board
Manual UpdateNo
Automatic UpdateAlways
Data ConceptsAlways
On-Demand OutputsNo
Automatic OutputsNo
Autonomous ProcessesAlways
| Section | Relationship | Notes |
|---|---|---|
| Manual Update | No | The manual trigger only starts the extraction; the data enters the system automatically, not through direct writing by the operator. |
| Automatic Update | Always | The incorporation into the system of data extracted from external sources. |
| Data Concepts | Always | The entities resulting from the extraction (products, offers, profiles, depending on the domain). |
| On-Demand Outputs | No | — |
| Automatic Outputs | No | — |
| Autonomous Processes | Always | Parsing the content to identify and separate the relevant information. Logging and tolerance to changes, when those capabilities are present. |
Digital Board example
Manual Update
—
Data Concepts
Job Posting
Company
On-Demand Outputs
—
Automatic Update
Scheduled Extraction from Company Websites
Autonomous Processes
HTML Parser
Extraction Logs
Automatic Outputs
—