Leveraging Wikipedia Data in Power BI: A Step-by-Step Guide to Web Scraping

Web scraping has emerged as a powerful technique for data analysts seeking to harness the vast information available online without relying on costly APIs. Among the richest sources of freely available data is Wikipedia, offering over 60 million articles that can be transformed into valuable insights through tools like Power BI.

Understanding Wikipedia as a Data Source

Wikipedia stands as one of the world’s most visited websites, containing a wealth of information across countless topics. Launched in 2001 by Jimmy Wales and Larry Singer, this collaborative encyclopedia has become a trusted source even for platforms like Google and ChatGPT. While the quality can vary between articles, it remains an invaluable resource for data analysis projects.

Web Scraping Prerequisites

Before diving into web scraping with Power BI, certain prerequisites should be considered:

• A stable internet connection
• Sufficient RAM (minimum 16GB recommended)
• Good knowledge of Power BI, especially Power Query
• Understanding of functions and parameters
• Familiarity with the Power Query editor

Additionally, it’s important to activate the “Web Table Inference” option in Power Query to utilize the older web connector (Web.Page) rather than the newer Web.BrowserContent, as the former provides greater flexibility for data extraction.

Manual Method: Extracting Population Data

For a practical demonstration, we can extract historical population data from Wikipedia pages of French cities. The manual approach involves:

1. Obtaining the URL of the Wikipedia page (e.g., for Lyon, Marseille, or Paris)
2. Using the Web connector in Power BI to access the page
3. Filtering for tables containing population data
4. Transforming the data through steps like column expansion and text cleaning
5. Adding a custom column to identify the city
6. Repeating the process for each city of interest
7. Combining the separate queries into a consolidated dataset

While effective, this manual method becomes tedious when working with multiple cities, requiring repetitive steps for each new location.

Automated Method: Using Parameters and Functions

A more efficient approach leverages Power BI’s parameters and functions:

1. Create a parameter that stores the city name (e.g., “City” with default value “Lyon”)
2. Modify the original query to use this parameter instead of hardcoded city names
3. Convert the parameterized query into a function (e.g., “GetPopulation”)
4. Create a table containing all cities of interest
5. Invoke the function on each city in the table
6. Expand the resulting data for analysis

This automated method allows for scaling the data collection to dozens or even hundreds of cities without additional manual effort.

Beyond Population Data

The same techniques can be applied to extract various other data points from Wikipedia, including:

• Information from the InfoBox (density, area, altitude, coordinates)
• Political information (mayor, prefecture)
• Climate data (temperatures, precipitation)

The possibilities are limited only by your imagination and the data available on the Wikipedia pages.

Benefits Over API Alternatives

While connecting to APIs might provide more structured data, they often come with usage limitations or costs through token systems. Wikipedia’s web scraping approach offers completely free access to regularly updated information without such restrictions.

Conclusion

Web scraping Wikipedia through Power BI provides an accessible and cost-effective method for gathering rich datasets. By mastering the techniques of parameters and functions, analysts can automate data collection across multiple sources, transforming public information into actionable insights without specialized programming knowledge.

Whether you’re tracking population trends, gathering geographic information, or collecting other publicly available data, these methods open up new possibilities for data-driven decision making using the vast knowledge base that Wikipedia provides.