Web Scraping for Finance

As companies become more data-driven, an increasing number of use cases are set up where the need to step beyond internal data sources – including publicly available open data, data provided by data brokers, and data stemming from API’s – becomes more apparent. However, the “holy grail” of rich data lies within the wider World Wide Web, as the Internet contains a treasure trove of unstructured data that can be extracted and included in subsequent analysis and decision making processes.

As such, web scraping – the extraction, conversion and management of data from the web – is undergoing a revival. It can be used to extract and scrutinize companies’ financial statements allowing for a more comprehensive assessment of their risk and performance metrics; to scrape massive amounts of data from news to rebalance investment portfolios; to perform competitor analysis; etc.

This intensive, hands-on course covers the definitions and history of web scraping, as well as exploring its current applications and examples from the financial domain. Practical, real-life workshops using Python programming language are applied and results are compared with available commercial products. The links between web scraping and other data science techniques (e.g. web crawling, artificial intelligence, text mining and robotic process automation) are also examined.

The programme concludes by addressing managerial and legal aspects, concerns and challenges that need to be taken into account when setting up web scraping initiatives.

• Data and analytics heads and project managers
• IT managers and developers
• Data scientists, analysts and architects
• Quant analysts and financial engineers
• Database administrators
• Information Security staff
• Compliance officers
• Consultants

• Understand what web scraping entails and the use cases it enables (with a focus on finance)
• Understand the building blocks of the web: HTTP, HTML, and CSS
• Learn how Python packages (e.g. “Requests”, “BeautifulSoup”) are used to extract data from the web
• Explore the tools to assist with PDF and other forms of content scraping
• Learn how to scrape more complex, modern websites with technologies like Selenium
• Gain insights into open-source and closed-source offerings available for web scraping
• Examine the links between web scraping and web crawling, AI, text mining, robotic process automation
• Learn about the key managerial and legal aspects and concerns associated with web scraping projects

• Basic computer usage and understanding of the web (e.g. HTML and HTTP)
• Prior programming knowledge (e.g. Python) preferred

Introduction

• What is web scraping: definitions, history, and when to use it
• Which use cases does web scraping enable?
• Practical examples in finance and beyond

Web Scraping with Python

• Why Python?
• The three core pillars: HTTP, HTML, and CSS
• A deep dive in HTTP
• HTTP in Python: using Requests

Workshop: Using Requests in Practice

• Parsing HTML with BeautifulSoup

Workshop: Scraping SEC Current Events

• Cascading Style Sheets (CSS)

Workshop: Scraping Londonfs.com

• HTTP POST Requests: working with forms
• HTTP headers and cookies

Workshop: Scraping Mortgage Rates

• Dealing with JavaScript
• Using Selenium
• Other libraries

Other Tools and Cross-overs

• Commercial products
• Scraping in other programming languages
• Web scraping vs. web crawling
• Web scraping vs. artificial intelligence (AI) and machine learning (ML)

Workshop: Scraping Board Member Connections

Workshop: Scraping News Articles

• Web scraping vs. robotic process automation (RPA)
• From web scraping to <blank> scraping

Workshop: Scraping Investment Product Information

Managerial Aspects

• Legal concerns
• Web scraping as part of your data science pipeline
• Buy or build?
• When not to opt for web scraping
• Managing a web scraping project

Conclusions

• Mitigation and counter-mitigation
• Best practices