Skip to main content
This example covers how to load HTML documents from a list of URLs into the Document format that we can use downstream.

Unstructured URL Loader

For the examples below, please install the unstructured library and see this guide for more instructions on setting up Unstructured locally, including setting up required system dependencies:
Pass in ssl_verify=False with headers=headers to get past ssl_verification errors.

Selenium URL Loader

This covers how to load HTML documents from a list of URLs using the SeleniumURLLoader. Using Selenium allows us to load pages that require JavaScript to render. To use the SeleniumURLLoader, you have to install selenium and unstructured.

Playwright URL Loader

Playwright is an open-source automation tool developed by Microsoft that allows you to programmatically control and automate web browsers. It is designed for end-to-end testing, scraping, and automating tasks across various web browsers such as Chromium, Firefox, and WebKit.
This covers how to load HTML documents from a list of URLs using the PlaywrightURLLoader. Playwright enables reliable end-to-end testing for modern web apps. As in the Selenium case, Playwright allows us to load and render the JavaScript pages. To use the PlaywrightURLLoader, you have to install playwright and unstructured. Additionally, you have to install the Playwright Chromium browser:
Currently, nly the async method supported:

Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.