docs/examples/quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0cba38e5",
   "metadata": {},
   "source": [
    "# Crawl4AI 🕷️🤖\n",
    "<a href=\"https://trendshift.io/repositories/11716\" target=\"_blank\"><img src=\"https://trendshift.io/api/badge/repositories/11716\" alt=\"unclecode%2Fcrawl4ai | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"/></a>\n",
    "\n",
    "[![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)\n",
    "![PyPI - Downloads](https://img.shields.io/pypi/dm/Crawl4AI)\n",
    "[![GitHub Forks](https://img.shields.io/github/forks/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/network/members)\n",
    "[![GitHub Issues](https://img.shields.io/github/issues/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/issues)\n",
    "[![GitHub Pull Requests](https://img.shields.io/github/issues-pr/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/pulls)\n",
    "[![License](https://img.shields.io/github/license/unclecode/crawl4ai)](https://github.com/unclecode/crawl4ai/blob/main/LICENSE)\n",
    "\n",
    "Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. 🆓🌐\n",
    "\n",
    "- GitHub Repository: [https://github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)\n",
    "- Twitter: [@unclecode](https://twitter.com/unclecode)\n",
    "- Website: [https://crawl4ai.com](https://crawl4ai.com)\n",
    "\n",
    "## 🌟 Meet the Crawl4AI Assistant: Your Copilot for Crawling\n",
    "Use the [Crawl4AI GPT Assistant](https://tinyurl.com/crawl4ai-gpt) as your AI-powered copilot! With this assistant, you can:\n",
    "- 🧑‍💻 Generate code for complex crawling and extraction tasks\n",
    "- 💡 Get tailored support and examples\n",
    "- 📘 Learn Crawl4AI faster with step-by-step guidance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41de6458",
   "metadata": {},
   "source": [
    "### **Quickstart with Crawl4AI**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1380e951",
   "metadata": {},
   "source": [
    "#### 1. **Installation**\n",
    "Install Crawl4AI and necessary dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05fecfad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%capture\n",
    "!pip install crawl4ai\n",
    "!pip install nest_asyncio\n",
    "!playwright install  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2c2a74c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3c558d7",
   "metadata": {},
   "source": [
    "#### 2. **Basic Setup and Simple Crawl**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "003376f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 1.49 seconds\n",
      "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.10 seconds\n",
      "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
      "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.10 seconds.\n",
      "IE 11 is not supported. For an optimal experience visit our site on another browser.\n",
      "\n",
      "[Morning Rundown: Trump and Harris' vastly different closing pitches, why Kim Jong Un is helping Russia, and an ancient city is discovered by accident](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)[](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)\n",
      "\n",
      "Skip to Content\n",
      "\n",
      "[NBC News Logo](https://www.nbcnews.com)\n",
      "\n",
      "Spon\n"
     ]
    }
   ],
   "source": [
    "import asyncio\n",
    "from crawl4ai import AsyncWebCrawler\n",
    "\n",
    "async def simple_crawl():\n",
    "    async with AsyncWebCrawler() as crawler:\n",
    "        result = await crawler.arun(\n",
    "            url=\"https://www.nbcnews.com/business\",\n",
    "            bypass_cache=True # By default this is False, meaning the cache will be used\n",
    "        )\n",
    "        print(result.markdown[:500])  # Print the first 500 characters\n",
    "        \n",
    "asyncio.run(simple_crawl())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9b4d50",
   "metadata": {},
   "source": [
    "#### 3. **Dynamic Content Handling**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5bb8c1e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
      "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
      "[LOG] 🕸️ Crawling https://www.nbcnews.com/business using AsyncPlaywrightCrawlerStrategy...\n",
      "[LOG] ✅ Crawled https://www.nbcnews.com/business successfully!\n",
      "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 4.52 seconds\n",
      "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.15 seconds\n",
      "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
      "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.15 seconds.\n",
      "IE 11 is not supported. For an optimal experience visit our site on another browser.\n",
      "\n",
      "[Morning Rundown: Trump and Harris' vastly different closing pitches, why Kim Jong Un is helping Russia, and an ancient city is discovered by accident](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)[](https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973)\n",
      "\n",
      "Skip to Content\n",
      "\n",
      "[NBC News Logo](https://www.nbcnews.com)\n",
      "\n",
      "Spon\n"
     ]
    }
   ],
   "source": [
    "async def crawl_dynamic_content():\n",
    "    # You can use wait_for to wait for a condition to be met before returning the result\n",
    "    # wait_for = \"\"\"() => {\n",
    "    #     return Array.from(document.querySelectorAll('article.tease-card')).length > 10;\n",
    "    # }\"\"\"\n",
    "\n",
    "    # wait_for can be also just a css selector\n",
    "    # wait_for = \"article.tease-card:nth-child(10)\"\n",
    "\n",
    "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
    "        js_code = [\n",
    "            \"const loadMoreButton = Array.from(document.querySelectorAll('button')).find(button => button.textContent.includes('Load More')); loadMoreButton && loadMoreButton.click();\"\n",
    "        ]\n",
    "        result = await crawler.arun(\n",
    "            url=\"https://www.nbcnews.com/business\",\n",
    "            js_code=js_code,\n",
    "            # wait_for=wait_for,\n",
    "            bypass_cache=True,\n",
    "        )\n",
    "        print(result.markdown[:500])  # Print first 500 characters\n",
    "\n",
    "asyncio.run(crawl_dynamic_content())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86febd8d",
   "metadata": {},
   "source": [
    "#### 4. **Content Cleaning and Fit Markdown**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e8ab01f",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def clean_content():\n",
    "    async with AsyncWebCrawler() as crawler:\n",
    "        result = await crawler.arun(\n",
    "            url=\"https://janineintheworld.com/places-to-visit-in-central-mexico\",\n",
    "            excluded_tags=['nav', 'footer', 'aside'],\n",
    "            remove_overlay_elements=True,\n",
    "            word_count_threshold=10,\n",
    "            bypass_cache=True\n",
    "        )\n",
    "        full_markdown_length = len(result.markdown)\n",
    "        fit_markdown_length = len(result.fit_markdown)\n",
    "        print(f\"Full Markdown Length: {full_markdown_length}\")\n",
    "        print(f\"Fit Markdown Length: {fit_markdown_length}\")\n",
    "        print(result.fit_markdown[:1000])\n",
    "        \n",
    "\n",
    "asyncio.run(clean_content())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55715146",
   "metadata": {},
   "source": [
    "#### 5. **Link Analysis and Smart Filtering**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2ae47c69",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 0.93 seconds\n",
      "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.11 seconds\n",
      "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
      "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.11 seconds.\n",
      "Found 107 internal links\n",
      "Found 58 external links\n",
      "Href: https://www.nbcnews.com/news/harris-speech-ellipse-ancient-mayan-city-morning-rundown-rcna177973\n",
      "Text: Morning Rundown: Trump and Harris' vastly different closing pitches, why Kim Jong Un is helping Russia, and an ancient city is discovered by accident\n",
      "\n",
      "Href: https://www.nbcnews.com\n",
      "Text: NBC News Logo\n",
      "\n",
      "Href: https://www.nbcnews.com/politics/2024-election/live-blog/kamala-harris-donald-trump-rally-election-live-updates-rcna177529\n",
      "Text: 2024 Election\n",
      "\n",
      "Href: https://www.nbcnews.com/politics\n",
      "Text: Politics\n",
      "\n",
      "Href: https://www.nbcnews.com/us-news\n",
      "Text: U.S. News\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "async def link_analysis():\n",
    "    async with AsyncWebCrawler() as crawler:\n",
    "        result = await crawler.arun(\n",
    "            url=\"https://www.nbcnews.com/business\",\n",
    "            bypass_cache=True,\n",
    "            exclude_external_links=True,\n",
    "            exclude_social_media_links=True,\n",
    "            # exclude_domains=[\"facebook.com\", \"twitter.com\"]\n",
    "        )\n",
    "        print(f\"Found {len(result.links['internal'])} internal links\")\n",
    "        print(f\"Found {len(result.links['external'])} external links\")\n",
    "\n",
    "        for link in result.links['internal'][:5]:\n",
    "            print(f\"Href: {link['href']}\\nText: {link['text']}\\n\")\n",
    "                \n",
    "\n",
    "asyncio.run(link_analysis())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80cceef3",
   "metadata": {},
   "source": [
    "#### 6. **Media Handling**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "1fed7f99",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business, success: True, time taken: 1.42 seconds\n",
      "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business, success: True, time taken: 0.11 seconds\n",
      "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business, Strategy: AsyncWebCrawler\n",
      "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business, time taken: 0.12 seconds.\n",
      "Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-762x508,f_auto,q_auto:best/rockcms/2024-10/241023-NM-Chilccare-jg-27b982.jpg, Alt: , Score: 4\n",
      "Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241030-china-ev-electric-mb-0746-cae05c.jpg, Alt: Volkswagen Workshop in Hefei, Score: 5\n",
      "Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241029-nyc-subway-sandwich-2021-ac-922p-a92374.jpg, Alt: A sub is prepared at a Subway restaurant in Manhattan, New York City, Score: 5\n",
      "Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241029-suv-gravity-ch-1618-752415.jpg, Alt: The Lucid Gravity car., Score: 5\n",
      "Image URL: https://media-cldnry.s-nbcnews.com/image/upload/t_focal-80x80,f_auto,q_auto:best/rockcms/2024-10/241029-dearborn-michigan-f-150-ford-ranger-trucks-assembly-line-ac-426p-614f0b.jpg, Alt: Ford Introduces new F-150 And Ranger Trucks At Their Dearborn Plant, Score: 5\n"
     ]
    }
   ],
   "source": [
    "async def media_handling():\n",
    "    async with AsyncWebCrawler() as crawler:\n",
    "        result = await crawler.arun(\n",
    "            url=\"https://www.nbcnews.com/business\", \n",
    "            bypass_cache=True,\n",
    "            exclude_external_images=False,\n",
    "            screenshot=True\n",
    "        )\n",
    "        for img in result.media['images'][:5]:\n",
    "            print(f\"Image URL: {img['src']}, Alt: {img['alt']}, Score: {img['score']}\")\n",
    "        \n",
    "asyncio.run(media_handling())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9290499a",
   "metadata": {},
   "source": [
    "#### 7. **Using Hooks for Custom Workflow**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d069c2b",
   "metadata": {},
   "source": [
    "Hooks in Crawl4AI allow you to run custom logic at specific stages of the crawling process. This can be invaluable for scenarios like setting custom headers, logging activities, or processing content before it is returned. Below is an example of a basic workflow using a hook, followed by a complete list of available hooks and explanations on their usage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "bc4d2fc8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Hook] Preparing to navigate...\n",
      "[LOG] 🚀 Crawling done for https://crawl4ai.com, success: True, time taken: 3.49 seconds\n",
      "[LOG] 🚀 Content extracted for https://crawl4ai.com, success: True, time taken: 0.03 seconds\n",
      "[LOG] 🔥 Extracting semantic blocks for https://crawl4ai.com, Strategy: AsyncWebCrawler\n",
      "[LOG] 🚀 Extraction done for https://crawl4ai.com, time taken: 0.03 seconds.\n",
      "[Crawl4AI Documentation](https://docs.crawl4ai.com/)\n",
      "\n",
      "  * [ Home ](.)\n",
      "  * [ Installation ](basic/installation/)\n",
      "  * [ Quick Start ](basic/quickstart/)\n",
      "  * [ Search ](#)\n",
      "\n",
      "\n",
      "\n",
      "  * Home\n",
      "  * [Installation](basic/installation/)\n",
      "  * [Quick Start](basic/quickstart/)\n",
      "  * Basic\n",
      "    * [Simple Crawling](basic/simple-crawling/)\n",
      "    * [Output Formats](basic/output-formats/)\n",
      "    * [Browser Configuration](basic/browser-config/)\n",
      "    * [Page Interaction](basic/page-interaction/)\n",
      "    * [Content Selection](basic/con\n"
     ]
    }
   ],
   "source": [
    "async def custom_hook_workflow():\n",
    "    async with AsyncWebCrawler() as crawler:\n",
    "        # Set a 'before_goto' hook to run custom code just before navigation\n",
    "        crawler.crawler_strategy.set_hook(\"before_goto\", lambda page: print(\"[Hook] Preparing to navigate...\"))\n",
    "        \n",
    "        # Perform the crawl operation\n",
    "        result = await crawler.arun(\n",
    "            url=\"https://crawl4ai.com\",\n",
    "            bypass_cache=True\n",
    "        )\n",
    "        print(result.markdown[:500])  # Display the first 500 characters\n",
    "\n",
    "asyncio.run(custom_hook_workflow())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ff45e21",
   "metadata": {},
   "source": [
    "List of available hooks and examples for each stage of the crawling process:\n",
    "\n",
    "- **on_browser_created**\n",
    "    ```python\n",
    "    async def on_browser_created_hook(browser):\n",
    "        print(\"[Hook] Browser created\")\n",
    "    ```\n",
    "\n",
    "- **before_goto**\n",
    "    ```python\n",
    "    async def before_goto_hook(page):\n",
    "        await page.set_extra_http_headers({\"X-Test-Header\": \"test\"})\n",
    "    ```\n",
    "\n",
    "- **after_goto**\n",
    "    ```python\n",
    "    async def after_goto_hook(page):\n",
    "        print(f\"[Hook] Navigated to {page.url}\")\n",
    "    ```\n",
    "\n",
    "- **on_execution_started**\n",
    "    ```python\n",
    "    async def on_execution_started_hook(page):\n",
    "        print(\"[Hook] JavaScript execution started\")\n",
    "    ```\n",
    "\n",
    "- **before_return_html**\n",
    "    ```python\n",
    "    async def before_return_html_hook(page, html):\n",
    "        print(f\"[Hook] HTML length: {len(html)}\")\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d56ebb1",
   "metadata": {},
   "source": [
    "#### 8. **Session-Based Crawling**\n",
    "\n",
    "When to Use Session-Based Crawling: \n",
    "Session-based crawling is especially beneficial when navigating through multi-page content where each page load needs to maintain the same session context. For instance, in cases where a “Next Page” button must be clicked to load subsequent data, the new data often replaces the previous content. Here, session-based crawling keeps the browser state intact across each interaction, allowing for sequential actions within the same session.\n",
    "\n",
    "Example: Multi-Page Navigation Using JavaScript\n",
    "In this example, we’ll navigate through multiple pages by clicking a \"Next Page\" button. After each page load, we extract the new content and repeat the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7bfebae",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def multi_page_session_crawl():\n",
    "    async with AsyncWebCrawler() as crawler:\n",
    "        session_id = \"page_navigation_session\"\n",
    "        url = \"https://example.com/paged-content\"\n",
    "\n",
    "        for page_number in range(1, 4):\n",
    "            result = await crawler.arun(\n",
    "                url=url,\n",
    "                session_id=session_id,\n",
    "                js_code=\"document.querySelector('.next-page-button').click();\" if page_number > 1 else None,\n",
    "                css_selector=\".content-section\",\n",
    "                bypass_cache=True\n",
    "            )\n",
    "            print(f\"Page {page_number} Content:\")\n",
    "            print(result.markdown[:500])  # Print first 500 characters\n",
    "\n",
    "# asyncio.run(multi_page_session_crawl())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad32a778",
   "metadata": {},
   "source": [
    "#### 9. **Using Extraction Strategies**\n",
    "\n",
    "**LLM Extraction**\n",
    "\n",
    "This example demonstrates how to use language model-based extraction to retrieve structured data from a pricing page on OpenAI’s site."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3011a7c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Extracting Structured Data with openai/gpt-4o-mini ---\n",
      "[LOG] 🌤️  Warming up the AsyncWebCrawler\n",
      "[LOG] 🌞 AsyncWebCrawler is ready to crawl\n",
      "[LOG] 🕸️ Crawling https://openai.com/api/pricing/ using AsyncPlaywrightCrawlerStrategy...\n",
      "[LOG] ✅ Crawled https://openai.com/api/pricing/ successfully!\n",
      "[LOG] 🚀 Crawling done for https://openai.com/api/pricing/, success: True, time taken: 1.29 seconds\n",
      "[LOG] 🚀 Content extracted for https://openai.com/api/pricing/, success: True, time taken: 0.13 seconds\n",
      "[LOG] 🔥 Extracting semantic blocks for https://openai.com/api/pricing/, Strategy: AsyncWebCrawler\n",
      "[LOG] Call LLM for https://openai.com/api/pricing/ - block index: 0\n",
      "[LOG] Extracted 26 blocks from URL: https://openai.com/api/pricing/ block index: 0\n",
      "[LOG] 🚀 Extraction done for https://openai.com/api/pricing/, time taken: 15.12 seconds.\n",
      "[{'model_name': 'gpt-4o', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-2024-08-06', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-audio-preview', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-audio-preview-2024-10-01', 'input_fee': '$2.50 / 1M input tokens', 'output_fee': '$10.00 / 1M output tokens', 'error': False}, {'model_name': 'gpt-4o-2024-05-13', 'input_fee': '$5.00 / 1M input tokens', 'output_fee': '$15.00 / 1M output tokens', 'error': False}]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/unclecode/devs/crawl4ai/venv/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:\n",
      "  Expected `PromptTokensDetails` but got `dict` - serialized value may not be as expected\n",
      "  return self.__pydantic_serializer__.to_python(\n"
     ]
    }
   ],
   "source": [
    "from crawl4ai.extraction_strategy import LLMExtractionStrategy\n",
    "from pydantic import BaseModel, Field\n",
    "import os, json\n",
    "\n",
    "class OpenAIModelFee(BaseModel):\n",
    "    model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n",
    "    input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n",
    "    output_fee: str = Field(\n",
    "        ..., description=\"Fee for output token for the OpenAI model.\"\n",
    "    )\n",
    "\n",
    "async def extract_structured_data_using_llm(provider: str, api_token: str = None, extra_headers: dict = None):\n",
    "    print(f\"\\n--- Extracting Structured Data with {provider} ---\")\n",
    "    \n",
    "    # Skip if API token is missing (for providers that require it)\n",
    "    if api_token is None and provider != \"ollama\":\n",
    "        print(f\"API token is required for {provider}. Skipping this example.\")\n",
    "        return\n",
    "\n",
    "    extra_args = {\"extra_headers\": extra_headers} if extra_headers else {}\n",
    "\n",
    "    async with AsyncWebCrawler(verbose=True) as crawler:\n",
    "        result = await crawler.arun(\n",
    "            url=\"https://openai.com/api/pricing/\",\n",
    "            word_count_threshold=1,\n",
    "            extraction_strategy=LLMExtractionStrategy(\n",
    "                provider=provider,\n",
    "                api_token=api_token,\n",
    "                schema=OpenAIModelFee.schema(),\n",
    "                extraction_type=\"schema\",\n",
    "                instruction=\"\"\"Extract all model names along with fees for input and output tokens.\"\n",
    "                \"{model_name: 'GPT-4', input_fee: 'US$10.00 / 1M tokens', output_fee: 'US$30.00 / 1M tokens'}.\"\"\",\n",
    "                **extra_args\n",
    "            ),\n",
    "            bypass_cache=True,\n",
    "        )\n",
    "        print(json.loads(result.extracted_content)[:5])\n",
    "\n",
    "# Usage:\n",
    "await extract_structured_data_using_llm(\"openai/gpt-4o-mini\", os.getenv(\"OPENAI_API_KEY\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6532db9d",
   "metadata": {},
   "source": [
    "**Cosine Similarity Strategy**\n",
    "\n",
    "This strategy uses semantic clustering to extract relevant content based on contextual similarity, which is helpful when extracting related sections from a single topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "ec079108",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LOG] Loading Extraction Model for mps device.\n",
      "[LOG] Loading Multilabel Classifier for mps device.\n",
      "[LOG] Model loaded sentence-transformers/all-MiniLM-L6-v2, models/reuters, took 5.193778038024902 seconds\n",
      "[LOG] 🚀 Crawling done for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, success: True, time taken: 1.37 seconds\n",
      "[LOG] 🚀 Content extracted for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, success: True, time taken: 0.07 seconds\n",
      "[LOG] 🔥 Extracting semantic blocks for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, Strategy: AsyncWebCrawler\n",
      "[LOG] 🚀 Assign tags using mps\n",
      "[LOG] 🚀 Categorization done in 0.55 seconds\n",
      "[LOG] 🚀 Extraction done for https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156, time taken: 6.63 seconds.\n",
      "[{'index': 1, 'tags': ['news_&_social_concern'], 'content': \"McDonald's 2024 combo: Inflation, a health crisis and a side of politics # McDonald's 2024 combo: Inflation, a health crisis and a side of politics\"}, {'index': 2, 'tags': ['business_&_entrepreneurs', 'news_&_social_concern'], 'content': 'Like many major brands, McDonald’s raked in big profits as the economy reopened from the pandemic. In October 2022, [executives were boasting](https://www.cnbc.com/2022/10/27/mcdonalds-mcd-earnings-q3-2022.html) that they’d been raising prices without crimping traffic, even as competitors began to warn that some customers were closing their wallets after inflation peaked above 9% that summer. Still, the U.S. had repeatedly dodged a much-forecast recession, and [Americans kept spending on nonessentials](https://www.nbcnews.com/business/economy/year-peak-inflation-travel-leisure-mostly-cost-less-rcna92760) like travel and dining out — despite regularly relaying to pollsters their dismal views of an otherwise solid economy. Even so, 64% of consumers said they noticed price increases at quick-service restaurants in September, more than at any other type of venue, according to a survey by Datassential, a food and beverage market researcher. Politicians are still drawing attention to fast-food costs, too, as the election season barrels toward a tumultuous finish. A group of Democratic senators this month [denounced McDonald’s for menu prices](https://www.nbcnews.com/news/us-news/democratic-senators-slam-mcdonalds-menu-price-hikes-rcna176380) that they said outstripped inflation, accusing the company of looking to profit “at the expense of people’s ability to put food on the table.” The financial results come toward the end of a humbling year for the nearly $213 billion restaurant chain, whose shares remained steady on the heels of its latest earnings. Kempczinski [sought to reassure investors](https://www.cnbc.com/2024/10/29/mcdonalds-e-coli-outbreak-ceo-comments.html) that [the E. coli outbreak](https://www.nbcnews.com/health/health-news/illnesses-linked-mcdonalds-e-coli-outbreak-rise-75-cdc-says-rcna177260), linked to Quarter Pounder burgers, was under control after the health crisis temporarily dented the company’s stock and caused U.S. foot traffic to drop nearly 10% in the days afterward, according to estimates by Gordon Haskett financial researchers. The fast-food giant [reported Tuesday](https://www.cnbc.com/2024/10/29/mcdonalds-mcd-earnings-q3-2024.html) that it had reversed its recent U.S. sales drop, posting a 0.3% uptick in the third quarter. Foot traffic was still down slightly, but the company said its summer of discounts was paying off. But by early this year, [photos of eye-watering menu prices](https://x.com/sam_learner/status/1681367351143301129) at some McDonald’s locations — including an $18 Big Mac combo at a Connecticut rest stop from July 2023 — went viral, bringing diners’ long-simmering frustrations to a boiling point that the company couldn’t ignore. On an earnings call in April, Kempczinski acknowledged that foot traffic had fallen. “We will stay laser-focused on providing an unparalleled experience with simple, everyday value and affordability that our consumers can count on as they continue to be mindful about their spending,” CEO Chris Kempczinski [said in a statement](https://www.prnewswire.com/news-releases/mcdonalds-reports-third-quarter-2024-results-302289216.html?Fds-Load-Behavior=force-external) alongside the earnings report.'}, {'index': 3, 'tags': ['food_&_dining', 'news_&_social_concern'], 'content': '![mcdonalds drive-thru economy fast food](https://media-cldnry.s-nbcnews.com/image/upload/t_fit-760w,f_auto,q_auto:best/rockcms/2024-10/241024-los-angeles-mcdonalds-drive-thru-ac-1059p-cfc311.jpg)McDonald’s has had some success leaning into discounts this year. Eric Thayer / Bloomberg via Getty Images file'}, {'index': 4, 'tags': ['business_&_entrepreneurs', 'food_&_dining', 'news_&_social_concern'], 'content': 'McDonald’s has faced a customer revolt over pricey Big Macs, an unsolicited cameo in election-season crossfire, and now an E. coli outbreak — just as the company had been luring customers back with more affordable burgers. Despite a difficult quarter, McDonald’s looks resilient in the face of various pressures, analysts say — something the company shares with U.S. consumers overall. “Consumers continue to be even more discriminating with every dollar that they spend,” he said at the time. Going forward, McDonald’s would be “laser-focused” on affordability.  “McDonald’s has also done a good job of embedding the brand in popular culture to enhance its relevance and meaning around fun and family. But it also needed to modify the product line to meet the expectations of a consumer who is on a tight budget,” he said. “The thing that McDonald’s had struggled with, and why I think we’re seeing kind of an inflection point, is a value proposition,” Senatore said. “McDonald’s menu price increases had run ahead of a lot of its restaurant peers. … Consumers are savvy enough to know that.” For many consumers, the fast-food giant’s menus serve as an informal gauge of the economy overall, said Sara Senatore, a Bank of America analyst covering restaurants. “The spotlight is always on McDonald’s because it’s so big” and something of a “bellwether,” she said. McDonald’s didn’t respond to requests for comment.'}, {'index': 5, 'tags': ['business_&_entrepreneurs', 'food_&_dining'], 'content': 'Mickey D’s’ $5 meal deal, which it launched in late June to jumpstart slumping sales, has given the company an appealing price point to advertise nationwide, Senatore said, speculating that it could open the door to a new permanent value offering. But before that promotion rolled out, the company’s reputation as a low-cost option had taken a bruising hit.'}]\n"
     ]
    }
   ],
   "source": [
    "from crawl4ai.extraction_strategy import CosineStrategy\n",
    "\n",
    "async def cosine_similarity_extraction():\n",
    "    async with AsyncWebCrawler() as crawler:\n",
    "        strategy = CosineStrategy(\n",
    "            word_count_threshold=10,\n",
    "            max_dist=0.2, # Maximum distance between two words\n",
    "            linkage_method=\"ward\", # Linkage method for hierarchical clustering (ward, complete, average, single)\n",
    "            top_k=3, # Number of top keywords to extract\n",
    "            sim_threshold=0.3, # Similarity threshold for clustering\n",
    "            semantic_filter=\"McDonald's economic impact, American consumer trends\", # Keywords to filter the content semantically using embeddings\n",
    "            verbose=True\n",
    "        )\n",
    "        \n",
    "        result = await crawler.arun(\n",
    "            url=\"https://www.nbcnews.com/business/consumer/how-mcdonalds-e-coli-crisis-inflation-politics-reflect-american-story-rcna177156\",\n",
    "            extraction_strategy=strategy\n",
    "        )\n",
    "        print(json.loads(result.extracted_content)[:5])\n",
    "\n",
    "asyncio.run(cosine_similarity_extraction())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff423629",
   "metadata": {},
   "source": [
    "#### 10. **Conclusion and Next Steps**\n",
    "\n",
    "You’ve explored core features of Crawl4AI, including dynamic content handling, link analysis, and advanced extraction strategies. Visit our documentation for further details on using Crawl4AI’s extensive features.\n",
    "\n",
    "- GitHub Repository: [https://github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)\n",
    "- Twitter: [@unclecode](https://twitter.com/unclecode)\n",
    "- Website: [https://crawl4ai.com](https://crawl4ai.com)\n",
    "\n",
    "Happy Crawling with Crawl4AI! 🕷️🤖\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d34c1d35",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}