forked from Archives/langchain
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
118 lines
12 KiB
Plaintext
118 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bf920da0",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Web Base\n",
|
|
"\n",
|
|
"This covers how to load all text from webpages into a document format that we can use downstream. For more custom logic for loading webpages look at some child class examples such as IMSDbLoader, AZLyricsLoader, and CollegeConfidentialLoader"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "00b6de21",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.document_loaders import WebBaseLoader"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "0231df35",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"loader = WebBaseLoader(\"https://www.espn.com/\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "f06bdc4e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"data = loader.load()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "a390d79f",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[Document(page_content=\"\\n\\n\\n\\n\\n\\n\\n\\n\\nESPN - Serving Sports Fans. Anytime. Anywhere.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n Skip to main content\\n \\n\\n Skip to navigation\\n \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n<\\n\\n>\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nMenuESPN\\n\\n\\nSearch\\n\\n\\n\\nscores\\n\\n\\n\\nNFLNBANHLNCAAMNCAAWSoccer…MLBNCAAFGolfTennisSports BettingBoxingCaribbean SeriesCFLNCAACricketF1HorseLLWSMMANASCARNBA G LeagueOlympic SportsRacingRN BBRN FBRugbyWNBAWWEX GamesXFLMore ESPNFantasyListenWatchESPN+\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n \\n\\nSUBSCRIBE NOW\\n\\n\\n\\n\\n\\nUFC 284: Makhachev vs. Volkanovski (ESPN+ PPV)\\n\\n\\n\\n\\n\\n\\n\\nMen's College Hoops: Select Games\\n\\n\\n\\n\\n\\n\\n\\nWomen's College Hoops: Select Games\\n\\n\\n\\n\\n\\n\\n\\nNHL: Select Games\\n\\n\\n\\n\\n\\n\\n\\nGerman Cup: Round of 16\\n\\n\\n\\n\\n\\n\\n\\n30 For 30: Bullies Of Baltimore\\n\\n\\n\\n\\n\\n\\n\\nMatt Miller's Two-Round NFL Mock Draft\\n\\n\\nQuick Links\\n\\n\\n\\n\\nSuper Bowl LVII\\n\\n\\n\\n\\n\\n\\n\\nSuper Bowl Betting\\n\\n\\n\\n\\n\\n\\n\\nNBA Trade Machine\\n\\n\\n\\n\\n\\n\\n\\nNBA All-Star Game\\n\\n\\n\\n\\n\\n\\n\\nFantasy Baseball: Sign Up\\n\\n\\n\\n\\n\\n\\n\\nHow To Watch NHL Games\\n\\n\\n\\n\\n\\n\\n\\nGames For Me\\n\\n\\n\\n\\n\\n\\nFavorites\\n\\n\\n\\n\\n\\n\\n Manage Favorites\\n \\n\\n\\n\\nCustomize ESPNSign UpLog InESPN Sites\\n\\n\\n\\n\\nESPN Deportes\\n\\n\\n\\n\\n\\n\\n\\nAndscape\\n\\n\\n\\n\\n\\n\\n\\nespnW\\n\\n\\n\\n\\n\\n\\n\\nESPNFC\\n\\n\\n\\n\\n\\n\\n\\nX Games\\n\\n\\n\\n\\n\\n\\n\\nSEC Network\\n\\n\\nESPN Apps\\n\\n\\n\\n\\nESPN\\n\\n\\n\\n\\n\\n\\n\\nESPN Fantasy\\n\\n\\nFollow ESPN\\n\\n\\n\\n\\nFacebook\\n\\n\\n\\n\\n\\n\\n\\nTwitter\\n\\n\\n\\n\\n\\n\\n\\nInstagram\\n\\n\\n\\n\\n\\n\\n\\nSnapchat\\n\\n\\n\\n\\n\\n\\n\\nYouTube\\n\\n\\n\\n\\n\\n\\n\\nThe ESPN Daily Podcast\\n\\n\\nAP Photo/Mark J. Terrilllive\\n\\n\\n\\nChristian Wood elevates for the big-time stuffChristian Wood elevates for the big-time stuff15m0:29\\n\\n\\nKyrie Irving nails the treyKyrie Irving nails the trey37m0:17\\n\\n\\nDwight Powell rises up for putback dunkDwight Powell throws down the putback dunk for the Mavericks.38m0:16\\n\\n\\nKyrie sinks his first basket with the MavericksKyrie Irving drains the jump shot early vs. the Clippers for his first points with the Mavericks.39m0:17\\n\\n\\nReggie Bullock pulls up for wide open 3Reggie Bullock is left wide open for the 3-pointer early vs. the Clippers.46m0:21\\n\\n\\n\\nTOP HEADLINESSources: Lakers get PG Russell in 3-team tradeTrail Blazers shipping Hart to Knicks, sources sayUConn loses two straight for first time in 30 yearsNFL's Goodell on officiating: Never been betterNFLPA's Smith: Get rid of 'intrusive' NFL combineAlex Morgan: 'Bizarre' for Saudis to sponsor WWCBills' Hamlin makes appearance to receive awardWWE Hall of Famer Lawler recovering from strokeWhich NFL team trades up to No. 1?NBA TRADE DEADLINE3 P.M. ET ON THURSDAYTrade grades: What to make of the three-team deal involving Russell Westbrook and D'Angelo RussellESPN NBA Insider Kevin Pelton is handing out grades for the biggest moves.2hLayne Murdoch Jr./NBAE via Getty ImagesNBA trade tracker: Grades, details for every deal for the 2022-23 seasonWhich players are finding new homes and which teams are making trades during the free-agency frenzy?59mESPN.comNBA trade deadline: Latest buzz and newsNBA SCOREBOARDWEDNESDAY'S GAMESSee AllCLEAR THE RUNWAYJalen Green soars for lefty alley-oop1h0:19Jarrett Allen skies to drop the hammer2h0:16Once the undisputed greatest, Joe Montana is still working things out15hWright ThompsonSUPER BOWL LVII6:30 P.M. ET ON SUNDAYBarbershop tales, a fistfight and brotherly love: Untold stories that explain the Kelce brothersJason and Travis Kelce will become the first brothers to face each other in a Super Bowl. Here are untold stories from people who know them best.16hTim McManus, +2 MoreEd Zurga/AP PhotoNFL experts predict Chiefs-Eagles: Our Super Bowl winner picksNFL writers, analysts and reporters take their best guesses on the Super Bowl LVII matchup.17hESPN staffBeware of Philadelphia's Rocky statue curseMadden sim predicts Eagles to win Super BowlTOP 10 TEAMS FALLCOLLEGE HOOPSUConn loses two straight for first time since 1993, falling to Marquette57m1:58Vandy drains 3 at buzzer to knock off Tennessee, fans storm the court1h0:54COLLEGE HOOPS SCORESMEN'S AND WOMEN'S TOP-25 GAMESMen's college hoops scoreboardWomen's college basketball scoresPROJECTING THE BUBBLEMEN'S COLLEGE HOOPSBubble Watch: Current situation? North Carolina has some work to doThe countdown to Selection Sunday on March 12 has begun. We will track which teams are locks and which ones can play their way into or out of the 2023 NCAA men's basketball tournament.6hJohn GasawayAP Photo/Matt Rourke Top HeadlinesSources: Lakers get PG Russell in 3-team tradeTrail Blazers shipping Hart to Knicks, sources sayUConn loses two straight for first time in 30 yearsNFL's Goodell on officiating: Never been betterNFLPA's Smith: Get rid of 'intrusive' NFL combineAlex Morgan: 'Bizarre' for Saudis to sponsor WWCBills' Hamlin makes appearance to receive awardWWE Hall of Famer Lawler recovering from strokeWhich NFL team trades up to No. 1?Favorites FantasyManage FavoritesFantasy HomeCustomize ESPNSign UpLog InICYMI1:54Orlovsky roasts Stephen A. for his top-5 players in the Super BowlDan Orlovsky lets Stephen A. Smith hear it after he lists his top five players in Super Bowl LVII. Best of ESPN+Michael Hickey/Getty ImagesBubble Watch 2023: Brace yourself for NCAA tournament dramaThe countdown to Selection Sunday on March 12 has begun. We will track which teams are locks and which ones can play their way into or out of the 2023 NCAA men's basketball tournament.Adam Pantozzi/NBAE via Getty ImagesLeBron's journey to the NBA scoring record in shot chartsTake a look at how LeBron James' on-court performance has changed during his march to 38,388 points.Illustration by ESPNRe-drafting first two rounds of 2022 NFL class: All 64 picksWe gave every NFL team a do-over for last year's draft, re-drafting the top 64 picks. Here's who rises and falls with the benefit of hindsight.AP Photo/David DermerWay-too-early 2023 MLB starting rotation rankingsThe Yanks' and Mets' rotations take two of the top three spots on our pre-spring training list. Where did they land -- and did another team sneak past one of 'em? Trending NowAP Photo/Jae C. HongStars pay tribute to LeBron James for securing NBA's all-time points recordLeBron James has passed Kareem Abdul-Jabbar for No. 1 on the all-time NBA scoring list, and other stars paid tribute to him on social media.Getty ImagesFans prepare for Rihanna's 2023 Super Bowl halftime showAs Rihanna prepares to make her highly anticipated return, supporters of all 32 teams are paying homage to the icon -- as only tormented NFL fans can.Photo by Cooper Neill/Getty ImagesWhen is the 2023 Super Bowl? Date, time for Chiefs vs. EaglesWe have you covered with seeding, scores and the full schedule for this season's playoffs -- and how to watch Super Bowl LVII.James Drake/Sports Illustrated via Getty ImagesNFL history: Super Bowl winners and resultsFrom the Packers' 1967 win over the Chiefs to the Rams' victory over the Bengals in 2022, we've got results for every Super Bowl.China Wong/NHLI via Getty ImagesBoston Bruins record tracker: Wins, points, milestonesThe B's are on pace for NHL records in wins and points, along with some individual superlatives as well. Follow along here with our updated tracker. Sports BettingPhoto by Kevin C. Cox/Getty ImagesSuper Bowl LVII betting: Everything you need to know to bet Eagles-ChiefsHere's your one-stop shop for all the information you need to help make your picks on the Philadelphia Eagles vs. Kansas City Chiefs in Super Bowl LVII. How to Watch on ESPN+(AP Photo/Koji Sasahara, File)How to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN, ESPN+Here's everything you need to know about how to watch the PGA Tour, Masters, PGA Championship and FedEx Cup playoffs on ESPN and ESPN+. \\n\\nESPN+\\n\\n\\n\\n\\nUFC 284: Makhachev vs. Volkanovski (ESPN+ PPV)\\n\\n\\n\\n\\n\\n\\n\\nMen's College Hoops: Select Games\\n\\n\\n\\n\\n\\n\\n\\nWomen's College Hoops: Select Games\\n\\n\\n\\n\\n\\n\\n\\nNHL: Select Games\\n\\n\\n\\n\\n\\n\\n\\nGerman Cup: Round of 16\\n\\n\\n\\n\\n\\n\\n\\n30 For 30: Bullies Of Baltimore\\n\\n\\n\\n\\n\\n\\n\\nMatt Miller's Two-Round NFL Mock Draft\\n\\n\\nQuick Links\\n\\n\\n\\n\\nSuper Bowl LVII\\n\\n\\n\\n\\n\\n\\n\\nSuper Bowl Betting\\n\\n\\n\\n\\n\\n\\n\\nNBA Trade Machine\\n\\n\\n\\n\\n\\n\\n\\nNBA All-Star Game\\n\\n\\n\\n\\n\\n\\n\\nFantasy Baseball: Sign Up\\n\\n\\n\\n\\n\\n\\n\\nHow To Watch NHL Games\\n\\n\\n\\n\\n\\n\\n\\nGames For Me\\n\\n\\nESPN Sites\\n\\n\\n\\n\\nESPN Deportes\\n\\n\\n\\n\\n\\n\\n\\nAndscape\\n\\n\\n\\n\\n\\n\\n\\nespnW\\n\\n\\n\\n\\n\\n\\n\\nESPNFC\\n\\n\\n\\n\\n\\n\\n\\nX Games\\n\\n\\n\\n\\n\\n\\n\\nSEC Network\\n\\n\\nESPN Apps\\n\\n\\n\\n\\nESPN\\n\\n\\n\\n\\n\\n\\n\\nESPN Fantasy\\n\\n\\nFollow ESPN\\n\\n\\n\\n\\nFacebook\\n\\n\\n\\n\\n\\n\\n\\nTwitter\\n\\n\\n\\n\\n\\n\\n\\nInstagram\\n\\n\\n\\n\\n\\n\\n\\nSnapchat\\n\\n\\n\\n\\n\\n\\n\\nYouTube\\n\\n\\n\\n\\n\\n\\n\\nThe ESPN Daily Podcast\\n\\n\\nTerms of UsePrivacy PolicyYour US State Privacy RightsChildren's Online Privacy PolicyInterest-Based AdsAbout Nielsen MeasurementDo Not Sell or Share My Personal InformationContact UsDisney Ad Sales SiteWork for ESPNCopyright: © ESPN Enterprises, Inc. All rights reserved.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\", lookup_str='', metadata={'source': 'https://www.espn.com/'}, lookup_index=0)]"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "878179f7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"\"\"\"\n",
|
|
"# Use this piece of code for testing new custom BeautifulSoup parsers\n",
|
|
"\n",
|
|
"import requests\n",
|
|
"from bs4 import BeautifulSoup\n",
|
|
"\n",
|
|
"html_doc = requests.get(\"{INSERT_NEW_URL_HERE}\")\n",
|
|
"soup = BeautifulSoup(html_doc.text, 'html.parser')\n",
|
|
"\n",
|
|
"# Beautiful soup logic to be exported to langchain.document_loaders.webpage.py\n",
|
|
"# Example: transcript = soup.select_one(\"td[class='scrtext']\").text\n",
|
|
"# BS4 documentation can be found here: https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
|
|
"\n",
|
|
"\"\"\";"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ca330a63",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.1"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|