langchain/docs/extras/integrations/document_loaders/rss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2dfc4698",
   "metadata": {},
   "source": [
    "# RSS Feeds\n",
    "\n",
    "This covers how to load HTML news articles from a list of RSS feed URLs into a document format that we can use downstream."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7c2cd52-c1f7-4a06-8539-b0117da91fba",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install feedparser newspaper3k listparser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "16c3699e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import RSSFeedLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "836fbac1",
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = [\"https://news.ycombinator.com/rss\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33089aba-ff74-4d00-8f40-9449c29587cc",
   "metadata": {},
   "source": [
    "Pass in urls to load them into Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00f46fda",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = RSSFeedLoader(urls=urls)\n",
    "data = loader.load()\n",
    "print(len(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b447468cc42266d0",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(next Rich)\n",
      "\n",
      "04 August 2023\n",
      "\n",
      "Rich Hickey\n",
      "\n",
      "It is with a mixture of heartache and optimism that I announce today my (long planned) retirement from commercial software development, and my employment at Nubank. It’s been thrilling to see Clojure and Datomic successfully applied at scale.\n",
      "\n",
      "I look forward to continuing to lead ongoing work maintaining and enhancing Clojure with Alex, Stu, Fogus and many others, as an independent developer once again. We have many useful things planned for 1.12 and beyond. The community remains friendly, mature and productive, and is taking Clojure into many interesting new domains.\n",
      "\n",
      "I want to highlight and thank Nubank for their ongoing sponsorship of Alex, Fogus and the core team, as well as the Clojure community at large.\n",
      "\n",
      "Stu will continue to lead the development of Datomic at Nubank, where the Datomic team grows and thrives. I’m particularly excited to see where the new free availability of Datomic will lead.\n",
      "\n",
      "My time with Cognitect remains the highlight of my career. I have learned from absolutely everyone on our team, and am forever grateful to all for our interactions. There are too many people to thank here, but I must extend my sincerest appreciation and love to Stu and Justin for (repeatedly) taking a risk on me and my ideas, and for being the best of partners and friends, at all times fully embodying the notion of integrity. And of course to Alex Miller - who possesses in abundance many skills I lack, and without whose indomitable spirit, positivity and friendship Clojure would not have become what it did.\n",
      "\n",
      "I have made many friends through Clojure and Cognitect, and I hope to nurture those friendships moving forward.\n",
      "\n",
      "Retirement returns me to the freedom and independence I had when originally developing Clojure. The journey continues!\n"
     ]
    }
   ],
   "source": [
    "print(data[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36d3b0d329faf2a",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "You can pass arguments to the NewsURLLoader which it uses to load articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "5fdada62470d3019",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error fetching or processing https://twitter.com/andrewmccalip/status/1687405505604734978, exception: You must `parse()` an article first!\n",
      "Error processing entry https://twitter.com/andrewmccalip/status/1687405505604734978, exception: list index out of range\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13\n"
     ]
    }
   ],
   "source": [
    "loader = RSSFeedLoader(urls=urls, nlp=True)\n",
    "data = loader.load()\n",
    "print(len(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "11d71963f7735c1d",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['nubank',\n",
       " 'alex',\n",
       " 'stu',\n",
       " 'taking',\n",
       " 'team',\n",
       " 'remains',\n",
       " 'rich',\n",
       " 'clojure',\n",
       " 'thank',\n",
       " 'planned',\n",
       " 'datomic']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].metadata['keywords']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "9fb64ba0e8780966",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'It’s been thrilling to see Clojure and Datomic successfully applied at scale.\\nI look forward to continuing to lead ongoing work maintaining and enhancing Clojure with Alex, Stu, Fogus and many others, as an independent developer once again.\\nThe community remains friendly, mature and productive, and is taking Clojure into many interesting new domains.\\nI want to highlight and thank Nubank for their ongoing sponsorship of Alex, Fogus and the core team, as well as the Clojure community at large.\\nStu will continue to lead the development of Datomic at Nubank, where the Datomic team grows and thrives.'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].metadata['summary']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ac26c488315bff",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "You can also use an OPML file such as a Feedly export.  Pass in either a URL or the OPML contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8b6f07ae526a897c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error fetching http://www.engadget.com/rss-full.xml, exception: Error fetching http://www.engadget.com/rss-full.xml, exception: document declared as us-ascii, but parsed as utf-8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    }
   ],
   "source": [
    "with open(\"example_data/sample_rss_feeds.opml\", \"r\") as f:\n",
    "    loader = RSSFeedLoader(opml=f.read())\n",
    "data = loader.load()\n",
    "print(len(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b68a26b3",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The electric vehicle startup Fisker made a splash in Huntington Beach last night, showing off a range of new EVs it plans to build alongside the Fisker Ocean, which is slowly beginning deliveries in Europe and the US. With shades of Lotus circa 2010, it seems there\\'s something for most tastes, with a powerful four-door GT, a versatile pickup truck, and an affordable electric city car.\\n\\n\"We want the world to know that we have big plans and intend to move into several different segments, redefining each with our unique blend of design, innovation, and sustainability,\" said CEO Henrik Fisker.\\n\\nStarting with the cheapest, the Fisker PEAR—a cutesy acronym for \"Personal Electric Automotive Revolution\"—is said to use 35 percent fewer parts than other small EVs. Although it\\'s a smaller car, the PEAR seats six thanks to front and rear bench seats. Oh, and it has a frunk, which the company is calling the \"froot,\" something that will satisfy some British English speakers like Ars\\' friend and motoring journalist Jonny Smith.\\n\\nBut most exciting is the price—starting at $29,900 and scheduled for 2025. Fisker plans to contract with Foxconn to build the PEAR in Lordstown, Ohio, meaning it would be eligible for federal tax incentives.\\n\\nAdvertisement\\n\\nThe Fisker Alaska is the company\\'s pickup truck, built on a modified version of the platform used by the Ocean. It has an extendable cargo bed, which can be as little as 4.5 feet (1,371 mm) or as much as 9.2 feet (2,804 mm) long. Fisker claims it will be both the lightest EV pickup on sale and the most sustainable pickup truck in the world. Range will be an estimated 230–240 miles (370–386 km).\\n\\nThis, too, is slated for 2025, and also at a relatively affordable price, starting at $45,400. Fisker hopes to build this car in North America as well, although it isn\\'t saying where that might take place.\\n\\nFinally, there\\'s the Ronin, a four-door GT that bears more than a passing resemblance to the Fisker Karma, Henrik Fisker\\'s 2012 creation. There\\'s no price for this one, but Fisker says its all-wheel drive powertrain will boast 1,000 hp (745 kW) and will hit 60 mph from a standing start in two seconds—just about as fast as modern tires will allow. Expect a massive battery in this one, as Fisker says it\\'s targeting a 600-mile (956 km) range.\\n\\n\"Innovation and sustainability, along with design, are our three brand values. By 2027, we intend to produce the world’s first climate-neutral vehicle, and as our customers reinvent their relationships with mobility, we want to be a leader in software-defined transportation,\" Fisker said.'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5a0cbe8-18a6-4af2-b447-7abb8b734451",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "poetry-venv",
   "language": "python",
   "name": "poetry-venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}