fix: ReadTheDocs loader main content filter (#2609)

It seems the main element wrapper changed in ReadTheDocs website or for
some reason it's different for me ?

This adds an extra filter for the main content wrapper if the first one
returns no text.


![2023-04-09-043315_1178x873_scrot](https://user-images.githubusercontent.com/210457/230751369-24b69cb9-1601-4540-b5f3-d115165f55f6.jpg)

Co-authored-by: blob42 <spike@w530>
fix_agent_callbacks
blob42 1 year ago committed by GitHub
parent aaac7071a3
commit 54b1645d13
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -45,6 +45,10 @@ class ReadTheDocsLoader(BaseLoader):
def _clean_data(data: str) -> str:
soup = BeautifulSoup(data, **self.bs_kwargs)
text = soup.find_all("main", {"id": "main-content"})
if len(text) == 0:
text = soup.find_all("div", {"role": "main"})
if len(text) != 0:
text = text[0].get_text()
else:

Loading…
Cancel
Save