mirror of
https://github.com/hwchase17/langchain
synced 2024-11-08 07:10:35 +00:00
community: fix issue with nested field extraction in MongodbLoader (#22801)
**Description:** This PR addresses an issue in the `MongodbLoader` where nested fields were not being correctly extracted. The loader now correctly handles nested fields specified in the `field_names` parameter. **Issue:** Fixes an issue where attempting to extract nested fields from MongoDB documents resulted in `KeyError`. **Dependencies:** No new dependencies are required for this change. **Twitter handle:** (Optional, your Twitter handle if you'd like a mention when the PR is announced) ### Changes 1. **Field Name Parsing**: - Added logic to parse nested field names and safely extract their values from the MongoDB documents. 2. **Projection Construction**: - Updated the projection dictionary to include nested fields correctly. 3. **Field Extraction**: - Updated the `aload` method to handle nested field extraction using a recursive approach to traverse the nested dictionaries. ### Example Usage Updated usage example to demonstrate how to specify nested fields in the `field_names` parameter: ```python loader = MongodbLoader( connection_string=MONGO_URI, db_name=MONGO_DB, collection_name=MONGO_COLLECTION, filter_criteria={"data.job.company.industry_name": "IT", "data.job.detail": { "$exists": True }}, field_names=[ "data.job.detail.id", "data.job.detail.position", "data.job.detail.intro", "data.job.detail.main_tasks", "data.job.detail.requirements", "data.job.detail.preferred_points", "data.job.detail.benefits", ], ) docs = loader.load() print(len(docs)) for doc in docs: print(doc.page_content) ``` ### Testing Tested with a MongoDB collection containing nested documents to ensure that the nested fields are correctly extracted and concatenated into a single page_content string. ### Note This change ensures backward compatibility for non-nested fields and improves functionality for nested field extraction. ### Output Sample ```python print(docs[:3]) ``` ```shell # output sample: [ Document( # Here in this example, page_content is the combined text from the fields below # "position", "intro", "main_tasks", "requirements", "preferred_points", "benefits" page_content='all combined contents from the requested fields in the document', metadata={'database': 'Your Database name', 'collection': 'Your Collection name'} ), ... ] ``` --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>
This commit is contained in:
parent
aeeda370aa
commit
2b87e330b0
@ -77,7 +77,19 @@ class MongodbLoader(BaseLoader):
|
|||||||
|
|
||||||
# Extract text content from filtered fields or use the entire document
|
# Extract text content from filtered fields or use the entire document
|
||||||
if self.field_names is not None:
|
if self.field_names is not None:
|
||||||
fields = {name: doc[name] for name in self.field_names}
|
fields = {}
|
||||||
|
for name in self.field_names:
|
||||||
|
# Split the field names to handle nested fields
|
||||||
|
keys = name.split(".")
|
||||||
|
value = doc
|
||||||
|
for key in keys:
|
||||||
|
if key in value:
|
||||||
|
value = value[key]
|
||||||
|
else:
|
||||||
|
value = ""
|
||||||
|
break
|
||||||
|
fields[name] = value
|
||||||
|
|
||||||
texts = [str(value) for value in fields.values()]
|
texts = [str(value) for value in fields.values()]
|
||||||
text = " ".join(texts)
|
text = " ".join(texts)
|
||||||
else:
|
else:
|
||||||
|
Loading…
Reference in New Issue
Block a user