From d25d32ed5f34e86b1f47de8a8194472badbfab64 Mon Sep 17 00:00:00 2001 From: Andrew Peng Date: Mon, 26 Feb 2024 17:51:50 -0800 Subject: [PATCH] Update chat finetune data prep notebook (#1074) Co-authored-by: andrewpeng@openai.com --- examples/Chat_finetuning_data_prep.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/Chat_finetuning_data_prep.ipynb b/examples/Chat_finetuning_data_prep.ipynb index 55ab9b02..c5a3cb7b 100644 --- a/examples/Chat_finetuning_data_prep.ipynb +++ b/examples/Chat_finetuning_data_prep.ipynb @@ -83,7 +83,7 @@ "1. **Data Type Check**: Checks whether each entry in the dataset is a dictionary (`dict`). Error type: `data_type`.\n", "2. **Presence of Message List**: Checks if a `messages` list is present in each entry. Error type: `missing_messages_list`.\n", "3. **Message Keys Check**: Validates that each message in the `messages` list contains the keys `role` and `content`. Error type: `message_missing_key`.\n", - "4. **Unrecognized Keys in Messages**: Logs if a message has keys other than `role`, `content`, and `name`. Error type: `message_unrecognized_key`.\n", + "4. **Unrecognized Keys in Messages**: Logs if a message has keys other than `role`, `content`, `weight`, `function_call`, and `name`. Error type: `message_unrecognized_key`.\n", "5. **Role Validation**: Ensures the `role` is one of \"system\", \"user\", or \"assistant\". Error type: `unrecognized_role`.\n", "6. **Content Validation**: Verifies that `content` has textual data and is a string. Error type: `missing_content`.\n", "7. **Assistant Message Presence**: Checks that each conversation has at least one message from the assistant. Error type: `example_missing_assistant_message`.\n", @@ -123,7 +123,7 @@ " if \"role\" not in message or \"content\" not in message:\n", " format_errors[\"message_missing_key\"] += 1\n", " \n", - " if any(k not in (\"role\", \"content\", \"name\", \"function_call\") for k in message):\n", + " if any(k not in (\"role\", \"content\", \"name\", \"function_call\", \"weight\") for k in message):\n", " format_errors[\"message_unrecognized_key\"] += 1\n", " \n", " if message.get(\"role\", None) not in (\"system\", \"user\", \"assistant\", \"function\"):\n",