Update chat finetune data prep notebook (#1074)

Co-authored-by: andrewpeng@openai.com <Andrew Peng>
pull/1021/head^2
Andrew Peng 2 months ago committed by GitHub
parent 1e1dd5a33d
commit d25d32ed5f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

@ -83,7 +83,7 @@
"1. **Data Type Check**: Checks whether each entry in the dataset is a dictionary (`dict`). Error type: `data_type`.\n",
"2. **Presence of Message List**: Checks if a `messages` list is present in each entry. Error type: `missing_messages_list`.\n",
"3. **Message Keys Check**: Validates that each message in the `messages` list contains the keys `role` and `content`. Error type: `message_missing_key`.\n",
"4. **Unrecognized Keys in Messages**: Logs if a message has keys other than `role`, `content`, and `name`. Error type: `message_unrecognized_key`.\n",
"4. **Unrecognized Keys in Messages**: Logs if a message has keys other than `role`, `content`, `weight`, `function_call`, and `name`. Error type: `message_unrecognized_key`.\n",
"5. **Role Validation**: Ensures the `role` is one of \"system\", \"user\", or \"assistant\". Error type: `unrecognized_role`.\n",
"6. **Content Validation**: Verifies that `content` has textual data and is a string. Error type: `missing_content`.\n",
"7. **Assistant Message Presence**: Checks that each conversation has at least one message from the assistant. Error type: `example_missing_assistant_message`.\n",
@ -123,7 +123,7 @@
" if \"role\" not in message or \"content\" not in message:\n",
" format_errors[\"message_missing_key\"] += 1\n",
" \n",
" if any(k not in (\"role\", \"content\", \"name\", \"function_call\") for k in message):\n",
" if any(k not in (\"role\", \"content\", \"name\", \"function_call\", \"weight\") for k in message):\n",
" format_errors[\"message_unrecognized_key\"] += 1\n",
" \n",
" if message.get(\"role\", None) not in (\"system\", \"user\", \"assistant\", \"function\"):\n",

Loading…
Cancel
Save