Merge pull request #350 from eugeneyan/main

Update the section token count from 1,000 to 1,600
pull/1077/head
Ted Sanders 1 year ago committed by GitHub
commit 82d81a9ac0

@ -411,7 +411,7 @@
"- Shorter sections allow more sections to be retrieved, which may help with recall\n",
"- Overlapping sections may help prevent answers from being cut by section boundaries\n",
"\n",
"Here, we'll use a simple approach and limit sections to 1,000 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible."
"Here, we'll use a simple approach and limit sections to 1,600 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible."
]
},
{

Loading…
Cancel
Save