fix iteration error (#208)

* fix iteration error

array iteration error: sentences up till max_len of token exceeds should be returned

* split list once instead of twice

split list once instead of twice and take parsed string up till max token length is exceeded
pull/1077/head
Swastik Banerjee 1 year ago committed by GitHub
parent 3f2ec8c9e5
commit da56bd7b54

@ -154,7 +154,7 @@
" for i, sentence in enumerate(sentences):\n",
" ntokens += 1 + count_tokens(sentence)\n",
" if ntokens > max_len:\n",
" return \". \".join(sentences[:i][:-1]) + \".\"\n",
" return \". \".join(sentences[:i]) + \".\"\n",
"\n",
" return long_text\n",
"\n",

Loading…
Cancel
Save