From da56bd7b54006815382e5145b5738425fb33a826 Mon Sep 17 00:00:00 2001 From: Swastik Banerjee Date: Wed, 22 Mar 2023 03:15:56 +0530 Subject: [PATCH] fix iteration error (#208) * fix iteration error array iteration error: sentences up till max_len of token exceeds should be returned * split list once instead of twice split list once instead of twice and take parsed string up till max token length is exceeded --- examples/fine-tuned_qa/olympics-1-collect-data.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/fine-tuned_qa/olympics-1-collect-data.ipynb b/examples/fine-tuned_qa/olympics-1-collect-data.ipynb index 2be05269..ce78cbac 100644 --- a/examples/fine-tuned_qa/olympics-1-collect-data.ipynb +++ b/examples/fine-tuned_qa/olympics-1-collect-data.ipynb @@ -154,7 +154,7 @@ " for i, sentence in enumerate(sentences):\n", " ntokens += 1 + count_tokens(sentence)\n", " if ntokens > max_len:\n", - " return \". \".join(sentences[:i][:-1]) + \".\"\n", + " return \". \".join(sentences[:i]) + \".\"\n", "\n", " return long_text\n", "\n",