From 155439e228811fb031654562cf34715d717af21c Mon Sep 17 00:00:00 2001 From: simon Date: Wed, 17 Aug 2022 17:29:04 +0700 Subject: [PATCH 1/6] add api Table of contents --- tubearchivist/api/README.md | 96 ++++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 33 deletions(-) diff --git a/tubearchivist/api/README.md b/tubearchivist/api/README.md index a5bc1b0a..615559ad 100644 --- a/tubearchivist/api/README.md +++ b/tubearchivist/api/README.md @@ -5,6 +5,36 @@ Note: - This is very early stages and will change! - Check the commit history to see if a documented feature is already in your release +## Table of contents +- [Authentication](#authentication) +- [Pagination](#pagination) + +**Video** +- [Video List](#video-list-view) +- [Video Single](#video-item-view) +- [Video Single Progress](#video-progress-view) +- [Video Single Sponsorblock](#sponsor-block-view) WIP + +**Channel** +- [Channel List](#channel-list-view) +- [Channel Single](#channel-item-view) +- [Channel Video List](#channel-videos-view) + +**Playlist** +- [Playlist List](#playlist-list-view) +- [Playlist Single](#playlist-item-view) +- [Playlist Videos List](#playlist-videos-view) + +**Download queue** +- [Download Queue List](#download-queue-list-view) +- [Download Queue Single](#download-queue-item-view) + +**Additional** +- [Login](#login-view) +- [Task](#task-view) WIP +- [Cookie](#cookie-view) +- [Ping](#ping-view) + ## Authentication API token will get automatically created, accessible on the settings page. Token needs to be passed as an authorization header with every request. Additionally session based authentication is enabled too: When you are logged into your TubeArchivist instance, you'll have access to the api in the browser for testing. @@ -25,35 +55,17 @@ response = requests.get(url, headers=headers) ## Pagination The list views return a paginate object with the following keys: -- page_size: int current page size set in config -- page_from: int first result idx -- prev_pages: array of ints of previous pages, if available -- current_page: int current page from query -- max_hits: reached: bool if max of 10k results is reached -- last_page: int of last page link -- next_pages: array of ints of next pages -- total_hits: int total results +- page_size: *int* current page size set in config +- page_from: *int* first result idx +- prev_pages: *array of ints* of previous pages, if available +- current_page: *int* current page from query +- max_hits: *bool* if max of 10k results is reached +- last_page: *int* of last page link +- next_pages: *array of ints* of next pages +- total_hits: *int* total results Pass page number as a query parameter: `page=2`. Defaults to *0*, `page=1` is redundant and falls back to *0*. If a page query doesn't return any results, you'll get `HTTP 404 Not Found`. -## Login View -Return token and user ID for username and password: -POST /api/login -```json -{ - "username": "tubearchivist", - "password": "verysecret" -} -``` - -after successful login returns -```json -{ - "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", - "user_id": 1 -} -``` - ## Video List View /api/video/ @@ -145,7 +157,7 @@ POST /api/channel/ ## Playlist List View /api/playlist/ -## Playlists Item View +## Playlist Item View /api/playlist/\/ ## Playlist Videos View @@ -192,15 +204,21 @@ Add to queue previously ignored video: DELETE /api/download/\/ Forget or delete from download queue -## Ping View -Validate your connection with the API -GET /api/ping +## Login View +Return token and user ID for username and password: +POST /api/login +```json +{ + "username": "tubearchivist", + "password": "verysecret" +} +``` -When valid returns message with user id: +after successful login returns ```json { - "response": "pong", - "user": 1 + "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", + "user_id": 1 } ``` @@ -273,3 +291,15 @@ Or returns status code 400 on failure: "cookie_validated": false } ``` + +## Ping View +Validate your connection with the API +GET /api/ping + +When valid returns message with user id: +```json +{ + "response": "pong", + "user": 1 +} +``` From 650ac94e6803299a72a068a56be475e540f59f40 Mon Sep 17 00:00:00 2001 From: simon Date: Wed, 17 Aug 2022 18:48:34 +0700 Subject: [PATCH 2/6] better wording for es permission error --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 93a6f779..e867e111 100644 --- a/README.md +++ b/README.md @@ -146,12 +146,12 @@ To apply the change permanently depends on your host operating system: - On any other platform look up in the documentation on how to pass kernel parameters. ### Permissions for elasticsearch -If you see a message similar to `AccessDeniedException[/usr/share/elasticsearch/data/nodes]` when initially starting elasticsearch, that means the container is not allowed to write files to the volume. -That's most likely the case when you run `docker-compose` as an unprivileged user. To fix that issue, shutdown the container and on your host machine run: +If you see a message similar to `failed to obtain node locks, tried [/usr/share/elasticsearch/data]` and `maybe these locations are not writable` when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume. +To fix that issue, shutdown the container and on your host machine run: ``` -chown 1000:0 /path/to/mount/point +chown 1000:0 -R /path/to/mount/point ``` -This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue. +This will match the permissions with the **UID** and **GID** of elasticsearch process within the container and should fix the issue. ### Disk usage The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`, [link](https://github.com/tubearchivist/tubearchivist#disk-usage). From 115db944c814a9854f44752e5e669a7ff5e61261 Mon Sep 17 00:00:00 2001 From: simon Date: Thu, 18 Aug 2022 18:35:09 +0700 Subject: [PATCH 3/6] add curl output for failed es connection --- docker_assets/run.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/docker_assets/run.sh b/docker_assets/run.sh index effe8435..f8c96a96 100644 --- a/docker_assets/run.sh +++ b/docker_assets/run.sh @@ -33,6 +33,7 @@ until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do if [[ $counter -eq 12 ]]; then # fail after 2 min echo "failed to connect to elastic search, exiting..." + curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL"?pretty exit 1 fi sleep 10 From 0e7e0a05604c2458d91ea83a04bb047ad3fab3f6 Mon Sep 17 00:00:00 2001 From: simon Date: Thu, 18 Aug 2022 18:45:46 +0700 Subject: [PATCH 4/6] fail open on ryd error --- tubearchivist/home/src/index/video.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tubearchivist/home/src/index/video.py b/tubearchivist/home/src/index/video.py index a68cae2c..c78cea82 100644 --- a/tubearchivist/home/src/index/video.py +++ b/tubearchivist/home/src/index/video.py @@ -320,15 +320,16 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle): def _get_ryd_stats(self): """get optional stats from returnyoutubedislikeapi.com""" + # pylint: disable=broad-except try: print(f"{self.youtube_id}: get ryd stats") result = ryd_client.get(self.youtube_id) - except requests.exceptions.ConnectionError: - print(f"{self.youtube_id}: failed to query ryd api, skipping") - return False + except Exception as err: + print(f"{self.youtube_id}: failed to query ryd api {err}") + return if result["status"] == 404: - return False + return dislikes = { "dislike_count": result.get("dislikes", 0), @@ -336,8 +337,6 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle): } self.json_data["stats"].update(dislikes) - return True - def _get_sponsorblock(self): """get optional sponsorblock timestamps from sponsor.ajay.app""" sponsorblock = SponsorBlock().get_timestamps(self.youtube_id) From 1896e982b9e7990f5e5520e81cf0b7d220805bde Mon Sep 17 00:00:00 2001 From: simon Date: Thu, 18 Aug 2022 18:51:11 +0700 Subject: [PATCH 5/6] codespell lint fixes --- tubearchivist/api/src/search_processor.py | 4 ++-- tubearchivist/home/src/index/filesystem.py | 2 +- tubearchivist/home/src/ta/helper.py | 4 ++-- tubearchivist/static/cast-videos.js | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tubearchivist/api/src/search_processor.py b/tubearchivist/api/src/search_processor.py index 07102310..7a41163c 100644 --- a/tubearchivist/api/src/search_processor.py +++ b/tubearchivist/api/src/search_processor.py @@ -22,7 +22,7 @@ class SearchProcess: self.processed = False def process(self): - """dedect type and process""" + """detect type and process""" if "_source" in self.response.keys(): # single self.processed = self._process_result(self.response) @@ -37,7 +37,7 @@ class SearchProcess: return self.processed def _process_result(self, result): - """dedect which type of data to process""" + """detect which type of data to process""" index = result["_index"] processed = False if index == "ta_video": diff --git a/tubearchivist/home/src/index/filesystem.py b/tubearchivist/home/src/index/filesystem.py index 4f6c88c0..e6042cca 100644 --- a/tubearchivist/home/src/index/filesystem.py +++ b/tubearchivist/home/src/index/filesystem.py @@ -363,7 +363,7 @@ class ImportFolderScanner: return new_path def get_mp4_thumb_type(self, media_path): - """dedect filetype of embedded thumbnail""" + """detect filetype of embedded thumbnail""" streams = self._get_streams(media_path) for stream in streams["streams"]: diff --git a/tubearchivist/home/src/ta/helper.py b/tubearchivist/home/src/ta/helper.py index 314ecd79..8e51e3ef 100644 --- a/tubearchivist/home/src/ta/helper.py +++ b/tubearchivist/home/src/ta/helper.py @@ -173,13 +173,13 @@ class UrlListParser: _ = self.find_valid_id(youtube_id) return youtube_id, "channel" - # dedect channel with yt_dlp + # detect channel with yt_dlp youtube_id = self.extract_channel_name(parsed.geturl()) return youtube_id, "channel" @staticmethod def find_valid_id(id_str): - """dedect valid id from length of string""" + """detect valid id from length of string""" str_len = len(id_str) if str_len == 11: id_type = "video" diff --git a/tubearchivist/static/cast-videos.js b/tubearchivist/static/cast-videos.js index 867093e3..4334d042 100644 --- a/tubearchivist/static/cast-videos.js +++ b/tubearchivist/static/cast-videos.js @@ -1,6 +1,6 @@ function initializeCastApi() { cast.framework.CastContext.getInstance().setOptions({ - receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in reciver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee. + receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in receiver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee. autoJoinPolicy: chrome.cast.AutoJoinPolicy.ORIGIN_SCOPED }); From e6b673832f58b3a676340f49e73ff2e3ed45a0e8 Mon Sep 17 00:00:00 2001 From: simon Date: Fri, 19 Aug 2022 17:53:18 +0700 Subject: [PATCH 6/6] bump yt-dlp --- tubearchivist/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tubearchivist/requirements.txt b/tubearchivist/requirements.txt index 0d9a260f..d511d8c1 100644 --- a/tubearchivist/requirements.txt +++ b/tubearchivist/requirements.txt @@ -10,4 +10,4 @@ requests==2.28.1 ryd-client==0.0.6 uWSGI==2.0.20 whitenoise==6.2.0 -yt_dlp==2022.8.14 +yt_dlp==2022.8.19