New search API endpoint, #build

Changed:
- Switched search form to API
- Additional validations for RYD
- Additional log output for curl ES startup check

README.md

@@ -146,12 +146,12 @@ To apply the change permanently depends on your host operating system:
 - On any other platform look up in the documentation on how to pass kernel parameters.
 
 ### Permissions for elasticsearch
-If you see a message similar to `AccessDeniedException[/usr/share/elasticsearch/data/nodes]` when initially starting elasticsearch, that means the container is not allowed to write files to the volume.  
-That's most likely the case when you run `docker-compose` as an unprivileged user. To fix that issue, shutdown the container and on your host machine run:
+If you see a message similar to `failed to obtain node locks, tried [/usr/share/elasticsearch/data]` and `maybe these locations are not writable` when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.  
+To fix that issue, shutdown the container and on your host machine run:
 ```
-chown 1000:0 /path/to/mount/point
+chown 1000:0 -R /path/to/mount/point
 ```
-This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
+This will match the permissions with the **UID** and **GID** of elasticsearch process within the container and should fix the issue.
 
 ### Disk usage
 The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`, [link](https://github.com/tubearchivist/tubearchivist#disk-usage).  

docker_assets/run.sh

@@ -33,6 +33,7 @@ until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
     if [[ $counter -eq 12 ]]; then
         # fail after 2 min
         echo "failed to connect to elastic search, exiting..."
+        curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL"?pretty
         exit 1
     fi
     sleep 10

tubearchivist/api/README.md

@@ -5,6 +5,36 @@ Note:
 - This is very early stages and will change!
 - Check the commit history to see if a documented feature is already in your release
 
+## Table of contents
+- [Authentication](#authentication)
+- [Pagination](#pagination)
+
+**Video**
+- [Video List](#video-list-view)
+- [Video Single](#video-item-view)
+- [Video Single Progress](#video-progress-view)
+- [Video Single Sponsorblock](#sponsor-block-view) WIP
+
+**Channel**
+- [Channel List](#channel-list-view)
+- [Channel Single](#channel-item-view)
+- [Channel Video List](#channel-videos-view)
+
+**Playlist**
+- [Playlist List](#playlist-list-view)
+- [Playlist Single](#playlist-item-view)
+- [Playlist Videos List](#playlist-videos-view)
+
+**Download queue**
+- [Download Queue List](#download-queue-list-view)
+- [Download Queue Single](#download-queue-item-view)
+
+**Additional**
+- [Login](#login-view)
+- [Task](#task-view) WIP
+- [Cookie](#cookie-view)
+- [Ping](#ping-view)
+
 ## Authentication
 API token will get automatically created, accessible on the settings page. Token needs to be passed as an authorization header with every request. Additionally session based authentication is enabled too: When you are logged into your TubeArchivist instance, you'll have access to the api in the browser for testing.
 
@@ -25,35 +55,17 @@ response = requests.get(url, headers=headers)
 
 ## Pagination
 The list views return a paginate object with the following keys:
-- page_size: int current page size set in config
-- page_from: int first result idx
-- prev_pages: array of ints of previous pages, if available
-- current_page: int current page from query
-- max_hits: reached: bool if max of 10k results is reached
-- last_page: int of last page link
-- next_pages: array of ints of next pages
-- total_hits: int total results
+- page_size: *int* current page size set in config
+- page_from: *int* first result idx
+- prev_pages: *array of ints* of previous pages, if available
+- current_page: *int* current page from query
+- max_hits: *bool* if max of 10k results is reached
+- last_page: *int* of last page link
+- next_pages: *array of ints* of next pages
+- total_hits: *int* total results
 
 Pass page number as a query parameter: `page=2`. Defaults to *0*, `page=1` is redundant and falls back to *0*. If a page query doesn't return any results, you'll get `HTTP 404 Not Found`.
 
-## Login View
-Return token and user ID for username and password:  
-POST /api/login
-```json
-{
-    "username": "tubearchivist",
-    "password": "verysecret"
-}
-```
-
-after successful login returns 
-```json
-{
-    "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
-    "user_id": 1
-}
-```
-
 ## Video List View
 /api/video/
 
@@ -145,7 +157,7 @@ POST /api/channel/
 ## Playlist List View
 /api/playlist/
 
-## Playlists Item View
+## Playlist Item View
 /api/playlist/\<playlist_id>/
 
 ## Playlist Videos View
@@ -192,15 +204,21 @@ Add to queue previously ignored video:
 DELETE /api/download/\<video_id>/  
 Forget or delete from download queue
 
-## Ping View
-Validate your connection with the API  
-GET /api/ping
-
-When valid returns message with user id: 
+## Login View
+Return token and user ID for username and password:  
+POST /api/login
 ```json
 {
-    "response": "pong",
-    "user": 1
+    "username": "tubearchivist",
+    "password": "verysecret"
+}
+```
+
+after successful login returns 
+```json
+{
+    "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
+    "user_id": 1
 }
 ```
 
@@ -273,3 +291,15 @@ Or returns status code 400 on failure:
     "cookie_validated": false
 }
 ```
+
+## Ping View
+Validate your connection with the API  
+GET /api/ping
+
+When valid returns message with user id: 
+```json
+{
+    "response": "pong",
+    "user": 1
+}
+```

tubearchivist/api/src/search_processor.py

@@ -22,7 +22,7 @@ class SearchProcess:
         self.processed = False
 
     def process(self):
-        """dedect type and process"""
+        """detect type and process"""
         if "_source" in self.response.keys():
             # single
             self.processed = self._process_result(self.response)
@@ -37,7 +37,7 @@ class SearchProcess:
         return self.processed
 
     def _process_result(self, result):
-        """dedect which type of data to process"""
+        """detect which type of data to process"""
         index = result["_index"]
         processed = False
         if index == "ta_video":

tubearchivist/home/src/index/filesystem.py

@@ -363,7 +363,7 @@ class ImportFolderScanner:
         return new_path
 
     def get_mp4_thumb_type(self, media_path):
-        """dedect filetype of embedded thumbnail"""
+        """detect filetype of embedded thumbnail"""
         streams = self._get_streams(media_path)
 
         for stream in streams["streams"]:

tubearchivist/home/src/index/video.py

@@ -320,15 +320,16 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
 
     def _get_ryd_stats(self):
         """get optional stats from returnyoutubedislikeapi.com"""
+        # pylint: disable=broad-except
         try:
             print(f"{self.youtube_id}: get ryd stats")
             result = ryd_client.get(self.youtube_id)
-        except requests.exceptions.ConnectionError:
-            print(f"{self.youtube_id}: failed to query ryd api, skipping")
-            return False
+        except Exception as err:
+            print(f"{self.youtube_id}: failed to query ryd api {err}")
+            return
 
         if result["status"] == 404:
-            return False
+            return
 
         dislikes = {
             "dislike_count": result.get("dislikes", 0),
@@ -336,8 +337,6 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         }
         self.json_data["stats"].update(dislikes)
 
-        return True
-
     def _get_sponsorblock(self):
         """get optional sponsorblock timestamps from sponsor.ajay.app"""
         sponsorblock = SponsorBlock().get_timestamps(self.youtube_id)

tubearchivist/home/src/ta/helper.py

@@ -173,13 +173,13 @@ class UrlListParser:
             _ = self.find_valid_id(youtube_id)
             return youtube_id, "channel"
 
-        # dedect channel with yt_dlp
+        # detect channel with yt_dlp
         youtube_id = self.extract_channel_name(parsed.geturl())
         return youtube_id, "channel"
 
     @staticmethod
     def find_valid_id(id_str):
-        """dedect valid id from length of string"""
+        """detect valid id from length of string"""
         str_len = len(id_str)
         if str_len == 11:
             id_type = "video"

tubearchivist/requirements.txt

@@ -10,4 +10,4 @@ requests==2.28.1
 ryd-client==0.0.6
 uWSGI==2.0.20
 whitenoise==6.2.0
-yt_dlp==2022.8.14
+yt_dlp==2022.8.19

tubearchivist/static/cast-videos.js

@@ -1,6 +1,6 @@
 function initializeCastApi() {
     cast.framework.CastContext.getInstance().setOptions({
-        receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in reciver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
+        receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in receiver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
         autoJoinPolicy: chrome.cast.AutoJoinPolicy.ORIGIN_SCOPED
     });