Improve error handling for failed downloads #95

deldesir · 2024-01-14T20:35:13Z

🚀 Pull Request Overview:

This pull request brings following enhancements:

Improved error handling with detailed error messages, including HTTP status codes and reasons, in case of download failures.
Set appropriate progress and messages when the download process encounters errors.

These updates aim to provide users with clearer information about download failures.

📋 Checklist:

Tested the changes on Ubuntu 23.10
Updated relevant documentation: https://github.com/iiab/calibre-web/wiki/Home/_compare/b612eaa9c7c890ccc5dcfe62c6130f0e9392b14b...ed8f1a8402c1ec526c143cf63e193c426be91d9a

🔗 Related Issues: #89

Some testing steps (thanks @EMG70):

Ensure that the script is running in an environment with internet access.
Initiate a download task with a valid YouTube or Vimeo URL
Confirm that the download progresses successfully.
Verify that the completion message reflects the successful download.
Check the Task column (in Tasks view) for success or error messages. Also, check the logs for the message "Successfully downloaded" or an error if any.

holta · 2024-01-14T23:23:06Z

Tangentially related:

PR Get lb-wrapper command directly : cps/tasks/download.py #94

EMG70 · 2024-01-15T01:04:34Z

New VM created in last 30mins.
iiab-diagnostics http://sprunge.us/1ZZzoZ?en

I have downloaded few videos and some that have failed in the past ,the ones that downloaded successfully showed progress for every video in the playlist.
A new message "almost done " now appears as playlist/video is being downloaded .
The failed downloads have error messages clearly explained in Adminstartion/viewlogs
https://www.youtube.com/playlist?list=PLg2tfDG3Ww4vsiMrgcS-wiU9boyII0z4A--- success
https://www.youtube.com/playlist?list=PLg2tfDG3Ww4t14_tIrNuqyJlAGmRn5KHu ---- success
https://www.youtube.com/watch?v=1RQVkzbU2jo&list=PLg2tfDG3Ww4uBdOZ-j-znfOarR7TKGs-w -----still fails but showing reasons for failure ok in viewlogs
https://vimeo.com/channels/staffpicks/893773360 ---Actual air vimeo staff picks still fails -reasons for failure showing in viewlogs

Overall very good progress ,implementers can see reasons for failure.

holta · 2024-01-15T02:11:07Z

iiab-diagnostics http://sprunge.us/1ZZzoZ?en

Please see above Lines 1079-1363.

(Starting here...)

https://sprunge.us/1ZZzoZ?en#n-1079

ASIDE:

PR iiab-diagnostics: Show Calibre-Web "version" (~50 recent commits) + systemd service status + 3 "log files" iiab#3697

holta · 2024-01-15T02:28:33Z

@deldesir the 100 last lines of /var/log/xklb.log appear to be counterproductive here — certainly not helping anybody in this example:

https://sprunge.us/1ZZzoZ?en#n-1261

QUESTIONS:

How do we eliminate this pollution?
If absolutely necessary, does iiab-diagnostics need to grep through xklb.log — so meaningful debugging hints are highlighted — and/or spammy pollution explicitly removed?

holta · 2024-01-15T02:58:31Z

100 last lines of /var/log/xklb.log

@deldesir I'm not talking about irqbalance. You're looking at Lines 1550-1649 which are off-topic.

I'm talking about the "100 last lines of /var/log/xklb.log" on Lines 1264-1363 here. See the direct link I provided earlier:

https://sprunge.us/1ZZzoZ?en#n-1261

holta · 2024-01-15T04:31:00Z

I see, ffmpeg output captured when using lb dl with -vv option. I agree it's verbose.

Should grep -v remove specific spammy patterns from xklb.log to make it much more concise?
Which patterns if so?

(iiab-diagnostics' approximately 100 commands each transmit a very limited number of lines!)

deldesir · 2024-01-15T04:42:46Z

Should grep -v remove specific spammy patterns from xklb.log to make it much more concise?

Let's start by removing ffmpeg ones

Which patterns if so?

All lines that contain ffmpeg

holta · 2024-01-15T05:10:19Z

Let's start by removing ffmpeg ones

It's an idea. Something like that definitely might be necessary in future.

But just for now, iiab-diagnostics will now collect the last 300 lines of /var/log/xklb.log towards helping us to understand the patterns first:

PR iiab-diagnostics: 300 last lines of /var/log/xklb.log for now iiab#3698

holta · 2024-01-15T05:28:57Z

@EMG70 this PR #95 is now merged: can you confirm below?

Some testing steps (thanks @EMG70):

Ensure that the script is running in an environment with internet access.

Initiate a download task with a valid YouTube or Vimeo URL

Confirm that the download progresses successfully.

Verify that the completion message reflects the successful download.

Check the Task column (in Tasks view) for success or error messages. Also, check the logs for the message "Successfully downloaded" or an error if any.

EMG70 · 2024-01-16T15:16:56Z

PR #95 Test as per newly added instructions to Calibre-web wiki
iiab-diagnostics
http://sprunge.us/5bxTq5?en
Ensure that the script is running in an environment with internet access. ✅
Initiate a download task with a valid YouTube or Vimeo URL ✅
Confirm that the download progresses successfully.✅
Verify that the completion message reflects the successful download.✅

Check the Task column (in Tasks view) for success or error messages. Also, check the logs for the message "Successfully downloaded" or an error if any.✅
This video https://www.youtube.com/watch?v=QXQrvT23rPw still shows 500 internal server error ,status failed although it successfully downloaded.It does not have a thumbnail (Picture not available)

EMG70 · 2024-01-16T15:24:05Z

Log error for failed status 👍

[2024-01-16 15:02:08,974] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download task for https://www.youtube.com/watch?v=QXQrvT23rPw
[2024-01-16 15:02:09,221]  INFO {cps.tasks.download:29} Starting download task for URL: https://www.youtube.com/watch?v=QXQrvT23rPw
[2024-01-16 15:02:09,221]  INFO {cps.tasks.download:38} Subprocess args: ['lb-wrapper', 'https://www.youtube.com/watch?v=QXQrvT23rPw']
[2024-01-16 15:02:28,276]  INFO {cps.editbooks:384} Requested files list: ['/library/downloads/calibre-web/Youtube/Leaf_of_Life/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQrvT23rPw].mp4', 'https://www.youtube.com/watch?v=QXQrvT23rPw']
[2024-01-16 15:02:28,276]  INFO {cps.editbooks:390} Processing file: <_io.BufferedReader name='/library/downloads/calibre-web/Youtube/Leaf_of_Life/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQrvT23rPw].mp4'>
[2024-01-16 15:02:28,276] DEBUG {cps.uploader:371} Temporary file: /tmp/calibre_web/2ecb7c6140e4252738fc807bb1a8aca8
[2024-01-16 15:02:28,279]  WARN {cps.uploader:98} cannot parse metadata, using default: list index out of range
[2024-01-16 15:02:28,286] DEBUG {cps.helper:543} Moving title: /tmp/calibre_web/2ecb7c6140e4252738fc807bb1a8aca8 to /library/calibre-web/Unknown/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQr (17)/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQr - Unknown
[2024-01-16 15:02:28,299] DEBUG {cps.services.worker:91} Add Task for user: Admin - Upload How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQrvT23rPw]
[2024-01-16 15:02:28,299] ERROR {cps:1414} Exception on /meta [GET]
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 473, in meta
    if move_mediafile(requested_files, current_user_name, shelf_title):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 386, in move_mediafile
    requested_file = open(requested_file, "rb")
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'https://www.youtube.com/watch?v=QXQrvT23rPw'
[2024-01-16 15:02:28,312] ERROR {cps.tasks.download:89} Failed to send the list of requested files to http://192.168.0.15/books/meta
[2024-01-16 15:17:44,575] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download task for https://www.youtube.com/watch?v=QXQrvT23rPw
[2024-01-16 15:17:44,576]  INFO {cps.tasks.download:29} Starting download task for URL: https://www.youtube.com/watch?v=QXQrvT23rPw
[2024-01-16 15:17:44,576]  INFO {cps.tasks.download:38} Subprocess args: ['lb-wrapper', 'https://www.youtube.com/watch?v=QXQrvT23rPw']
[2024-01-16 15:18:02,310]  INFO {cps.editbooks:384} Requested files list: ['/library/downloads/calibre-web/Youtube/Leaf_of_Life/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQrvT23rPw].mp4', 'https://www.youtube.com/watch?v=QXQrvT23rPw']
[2024-01-16 15:18:02,311]  INFO {cps.editbooks:390} Processing file: <_io.BufferedReader name='/library/downloads/calibre-web/Youtube/Leaf_of_Life/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQrvT23rPw].mp4'>
[2024-01-16 15:18:02,311] DEBUG {cps.uploader:371} Temporary file: /tmp/calibre_web/2ecb7c6140e4252738fc807bb1a8aca8
[2024-01-16 15:18:02,313]  WARN {cps.uploader:98} cannot parse metadata, using default: list index out of range
[2024-01-16 15:18:02,319] DEBUG {cps.helper:543} Moving title: /tmp/calibre_web/2ecb7c6140e4252738fc807bb1a8aca8 to /library/calibre-web/Unknown/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQr (18)/How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQr - Unknown
[2024-01-16 15:18:02,329] DEBUG {cps.services.worker:91} Add Task for user: Admin - Upload How_the_Sahara_Desert_is_Turning_into_a_Farmland_Oasis_-_GREENING_THE_DESERT_PROJECT_5.81M_[QXQrvT23rPw]
[2024-01-16 15:18:02,330] ERROR {cps:1414} Exception on /meta [GET]
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 473, in meta
    if move_mediafile(requested_files, current_user_name, shelf_title):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 386, in move_mediafile
    requested_file = open(requested_file, "rb")
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'https://www.youtube.com/watch?v=QXQrvT23rPw'
[2024-01-16 15:18:02,333] ERROR {cps.tasks.download:89} Failed to send the list of requested files to http://192.168.0.15/books/meta
[2024-01-16 15:20:28,771] DEBUG {cps.updater:329} Stable version: {'version': '0.6.22 Beta'}

EMG70 · 2024-01-16T15:37:51Z

Improve error handling

daef51f

deldesir added the enhancement New feature or request label Jan 14, 2024

deldesir requested a review from holta January 14, 2024 20:35

deldesir self-assigned this Jan 14, 2024

deldesir changed the title ~~Improve error handling~~ Improve error handling for failed downloads Jan 14, 2024

deldesir marked this pull request as draft January 14, 2024 20:36

Merge branch 'iiab:master' into deldesir-error-handling

47f07bb

deldesir marked this pull request as ready for review January 15, 2024 04:44

holta mentioned this pull request Jan 15, 2024

iiab-diagnostics: 300 last lines of /var/log/xklb.log for now iiab/iiab#3698

Merged

Remove flash import

b3f39b3

holta mentioned this pull request Jan 15, 2024

Add xklb version [and more, to friendlier & more efficient logging] #70

Closed

holta merged commit 659c00c into iiab:master Jan 15, 2024

avni mentioned this pull request Jan 20, 2024

Videos sometimes fail to "Download to IIAB" w/ message: 405 METHOD NOT ALLOWED #106

Closed

deldesir deleted the deldesir-error-handling branch July 1, 2024 21:42

Improve error handling for failed downloads #95

Improve error handling for failed downloads #95

Uh oh!

Conversation

deldesir commented Jan 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jan 14, 2024

Uh oh!

EMG70 commented Jan 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jan 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jan 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jan 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jan 15, 2024

Uh oh!

deldesir commented Jan 15, 2024

Uh oh!

holta commented Jan 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jan 15, 2024

Uh oh!

EMG70 commented Jan 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EMG70 commented Jan 16, 2024 • edited by holta Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EMG70 commented Jan 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deldesir commented Jan 14, 2024 •

edited

Loading

EMG70 commented Jan 15, 2024 •

edited

Loading

holta commented Jan 15, 2024 •

edited

Loading

holta commented Jan 15, 2024 •

edited

Loading

holta commented Jan 15, 2024 •

edited

Loading

holta commented Jan 15, 2024 •

edited

Loading

EMG70 commented Jan 16, 2024 •

edited

Loading

EMG70 commented Jan 16, 2024 •

edited by holta

Loading