Commit Graph

2266 Commits (e215a20a0ac3dd4a3141f842ec6dd90b54cb67c3)

Author SHA1 Message Date
Samantaz Fox f54e247eb4
Extractors: Add support for shorts
Fixes 
bbielsa b90bceb2dc Fix formatting of preferences.cr and videos.cr
bbielsa f31bd5ffb9 Use localization for save player position label in the preferences page
bbielsa 5abe7fe123 Rename 'remember_position' to 'save_player_pos' for clarity
bbielsa a6a0bbf398 Add remember_position field to the Preferences and VideoPreferences structs, and add a checkbox in the preferences page to toggle it
Samantaz Fox ee91effb7a
Merge pull request from SamantazFox/fix-locales-handling
Fix locales handling
Samantaz Fox f236a6872b
Merge pull request from SamantazFox/fix-likes-dislikes
Fix likes/dislikes
Samantaz Fox 3e0096f360
Merge pull request from iv-org/SamantazFox-patch-1
Fix 
Samantaz Fox 438b334320
Merge pull request from matthewmcgarvey/code-removal
Remove dead code
Samantaz Fox 4aa96ecab9
Use 'dig()' in 'find()' statements
Samantaz Fox 7b9d26d688
Fix
Fixes "Download widget replaces spaces in filename with +"
https://github.com/iv-org/invidious/issues/2670
matthewmcgarvey 8d4b4cd14c Remove dead code
Samantaz Fox 342fc202a7
Fix
Fix "Missing param name: "q" (KeyError)"
https://github.com/iv-org/invidious/issues/2682
Samantaz Fox 4436359d07
Use dig to get category contents
Co-authored-by: Matthew McGarvey <matthewmcgarvey14@gmail.com>
Samantaz Fox 91f8395222
Typo: missing '?' when looking for key in dislikes_button
Co-authored-by: Matthew McGarvey <matthewmcgarvey14@gmail.com>
Samantaz Fox de00e86cd5
Decompress the response body ourselves
Temp fix for 
Émilien Devos c6e086c6ff
Revert "Temporarily fix for #2612" ()
Samantaz Fox 82f3eda82b
Merge pull request from SamantazFox/fix-2549
extract_video_info: Make sure that the Android player response is valid
Samantaz Fox 05f9613e14
Merge pull request from SamantazFox/temp-decompression-fix
Temporarily fix for 
Samantaz Fox ceb1feb350
likes/dislikes: better fallback management
'.to_i64?' instead of '.to_i64' returns nil rather than raising
an exception when it's done on an empty string.

In some rare cases, rating can be equal to 5. In this case, the
value of player_response[videoDetails][averageRating] is an
Int and not a Float.
Samantaz Fox 2ea0590b03
i18n: return 'key' if 'key' is not in locales files
Samantaz Fox 80a513baa5
Use new techniques to get (dis)likes back
Samantaz Fox ba48f68fc3
allow multiple, successive content-encodings
Samantaz Fox 319587e2f1
extract_video_info: make sure that the Android player response is valid
Samantaz Fox bf7952d9c7
i18n: log a warning instead of rising an exception
This is more user-friendly.
TODO: maybe make a compile time flag for testing purposes
Samantaz Fox f29ab53aff
Add other missing translations
* on watch page and video cards (search results, playlists, etc...)
* on /feed/playlists
* in search filters (not normalized in order to avoid collisions with
an existing PR that reworks the search filters)
Samantaz Fox b5b0c58de7
Add missing translation for quality selectors
Samantaz Fox a1bb421eec
Remove useless 'hl' parameters on captions URL
Samantaz Fox 139786b9ef
i18n: pass only the ISO code string to 'translate()'
Don't use the whole Hash everywhere.
Also fall back nicely to english string if no translation exists.
Samantaz Fox 301444563b
i18n: Use language full name instead of ISO code
Fixes 
Samantaz Fox 9966c21c6b
i18n: Add list of language names
babababag fd54cf2d05
Escape video description
Samantaz Fox 2c447a42f2
Make sure to only apply fix if QUIC is disabled
Samantaz Fox dad8f9a0ce
Fix typo
Should be checking the returned headers, not the sent ones.
Samantaz Fox 2eac23a0b3
Temporary fix for
Don't rely on the auto compression/decompression provided by the crystal stdlib.
Samantaz Fox 00904ae3f2
Merge pull request from syeopite/only-use-redirect-endpoint-when-needed
Only use the /redirect endpoint when automatically redirecting to another instance
Émilien Devos d214a0b333
remove duplicate lsquic requirement
syeopite a120f143d7
Disable quic by default
See 
syeopite 65fbdbff6a
Remove of gzip header w/ use_quic config
Continuation of b0f127d4d8
syeopite 6ec4dcfafd
Fix handling for maxres thumbnail
syeopite 48191aca6e
Fix copy-paste error
syeopite 83556bace2
Allow thumbnail queries with QUIC disabled
syeopite 814c9e6c3a
Use https for storyboard image requests
syeopite 547abe17d9
Use https for ggpht requests
syeopite 6b8450558d
Allow storyboard queries with QUIC disabled
syeopite c3747c2d49
Allow ggpht queries with QUIC disabled
syeopite 245122104a
Respect use_quic param and fix typos
syeopite b0f127d4d8
Fix gzip decompression with HTTP::Client
syeopite d379a36c0e
Add compile-time flag to remove code for QUIC
Samantaz Fox 6cf0ff6b49
Remove useless auto_generated param from PlaylistVideo#to_xml
given the variables available in this function's context, 'author' and 'ucid'
provide the same data 'self.author' and 'self.ucid', respectively.

Given that fact, the variable `auto_generated` has no impact on the logic of
this function, and hence can be safely removed. this greatly simplifies the
code and makes it perfectly compatible with crystal's calling convention for
'#to_xml' methods.
Samantaz Fox 86f75758a7
Fix 'to_json' in struct PlaylistVideo
Samantaz Fox 0ec94405ce
Add TODO comments to other places
Samantaz Fox 33780f1995
Also fix 'to_json' in struct Video
Samantaz Fox 1cb715ac9f
serialized_yt_data: force datatype of 'locale'
Samantaz Fox f65b628bf3
serialized_yt_data: Remove default nil value in to_json
this will ensure that two parameters are passed and that it
doesn't collide with 'to_json(builder)'
Samantaz Fox 60fa552469
Fix an infinite recursion caused by
Changes in the aforementioned PR lead to change the behavior
of some old code.

The data type of the parameters aren't explicit enough, which
makes the compiler use the wrong method because of type infering.
syeopite 437bdedb07
Use env.request.resource for instance switch link
syeopite a531f4f057
Rebase error
syeopite bb396a310e
Remove unneeded redirect.invidious.io code in /redirect
syeopite 66e7285108
Only use /redirect when automatically redirecting
Samantaz Fox 2b81a82620
Merge pull request from DmitrySandalov/master
Feature Request: Localization for trending iv-org#331
Samantaz Fox b555df8377
Merge pull request from syeopite/ameba-fixes
Fix (some) issues detected by Ameba
Samantaz Fox 232730e909
Merge branch 'master' into master
Samantaz Fox 1e0712625a
Normalize translation key for preferences categories
categories normalized:
 - Miscellanous
 - Player
 - Visual
 - Administrator
 - Data
 - Subscription
Samantaz Fox 88752f32bd
Normalize translation key for user prefrerences
preferences normalized:
 - annotations
 - annotations_subscribed
 - automatic_instance_redirect
 - autoplay
 - captions
 - comments
 - continue
 - continue_autoplay
 - dark_mode
 - default_home
 - extend_desc
 - feed_menu
 - listen
 - local
 - locale
 - max_results
 - notifications_only
 - player_style
 - quality
 - quality_dash
 - related_videos
 - show_nick
 - sort
 - speed
 - thin_mode
 - unseen_only
 - video_loop
 - volume
 - vr_mode
syeopite e2bbc9a6fa
Switch to #sort_by! in pull_popular_videos job
syeopite 17e481c107
Reduce block verbosity further
syeopite b8f27a42a7
Optimize assigning of on_response_received_endpoints
syeopite 575c66efd3
Remove extra dot between [] from ambea copy-pasting.
syeopite 88ad7c8d8d
Update src/invidious/routes/playlists.cr
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
syeopite 1adcac175e
Update src/invidious/routes/channels.cr
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
syeopite dd8c412abc
Fix Style/IsAFilter issues
syeopite e969c1490a
Fix Performance/ChainedCallWithNoBang issues
syeopite e91421253e
Fix Style/VerboseBlock issues
syeopite 35d15c7c2b
Fix Style/VariableNames issues
syeopite 20cb751ff6
Fix Lint/UnusedArgument issues
syeopite 0614b52f03
Fix Lint/RedundantStringCoercion issues
Dmitry Sandalov a629521c37
Feature Request: Localization for trending (remove region from server config)
Dmitry Sandalov 0e17d026f2
Feature Request: Localization for trending (fix lint CONTENT_REGIONS end comma)
Dmitry Sandalov 1d78d67af3
Feature Request: Localization for trending (move TRENDING_REGIONS to i18n.cr as CONTENT_REGIONS)
Dmitry Sandalov d9c58c4837
Feature Request: Localization for trending iv-org#331
syeopite 129779a757
Merge pull request from iv-org/increase-channel-refresh-delay
Decrease channel refresh frequency (1 min -> 1 h)
syeopite c8a985cc77
Merge pull request from SamantazFox/move-prefs-and-config
Move user preferences and server config
Samantaz Fox cb9b84f940
Move 'extract_text()' to 'extractors_utils.cr'
Samantaz Fox 33d9be0ffb
Move 'extractors.cr' to 'yt_backend' folder
Samantaz Fox 2571e420f3
Move 'add_yt_headers()' to 'connection_pool.cr'
Samantaz Fox e17c8b1f4d
Move 'extract_videos' and 'extract_selected_tab' too
Samantaz Fox 8805ee7c8c
Add fetch_continuation_token back (required by )
Samantaz Fox 7df2fd0bc8
Add 'require' statement to 'invidious.cr'
Samantaz Fox f7f0910953
Remove fetch_continuation_token(): dead code
Samantaz Fox d300797e22
Move the YoutubeConnectionPool code to its own file
Samantaz Fox 56dbe15943
Move proxy & YoutubeAPI code to the yt_backend folder
syeopite 5f65e92500
Merge pull request from syeopite/lazy-load-images
Lazy load (some) images
syeopite 21b96a3159
Merge pull request from syeopite/handle-empty-playlist-tabs
Handle YT tabs without content
syeopite 888c7e2c29
Merge pull request from syeopite/extract-image-routes
Extract image routes
TheFrenchGhosty 22e8f7e287 Fix
TheFrenchGhosty 0ad2793b68 Link to invidious.io/donate/ in the footer
syeopite 585e4617e8
Lazy load (some) images
syeopite 4d44b2c3a4
Handle YT tabs without any content
syeopite b49b5fbda9
Support empty categories
syeopite 17e6093abb
Merge pull request from Yetangitu/videorendererparser_parse_missing_title
Handle missing title fields in VideoRendererParser.parse (fixes )
Frank de Lange 3dc980e800
Fix for - parse contents of search results of type=Category ()
* Fix for  - parse contents of search results of type=Category (returned on first page for universal (type=all) queries instead of returning an error.

* Moved content array walker to Category#to_json

As requested by reviewer this change moves the content array walker from the API endpoint to the Category class.

* Update src/invidious/helpers/serialized_yt_data.cr

Co-authored-by: syeopite <70992037+syeopite@users.noreply.github.com>
Frank de Lange c44c1003af Handle missing title fields in VideoRendererParser.parse
Samantaz Fox 678b10dbcf
Lookback 50 videos
diogo a1d6411f1f
propagate video_id field on getting playlists
diogo c4c8a10507
rename from continuation to video_id on get_playlist_videos
diogo ee94ccdeb0
update to new YoutubeAPI
diogo e3df9f9ead
use dig for getting the video index
diogo 62dc629337
linting
diogo dccdf38ce7
increase the max videos in a playlist
diogo 84124b837d
use v1/next instead of searching for the continuation index
diogo 6176da3cbb
linting
diogo 24bc3e2704
no need to normalize the offset
diogo 0a9e19646a
pass the api/v1/playlists with videos before the offset
diogo 65e45c4079
linting
diogo 440105976f
fix cases when high offset video from playlist has no offset in url
diogo 7eba7fbcc7
add index to playlist item
diogo d9bfb3d305
playlist starts at the offset
syeopite 4246c7a523
Extract image routes
Samantaz Fox 57ed047025
Add mising 'require' statement to 'invidious.cr'
Samantaz Fox bda3a26423
Move server structure to a separate file
Samantaz Fox 080d9a8dc7
move theme converter function to a separate file
Samantaz Fox 984a4acc7b
Move user preferences structure to a separate file
syeopite b10f37bea9
Use kemal in production mode ()
mastihios 0947c26612
Fix URL-encoding in href strings ()
* hrefs: replace HTML.escape w/ URI.encode_www_form

* Fix search_query_encoded
Samantaz Fox c6f088d6ca
Reduce refresh delay, increase backoff start duration
syeopite 21e29411af
Fix extractor bugs ()
* Add debug/trace logging to extract_items
* Handle invalid timestamps for livestreams extraction
* Make use of author_fallback in playlist extractor
* Don't use extract_text for video length extraction

The extract_text function attempts to extract from both the simpleText and
the runs route. This is typically what we'd want for text extraction as
it could appear in both locations. However, while this still holds true,
the thumbnailOverlayTimeStatusRenderer writes a numerical length (when
present on the video) to the simpleText route and uses runs for a
text overlay like "LIVE" or "PREMIERE".

Therefore, when a video has a text overlay instead of a numerical one,
Invidious still passes it onto decode_length_seconds, which obviously
raises since it cannot be converted into integers.

In the future, if more routes requires one text route over the other, we
should go ahead and add an argument to extract_text itself. Though for
now, this is sufficient.

* Handle unsupported "special" categories
Samantaz Fox adc12addfa
Add config option to display source code URL in footer ()
Samantaz Fox d806310665
Revert "Fix typo ()" ()
This reverts commit 4982bff74d.
Jorge Maldonado Ventura 4982bff74d
Fix typo ()
syeopite 81b12b8001
Add support to make invidious API-only via flag ()
syeopite 3e781b7d13
Merge pull request from iv-org/syeopite-patch-1
Use make_client when querying instance api
syeopite 9c44e41a4c
Merge pull request from syeopite/extract-items-overhaul
Overhaul extract_item(s) functions and add infrastructure for parsing YouTube categories
syeopite 73eabb6ca2
Actually fix lint
Samantaz Fox 5a52b4fe45
Fix lint
Samantaz Fox caa08a6379
Disable locales with less than 50% of strings translated
Samantaz Fox cd02078e26
Add Portuguese (pt) to locales
follow up to 
syeopite 77131cff91
Use make_client when querying instance api
mastihios 7b2aa5f98e
add icon-buttons to playlist items ()
syeopite 26b28cea49
Use break instead of short-circuit return
syeopite 23049e026f
Improve readabltiy of SearchChannel auto-gen detect
syeopite 9ab242ca2e
Optimize routing logic of extract_item(s) funcs
syeopite aa59925374
Rename get_browse_endpoint to get_browse_id
syeopite 43ea8fa706
Convert nil for AuthorFallback to empty strings
syeopite 6df85718e6
Apply suggestions from code review
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
Samantaz Fox 9ba3e1cdb4
Decrease channel refresh frequency (1 min -> 1 h)
This is a temporary fix to reduce load on instances with many channels and avoid IP being flagged by Google.
syeopite b20f72b963
Use default timeout (5 seconds) for YT pool ()
syeopite 092b8a4e52
Add documentation to extractors.cr
syeopite e5f07dedbf
Typos and tiny styling changes
syeopite ca9eb0d539
Bountiful extractor changes
- Add extract_text to simplify extraction of InnerTube texts
- Add helper extractor methods to reduce repetition in parsing InnerTube
- Change [] more than 2 blocks long to use #dig or #dig?
- Remove useless ?.try blocks for items that always exists
- Add (some) documentation to VideoRendererParser
syeopite 142317c2be
Overhaul extractors.cr to use modules
syeopite 3dea670091
Switch to structs in extractors.cr for performance
syeopite 8435e79913
Improve documentation for extract_item(s) funcs
syeopite 30e85b40f9
Fix extract_videos
syeopite be1a43a337
Manually extract category refactor from 1b569bbc99
Also fixes some errors caused by cherry-picking
syeopite abca8f7a7c
Rename invidiousitems.cr
syeopite 7b60dac526
Add description_html field to Category
(cherry picked from commit aa8f15f795)
syeopite ea6434662d
Change typing of Category contents to only Array
(cherry picked from commit d3384e17f1)
syeopite 0b7a108a59
Move continuation_token out of Category struct
(cherry picked from commit 0e96eda28f)
syeopite 57c63f3598
Rename "items_without_cate_items" to reflect usage
syeopite ae30f32c36
Unpack search items that are embedded in categories
This is a squash of a bunch of commits
cherry-picked commits

Fix category parse error on search

(cherry picked from commit cc02fed4e6)

Fix category items not being extracted in search

(cherry picked from commit 2605b9c609)

Make search not include category items for now

(cherry picked from commit ca4afd59f4)

Change behavior of categories in search results

(cherry picked from commit cc10675610)

Fix missing search results in extraction

(cherry picked from commit abda6840d5)

Fix miscount of search results

(cherry picked from commit 491e33450e)
syeopite a50f64f6e9
Add parser for categories (shelfRenderer)
This commit adds a new parser for YT's shelfRenderers which are
typically used to denote different categories.The code for featured
channels parsing has also been moved to use the new parser but some
additional refactoring are needed there.

The ContinuationExtractor has also been improved and is now capable of
extraction continuation data that is packaged under
"appendContinuationItemsAction"

In additional this commit adds some useful helper functions to extract
the current selected tab the continuation token. This is to mainly
reduce code size and repetition.
--
This cherry-picked commit also removes the code for parsing featured
channels present on the original.

(cherry picked from commit 8000d538db)
syeopite 1323b94b7a
Rewrite extract_item and extract_items functions
This commit completely rewrites the extract_item and extract_items
function. Before this commit these two function were an unreadable
mess. The extract_item function was a lengthy if-elsif chain
while the extract_items function contained an incomprehensible
mess of .try, else and ||.

With this commit both of these functions have been pulled into a
separate file with the internal logic being moved to a few classes.

This significantly reduces the size of these two methods, enhances
readability and makes adding new extraction/parse rules much simpler.

See diff for details.

--
This cherry-picked commit also removes the code for parsing featured
channels present on the original.

(cherry picked from commit a027fbf7af)
syeopite 6d68fbc31d
Fix livestream regex regression caused by
Closes 

Special thanks to @WaywardHeart for finding this issue!
syeopite 86ca568d6d
Remove login type button from frontend ()
Walkyst 2b0bb69a4f
Fix mixes route ()
syeopite 87f46a7532
Unregister captcha job ()
syeopite 4a0359c04e
Pass success msg from dep script during compile-time
syeopite 02431b3f98
Use correct videojs-vtt-thumbnails location in licences.ecr
syeopite 0323202a03
Revert to iv-org fork of silvermine-videojs-quality-selector
Upstream requires at least two additional sources. Whereas Invidious needs it to be
able to display a single additional source for normal (dashless)
qualites. Aka medium and hd720.
syeopite 62c1991b88
Typo
syeopite 3a6085ad31
Readd player.css
syeopite 2451497b31
Typo
syeopite 06a1d2ac41
Rename fetch_videojs* to fetch_player*
syeopite 262131f68e
Add script to resolve and fetch VideoJS files
Samantaz Fox e655af251c
Try not to log search queries ()
syeopite 5054510d15
Prevent VR from being initialized in listen mode ()
Samantaz Fox 947fe4fbb3
HTML escape video mimetype
Video mimetype may contain code information between double quotes.
If not properly escaped, it breaks the browser's parser. E.g:
```
type="video/mp4; codecs=" avc1.64001f,="" mp4a.40.2""=""
```

Thank Robin for catching this!
syeopite 50c8afb525
Handle equirectangular projections for VR ()
syeopite a539de4f97
Bump dependencies ()
* Upgrade to Kemal v1.1.0

* Bump postgres driver
TheFrenchGhosty 8d5a867271
Merge pull request from syeopite/fix-invidious-showing-yt-redirect-links
Fix invidious showing yt redirect links
syeopite 387bddb51b
Improve detection and handling of yt redirect links
Emilien Devos 41ba19b615 fix comment replies
syeopite a28945273d
Propagate replacing yout.be links to /watch to RSS
syeopite fd6f03655e
Fix typo causing links to be youtube.com/redirect
Émilien Devos 8b62c05fe2
remove 3gp only from the player ()
+ video quality precedence on default player when js is not enabled
syeopite a1001ada47
Properly transform youtu.be links to be /watch routes in comments and descriptions ()
TheFrenchGhosty 8e3ff79f22 Remove the mention of 'Omar Roth' from the footer
syeopite 4fcd0964cd
Extract API routes ()
* Extract API routes from invidious.cr
* Remove deprecated APIs
  - insights
  - top feed
syeopite 5005212bec
Extract feed routes ()
* Extract feed routes from invidious.cr
* Removes the deprecated route for /feed/top
* Deprecate /view_all_playlist & use /feed/playlists
* Move feed views into their own directory

* Add haltf method to halt current route context
* Change status_code + return blocks to use haltf

* Set appropriate response headers for RSS routes
Émilien Devos a279d6f433
Fix livestream parsing URLs ()
syeopite fceb8093f1
Use `athena-negotiation` to detect language through Accept-Language header ()
Detect language through Accept-Language header
syeopite d984a898d4
Remove usage of haltf in /api/v1/channels/:ucid/comments
syeopite 52688106e4
Fix /api/v1/search/suggestions route link
syeopite 66b45a8fe2
Bountiful changes
- Use haltf in more locations
 - Fix wrong URL params
 - Rename API modules
 - Remove API routing file and move everything to general iv routing
   file
Émilien Devos b5d2eb5c70
fetch with innertube api when video is unavailable ()
+ rename some client type to better names
+ fix thirdParty hack
syeopite 25362f16a0
Readd paid attribute for videos ()
syeopite 39b34eece8
Extract API routes from invidious.cr (3/3)
- Auth (excluding notifications*) APIs
- Mixes

*Notifications currently require the "connection_channel" channel
for talking with the notifications job. Unfortunately, we cannot
access that within the route modules yet.
syeopite b3426fdc94
Restructure API routes to use more namespaces
syeopite 6aa65593ef
Extract API routes from invidious.cr (2/?)
- Video playback endpoints
- Search feed api
- Video info api