* Add debug/trace logging to extract_items
* Handle invalid timestamps for livestreams extraction
* Make use of author_fallback in playlist extractor
* Don't use extract_text for video length extraction
The extract_text function attempts to extract from both the simpleText and
the runs route. This is typically what we'd want for text extraction as
it could appear in both locations. However, while this still holds true,
the thumbnailOverlayTimeStatusRenderer writes a numerical length (when
present on the video) to the simpleText route and uses runs for a
text overlay like "LIVE" or "PREMIERE".
Therefore, when a video has a text overlay instead of a numerical one,
Invidious still passes it onto decode_length_seconds, which obviously
raises since it cannot be converted into integers.
In the future, if more routes requires one text route over the other, we
should go ahead and add an argument to extract_text itself. Though for
now, this is sufficient.
* Handle unsupported "special" categories
- Add extract_text to simplify extraction of InnerTube texts
- Add helper extractor methods to reduce repetition in parsing InnerTube
- Change [] more than 2 blocks long to use #dig or #dig?
- Remove useless ?.try blocks for items that always exists
- Add (some) documentation to VideoRendererParser
This is a squash of a bunch of commits
cherry-picked commits
Fix category parse error on search
(cherry picked from commit cc02fed4e6)
Fix category items not being extracted in search
(cherry picked from commit 2605b9c609)
Make search not include category items for now
(cherry picked from commit ca4afd59f4)
Change behavior of categories in search results
(cherry picked from commit cc10675610)
Fix missing search results in extraction
(cherry picked from commit abda6840d5)
Fix miscount of search results
(cherry picked from commit 491e33450e)
This commit adds a new parser for YT's shelfRenderers which are
typically used to denote different categories.The code for featured
channels parsing has also been moved to use the new parser but some
additional refactoring are needed there.
The ContinuationExtractor has also been improved and is now capable of
extraction continuation data that is packaged under
"appendContinuationItemsAction"
In additional this commit adds some useful helper functions to extract
the current selected tab the continuation token. This is to mainly
reduce code size and repetition.
--
This cherry-picked commit also removes the code for parsing featured
channels present on the original.
(cherry picked from commit 8000d538db)
This commit completely rewrites the extract_item and extract_items
function. Before this commit these two function were an unreadable
mess. The extract_item function was a lengthy if-elsif chain
while the extract_items function contained an incomprehensible
mess of .try, else and ||.
With this commit both of these functions have been pulled into a
separate file with the internal logic being moved to a few classes.
This significantly reduces the size of these two methods, enhances
readability and makes adding new extraction/parse rules much simpler.
See diff for details.
--
This cherry-picked commit also removes the code for parsing featured
channels present on the original.
(cherry picked from commit a027fbf7af)
Video mimetype may contain code information between double quotes.
If not properly escaped, it breaks the browser's parser. E.g:
```
type="video/mp4; codecs=" avc1.64001f,="" mp4a.40.2""=""
```
Thank Robin for catching this!
* Extract feed routes from invidious.cr
* Removes the deprecated route for /feed/top
* Deprecate /view_all_playlist & use /feed/playlists
* Move feed views into their own directory
* Add haltf method to halt current route context
* Change status_code + return blocks to use haltf
* Set appropriate response headers for RSS routes
- Auth (excluding notifications*) APIs
- Mixes
*Notifications currently require the "connection_channel" channel
for talking with the notifications job. Unfortunately, we cannot
access that within the route modules yet.
* use the new youtube api for comments
* remove PG_DB & action parameter + allow force region
* support new comments data with onResponseReceivedEndpoints
* Extract primary channel routes from invidious.cr
Also removes timedtext_video stub since all it does is redirect to the
homepage. However, Invidious's 404 handler already does this.
--
As the template for the channel about page doesn't exist yet, the
behavior for the /channel/:ucid/about endpoint has been changed to be
the same as what's currently present on Invidious
(cherry picked from commit 8fad19d805)
* Manually extract brand_redirect from 1b569bbc99
This commit manually extracts the brand_redirect function from the
commit mentioned.
However, the redirect to the `.../about` endpoint is removed due to the
fact that it doesn't exist yet.
This commit is also mainly just a bridge for the next few cherry picks from
\#2215
* Update brand_redirect to use youtubei resolve_url
(cherry picked from commit 53335fe7cf)
* Add additional channel endpoints to brand_redirect
(cherry picked from commit 8fc6f3add6)
* Add separate handler for /profile endpoint
* Add /channel/:ucid/home route
* Document all channel brand_urls
* Move Crystal stdlib classes overrides to a separate file
* Document known crystal overrides
* Update crystal overrides for HTTP::Client socket
* Update shard.yml to restrict crystal versions
* Fix compilation error in Crystal 1.1.x (See
https://github.com/crystal-lang/crystal/issues/10965
for more details about this issue).
The private `_post_json` method of the YoutubeAPI requires a ClientConfig
as the third parameter. This was passed in all Youtube API methods except the
`#resolve_url` method.
* Put youtube API functions under the YoutubeAPI namespace
* Implement the following endpoints:
- `next`
- `player`
- `resolve_url`
* Allow a ClientConfig to be passed to YoutubeAPI endpoint handlers.
* Add constants for many new clients
* Fix documentation of YoutubeAPI.browse(): Comments and search
result aren't returned by the browse() endpoint but by the next()
and search() endpoints, respectively.
* Accept gzip compressed data, to help save on bandwidth
* Add debug/trace logging
* Other minor fixes
Fixes:
* Sanitize user-provided content in HTML (Fixes#2193)
* Fix encoding of search query in prev/next pages (Fixes#2229)
* Fix some issues introduced with #2196:
- Fix alignment of all <h3> elements (Move the inline style from the parent to the <h3> element)
- Add missing comma on 'dir' HTML attribute (Typo introduced by PR #2196)
Code cleaning:
* Remove unnecessary 'each_sclice' + 'each' double loop in ECR files
* Clean the player's <source> list generation code (in player.ecr)
Related to #1416, it doesn't really fix the real error, but instead mutes the exception message.
Like explained in #1416, this "exception Error" while flushing the client data doesn't harm the client-server connection. However, this exception message continuously spams the logs and makes debugging and error finding really difficult.
Cherry picked from ui overhaul branch with a few modifications:
- channel folder is renamed to channels
- parsing for channel home and featured channels are removed due to
lack of infrastructure from other commits
(cherry picked from commit 44d18b8e14)
mastihios