Commit Graph

1958 Commits

Author SHA1 Message Date
syeopite
142317c2be
Overhaul extractors.cr to use modules 2021-09-24 21:07:07 -07:00
syeopite
3dea670091
Switch to structs in extractors.cr for performance 2021-09-24 21:07:07 -07:00
syeopite
8435e79913
Improve documentation for extract_item(s) funcs 2021-09-24 21:07:07 -07:00
syeopite
30e85b40f9
Fix extract_videos 2021-09-24 21:07:07 -07:00
syeopite
be1a43a337
Manually extract category refactor from 1b569bbc99
Also fixes some errors caused by cherry-picking
2021-09-24 21:07:07 -07:00
syeopite
abca8f7a7c
Rename invidiousitems.cr 2021-09-24 21:07:07 -07:00
syeopite
7b60dac526
Add description_html field to Category
(cherry picked from commit aa8f15f795)
2021-09-24 21:07:07 -07:00
syeopite
ea6434662d
Change typing of Category contents to only Array
(cherry picked from commit d3384e17f1)
2021-09-24 21:07:06 -07:00
syeopite
0b7a108a59
Move continuation_token out of Category struct
(cherry picked from commit 0e96eda28f)
2021-09-24 21:07:06 -07:00
syeopite
57c63f3598
Rename "items_without_cate_items" to reflect usage 2021-09-24 21:07:06 -07:00
syeopite
ae30f32c36
Unpack search items that are embedded in categories
This is a squash of a bunch of commits
cherry-picked commits

Fix category parse error on search

(cherry picked from commit cc02fed4e6)

Fix category items not being extracted in search

(cherry picked from commit 2605b9c609)

Make search not include category items for now

(cherry picked from commit ca4afd59f4)

Change behavior of categories in search results

(cherry picked from commit cc10675610)

Fix missing search results in extraction

(cherry picked from commit abda6840d5)

Fix miscount of search results

(cherry picked from commit 491e33450e)
2021-09-24 21:07:06 -07:00
syeopite
a50f64f6e9
Add parser for categories (shelfRenderer)
This commit adds a new parser for YT's shelfRenderers which are
typically used to denote different categories.The code for featured
channels parsing has also been moved to use the new parser but some
additional refactoring are needed there.

The ContinuationExtractor has also been improved and is now capable of
extraction continuation data that is packaged under
"appendContinuationItemsAction"

In additional this commit adds some useful helper functions to extract
the current selected tab the continuation token. This is to mainly
reduce code size and repetition.
--
This cherry-picked commit also removes the code for parsing featured
channels present on the original.

(cherry picked from commit 8000d538db)
2021-09-24 21:07:06 -07:00
syeopite
1323b94b7a
Rewrite extract_item and extract_items functions
This commit completely rewrites the extract_item and extract_items
function. Before this commit these two function were an unreadable
mess. The extract_item function was a lengthy if-elsif chain
while the extract_items function contained an incomprehensible
mess of .try, else and ||.

With this commit both of these functions have been pulled into a
separate file with the internal logic being moved to a few classes.

This significantly reduces the size of these two methods, enhances
readability and makes adding new extraction/parse rules much simpler.

See diff for details.

--
This cherry-picked commit also removes the code for parsing featured
channels present on the original.

(cherry picked from commit a027fbf7af)
2021-09-24 21:06:46 -07:00
syeopite
6d68fbc31d
Fix livestream regex regression caused by #2271
Closes #2352

Special thanks to @WaywardHeart for finding this issue!
2021-09-25 02:47:52 +00:00
syeopite
86ca568d6d
Remove login type button from frontend (#2423) 2021-09-23 08:44:26 +02:00
Walkyst
2b0bb69a4f
Fix mixes route (#2421) 2021-09-20 22:39:32 -07:00
syeopite
87f46a7532
Unregister captcha job (#2390) 2021-09-19 10:30:19 +00:00
Samantaz Fox
e655af251c
Try not to log search queries (#2362) 2021-09-16 09:36:22 +00:00
syeopite
5054510d15
Prevent VR from being initialized in listen mode (#2396) 2021-09-15 01:37:23 +02:00
Samantaz Fox
947fe4fbb3
HTML escape video mimetype
Video mimetype may contain code information between double quotes.
If not properly escaped, it breaks the browser's parser. E.g:
```
type="video/mp4; codecs=" avc1.64001f,="" mp4a.40.2""=""
```

Thank Robin for catching this!
2021-09-13 18:20:11 +02:00
syeopite
50c8afb525
Handle equirectangular projections for VR (#2379) 2021-09-10 07:42:39 +00:00
syeopite
a539de4f97
Bump dependencies (#2378)
* Upgrade to Kemal v1.1.0

* Bump postgres driver
2021-09-10 07:42:15 +00:00
TheFrenchGhosty
8d5a867271
Merge pull request #2380 from syeopite/fix-invidious-showing-yt-redirect-links
Fix invidious showing yt redirect links
2021-09-06 11:15:08 +02:00
syeopite
387bddb51b
Improve detection and handling of yt redirect links 2021-09-05 13:13:37 -07:00
Emilien Devos
41ba19b615 fix comment replies 2021-09-04 15:37:16 +02:00
syeopite
a28945273d
Propagate replacing yout.be links to /watch to RSS 2021-09-03 03:33:49 -07:00
syeopite
fd6f03655e
Fix typo causing links to be youtube.com/redirect 2021-09-03 03:30:36 -07:00
Émilien Devos
8b62c05fe2
remove 3gp only from the player (#2376)
+ video quality precedence on default player when js is not enabled
2021-09-03 09:39:11 +02:00
syeopite
a1001ada47
Properly transform youtu.be links to be /watch routes in comments and descriptions (#2365) 2021-09-01 12:59:47 +02:00
TheFrenchGhosty
8e3ff79f22 Remove the mention of 'Omar Roth' from the footer 2021-09-01 12:23:50 +02:00
syeopite
4fcd0964cd
Extract API routes (#2271)
* Extract API routes from invidious.cr
* Remove deprecated APIs
  - insights
  - top feed
2021-08-30 18:27:47 +02:00
syeopite
5005212bec
Extract feed routes (#2269)
* Extract feed routes from invidious.cr
* Removes the deprecated route for /feed/top
* Deprecate /view_all_playlist & use /feed/playlists
* Move feed views into their own directory

* Add haltf method to halt current route context
* Change status_code + return blocks to use haltf

* Set appropriate response headers for RSS routes
2021-08-30 16:58:24 +02:00
Émilien Devos
a279d6f433
Fix livestream parsing URLs (#2356) 2021-08-26 21:02:26 +00:00
syeopite
fceb8093f1
Use athena-negotiation to detect language through Accept-Language header (#2324)
Detect language through Accept-Language header
2021-08-24 19:59:27 +00:00
syeopite
d984a898d4
Remove usage of haltf in /api/v1/channels/:ucid/comments 2021-08-23 17:07:45 -07:00
syeopite
52688106e4
Fix /api/v1/search/suggestions route link 2021-08-23 16:38:29 -07:00
syeopite
66b45a8fe2
Bountiful changes
- Use haltf in more locations
 - Fix wrong URL params
 - Rename API modules
 - Remove API routing file and move everything to general iv routing
   file
2021-08-23 16:28:30 -07:00
Émilien Devos
b5d2eb5c70
fetch with innertube api when video is unavailable (#2329)
+ rename some client type to better names
+ fix thirdParty hack
2021-08-16 10:41:16 -07:00
syeopite
25362f16a0
Readd paid attribute for videos (#2330) 2021-08-15 10:38:30 +02:00
syeopite
39b34eece8
Extract API routes from invidious.cr (3/3)
- Auth (excluding notifications*) APIs
- Mixes

*Notifications currently require the "connection_channel" channel
for talking with the notifications job. Unfortunately, we cannot
access that within the route modules yet.
2021-08-14 00:18:25 -07:00
syeopite
b3426fdc94
Restructure API routes to use more namespaces 2021-08-14 00:18:25 -07:00
syeopite
6aa65593ef
Extract API routes from invidious.cr (2/?)
- Video playback endpoints
- Search feed api
- Video info api
2021-08-14 00:18:24 -07:00
syeopite
66becbf46f
Restructure API route organisation 2021-08-14 00:18:24 -07:00
syeopite
cbf3d75087
Extract API routes from invidious.cr (1/?) 2021-08-14 00:18:24 -07:00
syeopite
0b0036813f
Remove deprecated APIs
- insights
- top feed
2021-08-14 00:18:03 -07:00
Émilien Devos
c06a20e085
Switch to innertube API for videos (#2220)
Commit history:
- Switch to innertube API for videos
- fix hardcoded strings + fix bypass geo-restriction
2021-08-13 20:29:43 +00:00
Émilien Devos
12b46bbd41
switch to innertube API for about channels (#2255) 2021-08-12 16:00:26 -07:00
syeopite
cea38de4ad
Check for existance of "adaptiveFormats" in videos
Combined from commits:
- Check for existance of "adaptiveFormats" in videos
- Remove usage of ternary conditional
2021-08-12 19:26:50 +00:00
Emilien Devos
6745ca7775 fix comments using dig 2021-08-12 20:00:14 +02:00
Émilien Devos
88c5e3b6fa
Use the new youtube api for comments (#2217)
* use the new youtube api for comments
* remove PG_DB & action parameter + allow force region
* support new comments data with onResponseReceivedEndpoints
2021-08-12 19:14:30 +02:00