Commit Graph

3108 Commits

Author SHA1 Message Date
syeopite
4bc77b81bf
Move YTIMG_POOLS to connection_pool.cr 2024-08-24 19:47:40 -07:00
syeopite
06e1a508e8
Fix headers not being added in image requests
Regression from #2364
2024-08-24 19:45:52 -07:00
syeopite
52bc9aa328
Refactor duplicate logic in image routes 2024-08-24 19:45:52 -07:00
syeopite
480e073fa9
Use HTTP pools for image requests to YouTube 2024-08-24 19:45:52 -07:00
syeopite
6e39b9b303
make_client: add YouTube headers on *.youtube.com 2024-08-24 19:41:39 -07:00
syeopite
46c58bd84c
Pool: Use force_resolve in fallback new client 2024-08-24 19:41:23 -07:00
syeopite
7521902e88
Ensure IP family is always used when force_resolve 2024-08-24 19:41:22 -07:00
syeopite
bd48af825c
Search API: Fix named arg syntax to make_client 2024-08-24 19:34:09 -07:00
syeopite
ee89db49ba
Typo
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2024-08-24 19:34:09 -07:00
syeopite
3af6681869
Fix typo in argument to make_client
Co-authored-by: ChunkyProgrammer <78101139+ChunkyProgrammer@users.noreply.github.com>
2024-08-24 19:34:09 -07:00
syeopite
1124dd645d
Use make_client instead of calling HTTP::Client
Using `make_client` to create `HTTP::Client`, allows for a simple way to
easily add logic to all `HTTP::Client` initialized within Invidious.
2024-08-24 19:34:09 -07:00
Samantaz Fox
c5fdd9ea65
HTML: Sort playlists alphabetically in watch page drop down (#4853)
Closes issue 4708
2024-08-24 20:50:46 +02:00
Samantaz Fox
2876ee0f9f
HTML: Fix XSS vulnerability in description/comments (#4852)
Before this PR, the comment/description content was not HTML escaped when 'parse_description()'
was called with a JSON object lacking the "commandRuns" entry.

Closes issue 4727
2024-08-24 20:50:05 +02:00
Samantaz Fox
0699e5fc27
YtAPI: Bump client versions (#4849)
This might help reducing the amount of playback errors.

No related issue
2024-08-24 20:47:01 +02:00
Samantaz Fox
15669acccf
SigHelper: Fix inverted time comparison in 'check_update' (#4845)
Closes issue 4840
2024-08-24 20:44:52 +02:00
Samantaz Fox
cd2daf4adb
Storyboards: Various fixes and code cleaning (#4153)
Closes issue 3441
2024-08-24 20:43:05 +02:00
syeopite
ccecc6d318
Fix lint errors introduced in #4146 and #4295 (#4876)
* Ameba: Fix Naming/VariableNames

Introduced in #4295

* Ameba: Fix Naming/PredicateName

Introduced in #4146
2024-08-24 18:11:11 +00:00
Samantaz Fox
3c6a662aaf
Search: Add support for Youtube URLs (#4146)
Closes issue 3300
2024-08-24 19:44:59 +02:00
Samantaz Fox
9e55799269
Channel: Render age restricted channels (#4295)
This PR:
 * gets thumbnail and channel name from the initial request
 * gets videos, shorts and streams via autogenerated channel playlists

Test Url: /channel/UCbfnHqxXs_K3kvaH-WlNlig

Closes issue 3513
2024-08-24 19:43:59 +02:00
Samantaz Fox
da70c9b7b0
Ameba: Miscellaneous fixes (#4807)
End of a series of PRs meant to improve code quality.

Related to issue 2231
2024-08-24 19:42:10 +02:00
Samantaz Fox
828da3c6ce
API: Proxy formatStreams URLs too (#4859)
The /api/v1/videos endpoint does not proxy the formatStreams URLs when
'local=true' is passed, whereas the adaptiveFormats URLs are correctly proxied.

The Web UI does proxy when clicking "Download" with 'fmt=18' for example, so
this is probably an oversight. This PR aims to fix that.

No related issue
2024-08-24 19:39:36 +02:00
Samantaz Fox
febf18cbf7
UI: Add search button to search bar (#4706)
Closes issue 529
2024-08-24 19:38:48 +02:00
Samantaz Fox
b2133c6b2c
Videos: Convert URL before putting result into cache 2024-08-24 18:01:56 +02:00
Samantaz Fox
21ab5dc668
Storyboard: Revert cue timing "fix" 2024-08-22 00:29:15 +02:00
syeopite
ecbea0b67b
Ameba: Fix Lint/ShadowingOuterLocalVar 2024-08-21 02:43:26 -07:00
syeopite
d1cd790388
Ameba: Fix Lint/RedundantStringCoercion 2024-08-21 02:43:26 -07:00
syeopite
f66068976e
Ameba: Fix Naming/PredicateName 2024-08-21 02:43:08 -07:00
syeopite
22b35c453e
Ameba: Fix Style/WhileTrue 2024-08-21 02:43:08 -07:00
Colin Leroy-Mira
c606465708 Proxify formatStreams URLs too 2024-08-19 09:37:24 +02:00
Samantaz Fox
85deea5aca
Search: Change smart search inhibitor to a backslash 2024-08-17 19:22:40 +02:00
Samantaz Fox
78c5ba93c7
Misc: Clean some code in UrlSanitizer 2024-08-17 19:22:40 +02:00
Samantaz Fox
31a80420ec
Search: Add URL search inhibition logic 2024-08-17 19:22:40 +02:00
Samantaz Fox
4c0b5c314d
Search: Add support for youtu.be and youtube.com URLs 2024-08-17 19:22:40 +02:00
Samantaz Fox
eb0f651812
Add a youtube URL sanitizer 2024-08-17 19:22:40 +02:00
Samantaz Fox
764965c441
Storyboards: Fix lint error 2024-08-17 12:20:53 +02:00
Samantaz Fox
b795bdf2a4
HTML: Sort playlists alphabetically in watch page drop down 2024-08-16 12:10:22 +02:00
Samantaz Fox
5b05f3bd14
Storyboards: Workarounds for videojs-vtt-thumbnails
The workarounds are as follow:
  * Unescape HTML entities
  * Always use 0:00:00.000 for cue start/end
2024-08-16 11:36:01 +02:00
Samantaz Fox
a335bc0814
Storyboards: Fix some small logic mistakes 2024-08-16 10:05:49 +02:00
Samantaz Fox
7b50388eaf
Storyboards: Fix broken first storyboard 2024-08-16 10:05:48 +02:00
Samantaz Fox
da3d58f03c
Storyboards: Cleanup and document code 2024-08-16 10:05:47 +02:00
Samantaz Fox
8327862697
Storyboards: Use replace the NamedTuple by a struct 2024-08-16 10:04:40 +02:00
Samantaz Fox
6878822c4d
Storyboards: Move parser to its own file 2024-08-16 10:02:52 +02:00
Samantaz Fox
0b28054f8a
videos: Fix XSS vulnerability in description/comments
Patch provided by e-mail, thanks to an anonymous user whose cats are named
Yoshi and Yasuo.

Comment is mine
2024-08-15 18:26:17 +02:00
Samantaz Fox
cc33d3f074
YtAPI: Also update User-Agent string 2024-08-15 18:14:29 +02:00
Samantaz Fox
acbb625866
YtAPI: Update clients to latest version 2024-08-15 12:57:36 +02:00
Samantaz Fox
466bfbb306
SigHelper: Fix inverted time comparison in 'check_update' 2024-08-14 21:43:37 +02:00
ChunkyProgrammer
e31053e812 Use dig to get properties
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2024-08-13 15:09:16 -04:00
ChunkyProgrammer
96ade642fa Channel: Render age restricted channels 2024-08-13 15:09:16 -04:00
Samantaz Fox
e319c35f09
Videos: use intermediary variable when using CONFIG.po_token 2024-08-13 20:56:09 +02:00
Samantaz Fox
2d18ff1f80
Add ability to set po_token and visitordata ID (#4789)
This PR adds two new config option, to pass a PO token (config 'po_token') and
a visitor ID (config 'visitor_data') to Youtube. These two strings are required
to play videos using the WEB client.

Warning: These strings gives much more identifiable information to Google!

If the po_token setting is filled in, then the WEB client is used. If not, the
Android client is used. TvHtml5ScreenEmbed will still be used as a fallback.

Script for generating po_token and visitor_data:
https://github.com/iv-org/youtube-trusted-session-generator

Helps with issue 4734
2024-08-13 20:35:43 +02:00
Samantaz Fox
2d7869b48b
Add support for an external signature server (#4772)
This PR adds support for inv_sig_helper, which offloads the player fetching,
function extraction and signature parsing, which in turn allows to use the
web client to watch videos.

When the new config option "signature_server" is not set, the logic for the
external signature server is not enabled and invidious behaves like before.

This PR also updates the crystal overrides because the stdlib changed quite
a while ago (See issue 11049 at crystal-lang/crystal) and those were required
to properly use TCP/unix sockets.

Closes issue 4649
2024-08-13 20:26:59 +02:00
Samantaz Fox
88b9f17388
Ameba: Fix Naming/VariableNames (#4790)
Related to issue 2231
2024-08-13 20:26:15 +02:00
Emilien Devos
e6c39f9e3a add pot= parameter now required by youtube 2024-08-13 14:37:35 +02:00
Samantaz Fox
5e38ef59da
Ameba: Fix Lint/UselessAssign (#4795)
Related to issue 2231
2024-08-11 13:38:29 +02:00
Samantaz Fox
80ffc442f2
HTML: Add rel="noreferrer noopener" to external links (#4667)
Note: Does not add rel="noreferrer noopener" to:
 * links in channel description
 * links in video descriptions
 * links in video comments

Related to issue 4267
2024-08-11 13:35:57 +02:00
Samantaz Fox
9bf754ed4f
Remove unused methods in Invidious::LogHandler (#4812)
Closes issue 4791
2024-08-11 11:45:56 +02:00
Samantaz Fox
fa6c5158c5
Ameba: Fix Lint/NotNilAfterNoBang (#4796)
Related to issue 2231
2024-08-11 11:45:05 +02:00
Samantaz Fox
b45310c7d4
Ameba: Fix unused argument Lint warnings (#4805)
Related to issue 2231
2024-08-11 11:43:56 +02:00
Samantaz Fox
eb2dfe0ab1
Ameba: i18next.cr fixes (#4806)
Related to issue 2231
2024-08-11 11:41:36 +02:00
Samantaz Fox
cc36a82933
SigHelper: Fix some logic errors raised during code review 2024-08-07 23:26:10 +02:00
Samantaz Fox
7798faf234
SigHelper: Make signature server optional and configurable 2024-08-07 23:25:35 +02:00
Samantaz Fox
ec1bb5db87
SigHelper: Add support for PLAYER_UPDATE_TIMESTAMP opcode 2024-08-07 23:25:32 +02:00
Samantaz Fox
3b7e45b7bc
SigHelper: Small fixes + suggestions from code review 2024-08-07 23:12:38 +02:00
syeopite
e098c27a45
Remove unused methods in Invidious::LogHandler 2024-07-28 16:44:30 -07:00
syeopite
6506b8dbfc
Ameba: Fix Naming/PredicateName 2024-07-25 20:08:26 -07:00
Samantaz Fox
61d75050e4
SigHelper: Use 'URI.parse' instead of 'URI.new'
Co-authored-by: Brahim Hadriche <brahim.hadriche@gmail.com>
2024-07-25 22:13:08 +02:00
Samantaz Fox
10e5788c21
Videos: Send player sts when required 2024-07-25 22:13:08 +02:00
Samantaz Fox
b509aa91d5
SigHelper: Fix many issues 2024-07-25 22:13:08 +02:00
Samantaz Fox
ec8b7916fa
Videos: Make use of the video decoding 2024-07-25 22:13:08 +02:00
Samantaz Fox
56a7488161
Helpers: Add inv_sig_helper client 2024-07-25 22:13:08 +02:00
Samantaz Fox
a845752fff
Jobs: Remove the signature function update job 2024-07-25 22:13:08 +02:00
Samantaz Fox
63a729998b
Misc: Sync crystal overrides with current stdlib 2024-07-25 22:13:07 +02:00
syeopite
205f988491
Ameba: Fix Naming/MethodNames 2024-07-24 20:04:44 -07:00
syeopite
0db3b830b7
Ameba: Fix Lint/HashDuplicatedKey 2024-07-24 20:03:41 -07:00
syeopite
c8fb75e6fd
Ameba: Fix Lint/UnusedBlockArgument 2024-07-24 19:59:20 -07:00
syeopite
636a6d0be2
Ameba: Fix Lint/UnusedArgument 2024-07-24 19:57:54 -07:00
syeopite
3415507e4a
Ameba: undo Lint/NotNilAfterNoBang in signatures.cr
File is set to be removed with #4772
2024-07-24 19:48:34 -07:00
Emilien Devos
53223f99b0 Add ability to set po_token and visitordata ID 2024-07-24 19:28:47 +02:00
Samantaz Fox
325561e755
Channel: parse subscriber count and channel banner (#4785)
This PR adds support for parsing the newer channel header format
(banner + subscription parsing)

Before this change:
* 0 subscribers
* No banner image

After this change:
* Example with Mr Breast channel: 299M
* Image banner is visible

Closes issue 4783
2024-07-21 17:24:09 +02:00
Samantaz Fox
7fdbda612f
Videos: Fix genre url being unusable (#4717)
Closes issue 4700
2024-07-21 17:24:03 +02:00
Samantaz Fox
4f60feee17
API: Fix out of bound error on empty playlists (#4696)
Before this PR, Invidious assumed that every playlist had at least one video.
When a playlist had no videos, Invidious was throwing an "Index out of bounds"
exception.

The following API endpoints were impacted:
* api/v1/playlists/:plid
* api/v1/auth/playlists/:plid

Fixes issue 4679
2024-07-21 17:24:01 +02:00
Samantaz Fox
733bd27a5c
Handle playlists cataloged as Podcast (#4695)
Videos of a playlist cataloged as podcast are called "episodes" therefore
Invidious was not able to find video in the text value inside the stats array.

Test case: "/playlist?list=PLDu-Eh5lUs1a4irCbnxMIB6FrUMaTXgVF"

Fixes issue 4688
2024-07-21 17:23:58 +02:00
Samantaz Fox
1ff0775f4b
API: Fix duplicated query parameters in proxied video URLs (#4587)
This pull request fixes that bug that was causing the query parameters to get
doubled in the streaming URLs when '?local=true' is passed to the
'/api/v1/videos/{id}' API endpoint.

Before: host/path?parameters?parameters
After: host/path?parameters

No associated open issue
2024-07-21 17:23:53 +02:00
Samantaz Fox
e62d4db752
API: Return actual stream height, width and fps (#4586)
At the moment Invidious will return hardcoded data for the 'size',
'qualityLabel' and 'fps' fields for streams, when such hardcoded data is
available, otherwise it just omits those fields from the response (e.g. with
the AV1 formats). Those issues are especially noticable when Invidious claims
that 50fps streams have 60fps and when it claims that the dimensions for a
vertical video are landscape. The DASH manifests that Invidious generates
already use the correct information.

This pull request corrects that issue by returning the information that
YouTube provides instead of hardcoded values and also fixes the long
standing bug of Invidious claiming that audio streams have 30 fps.

Here are two test cases:
50/25/13fps: https://youtu.be/GbXYZwUigCM (/api/v1/videos/GbXYZwUigCM)
vertical video: https://youtu.be/hxQwWEOOyU8 (/api/v1/videos/hxQwWEOOyU8)

Originally these problems were going to be solved by the complete refactor
of stream handling in 3620, but as that pull request got closed by the stale
bot over a month ago and has such a massive scope that it would require a
massive amount of work to complete it, I decided to open this pull request
that takes a less radical approach of just fixing bugs instead of a full
on refactoring.

FreeTube generates it's own DASH manifests instead of using Invidious' one,
so that it can support multiple audio tracks and HDR. Unfortunately due to
the missing and inaccurate information in the API responses, FreeTube has
to request the DASH manifest from Invidious to extract the height, width and
fps. With this pull request FreeTube could rely just on the API response,
saving that extra request to the Invidious instance. It would also make it
possible for FreeTube to use the vp9 streams with Invidious, which would
reduce the load on the video proxies.

Closes issue 4131
2024-07-21 17:23:50 +02:00
Samantaz Fox
8b1da2001e
Preferences: Fix handling of modified source code URL(#4437)
Before this PR, setting the modified code repo URL through the preferences
page in Invidious was broken:

* the HTML input tag for this field had invalid type "input"
  (though browser falls back on text input)

* the URL was used to set the "checked" property and not as a plain value,
  which makes no sense for a text-based input (and resulted in a blank field)

* when the submitted field is empty, the retrieved value was an empty 'String'
  instead of 'nil', causing the "modified source code URL" to be an empty
  'href' link which just pointed to the current page

No associated open issue
2024-07-21 17:23:48 +02:00
Samantaz Fox
5a12005b48
API: Fix URL for vtt subtitles (#4221)
For 'fmt=vtt' to work, the 'fmt' parameter needs to be replaced
in the original caption api URL.

No associated open issue
2024-07-21 17:23:44 +02:00
syeopite
fad0a4f52d
Ameba: Fix Lint/UselessAssign 2024-07-17 12:39:40 -07:00
syeopite
fa50e0abf4
Simplify last_node retrieval
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2024-07-17 12:21:48 -07:00
syeopite
8258062ec5
Ameba: Fix Lint/NotNilAfterNoBang 2024-07-15 17:36:00 -07:00
syeopite
8a90add310
Ameba: Fix Naming/VariableNames
Fix Naming/VariableNames in comment renderer

Fix Naming/VariableNames in helpers/utils

Fix Naming/VariableNames in api/v1/misc.cr
2024-07-11 20:56:28 -07:00
Samantaz Fox
bad92093bf
Channels: Add sort options to streams (#4224) 2024-07-10 22:28:22 +02:00
Samantaz Fox
436a61e3bb
API: Fix error code for disabled popular endpoint (#4296)
When visiting /api/v1/popular and popular endpoint is disabled
Before:

500 {"error":"Closed stream"}

After

403 {"error":"Administrator has disabled this endpoint."}
2024-07-10 22:25:31 +02:00
Samantaz Fox
5e0f55333a
Allow embedding videos in local HTML files (#4450)
The current Content Security Policy does not allow to embed videos
inside local HTML files which are viewed in the browser via the file
protocol. This commit adds the file protocol to the allowed frame
ancestors, so that the embedded videos load correctly in local HTML
files.

This behaviour is consistent which how the official YouTube website
allows to embed videos from itself.

Closes issue 4448
2024-07-10 22:24:18 +02:00
Samantaz Fox
99c7e9e800
YtAPI: Remove API keys like official clients (#4655)
This PR removes API keys from innertube requests, as the official clients
did it too.
2024-07-10 22:19:51 +02:00
Samantaz Fox
e9bab06e90
HTML: Use full URL in the og:image property (#4675)
Some opengraph implementations don't support a URL without the domain
therefore failing to fetch the video thumbnail and channel image.
This pull request basically fixes that.
2024-07-10 22:17:45 +02:00
ChunkyProgrammer
911dad6935 Channel: parse subscriber count and channel banner 2024-07-09 14:43:14 -04:00
meatball
248df785d7 Update spec and rollback to last commits changes 2024-06-18 20:55:14 +02:00
syeopite
c24ed85110
Fix named arg syntax when passing force_resolve 2024-06-16 14:49:48 -07:00
syeopite
f466116cd7
Extract label for transcript in YouTube response 2024-06-13 09:07:20 -07:00
syeopite
5b519123a7
Raise error when transcript does not exist 2024-06-11 18:46:34 -07:00