musicfetch/README.md

# 🎵 MusicFetch

**MusicFetch** is a smart command-line utility that finds music by searching
**Lidarr** (your music collection manager) and **YouTube Music** at the same
time, shows you the top hits in an interactive picker, and downloads/queues
whatever you choose. It accepts:

- A **free-form query**: an artist, an album, a track title, or combos like
  `"Artist - Title"` or `"Artist - Album"` (e.g. `"ODESZA - Bloom"`, `"Daft Punk"`, `"Discovery"`).
- A **URL** (e.g. `"https://music.youtube.com/watch?v=..."` or a regular YouTube URL).
- **Any streaming link** (Spotify, Apple Music, Tidal, Deezer, …): resolved to
  metadata via [Odesli/song.link](https://odesli.co), then searched on Lidarr
  first and downloaded from the matching YouTube track if no Lidarr release is
  available. YouTube and SoundCloud links download directly.

Lidarr is tried first by default. If you pick a Lidarr album but **no indexer
release is available**, MusicFetch automatically falls through to the top
YouTube hit. YouTube downloads prefer **YouTube Music URLs** so album art and
tags are correct.

---

## 🚀 Features

- One unified picker showing the top hits from **Lidarr and YouTube together**, with matching keywords **bolded**.
- Lidarr-first flow: pick an album → adds artist+album (monitored) → interactive indexer search → falls through to YouTube only if no release is found.
- Accurate YouTube metadata via `ytmusicapi` (real artist / album / year / album art), with `yt-dlp` scraping as a fallback.
- Explicit tag overrides on download so files are tagged from the chosen hit, not from scraped titles.
- Non-interactive, YouTube-first, dry-run, quality, limit, and source-restriction flags.

---

## 📦 Dependencies & Installation

### 🐍 Python

- Python 3.10+
- `requests`
- `ytmusicapi` (recommended — accurate YouTube Music metadata)
- `rich` (recommended — nicer picker table + bold keyword matching)

```bash
pip install requests ytmusicapi rich
```

> **Note:** if you hit `ModuleNotFoundError: No module named 'idna'` from
> `requests`, repair it with:
> ```bash
> pip install --force-reinstall idna requests
> ```

`ytmusicapi` and `rich` are optional — without them MusicFetch falls back to
`yt-dlp` search scraping and a plain ANSI picker.

### 📼 External Tools

- `yt-dlp` (audio download/extraction) and `ffmpeg` (for `-x` extraction / embedding).

```bash
pip install -U yt-dlp
sudo apt install ffmpeg      # or your distro's equivalent
```

---

## ⚙️ Configuration

Set via environment variables:

| Variable          | Default                 | Purpose                          |
|-------------------|-------------------------|----------------------------------|
| `LIDARR_API_KEY`  | *(required for Lidarr)* | Lidarr API key.                  |
| `LIDARR_URL`      | `http://localhost:8686` | Lidarr base URL.                 |
| `MUSICFETCH_ROOT` | `/media/music`          | Default output root folder.      |

```bash
export LIDARR_API_KEY="your-lidarr-api-key"
```

---

## 🧑‍💻 Usage

```bash
./musicfetch [OPTIONS] QUERY...
```

### Options

| Flag | Description |
|------|-------------|
| `-n`, `--noninteractive` | Auto-pick the top hit (no prompt). |
| `-s`, `--ytsearch` | Search/select YouTube first instead of Lidarr first. |
| `-d`, `--dry-run` | Show every action without executing it. |
| `-q`, `--quality {best,320,m4a,opus,flac}` | Audio quality/format (default `best`). |
| `--limit N` | Hits per source (default 10). |
| `--lidarr-only` | Skip YouTube. |
| `--yt-only` | Skip Lidarr. |
| `-o`, `--root PATH` | Output root folder (default `/media/music`). |
| `--search-all` | Search all albums when adding an artist to Lidarr. |
| `--pick-release` | Show Lidarr's indexer releases and choose one, instead of letting Lidarr grab the best automatically. |
| `--repair` | Re-tag existing downloads under `--root` from source metadata (see below). |
| `--workers N` | Parallel metadata fetches during `--repair` (default 4). |
| `--cookies FILE` | yt-dlp `cookies.txt` for authenticated YouTube (avoids bot-check / rate limits). |
| `--cookies-from-browser BROWSER` | Load YouTube cookies from a local browser (e.g. `firefox`). |
| `--remote-components SPEC` | yt-dlp EJS challenge-solver source (default `ejs:github`; needs `deno`). Fixes "signature/n challenge solving failed". `''` to disable. |
| `--retag-from-path` | Offline: re-tag artist/title from folder + filename (see below). |
| `-x`, `--exclude NAME` | Folder under `--root` to skip during `--repair`/`--retag-from-path` (repeatable). |
| `--debug` | Verbose output. |

### Examples

```bash
# Interactive: combined Lidarr + YouTube picker
./musicfetch "ODESZA - Bloom"

# Just an artist / just an album / just a title all work
./musicfetch "Daft Punk"
./musicfetch "Discovery"

# YouTube first, auto-pick top hit
./musicfetch -s -n "Daft Punk - Harder Better Faster Stronger"

# Dry run — see what would happen, change nothing
./musicfetch -d "ODESZA - Bloom"

# YouTube only, lossless preferred
./musicfetch --yt-only -q flac "Bonobo - Kerala"

# Download by URL (single track or playlist/set/album, any yt-dlp site)
./musicfetch "https://music.youtube.com/watch?v=xxxxxxxxxxx"
./musicfetch "https://soundcloud.com/artist/sets/my-mix"
```

### 🔧 Repair existing tags

`--repair` walks `<root>/<artist>/<source>/` (the `youtube`/`soundcloud`/… download
folders — Lidarr album folders are skipped), re-fetches authoritative metadata for each
file using the `[id]` in its filename, and fixes tags. Useful when downloads landed with
missing album or wrong year.

It is deliberately **conservative**: it overwrites **album** and **year** (the usual
breakage), and fills in **artist**/**title** when they are missing *or* a known-bogus
placeholder (`NA`, `Unknown Album`, `Unknown Artist` — left behind by older buggy tagging) —
but it never overwrites a genuine existing artist/title with a channel name or decorated video
title. A bogus `NA [<id>].<ext>` filename is renamed to the recovered title, and a literal
`NA` album with no source album is normalised to `Unknown Album`.

Each file is its own yt-dlp network round-trip, so repair runs them in a thread pool;
`--workers N` (default 4) caps concurrency. Progress prints every 100 files. Requires
`mutagen` (a yt-dlp dependency, usually already present). CLI-only — not exposed via the REST API.

**Cookies (important for bulk repair).** Unauthenticated YouTube requests get throttled fast —
a large `--repair` (or even a `--dry-run`, which still fetches) will trip *"Sign in to confirm
you're not a bot"* (HTTP 429) and every subsequent call fails until the IP-level flag clears.
Pass authenticated cookies to avoid it:

```bash
./musicfetch --repair --cookies /path/cookies.txt -o /media/music        # exported cookies.txt
./musicfetch --repair --cookies-from-browser firefox -o /media/music     # or read from a browser
```

With cookies you can raise `--workers`; without them keep it low (≤4) and expect occasional
throttling. Cookies also apply to normal fetches/downloads. The same can be set for the API
container via `$YTDLP_COOKIES` / `$YTDLP_COOKIES_FROM_BROWSER`. If you do get flagged, **stop** —
retrying extends it; wait ~30-60 min (429) or longer for a bot-check.

**JS challenge solving (not a cookie issue).** Separately from auth, YouTube gates media formats
behind signature / "n" JS challenges. If yt-dlp warns *"Signature solving failed"* / *"n challenge
solving failed: Some formats may be missing"*, downloads break even with valid cookies. musicfetch
passes `--remote-components ejs:github` by default so yt-dlp fetches the [EJS](https://github.com/yt-dlp/yt-dlp/wiki/EJS)
solver and runs it via a local JS runtime — install **[`deno`](https://deno.land)** and use a recent
yt-dlp. Override with `--remote-components` / `$YTDLP_REMOTE_COMPONENTS` (empty disables).

#### Getting YouTube cookies

> ⚠️ Use a **throwaway / secondary Google account**, not your main one — bulk automated
> requests can get the account flagged. You must be **logged in to YouTube** in the browser
> first.

**Option A — read straight from the browser (simplest, host CLI only).**
`--cookies-from-browser` reads the browser's own cookie store, so there's nothing to export:

```bash
./musicfetch --repair --cookies-from-browser firefox -o /media/music
./musicfetch --repair --cookies-from-browser chrome  -o /media/music
```

- **Firefox:** works while open; just be logged in to YouTube.
- **Chrome / Chromium / Brave / Edge:** must be **fully quit** when you run this (Chrome locks
  its cookie DB, and newer versions encrypt it — close the browser entirely first). On Linux a
  running Chrome will usually fail with a "could not copy cookie database / locked" error.
- Specify a profile if not the default, e.g. `--cookies-from-browser "chrome:Profile 1"`.

This only works where the browser lives (your host), **not** inside the Docker container.

**Option B — export a `cookies.txt` (works anywhere, incl. the container/server).**
Use a Netscape-format cookie exporter, then point `--cookies` / `$YTDLP_COOKIES` at the file:

1. Install a cookies exporter extension:
   - Firefox: *"cookies.txt"* (a.k.a. *Export Cookies*).
   - Chrome: *"Get cookies.txt LOCALLY"* (pick a **LOCALLY**-running one — avoid extensions that
     upload your cookies anywhere).
2. Log in to <https://www.youtube.com>, click the extension, **Export** → save `cookies.txt`.
3. Use it:

   ```bash
   ./musicfetch --repair --cookies ~/cookies.txt -o /media/music
   ```

   For the API container, just point it at the host file via `.env` — no compose
   edits needed (see `server/.env.example`):

   ```bash
   # server/.env
   YTDLP_COOKIES_HOST=/host/path/cookies.txt
   ```

   The compose mounts it read-only at `/cookies.txt` and sets `YTDLP_COOKIES`
   automatically. Leave `YTDLP_COOKIES_HOST` unset to run without cookies.

Cookies expire — if YouTube starts rejecting them, re-export. Treat `cookies.txt` like a
password (it *is* your logged-in session); keep it out of git (`.gitignore` it).

```bash
# Preview what would change (writes nothing)
./musicfetch --repair -d

# Apply fixes under a specific root
./musicfetch --repair -o /media/music
```

**`--retag-from-path`** is an offline companion: it derives **artist** and **title** purely
from the folder name + filename (stripping `(Official Video)` / `(Lyrics)`-style decorations,
and treating an `Artist - Title` filename correctly), with no network. Use it to undo bad
tags — e.g. titles/artists clobbered by an earlier `--repair` on music videos. It overwrites
artist/title and leaves album/year alone.

```bash
./musicfetch --retag-from-path -d            # preview
./musicfetch --retag-from-path -o /media/music

# Skip folders (e.g. hand-curated playlists you don't want re-tagged)
./musicfetch --repair -x Unsorted -x playlists
```

### 📁 Output Structure

```text
<root>/
├── Artist Name/
│   ├── Album Name/    (managed by Lidarr)
│   ├── youtube/       (YouTube / YouTube Music downloads)
│   ├── soundcloud/    (SoundCloud downloads)
│   └── <source>/      (one folder per yt-dlp source)
```

---

## ❓ Troubleshooting

- **No Lidarr hits / "LIDARR_API_KEY not set":** export your key and confirm `LIDARR_URL` is reachable.
- **Wrong album art from YouTube:** install `ytmusicapi` so MusicFetch can resolve proper YouTube Music URLs and metadata.
- **`yt-dlp` errors:** update with `yt-dlp -U`; ensure `ffmpeg` is installed for extraction/embedding.
- **`idna` import error:** `pip install --force-reinstall idna requests`.
- **Permission denied writing files:** ensure the output root exists and is writable (`-o`/`--root` or `MUSICFETCH_ROOT`).

---

## 🌐 REST API (Docker)

Run MusicFetch as an authenticated HTTP service inside your Lidarr Docker stack.
A client POSTs a query; the server grabs the top hit non-interactively and runs
the download as a background job you can poll. Every response includes a
human-readable `message` (handy for Siri).

### Configure & run

Set the network name in `server/docker-compose.yml` to your existing Lidarr
stack network, then:

```bash
export LIDARR_API_KEY="your-lidarr-key"
export MUSICFETCH_API_KEY="a-long-random-secret"
docker compose -f server/docker-compose.yml up -d --build
```

| Env var | Default | Purpose |
| --- | --- | --- |
| `MUSICFETCH_API_KEY` | *(required)* | Shared secret clients send as `X-API-Key`. |
| `MUSICFETCH_PORT` | `6769` | Listen port. |
| `LIDARR_URL` | `http://lidarr:8686` | Lidarr base URL (stack network). |
| `LIDARR_API_KEY` | *(required for Lidarr)* | Lidarr API key. |
| `MUSICFETCH_ROOT` | `/media/music` | Music output root (bind-mounted). |

TLS is expected to be handled by your upstream reverse proxy; the container
serves plain HTTP on `6769`.

### Endpoints

| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/health` | no | Liveness check. |
| `POST` | `/fetch?q=...` | yes | The one entry point. Returns a `job_id`. |
| `POST` | `/choose?job_id=...&n=...` | yes | Send the chosen release to the download client. |
| `GET` | `/jobs/{id}` | yes | Poll job status. |

`POST /fetch` decides what to do from `q`:

- **`q` is a URL** → grab it directly. No prompt. A direct YouTube or SoundCloud
  link downloads at once; a Spotify or Apple Music link resolves through Odesli
  first.
- **`q` is free text** → go through Lidarr. The artist and the album land in the
  library, Lidarr runs an interactive search, and the job parks at
  `awaiting_choice` with the releases. Pick one with `POST /choose`.

`POST /fetch` params: `q` (required), `quality` (`best,320,m4a,opus,flac`),
`source` (`auto,lidarr,youtube`), `auto` (`false` by default — `true` grabs the
best release and never asks), `limit` (1–25, default 10).

- `source=youtube` skips Lidarr. The top YouTube hit downloads at once.
- `source=lidarr` drops the YouTube fallback. No release means `no_release`.
- `auto=true` never parks the job. A URL always behaves this way.

`POST /choose` params: `job_id` and `n` (both required). `n` is the number of a
choice from the same job.

Job `status` is one of `queued`, `running`, `awaiting_choice`, `done`, `failed`
or `expired`. A job at `awaiting_choice` carries a `choices` array and waits for
`POST /choose`. It expires after 15 minutes (`MUSICFETCH_CHOICE_TTL`).

Every `message` states what really happened. An artist add says so — it does not
claim a download. A Lidarr grab says the release went to the download client.

### curl examples

```bash
# A URL: grabbed directly, no prompt.
curl -X POST 'https://mf.izebra.net/fetch?q=https://youtu.be/abc123' \
  -H 'X-API-Key: a-long-random-secret'
# -> {"message":"Fetching '...'. Downloading now.","job_id":"a1b2c3","status":"queued"}

# Poll any job.
curl 'https://mf.izebra.net/jobs/a1b2c3' -H 'X-API-Key: a-long-random-secret'
# -> {"message":"Downloaded '...'.","status":"done","result":{...}}

# Text, and take whatever Lidarr thinks is best.
curl -X POST 'https://mf.izebra.net/fetch?auto=true&q=Overgrown%20-%20Lay%20Cold' \
  -H 'X-API-Key: a-long-random-secret'
```

### Choose the release yourself

```bash
# 1. Free text. Lidarr adds the artist only if it is missing, monitors the
#    album, then runs an interactive search. That takes up to a minute, so the
#    search is part of the job.
curl -X POST 'https://mf.izebra.net/fetch?q=Overgrown%20-%20Lay%20Cold' \
  -H 'X-API-Key: a-long-random-secret'
# -> {"message":"Found the album 'Lay Cold' by Overgrown in Lidarr. Searching indexers now.",
#     "job_id":"a1b2c3","status":"queued","hit":{...}}

# 2. Poll until the job asks you a question.
curl 'https://mf.izebra.net/jobs/a1b2c3' -H 'X-API-Key: a-long-random-secret'
# -> {"status":"awaiting_choice",
#     "message":"Overgrown is already in Lidarr. I found 3 releases for 'Lay Cold' by Overgrown. Which one?",
#     "choices":[{"n":1,"label":"FLAC, 312 MB, 24 seeders, Redacted",
#                 "pick_label":"1. FLAC, 312 MB, 24 seeders, Redacted",
#                 "quality":"FLAC","size_mb":312,
#                 "seeders":24,"indexer":"Redacted","rejected":false}, ...]}

# 3. Pick one. Lidarr sends that exact release to the download client.
curl -X POST 'https://mf.izebra.net/choose?job_id=a1b2c3&n=1' \
  -H 'X-API-Key: a-long-random-secret'
# -> {"message":"Sending 'Lay Cold' by Overgrown to the download client.","status":"queued"}
```

A job never parks when no indexer has the album. It falls back to YouTube Music
instead, and says so.

### 🗣️ Siri Shortcuts integration

Make a shortcut that fetches music by voice ("Hey Siri, fetch music").

1. **Shortcuts app → New Shortcut.**
2. Add **Ask for Input** → Input Type **Text**, prompt "What should I fetch?".
   (Or use **Dictate Text** for fully spoken input.)
3. Add **Text** action, set it to: `https://mf.izebra.net/fetch?auto=true&q=`
   then insert the **Provided Input** variable at the end. (Shortcuts
   URL-encodes query variables automatically.) `auto=true` keeps this shortcut
   one-shot — without it, free text stops and asks which release you want.
4. Add **Get Contents of URL**:
   - **URL:** the Text variable from step 3.
   - **Method:** `POST`.
   - **Headers:** add one — key `X-API-Key`, value your `MUSICFETCH_API_KEY`.
   - **Request Body:** leave as is (the query is in the URL).
5. Add **Get Dictionary Value** → Get Value for **message** in **Contents of URL**.
6. Add **Speak Text** → the Dictionary Value. Siri reads back
   "Found '…' … Downloading now."
7. (Optional) To confirm completion: add **Get Dictionary Value** for `job_id`,
   **Wait** ~20 seconds, **Get Contents of URL** on
   `https://mf.izebra.net/jobs/<job_id>` (same `X-API-Key` header), then
   **Get Dictionary Value** `message` → **Speak Text** again.

Rename the shortcut (e.g. "Fetch Music") — that phrase becomes the Siri trigger.

#### Shortcut 2: choose the release yourself

This shortcut lists the indexer releases and lets you pick one.

Shortcuts has no "exit loop" action and no **OR** in an **If**, so the poll uses
a `Keep` flag and one **is** test per status.

**Start the search**

1. **Ask for Input** → Text, prompt "What should I find?".
2. **Text**: `https://mf.izebra.net/fetch?q=` + the **Provided Input**.
3. **Get Contents of URL** → Method `POST`, header `X-API-Key`.
4. **Get Dictionary Value** `job_id` → **Set Variable** `JobID`.
5. **Set Variable** `Keep` to `1`.

**Wait for the question** (the interactive search takes up to 90 seconds)

6. **Repeat** 20 times, with this inside:
   - **If** `Keep` **is** `1`:
     - **Text**: `https://mf.izebra.net/jobs/` + `JobID`.
     - **Get Contents of URL** → Method `GET`, header `X-API-Key`.
     - **Set Variable** `Reply` to **Contents of URL**.
     - **Get Dictionary Value** `status` from `Reply` → **Set Variable** `Status`.
     - **Set Variable** `Keep` to `0`.
     - **If** `Status` **is** `queued` → **Set Variable** `Keep` to `1`.
     - **If** `Status` **is** `running` → **Set Variable** `Keep` to `1`.
     - **If** `Keep` **is** `1` → **Wait** 5 seconds.

**Read the answer**

7. **Get Dictionary Value** `message` from `Reply` → **Speak Text**.
8. **If** `Status` **is not** `awaiting_choice` → **Stop Shortcut**. (The job
   finished or failed on its own. Step 7 already said why.)

**Pick a release**

9. **Get Dictionary Value** `choices` from `Reply`.
10. **Repeat with Each** item in it:
    - **Get Dictionary Value** `pick_label` → **Add to Variable** `Labels`.
11. **Choose from List** `Labels`. Each row reads
    `1. FLAC, 312 MB, 24 seeders, Redacted`.
12. **Split Text** the chosen row by Custom separator `. ` → **Get Item 1 from
    List**. That is the number `n`.
13. **Text**: `https://mf.izebra.net/choose?job_id=` + `JobID` + `&n=` +
    the number from step 12.
14. **Get Contents of URL** → Method `POST`, header `X-API-Key`.
15. **Get Dictionary Value** `message` → **Speak Text**
    ("Sending 'Lay Cold' … to the download client.").

The choice list stays valid for 15 minutes. After that, `/choose` answers `409`
and the job reads `expired`.

---

## 🛠️ Contributing

PRs welcome. This script is middleware around Lidarr + yt-dlp, not a Lidarr
replacement. Keep it a single bash-friendly executable.

---

## 📜 License

GPL V3.0