alteropen/streamyfin
1
0
Fork 0
mirror of https://github.com/streamyfin/streamyfin.git synced 2026-02-19 18:42:25 +00:00
Code Issues Actions 7 Packages Projects Releases Wiki Activity

wip

Browse Source
This commit is contained in:
Fredrik Burmester
2026-01-19 08:21:55 +01:00
parent a8c07a31d3
commit a173db9180
14 changed files with 1718 additions and 262 deletions
Download Patch File Download Diff File Expand all files Collapse all files

580
docs/research/hdr-mpv.md
View File

@@ -4,11 +4,13 @@
HDR content appears washed out on Apple TV when using the mpv-based player. The TV doesn't show an HDR indicator and colors look flat compared to other apps like Infuse.
## Current Implementation
**Key Discovery:** HDR works correctly on iPhone but not on tvOS, despite using the same mpv player.
Streamyfin uses MPVKit with the `vo_avfoundation` video output driver, which renders video to `AVSampleBufferDisplayLayer`. This enables Picture-in-Picture (PiP) support but has HDR limitations.
---
**Current code in `MpvPlayerView.swift`:**
## Why HDR Works on iPhone
In `MpvPlayerView.swift`:
```swift
#if !os(tvOS)
if #available(iOS 17.0, *) {
@@ -17,141 +19,392 @@ if #available(iOS 17.0, *) {
#endif
```
The tvOS exclusion was intentional because `wantsExtendedDynamicRangeContent` was believed to be iOS-only, but this may not be accurate for tvOS 17.0+.
On iOS 17+, setting `wantsExtendedDynamicRangeContent = true` on `AVSampleBufferDisplayLayer` enables Extended Dynamic Range (EDR). This tells the display layer to preserve HDR metadata and render in high dynamic range.
**This API does not exist on tvOS.** Attempting to use it results in:
> 'wantsExtendedDynamicRangeContent' is unavailable in tvOS
tvOS uses a different HDR architecture designed for external displays via HDMI.
---
## Research Findings
## tvOS HDR Architecture
### 1. This is a Known Industry-Wide Limitation
Unlike iPhone (integrated display), Apple TV connects to external TVs. Apple expects apps to:
The same HDR issue exists in multiple projects:
1. **Use `AVDisplayCriteria`** to request display mode changes
2. **Attach proper colorspace metadata** to pixel buffers
3. **Let the TV handle HDR rendering** via HDMI passthrough
This is how Netflix, Infuse, and the TV app work - they signal "I'm playing HDR10 at 24fps" and tvOS switches the TV to that mode.
---
## MPVKit vo_avfoundation Analysis
**Location:** `/MPVKit/Sources/BuildScripts/patch/libmpv/0004-avfoundation-video-output.patch`
### Existing HDR Infrastructure
The driver has comprehensive HDR support already built in:
#### 1. HDR Metadata Copy Function (lines 253-270)
```c
static void copy_hdr_metadata(CVPixelBufferRef src, CVPixelBufferRef dst)
{
const CFStringRef keys[] = {
kCVImageBufferTransferFunctionKey, // PQ for HDR10, HLG for HLG
kCVImageBufferColorPrimariesKey, // BT.2020 for HDR
kCVImageBufferYCbCrMatrixKey,
kCVImageBufferMasteringDisplayColorVolumeKey, // HDR10 static metadata
kCVImageBufferContentLightLevelInfoKey, // MaxCLL, MaxFALL
};
for (size_t i = 0; i < MP_ARRAY_SIZE(keys); i++) {
CFTypeRef value = CVBufferGetAttachment(src, keys[i], NULL);
if (value) {
CVBufferSetAttachment(dst, keys[i], value, kCVAttachmentMode_ShouldPropagate);
}
}
}
```
#### 2. 10-bit HDR Format Support (lines 232-247)
```c
// For 10-bit HDR content (P010), use RGBA half-float to preserve HDR precision
if (format == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange ||
format == kCVPixelFormatType_420YpCbCr10BiPlanarFullRange) {
outputFormat = kCVPixelFormatType_64RGBAHalf;
}
```
#### 3. HDR-Safe GPU Compositing (lines 694-695)
```c
CGColorSpaceRef workingColorSpace = CGColorSpaceCreateWithName(
kCGColorSpaceExtendedLinearDisplayP3);
```
### The Problem: Metadata Not Attached in Main Code Path
**Critical Finding:** `copy_hdr_metadata()` is only called during OSD compositing (line 609-610):
```c
// In composite mode, render OSD and composite onto frame
if (p->composite_osd) {
render_osd(vo, pts);
CVPixelBufferRef composited = composite_frame(vo, pixbuf);
// copy_hdr_metadata() called inside composite_frame()
}
```
If `composite_osd` is false (default), **HDR metadata is never attached**.
### Frame Flow Analysis
```
draw_frame() called
│
├─► Hardware decoded (IMGFMT_VIDEOTOOLBOX)
│ └─► pixbuf = mpi->planes[3] // Direct from VideoToolbox
│ └─► Metadata SHOULD be attached by decoder, but not verified
│
└─► Software decoded (NV12, 420P, P010)
└─► upload_software_frame()
└─► Creates new CVPixelBuffer
└─► Copies pixel data only
└─► ❌ NO colorspace metadata attached!
▼
CMVideoFormatDescriptionCreateForImageBuffer(finalBuffer)
└─► Format description created FROM pixel buffer
└─► If buffer lacks HDR metadata, format won't have it
▼
[displayLayer enqueueSampleBuffer:buf]
└─► Sent to display layer without HDR signal
```
---
## Root Cause Summary
| Issue | Impact |
|-------|--------|
| `wantsExtendedDynamicRangeContent` unavailable on tvOS | Can't use iOS EDR approach |
| `copy_hdr_metadata()` only runs during OSD compositing | Main playback path skips HDR metadata |
| Software decoded frames get no colorspace attachments | mpv knows colorspace but doesn't pass it to pixel buffer |
| VideoToolbox metadata not verified | May or may not have HDR attachments |
---
## mp_image Colorspace Structures
mpv uses libplacebo's colorspace structures. Here's how colorspace info flows:
### Structure Hierarchy
```
mp_image (video/mp_image.h)
└─► params: mp_image_params
└─► color: pl_color_space
│ ├─► primaries: pl_color_primaries (BT.2020, etc.)
│ ├─► transfer: pl_color_transfer (PQ, HLG, etc.)
│ └─► hdr: pl_hdr_metadata (MaxCLL, MaxFALL, etc.)
└─► repr: pl_color_repr
├─► sys: pl_color_system (BT.2100_PQ, etc.)
└─► levels: pl_color_levels (TV/Full range)
```
### Key Enums
#### Color Primaries (`enum pl_color_primaries`)
```c
PL_COLOR_PRIM_UNKNOWN = 0,
PL_COLOR_PRIM_BT_709, // HD/SDR standard
PL_COLOR_PRIM_BT_2020, // UHD/HDR wide gamut ← HDR
PL_COLOR_PRIM_DCI_P3, // DCI P3 (cinema)
PL_COLOR_PRIM_DISPLAY_P3, // Display P3 (Apple)
// ... more
```
#### Transfer Functions (`enum pl_color_transfer`)
```c
PL_COLOR_TRC_UNKNOWN = 0,
PL_COLOR_TRC_BT_1886, // SDR gamma
PL_COLOR_TRC_SRGB, // sRGB
PL_COLOR_TRC_PQ, // SMPTE 2084 PQ (HDR10/DolbyVision) ← HDR
PL_COLOR_TRC_HLG, // ITU-R BT.2100 HLG ← HDR
// ... more
```
#### Color Systems (`enum pl_color_system`)
```c
PL_COLOR_SYSTEM_BT_709, // HD/SDR
PL_COLOR_SYSTEM_BT_2020_NC, // UHD (non-constant luminance)
PL_COLOR_SYSTEM_BT_2100_PQ, // HDR10 ← HDR
PL_COLOR_SYSTEM_BT_2100_HLG, // HLG ← HDR
PL_COLOR_SYSTEM_DOLBYVISION, // Dolby Vision ← HDR
// ... more
```
### HDR Metadata Structure (`struct pl_hdr_metadata`)
```c
struct pl_hdr_metadata {
struct pl_raw_primaries prim; // CIE xy primaries
float min_luma, max_luma; // Luminance range (cd/m²)
float max_cll; // Maximum Content Light Level
float max_fall; // Maximum Frame-Average Light Level
// ... more
};
```
### Accessing Colorspace in vo_avfoundation
```c
// In draw_frame():
struct mp_image *mpi = frame->current;
// Color primaries
enum pl_color_primaries prim = mpi->params.color.primaries;
// Transfer function
enum pl_color_transfer trc = mpi->params.color.transfer;
// HDR metadata
struct pl_hdr_metadata hdr = mpi->params.color.hdr;
// HDR detection
bool is_hdr = (trc == PL_COLOR_TRC_PQ || trc == PL_COLOR_TRC_HLG);
bool is_wide_gamut = (prim == PL_COLOR_PRIM_BT_2020);
```
---
## The Fix
### Required Changes in vo_avfoundation
**File:** `MPVKit/Sources/BuildScripts/patch/libmpv/0004-avfoundation-video-output.patch`
**Location:** After line 821 in `draw_frame()`, before the sample buffer is created.
#### Add Required Include
```c
#include "video/csputils.h" // For pl_color_* enums (if not already included)
```
#### Add HDR Metadata Attachment Function
```c
// Add after copy_hdr_metadata() function (around line 270)
static void attach_hdr_metadata(struct vo *vo, CVPixelBufferRef pixbuf,
struct mp_image *mpi)
{
enum pl_color_primaries prim = mpi->params.color.primaries;
enum pl_color_transfer trc = mpi->params.color.transfer;
// Attach BT.2020 color primaries (HDR wide color gamut)
if (prim == PL_COLOR_PRIM_BT_2020) {
CVBufferSetAttachment(pixbuf, kCVImageBufferColorPrimariesKey,
kCVImageBufferColorPrimaries_ITU_R_2020,
kCVAttachmentMode_ShouldPropagate);
CVBufferSetAttachment(pixbuf, kCVImageBufferYCbCrMatrixKey,
kCVImageBufferYCbCrMatrix_ITU_R_2020,
kCVAttachmentMode_ShouldPropagate);
MP_VERBOSE(vo, "HDR: Attached BT.2020 color primaries\n");
}
// Attach PQ transfer function (HDR10/Dolby Vision)
if (trc == PL_COLOR_TRC_PQ) {
CVBufferSetAttachment(pixbuf, kCVImageBufferTransferFunctionKey,
kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ,
kCVAttachmentMode_ShouldPropagate);
MP_VERBOSE(vo, "HDR: Attached PQ transfer function (HDR10)\n");
}
// Attach HLG transfer function
else if (trc == PL_COLOR_TRC_HLG) {
CVBufferSetAttachment(pixbuf, kCVImageBufferTransferFunctionKey,
kCVImageBufferTransferFunction_ITU_R_2100_HLG,
kCVAttachmentMode_ShouldPropagate);
MP_VERBOSE(vo, "HDR: Attached HLG transfer function\n");
}
// Attach HDR static metadata if available
struct pl_hdr_metadata hdr = mpi->params.color.hdr;
if (hdr.max_cll > 0 || hdr.max_fall > 0) {
// ContentLightLevelInfo is a 4-byte structure:
// - 2 bytes: MaxCLL (max content light level)
// - 2 bytes: MaxFALL (max frame-average light level)
uint16_t cll_data[2] = {
(uint16_t)fminf(hdr.max_cll, 65535.0f),
(uint16_t)fminf(hdr.max_fall, 65535.0f)
};
CFDataRef cllInfo = CFDataCreate(NULL, (const UInt8 *)cll_data, sizeof(cll_data));
if (cllInfo) {
CVBufferSetAttachment(pixbuf, kCVImageBufferContentLightLevelInfoKey,
cllInfo, kCVAttachmentMode_ShouldPropagate);
CFRelease(cllInfo);
MP_VERBOSE(vo, "HDR: Attached CLL metadata (MaxCLL=%d, MaxFALL=%d)\n",
cll_data[0], cll_data[1]);
}
}
}
```
#### Call the Function in draw_frame()
```c
// In draw_frame(), after line 821 (after getting pixbuf), add:
// Attach HDR colorspace metadata to pixel buffer
// This ensures the display layer receives proper HDR signaling
attach_hdr_metadata(vo, pixbuf, mpi);
```
### Complete draw_frame() Modification
The modified section should look like:
```c
CVPixelBufferRef pixbuf = NULL;
bool pixbufNeedsRelease = false;
// Handle different input formats
if (mpi->imgfmt == IMGFMT_VIDEOTOOLBOX) {
// Hardware decoded: zero-copy passthrough
pixbuf = (CVPixelBufferRef)mpi->planes[3];
} else {
// Software decoded: upload to CVPixelBuffer
pixbuf = upload_software_frame(vo, mpi);
if (!pixbuf) {
MP_ERR(vo, "Failed to upload software frame\n");
mp_image_unrefp(&mpi);
return false;
}
pixbufNeedsRelease = true;
}
// >>> NEW: Attach HDR colorspace metadata <<<
attach_hdr_metadata(vo, pixbuf, mpi);
CVPixelBufferRef finalBuffer = pixbuf;
bool needsRelease = false;
// ... rest of the function
```
---
## Alternative Solutions
### Option A: Enable composite_osd Mode (Quick Test)
Since `copy_hdr_metadata()` works in composite mode, try enabling it:
```
--avfoundation-composite-osd=yes
```
This would trigger the existing HDR metadata path. Downside: OSD compositing has performance overhead.
### Option B: Full vo_avfoundation Fix (Recommended)
Modify the driver to always attach colorspace metadata based on `mp_image` params. This is the implementation described above.
### Option C: Dual Player Approach
Use AVPlayer for HDR content, mpv for everything else. This is what Swiftfin does.
---
## Implementation Checklist
- [ ] Clone MPVKit fork
- [ ] Modify `0004-avfoundation-video-output.patch`:
- [ ] Add `attach_hdr_metadata()` function
- [ ] Call it in `draw_frame()` after getting pixbuf
- [ ] Add necessary includes if needed
- [ ] Rebuild MPVKit
- [ ] Test with HDR10 content on tvOS
- [ ] Verify TV shows HDR indicator
- [ ] Test with HLG content
- [ ] Test with Dolby Vision content (may need additional work)
---
## Current Implementation Status
**What's implemented in Streamyfin:**
1. **HDR Detection** (`MPVLayerRenderer.swift`)
- Reads `video-params/primaries` and `video-params/gamma` from mpv
- Detects HDR10 (bt.2020 + pq), HLG, Dolby Vision
2. **AVDisplayCriteria** (`MpvPlayerView.swift`)
- Sets `preferredDisplayCriteria` on tvOS 17.0+ when HDR detected
- Creates CMFormatDescription with HDR color extensions
3. **target-colorspace-hint** (`MPVLayerRenderer.swift`)
- Added `target-colorspace-hint=yes` for tvOS
**What's NOT working:**
- TV doesn't show HDR indicator
- Colors appear washed out
- The pixel buffers lack HDR metadata attachments ← **This is what the fix addresses**
---
## Industry Context
| Project | Player | HDR Status |
|---------|--------|------------|
| [Swiftfin](https://github.com/jellyfin/Swiftfin/issues/811) | VLCKit | Washed out, no HDR signal |
| [Plex](https://freetime.mikeconnelly.com/archives/8360) | mpv (Enhanced Player) | No HDR support |
| [Swiftfin](https://github.com/jellyfin/Swiftfin/issues/811) | VLCKit | Washed out, uses AVPlayer for HDR |
| [Plex](https://freetime.mikeconnelly.com/archives/8360) | mpv | No HDR support |
| Infuse | Custom Metal engine | Works correctly |
**Key quote from mpv maintainer** ([issue #9633](https://github.com/mpv-player/mpv/issues/9633)):
> "mpv doesn't support metal at all (likely won't ever)"
### 2. Why Infuse Works
Infuse uses a **custom Metal-based video rendering engine** built from scratch - not mpv, not VLCKit, not AVPlayer. This allows them to properly handle HDR passthrough with the correct pixel formats and color spaces.
Source: [Firecore Community](https://community.firecore.com/t/what-player-does-infuse-use/38003)
### 3. Swiftfin's Solution
Swiftfin offers two players:
- **VLCKit player** (default) - No HDR support
- **Native player (AVPlayer)** - HDR works correctly
When using the native player with proper stream configuration (HEVC `hvc1`, fMP4-HLS), Apple TV correctly switches to HDR mode.
Source: [Swiftfin Issue #331](https://github.com/jellyfin/Swiftfin/issues/331)
### 4. Apple's HDR Requirements
According to [WWDC22: Display HDR video in EDR with AVFoundation and Metal](https://developer.apple.com/videos/play/wwdc2022/110565/):
**Required for EDR (Extended Dynamic Range):**
```swift
// On CAMetalLayer
layer.wantsExtendedDynamicRangeContent = true
layer.pixelFormat = MTLPixelFormatRGBA16Float
layer.colorspace = kCGColorSpaceExtendedLinearDisplayP3
```
**For AVPlayerItemVideoOutput:**
```swift
let videoColorProperties = [
AVVideoColorPrimariesKey: AVVideoColorPrimaries_P3_D65,
AVVideoTransferFunctionKey: AVVideoTransferFunction_Linear,
AVVideoYCbCrMatrixKey: AVVideoYCbCrMatrix_ITU_R_2020
]
```
### 5. CVPixelBuffer HDR Metadata
For pixel buffers to be recognized as HDR, they need colorspace attachments:
```c
CVBufferSetAttachment(pixelBuffer, kCVImageBufferColorPrimariesKey,
kCVImageBufferColorPrimaries_ITU_R_2020,
kCVAttachmentMode_ShouldPropagate);
CVBufferSetAttachment(pixelBuffer, kCVImageBufferTransferFunctionKey,
kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ, // HDR10
kCVAttachmentMode_ShouldPropagate);
CVBufferSetAttachment(pixelBuffer, kCVImageBufferYCbCrMatrixKey,
kCVImageBufferYCbCrMatrix_ITU_R_2020,
kCVAttachmentMode_ShouldPropagate);
```
### 6. macOS EDR Research
From [mpv issue #7341](https://github.com/mpv-player/mpv/issues/7341), testing showed:
- **mpv playback**: `maximumExtendedDynamicRangeColorComponentValue: 1.0`
- **QuickTime playback**: `maximumExtendedDynamicRangeColorComponentValue: 2.0`
QuickTime uses `CAOpenGLLayer.wantsExtendedDynamicRangeContent` to enable EDR.
### 7. iOS/tvOS OpenGL Limitations
From [mpv issue #8467](https://github.com/mpv-player/mpv/issues/8467):
> "Disabling HDR peak computation (one or more of the following is not supported: compute shaders=0, SSBO=0)"
Apple's OpenGL implementation lacks compute shader and SSBO support, making traditional HDR peak computation impossible.
---
## Potential Solutions
### ~~Option A: Enable EDR on tvOS (Quick Test)~~ - RULED OUT
**Status:** Tested and failed - `wantsExtendedDynamicRangeContent` is **not available on tvOS**.
The API is iOS-only. Attempting to use it on tvOS results in:
> 'wantsExtendedDynamicRangeContent' is unavailable in tvOS
This confirms tvOS requires a different approach for HDR.
### Option B: Modify vo_avfoundation in MPVKit
Add HDR colorspace metadata when creating CVPixelBuffers.
**Location:** `streamyfin/MPVKit` fork, vo_avfoundation driver
**Required changes:**
1. Detect HDR content (check video color primaries/transfer function)
2. Attach colorspace metadata to pixel buffers
3. Possibly configure AVSampleBufferDisplayLayer for HDR
**Pros:** Fixes the root cause
**Cons:** Requires modifying and rebuilding MPVKit
### Option C: Dual Player Approach (Like Swiftfin)
Implement AVPlayer-based playback for HDR content, keep mpv for everything else.
**Pros:** Proven solution, full HDR/DV support
**Cons:** Significant development effort, two player implementations to maintain
### Option D: Accept Limitation
Document that HDR passthrough is not supported with the mpv player on tvOS.
**Pros:** No development work
**Cons:** Poor user experience for HDR content
---
## Recommended Approach
1. **First:** Try Option A (enable EDR on tvOS) - simple test
2. **If that fails:** Investigate Option B (modify vo_avfoundation in MPVKit)
3. **Long-term:** Consider Option C (dual player) for full HDR support
**Key insight:** No mpv-based player has solved HDR on tvOS. This fix could be a first.
---
@@ -161,60 +414,23 @@ Document that HDR passthrough is not supported with the mpv player on tvOS.
- [AVDisplayManager](https://developer.apple.com/documentation/avkit/avdisplaymanager)
- [AVDisplayCriteria](https://developer.apple.com/documentation/avkit/avdisplaycriteria)
- [WWDC22: Display HDR video in EDR](https://developer.apple.com/videos/play/wwdc2022/110565/)
- [WWDC20: Edit and play back HDR video](https://developer.apple.com/videos/play/wwdc2020/10009/)
### Related Projects
- [Swiftfin HDR Issues](https://github.com/jellyfin/Swiftfin/issues/811)
- [Swiftfin Match Content](https://github.com/jellyfin/Swiftfin/issues/331)
- [mpv HDR on iOS](https://github.com/mpv-player/mpv/issues/9633)
- [mpv macOS EDR](https://github.com/mpv-player/mpv/issues/7341)
- [mpv HDR passthrough](https://github.com/mpv-player/mpv/issues/11812)
### CVImageBuffer Keys
- `kCVImageBufferColorPrimariesKey` - Color gamut (BT.709, BT.2020, P3)
- `kCVImageBufferTransferFunctionKey` - Transfer function (sRGB, PQ, HLG)
- `kCVImageBufferYCbCrMatrixKey` - YCbCr conversion matrix
- `kCVImageBufferMasteringDisplayColorVolumeKey` - Mastering display metadata
- `kCVImageBufferContentLightLevelInfoKey` - MaxCLL/MaxFALL
### Articles
- [Plex's mpv Player](https://freetime.mikeconnelly.com/archives/8360)
- [Rendering HDR Video with AVFoundation and Metal](https://metalbyexample.com/hdr-video/)
### mpv/libplacebo Source
- mp_image struct: `video/mp_image.h`
- Colorspace enums: libplacebo `pl_color.h`
- vo_avfoundation: `MPVKit/Sources/BuildScripts/patch/libmpv/0004-avfoundation-video-output.patch`
---
## Current Implementation Status
**What we've implemented so far:**
1. **HDR Detection** (`MPVLayerRenderer.swift`)
- Reads `video-params/primaries` and `video-params/gamma` from mpv
- Detects HDR10 (bt.2020 + pq), HLG, Dolby Vision
- Logs: `HDR Detection - primaries: bt.2020, gamma: pq, fps: 23.976`
2. **AVDisplayCriteria** (`MpvPlayerView.swift`)
- Sets `preferredDisplayCriteria` on tvOS 17.0+ when HDR detected
- Creates CMFormatDescription with proper HDR color extensions
- Logs: `🎬 HDR: Setting display criteria to hdr10, fps: 23.976`
3. **target-colorspace-hint** (`MPVLayerRenderer.swift`)
- Added `target-colorspace-hint=yes` for tvOS to signal colorspace to display
**What's NOT working:**
- TV doesn't show HDR indicator
- Colors still appear washed out
- The display mode switch may not be happening
---
## Next Steps for Investigation
1. **Verify AVDisplayCriteria is being honored:**
- Check if Apple TV settings allow app-requested display mode changes
- Verify the CMFormatDescription is correctly formed
2. **Examine vo_avfoundation pixel buffer creation:**
- Clone MPVKit source
- Find where CVPixelBuffers are created
- Check if colorspace attachments are being set
3. **Test with AVSampleBufferDisplayLayer debugging:**
- Log pixel buffer attachments
- Verify layer configuration
4. **Consider testing with VLCKit:**
- Swiftfin's VLCKit has same issue
- Their solution: use AVPlayer for HDR content
### Key Functions in vo_avfoundation
| Function | Line | Purpose |
|----------|------|---------|
| `draw_frame()` | 781 | Main frame rendering |
| `copy_hdr_metadata()` | 253 | Copy HDR metadata between buffers |
| `upload_software_frame()` | 295 | Upload SW frames to CVPixelBuffer |
| `composite_frame()` | 582 | OSD compositing with HDR support |