mirror of
https://github.com/streamyfin/streamyfin.git
synced 2026-07-16 17:33:00 +01:00
docs: hdr
This commit is contained in:
Fredrik Burmester
220
docs/research/hdr-mpv.md
Normal file
220
docs/research/hdr-mpv.md
Normal file
@@ -0,0 +1,220 @@
|
|||||||
|
# HDR Support on tvOS with mpv - Research Document
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
HDR content appears washed out on Apple TV when using the mpv-based player. The TV doesn't show an HDR indicator and colors look flat compared to other apps like Infuse.
|
||||||
|
|
||||||
|
## Current Implementation
|
||||||
|
|
||||||
|
Streamyfin uses MPVKit with the `vo_avfoundation` video output driver, which renders video to `AVSampleBufferDisplayLayer`. This enables Picture-in-Picture (PiP) support but has HDR limitations.
|
||||||
|
|
||||||
|
**Current code in `MpvPlayerView.swift`:**
|
||||||
|
```swift
|
||||||
|
#if !os(tvOS)
|
||||||
|
if #available(iOS 17.0, *) {
|
||||||
|
displayLayer.wantsExtendedDynamicRangeContent = true
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
```
|
||||||
|
|
||||||
|
The tvOS exclusion was intentional because `wantsExtendedDynamicRangeContent` was believed to be iOS-only, but this may not be accurate for tvOS 17.0+.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Research Findings
|
||||||
|
|
||||||
|
### 1. This is a Known Industry-Wide Limitation
|
||||||
|
|
||||||
|
The same HDR issue exists in multiple projects:
|
||||||
|
|
||||||
|
| Project | Player | HDR Status |
|
||||||
|
|---------|--------|------------|
|
||||||
|
| [Swiftfin](https://github.com/jellyfin/Swiftfin/issues/811) | VLCKit | Washed out, no HDR signal |
|
||||||
|
| [Plex](https://freetime.mikeconnelly.com/archives/8360) | mpv (Enhanced Player) | No HDR support |
|
||||||
|
| Infuse | Custom Metal engine | Works correctly |
|
||||||
|
|
||||||
|
**Key quote from mpv maintainer** ([issue #9633](https://github.com/mpv-player/mpv/issues/9633)):
|
||||||
|
> "mpv doesn't support metal at all (likely won't ever)"
|
||||||
|
|
||||||
|
### 2. Why Infuse Works
|
||||||
|
|
||||||
|
Infuse uses a **custom Metal-based video rendering engine** built from scratch - not mpv, not VLCKit, not AVPlayer. This allows them to properly handle HDR passthrough with the correct pixel formats and color spaces.
|
||||||
|
|
||||||
|
Source: [Firecore Community](https://community.firecore.com/t/what-player-does-infuse-use/38003)
|
||||||
|
|
||||||
|
### 3. Swiftfin's Solution
|
||||||
|
|
||||||
|
Swiftfin offers two players:
|
||||||
|
- **VLCKit player** (default) - No HDR support
|
||||||
|
- **Native player (AVPlayer)** - HDR works correctly
|
||||||
|
|
||||||
|
When using the native player with proper stream configuration (HEVC `hvc1`, fMP4-HLS), Apple TV correctly switches to HDR mode.
|
||||||
|
|
||||||
|
Source: [Swiftfin Issue #331](https://github.com/jellyfin/Swiftfin/issues/331)
|
||||||
|
|
||||||
|
### 4. Apple's HDR Requirements
|
||||||
|
|
||||||
|
According to [WWDC22: Display HDR video in EDR with AVFoundation and Metal](https://developer.apple.com/videos/play/wwdc2022/110565/):
|
||||||
|
|
||||||
|
**Required for EDR (Extended Dynamic Range):**
|
||||||
|
```swift
|
||||||
|
// On CAMetalLayer
|
||||||
|
layer.wantsExtendedDynamicRangeContent = true
|
||||||
|
layer.pixelFormat = MTLPixelFormatRGBA16Float
|
||||||
|
layer.colorspace = kCGColorSpaceExtendedLinearDisplayP3
|
||||||
|
```
|
||||||
|
|
||||||
|
**For AVPlayerItemVideoOutput:**
|
||||||
|
```swift
|
||||||
|
let videoColorProperties = [
|
||||||
|
AVVideoColorPrimariesKey: AVVideoColorPrimaries_P3_D65,
|
||||||
|
AVVideoTransferFunctionKey: AVVideoTransferFunction_Linear,
|
||||||
|
AVVideoYCbCrMatrixKey: AVVideoYCbCrMatrix_ITU_R_2020
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. CVPixelBuffer HDR Metadata
|
||||||
|
|
||||||
|
For pixel buffers to be recognized as HDR, they need colorspace attachments:
|
||||||
|
|
||||||
|
```c
|
||||||
|
CVBufferSetAttachment(pixelBuffer, kCVImageBufferColorPrimariesKey,
|
||||||
|
kCVImageBufferColorPrimaries_ITU_R_2020,
|
||||||
|
kCVAttachmentMode_ShouldPropagate);
|
||||||
|
CVBufferSetAttachment(pixelBuffer, kCVImageBufferTransferFunctionKey,
|
||||||
|
kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ, // HDR10
|
||||||
|
kCVAttachmentMode_ShouldPropagate);
|
||||||
|
CVBufferSetAttachment(pixelBuffer, kCVImageBufferYCbCrMatrixKey,
|
||||||
|
kCVImageBufferYCbCrMatrix_ITU_R_2020,
|
||||||
|
kCVAttachmentMode_ShouldPropagate);
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. macOS EDR Research
|
||||||
|
|
||||||
|
From [mpv issue #7341](https://github.com/mpv-player/mpv/issues/7341), testing showed:
|
||||||
|
- **mpv playback**: `maximumExtendedDynamicRangeColorComponentValue: 1.0`
|
||||||
|
- **QuickTime playback**: `maximumExtendedDynamicRangeColorComponentValue: 2.0`
|
||||||
|
|
||||||
|
QuickTime uses `CAOpenGLLayer.wantsExtendedDynamicRangeContent` to enable EDR.
|
||||||
|
|
||||||
|
### 7. iOS/tvOS OpenGL Limitations
|
||||||
|
|
||||||
|
From [mpv issue #8467](https://github.com/mpv-player/mpv/issues/8467):
|
||||||
|
> "Disabling HDR peak computation (one or more of the following is not supported: compute shaders=0, SSBO=0)"
|
||||||
|
|
||||||
|
Apple's OpenGL implementation lacks compute shader and SSBO support, making traditional HDR peak computation impossible.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Potential Solutions
|
||||||
|
|
||||||
|
### ~~Option A: Enable EDR on tvOS (Quick Test)~~ - RULED OUT
|
||||||
|
|
||||||
|
**Status:** Tested and failed - `wantsExtendedDynamicRangeContent` is **not available on tvOS**.
|
||||||
|
|
||||||
|
The API is iOS-only. Attempting to use it on tvOS results in:
|
||||||
|
> 'wantsExtendedDynamicRangeContent' is unavailable in tvOS
|
||||||
|
|
||||||
|
This confirms tvOS requires a different approach for HDR.
|
||||||
|
|
||||||
|
### Option B: Modify vo_avfoundation in MPVKit
|
||||||
|
|
||||||
|
Add HDR colorspace metadata when creating CVPixelBuffers.
|
||||||
|
|
||||||
|
**Location:** `streamyfin/MPVKit` fork, vo_avfoundation driver
|
||||||
|
|
||||||
|
**Required changes:**
|
||||||
|
1. Detect HDR content (check video color primaries/transfer function)
|
||||||
|
2. Attach colorspace metadata to pixel buffers
|
||||||
|
3. Possibly configure AVSampleBufferDisplayLayer for HDR
|
||||||
|
|
||||||
|
**Pros:** Fixes the root cause
|
||||||
|
**Cons:** Requires modifying and rebuilding MPVKit
|
||||||
|
|
||||||
|
### Option C: Dual Player Approach (Like Swiftfin)
|
||||||
|
|
||||||
|
Implement AVPlayer-based playback for HDR content, keep mpv for everything else.
|
||||||
|
|
||||||
|
**Pros:** Proven solution, full HDR/DV support
|
||||||
|
**Cons:** Significant development effort, two player implementations to maintain
|
||||||
|
|
||||||
|
### Option D: Accept Limitation
|
||||||
|
|
||||||
|
Document that HDR passthrough is not supported with the mpv player on tvOS.
|
||||||
|
|
||||||
|
**Pros:** No development work
|
||||||
|
**Cons:** Poor user experience for HDR content
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended Approach
|
||||||
|
|
||||||
|
1. **First:** Try Option A (enable EDR on tvOS) - simple test
|
||||||
|
2. **If that fails:** Investigate Option B (modify vo_avfoundation in MPVKit)
|
||||||
|
3. **Long-term:** Consider Option C (dual player) for full HDR support
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Technical References
|
||||||
|
|
||||||
|
### Apple Documentation
|
||||||
|
- [AVDisplayManager](https://developer.apple.com/documentation/avkit/avdisplaymanager)
|
||||||
|
- [AVDisplayCriteria](https://developer.apple.com/documentation/avkit/avdisplaycriteria)
|
||||||
|
- [WWDC22: Display HDR video in EDR](https://developer.apple.com/videos/play/wwdc2022/110565/)
|
||||||
|
- [WWDC20: Edit and play back HDR video](https://developer.apple.com/videos/play/wwdc2020/10009/)
|
||||||
|
|
||||||
|
### Related Projects
|
||||||
|
- [Swiftfin HDR Issues](https://github.com/jellyfin/Swiftfin/issues/811)
|
||||||
|
- [Swiftfin Match Content](https://github.com/jellyfin/Swiftfin/issues/331)
|
||||||
|
- [mpv HDR on iOS](https://github.com/mpv-player/mpv/issues/9633)
|
||||||
|
- [mpv macOS EDR](https://github.com/mpv-player/mpv/issues/7341)
|
||||||
|
- [mpv HDR passthrough](https://github.com/mpv-player/mpv/issues/11812)
|
||||||
|
|
||||||
|
### Articles
|
||||||
|
- [Plex's mpv Player](https://freetime.mikeconnelly.com/archives/8360)
|
||||||
|
- [Rendering HDR Video with AVFoundation and Metal](https://metalbyexample.com/hdr-video/)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Implementation Status
|
||||||
|
|
||||||
|
**What we've implemented so far:**
|
||||||
|
|
||||||
|
1. **HDR Detection** (`MPVLayerRenderer.swift`)
|
||||||
|
- Reads `video-params/primaries` and `video-params/gamma` from mpv
|
||||||
|
- Detects HDR10 (bt.2020 + pq), HLG, Dolby Vision
|
||||||
|
- Logs: `HDR Detection - primaries: bt.2020, gamma: pq, fps: 23.976`
|
||||||
|
|
||||||
|
2. **AVDisplayCriteria** (`MpvPlayerView.swift`)
|
||||||
|
- Sets `preferredDisplayCriteria` on tvOS 17.0+ when HDR detected
|
||||||
|
- Creates CMFormatDescription with proper HDR color extensions
|
||||||
|
- Logs: `🎬 HDR: Setting display criteria to hdr10, fps: 23.976`
|
||||||
|
|
||||||
|
3. **target-colorspace-hint** (`MPVLayerRenderer.swift`)
|
||||||
|
- Added `target-colorspace-hint=yes` for tvOS to signal colorspace to display
|
||||||
|
|
||||||
|
**What's NOT working:**
|
||||||
|
- TV doesn't show HDR indicator
|
||||||
|
- Colors still appear washed out
|
||||||
|
- The display mode switch may not be happening
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps for Investigation
|
||||||
|
|
||||||
|
1. **Verify AVDisplayCriteria is being honored:**
|
||||||
|
- Check if Apple TV settings allow app-requested display mode changes
|
||||||
|
- Verify the CMFormatDescription is correctly formed
|
||||||
|
|
||||||
|
2. **Examine vo_avfoundation pixel buffer creation:**
|
||||||
|
- Clone MPVKit source
|
||||||
|
- Find where CVPixelBuffers are created
|
||||||
|
- Check if colorspace attachments are being set
|
||||||
|
|
||||||
|
3. **Test with AVSampleBufferDisplayLayer debugging:**
|
||||||
|
- Log pixel buffer attachments
|
||||||
|
- Verify layer configuration
|
||||||
|
|
||||||
|
4. **Consider testing with VLCKit:**
|
||||||
|
- Swiftfin's VLCKit has same issue
|
||||||
|
- Their solution: use AVPlayer for HDR content
|
||||||
Reference in New Issue
Block a user