1
2
3

连续信号:  ∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿∿  (无限多点)
采样后:      ●     ●     ●     ●     (有限个点)
           t=0   t=T  t=2T  t=3T

1
2
3

时域:  ↑     ↑     ↑     ↑     (冲激串，周期 T)
       ↓ 傅里叶变换 ↓
频域:  ↑     ↑     ↑     ↑     (冲激串，周期 Ωs = 2π/T)

1
2
3
4
5
6

     ╱╲       ╱╲       ╱╲       ╱╲
    ╱  ╲     ╱  ╲     ╱  ╲     ╱  ╲
   ╱    ╲   ╱    ╲   ╱    ╲   ╱    ╲
──╱──────╲─╱──────╲─╱──────╲─╱──────╲──→ Ω
              ↑
         间隔存在，可完美重建

1
2
3
4
5
6

     ╱╲     ╱╲
    ╱  ╲╱╲╱  ╲
   ╱   ╳╳╳╳   ╲       ← 重叠！原始频谱被破坏
──╱──╱──────╲──╲──→ Ω
         ↑
    高低频信息混在一起无法分离

1
2
3
4
5
6

原始频谱: 0~30Hz 有能量

采样后频谱复制 (每 40Hz 一份):
  复制品0:  [0 ~ 30Hz]       ← 原始
  复制品1:  [40 ~ 70Hz]      ← 40+0 ~ 40+30
  复制品-1: [-40 ~ -10Hz]    ← 折叠到正频率变成 10~40Hz

1
2
3
4
5
6

频域视角：

采样信号频谱:    |原始|  复制品  |  复制品  |  复制品
                      ↑
                 只提取这一块
                 = 重建 = 低通滤波 = 去除复制品

1
2
3

sinc(t/T)          →  截止频率 = π/T = Ωs/2
sinc(2Wt)          →  截止频率 = W    (角频率)
sinc(2πf_c · t)    →  截止频率 = f_c  (Hz)

1
2
3
4
5
6
7

原始 Nyquist = Fs/2
新 Nyquist   = M·Fs/2

插零后频谱:   |原始|  镜像  |  镜像  |  镜像
                     ↑
              sinc 滤波器只保留这里
              截止 = 新采样率的 Nyquist = M·Fs/2

1
2
3
4
5
6
7

原始 Nyquist = Fs/2
新 Nyquist   = Fs/(2M)

原始频谱:     |████████████████|
                      ↑
              只保留这部分，截止 = 新采样率的 Nyquist = Fs/(2M)
              然后再抽取

1
2
3
4

原始连续信号:  ~~~波形~~~
采样点:          ●    ●    ●    ●
                  ↖   ↗   ↖   ↗
              每个点"抓住"一个sinc,叠加重建波形

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import numpy as np

def naive_resample(x, fs_in, fs_out, num_samples_out):
    """最原始的重采样：直接算 sinc"""
    output = np.zeros(num_samples_out)

    for m in range(num_samples_out):
        t = m / fs_out
        pos = t * fs_in  # 在输入序列中的"虚拟位置"

        n_center = int(np.floor(pos))
        half_len = 16  # 只取 ±16 个点（截断 sinc）

        s = 0.0
        for n in range(n_center - half_len, n_center + half_len + 1):
            if 0 <= n < len(x):
                sinc_val = np.sinc(pos - n)  # sin(πx)/(πx)
                s += x[n] * sinc_val
        output[m] = s

    return output

1
2
3
4
5
6
7
8

对每个输出样本:
  → 遍历 2 × half_len 个输入样本
  → 每个都要调用 np.sinc()（内部算 sin + 除法）

总计算量: num_out × 2 × half_len × (sin + 除法)

示例: 10 秒 48kHz 音频，half_len=32
  480000 × 64 = 3072 万次 sin() 调用

1
2

优化前:  每次算 sin(πx)/(πx)     → 贵（三角函数 + 除法）
优化后:  直接查表 sinc_table[i]   → 快（一次内存访问）

1
2
3
4
5
6

目标: sinc(3.7)
表里有: sinc(3.50), sinc(3.75)   ← 间隔 1/4

选项A: 线性插值   → sinc(3.7) ≈ 0.4×sinc(3.50) + 0.6×sinc(3.75)
选项B: 三次插值   → 用相邻 4 个点做 cubic 插值（更准）
选项C: 存更大的表  → oversample=16 或 32，直接取最近的（但费内存）

1
2
3
4
5

输出样本 0: sinc 偏移 0/4 → 系数 [a0, a1, a2, a3]
输出样本 1: sinc 偏移 1/4 → 系数 [b0, b1, b2, b3]
输出样本 2: sinc 偏移 2/4 → 系数 [c0, c1, c2, c3]
输出样本 3: sinc 偏移 3/4 → 系数 [d0, d1, d2, d3]
输出样本 4: sinc 偏移 4/4 = 0/4 → 系数 [a0, a1, a2, a3]  ← 重复！

1
2
3
4
5
6

输入 x[n] ───┬──→ [滤波器组 a] ──→ 输出 t=0/4
             ├──→ [滤波器组 b] ──→ 输出 t=1/4
             ├──→ [滤波器组 c] ──→ 输出 t=2/4
             └──→ [滤波器组 d] ──→ 输出 t=3/4

4 组滤波器，每组只对同一个输入段计算一次

1
2
3
4
5
6
7
8
9

den_rate = 160 (分母)
filt_len = 64  (滤波器长度)

sinc 表大小 = 64 × 160 = 10240 个系数
内存 ≈ 10240 × 2 bytes = 20KB

查找: sinc_table[j * den_rate + samp_frac_num]
      ↑                ↑
      第 j 个抽头      当前分数位置

1

filt_len=64, den_rate=48000 → 64 × 48000 = 307 万个系数 → 6MB 内存

1
2
3
4
5
6
7

输出位置 0.000:  用 sinc 在 ..., -2.000, -1.000, 0.000, 1.000, ... 处的值
输出位置 0.001:  用 sinc 在 ..., -1.999, -0.999, 0.001, 1.001, ... 处的值
输出位置 0.002:  用 sinc 在 ..., -1.998, -0.998, 0.002, 1.002, ... 处的值
...

den_rate=1000 时，需要为 1000 种不同的偏移各存 filt_len 个系数
表大小 = filt_len × 1000

1
2
3
4
5
6
7
8
9

oversample=16 时，只存 16 种偏移的系数：

偏移 0/16:  sinc 在 0.000, 1.000, 2.000, ... 处的值
偏移 1/16:  sinc 在 0.0625, 1.0625, 2.0625, ... 处的值
偏移 2/16:  sinc 在 0.125, 1.125, 2.125, ... 处的值
...
偏移 15/16: sinc 在 0.9375, 1.9375, 2.9375, ... 处的值

表大小 = filt_len × 16

1
2
3
4
5

信号: 48kHz → 44.1kHz 重采样
filt_len = 8  (滤波器长度，取 8 个输入样本)
oversample = 16

sinc 表大小 = 8 × 16 + 8 = 136 个系数

1
2
3
4

输入位置 pos = m × (fs_in / fs_out) = 100 × (48000/44100) = 108.8480...

整数部分: n_center = 108
小数部分: frac_input = 0.8480...

1
2
3
4

oversample_index = frac_input × oversample = 0.8480 × 16 = 13.568

整数偏移: offset = 13        ← 在 sinc 表中的基准位置
小数偏移: frac = 0.568       ← cubic 插值的输入

1
2
3
4

interp[0] = -0.16667*frac + 0.16667*frac*frac*frac;
interp[1] =  frac + 0.5*frac*frac - 0.5*frac*frac*frac;
interp[3] = -0.33333*frac + 0.5*frac*frac - 0.16667*frac*frac*frac;
interp[2] =  1.f - interp[0] - interp[1] - interp[3];

1
2

线性插值:  在两个点之间拉直线 → 对 sinc 这种振荡形状误差大
Cubic 插值: 用三次曲线拟合 → 高频失真更小，接近直接查表精度

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

场景: 48kHz → 44.1kHz, filt_len=64

Direct (den_rate=48000):
  表大小 = 64 × 48000 = 3,072,000 个系数
  内存 = 3,072,000 × 2 bytes = 6 MB

Interpolated (oversample=16):
  表大小 = 64 × 16 + 8 = 1,032 个系数
  内存 = 1,032 × 2 bytes = 2 KB

节省: 6MB → 2KB，缩小 3000 倍
代价: cubic 插值引入微小误差（~0.01dB，人耳不可感知）

1
2

// 朴素：每个样本都做一次除法
frac_pos = (m * fs_in) % fs_out / fs_out;

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// 预计算一次（整数除法，只在初始化时执行）
int_advance  = num_rate / den_rate;     // 整数部分
frac_advance = num_rate % den_rate;     // 分数累加步长

// 每个输出样本（只有整数加法和比较）
last_sample  += int_advance;            // 整数步进
samp_frac_num += frac_advance;          // 分数累加
if (samp_frac_num >= den_rate) {        // 进位
    samp_frac_num -= den_rate;
    last_sample++;
}

1
2
3
4
5
6

// 浮点版
sum += sinc_val * input_val;

// 定点版 (Q15 格式: 1 位符号 + 15 位小数)
sum += MULT16_16(sinc_val_q15, input_val_q15);
// 其中 MULT16_16 = (int32_t)a * (int32_t)b >> 15

1

audio (连续信号)  →  text (离散 token)

1

Encoder → Attention → Decoder → Output

1
2
3
4
5
6

输出: "hello world"
生成过程:
  1. 生成 "h"
  2. 基于 "h" 生成 "he"
  3. 基于 "he" 生成 "hel"
  4. ...

1
2
3

输出: "hello world"
生成过程:
  1. 直接输出完整句子 "hello world"

1
2
3
4
5
6
7

<key>NSExtension</key>
<dict>
    <key>NSExtensionPointIdentifier</key>
    <string>com.apple.broadcast-services-upload</string>
    <key>NSExtensionPrincipalClass</key>
    <string>$(PRODUCT_MODULE_NAME).SampleHandler</string>
</dict>

1
2
3
4

<key>UIBackgroundModes</key>
<array>
    <string>audio</string>
</array>

1
2
3
4
5
6

let picker = RPSystemBroadcastPickerView(
    frame: CGRect(x: 0, y: 0, width: 44, height: 44)
)
picker.preferredExtension = "com.xxx.broadcast"
picker.showsMicrophoneButton = true
view.addSubview(picker)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
    guard case .audioApp = sampleBufferType else { return }

    guard let formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer),
          let streamDesc = CMAudioFormatDescriptionGetStreamBasicDescription(formatDescription)?.pointee else {
        return
    }

    let inputSampleRate = streamDesc.mSampleRate
    let channelCount = Int(streamDesc.mChannelsPerFrame)
    let bitsPerChannel = streamDesc.mBitsPerChannel
    let formatFlags = streamDesc.mFormatFlags
    let isFloat = (formatFlags & kAudioFormatFlagIsFloat) != 0
    let isNonInterleaved = (formatFlags & kAudioFormatFlagIsNonInterleaved) != 0
    let isBigEndian = (formatFlags & kAudioFormatFlagIsBigEndian) != 0

    // 提取音频数据...
}

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146

private let targetSampleRate: Double = 16000.0

/// 处理音频样本buffer
private func processAudioBuffer(_ sampleBuffer: CMSampleBuffer, source: String) {
    guard let formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer),
          let streamDesc = CMAudioFormatDescriptionGetStreamBasicDescription(formatDescription)?.pointee else {
        return
    }

    let inputSampleRate = streamDesc.mSampleRate
    let channelCount = Int(streamDesc.mChannelsPerFrame)
    let bitsPerChannel = streamDesc.mBitsPerChannel
    let formatFlags = streamDesc.mFormatFlags
    let isFloat = (formatFlags & kAudioFormatFlagIsFloat) != 0
    let isNonInterleaved = (formatFlags & kAudioFormatFlagIsNonInterleaved) != 0
    let isBigEndian = (formatFlags & kAudioFormatFlagIsBigEndian) != 0

    // 获取 AudioBufferList
    var audioBufferList = AudioBufferList()
    var blockBuffer: CMBlockBuffer?

    let status = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
        sampleBuffer,
        bufferListSizeNeededOut: nil,
        bufferListOut: &audioBufferList,
        bufferListSize: MemoryLayout<AudioBufferList>.size,
        blockBufferAllocator: nil,
        blockBufferMemoryAllocator: nil,
        flags: kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
        blockBufferOut: &blockBuffer
    )

    guard status == noErr else { return }

    let audioBufferListPointer = UnsafeMutableAudioBufferListPointer(
        UnsafeMutablePointer<AudioBufferList>.allocate(capacity: 1)
    )
    defer {
        audioBufferListPointer.unsafeMutablePointer.deallocate()
    }

    let numBuffers = audioBufferListPointer.count
    let frameCount = CMSampleBufferGetNumSamples(sampleBuffer)
    var floatSamples: [Float] = []

    // 处理非交错格式
    if isNonInterleaved {
        var channelData: [[Float]] = []

        for bufferIndex in 0..<numBuffers {
            let buffer = audioBufferListPointer[bufferIndex]
            guard let data = buffer.mData else { continue }

            let dataByteSize = Int(buffer.mDataByteSize)
            var channelSamples: [Float] = []

            if isFloat && bitsPerChannel == 32 {
                let floatPtr = data.assumingMemoryBound(to: Float.self)
                let count = dataByteSize / MemoryLayout<Float>.size
                for i in 0..<count {
                    var value = floatPtr[i]
                    if isBigEndian {
                        value = Float(bitPattern: value.bitPattern.bigEndian)
                    }
                    channelSamples.append(value)
                }
            } else if bitsPerChannel == 16 {
                let int16Ptr = data.assumingMemoryBound(to: Int16.self)
                let count = dataByteSize / MemoryLayout<Int16>.size
                for i in 0..<count {
                    var value = int16Ptr[i]
                    if isBigEndian { value = value.bigEndian }
                    channelSamples.append(Float(value) / 32768.0)
                }
            }

            channelData.append(channelSamples)
        }

        // 混音为单声道
        if let firstChannel = channelData.first {
            if channelData.count == 1 {
                floatSamples = firstChannel
            } else {
                for i in 0..<firstChannel.count {
                    var sum: Float = 0
                    for ch in channelData where i < ch.count {
                        sum += ch[i]
                    }
                    floatSamples.append(sum / Float(channelData.count))
                }
            }
        }
    } else {
        // 交错格式处理...
    }

    // 重采样到16kHz
    if inputSampleRate != targetSampleRate {
        floatSamples = resample(floatSamples, from: inputSampleRate, to: targetSampleRate)
    }

    sharedBuffer.writeAudioSamples(floatSamples)
}

/// 使用AVAudioConverter重采样
private func resample(_ samples: [Float], from inputRate: Double, to outputRate: Double) -> [Float] {
    guard inputRate > 0 && outputRate > 0, inputRate != outputRate, !samples.isEmpty else {
        return samples
    }

    guard let inputFormat = AVAudioFormat(
        commonFormat: .pcmFormatFloat32, sampleRate: inputRate, channels: 1, interleaved: false
    ),
    let outputFormat = AVAudioFormat(
        commonFormat: .pcmFormatFloat32, sampleRate: outputRate, channels: 1, interleaved: false
    ),
    let converter = AVAudioConverter(from: inputFormat, to: outputFormat) else {
        return samples
    }

    guard let inputBuffer = AVAudioPCMBuffer(pcmFormat: inputFormat, frameCapacity: AVAudioFrameCount(samples.count)),
          let outputBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat) else {
        return samples
    }

    inputBuffer.frameLength = AVAudioFrameCount(samples.count)
    let inputData = inputBuffer.floatChannelData!
    for i in 0..<samples.count { inputData[0][i] = samples[i] }

    let ratio = outputRate / inputRate
    let outputFrameCount = Int(ceil(Double(samples.count) * ratio))
    outputBuffer.frameCapacity = AVAudioFrameCount(outputFrameCount)

    var error: NSError?
    let status = converter.convert(to: outputBuffer, error: &error) { _, outStatus in
        outStatus.pointee = .haveData
        return inputBuffer
    }

    guard status == .haveData, error == nil else { return samples }

    let outputData = outputBuffer.floatChannelData!
    let outputLength = Int(outputBuffer.frameLength)
    return (0..<outputLength).map { outputData[0][$0] }
}

1
2
3
4

<key>com.apple.security.application-groups</key>
<array>
    <string>group.com.xxx.shared</string>
</array>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

class AudioSharedBuffer {
    static let appGroupId = "group.com.xxx.shared"
    private let sharedContainerURL = FileManager.default.containerURL(
        forSecurityApplicationGroupIdentifier: Self.appGroupId
    )

    // Extension写入处理后的音频
    func writeAudioSamples(_ samples: [Float]) {
        guard let url = sharedContainerURL?.appendingPathComponent("audio.raw") else { return }

        let data = samples.withUnsafeBufferPointer { Data(buffer: $0) }

        if FileManager.default.fileExists(atPath: url.path) {
            let handle = try? FileHandle(forWritingTo: url)
            handle?.seekToEndOfFile()
            handle?.write(data)
            handle?.closeFile()
        } else {
            try? data.write(to: url)
        }

        postDarwinNotification("com.xxx.newAudioData")
    }

    // 主应用读取音频
    func readAudioSamples() -> [Float]? {
        guard let url = sharedContainerURL?.appendingPathComponent("audio.raw"),
              FileManager.default.fileExists(atPath: url.path) else { return nil }

        let data = try? Data(contentsOf: url)
        try? FileManager.default.removeItem(at: url)

        let floatCount = data?.count ?? 0 / MemoryLayout<Float>.size
        var samples = [Float](repeating: 0, count: floatCount)
        data?.copyBytes(to: samples.withUnsafeMutableBufferPointer { $0 })
        return samples
    }

    private func postDarwinNotification(_ name: String) {
        let center = CFNotificationCenterGetDarwinNotifyCenter()
        CFNotificationCenterPostNotification(center, CFNotificationName(name as CFString),
            nil, nil, true)
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

func startListening() {
    CFNotificationCenterAddObserver(
        CFNotificationCenterGetDarwinNotifyCenter(),
        Unmanaged.passUnretained(self).toOpaque(),
        { _, observer, _, _, _ in
            guard let observer = observer else { return }
            let selfPtr = Unmanaged<YourClass>.fromOpaque(observer).takeUnretainedValue()
            if let samples = selfPtr.audioBuffer.readAudioSamples() {
                selfPtr.onAudioReceived?(samples)
            }
        },
        "com.xxx.newAudioData" as CFString,
        nil,
        .deliverImmediately
    )
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

class SampleHandler: RPBroadcastSampleHandler {
    private let sharedBuffer = AudioSharedBuffer()
    private let targetSampleRate: Double = 16000.0

    override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
        sharedBuffer.clearAudioData()
    }

    override func broadcastPaused() {}

    override func broadcastResumed() {}

    override func broadcastFinished() {}

    override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer,
                                     with sampleBufferType: RPSampleBufferType) {
        switch sampleBufferType {
        case .audioApp:
            // 处理应用音频（系统播放的音频）
            let samples = convertTo16kMono(sampleBuffer)
            sharedBuffer.writeAudioSamples(samples)
        case .audioMic:
            // 忽略麦克风音频
            break
        case .video:
            // 忽略视频
            break
        }
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

   ┌───────────────────────────────┐
   │ OrtEnv （运行时环境）         │
   │ └─ 管理全局资源、线程池等     │
   └──────────────┬────────────────┘
                  │
   ┌──────────────┴────────────────┐
   │ OrtSession （推理会话）        │
   │ └─ 持有已加载的 ONNX 模型      │
   └──────────────┬────────────────┘
                  │
   ┌──────────────┴────────────────────────┐
   │ OrtRun（一次推理调用）                │
   │ ├─ 输入 OrtValue (Tensor 等)           │
   │ ├─ 输出 OrtValue                      │
   │ └─ 在 Env/Session 的线程池中执行      │
   └────────────────────────────────────────┘

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#include <onnxruntime_c_api.h>

// 创建运行时环境
OrtEnv* env = NULL;
OrtStatus* status = OrtCreateEnv(ORT_LOGGING_LEVEL_WARNING, "ORT", &env);
if (status != NULL) {
    // 错误处理
    const char* msg = OrtGetErrorMessage(status);
    OrtReleaseStatus(status);
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#include <onnxruntime_c_api.h>

// 1. 创建会话选项
OrtSessionOptions* session_options = NULL;
OrtCreateSessionOptions(&session_options);

// 2. 创建推理会话
OrtSession* session = NULL;
const char* model_path = "denoise_model.onnx";
status = OrtCreateSession(env, model_path, session_options, &session);
if (status != NULL) {
    // 错误处理
    const char* msg = OrtGetErrorMessage(status);
    OrtReleaseStatus(status);
}

// 3. 释放资源（使用完毕后）
OrtReleaseSessionOptions(session_options);
OrtReleaseSession(session);
OrtReleaseEnv(env);

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// CPU执行（默认，无需显式添加）
// 直接创建会话即可使用CPU

// CUDA执行（需要NVIDIA GPU）
OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, 0);

// TensorRT执行（需要NVIDIA GPU和TensorRT）
OrtTensorRTProviderOptions trt_options = {0};
OrtSessionOptionsAppendExecutionProvider_TensorRT(session_options, &trt_options);

// CoreML执行（macOS/iOS）
OrtSessionOptionsAppendExecutionProvider_CoreML(session_options, 0);

// 创建会话（会按顺序尝试EP，失败则回退到下一个）
OrtCreateSession(env, model_path, session_options, &session);

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

// 1. 获取输入输出信息
size_t num_input_nodes;
OrtStatus* status = OrtSessionGetInputCount(session, &num_input_nodes);

const char* input_name;
OrtTypeInfo* input_type_info;
OrtSessionGetInputName(session, 0, allocator, &input_name);
OrtSessionGetInputTypeInfo(session, 0, &input_type_info);

// 2. 准备输入数据
float input_data[] = { /* audio_features数据 */ };
int64_t input_shape[] = {1, 480};  // batch_size, feature_dim
size_t input_tensor_size = 480;

OrtValue* input_tensor = NULL;
OrtMemoryInfo* memory_info;
OrtCreateCpuMemoryInfo(OrtArenaAllocator, OrtMemTypeDefault, &memory_info);
OrtCreateTensorWithDataAsOrtValue(
    memory_info,
    input_data, input_tensor_size * sizeof(float),
    input_shape, 2,
    ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT,
    &input_tensor
);

// 3. 执行推理
const char* input_names[] = {input_name};
const char* output_names[] = {"output"};  // 根据模型实际输出名称
OrtValue* output_tensor = NULL;

status = OrtRun(session, NULL,
    input_names, &input_tensor, 1,
    output_names, 1, &output_tensor);

// 4. 获取输出数据
float* output_data;
OrtGetTensorMutableData(output_tensor, (void**)&output_data);
// 使用output_data...

// 5. 释放资源
OrtReleaseValue(output_tensor);
OrtReleaseValue(input_tensor);
OrtReleaseMemoryInfo(memory_info);

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

OrtSessionOptions* session_options = NULL;
OrtCreateSessionOptions(&session_options);

// 图优化级别
// ORT_DISABLE_ALL, ORT_ENABLE_BASIC, ORT_ENABLE_EXTENDED, ORT_ENABLE_ALL
OrtSetSessionGraphOptimizationLevel(session_options, ORT_ENABLE_ALL);

// 线程数设置
OrtSetIntraOpNumThreads(session_options, 4);  // 算子内部并行线程数
OrtSetInterOpNumThreads(session_options, 2); // 算子间并行线程数

// 内存模式
OrtEnableMemPattern(session_options);  // 启用内存模式优化
OrtEnableCpuMemArena(session_options); // 启用CPU内存池

// 执行模式
OrtSetSessionExecutionMode(session_options, ORT_SEQUENTIAL);  // 顺序执行
// OrtSetSessionExecutionMode(session_options, ORT_PARALLEL);  // 并行执行

// 优化配置文件（可选，用于更精细的控制）
// OrtSetOptimizedModelFilePath(session_options, "optimized_model.onnx");

// 创建会话时应用这些选项
OrtCreateSession(env, model_path, session_options, &session);

// 使用完毕后释放
OrtReleaseSessionOptions(session_options);

1
2
3

# PyTorch训练时的行为
lstm = nn.LSTM(input_size, hidden_size)
output, (hidden, cell) = lstm(input, (hidden, cell))  # 状态在层内部管理

1
2
3

// ONNX模型结构
// 输入: [audio_features, hidden_state, cell_state]  // 显式输入
// 输出: [denoised_features, new_hidden_state, new_cell_state]  // 显式输出

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247

import keras.backend as K
from keras.constraints import Constraint
from keras.layers import Input, Dense, GRU, concatenate
from keras.models import Model


def my_crossentropy(y_true, y_pred):
    return K.mean(2 * K.abs(y_true - 0.5) * K.binary_crossentropy(y_pred, y_true), axis=-1)


def mymask(y_true):
    return K.minimum(y_true + 1.0, 1.0)


def msse(y_true, y_pred):
    return K.mean(mymask(y_true) * K.square(K.sqrt(y_pred) - K.sqrt(y_true)), axis=-1)


def mycost(y_true, y_pred):
    return K.mean(
        mymask(y_true)
        * (
            10 * K.square(K.square(K.sqrt(y_pred) - K.sqrt(y_true)))
            + K.square(K.sqrt(y_pred) - K.sqrt(y_true))
            + 0.01 * K.binary_crossentropy(y_pred, y_true)
        ),
        axis=-1,
    )


def my_accuracy(y_true, y_pred):
    return K.mean(2 * K.abs(y_true - 0.5) * K.equal(y_true, K.round(y_pred)), axis=-1)


class WeightClip(Constraint):
    # Accept **kwargs to be compatible with Keras deserialization that may pass 'name' etc.
    def __init__(self, c=2, **kwargs):  # kwargs may include 'name'
        super().__init__()
        self.c = c

    def __call__(self, p):
        return K.clip(p, -self.c, self.c)

    def get_config(self):
        return {'name': self.__class__.__name__, 'c': self.c}


CUSTOM_OBJECTS = {
    'my_crossentropy': my_crossentropy,
    'mymask': mymask,
    'msse': msse,
    'mycost': mycost,
    'my_accuracy': my_accuracy,
    'WeightClip': WeightClip,
}


def rebuild_model_with_states(training_model: Model) -> Model:
    """
    自动重建模型，添加GRU隐状态输入/输出端口。
    如果模型已经有GRU状态端口，直接返回原模型。
    """
    # 检查是否已有GRU状态端口
    if len(training_model.inputs) == 4 and len(training_model.outputs) == 5:
        print("  Model already has GRU state ports, skipping rebuild")
        return training_model
    
    print("  Rebuilding model with GRU state inputs/outputs...")
    
    # 新的推理输入（带状态）
    features_in = Input(shape=(None, 42), name='features')
    vad_state_in = Input(shape=(24,), name='vad_gru_state')
    noise_state_in = Input(shape=(48,), name='noise_gru_state')
    denoise_state_in = Input(shape=(96,), name='denoise_gru_state')

    # 复制训练模型的层配置并加载权重
    # 1) input_dense
    input_dense_src = training_model.get_layer('input_dense')
    input_dense = Dense(24, activation='tanh', name='input_dense_export',
                        kernel_constraint=input_dense_src.kernel_constraint,
                        bias_constraint=input_dense_src.bias_constraint)
    tmp_export = input_dense(features_in)
    input_dense.set_weights(input_dense_src.get_weights())

    # 2) vad_gru (return_sequences+return_state)
    vad_gru_src = training_model.get_layer('vad_gru')
    vad_gru_exp = GRU(24, activation='tanh', recurrent_activation='sigmoid',
                      return_sequences=True, return_state=True, name='vad_gru_export',
                      kernel_regularizer=vad_gru_src.kernel_regularizer,
                      recurrent_regularizer=vad_gru_src.recurrent_regularizer,
                      kernel_constraint=vad_gru_src.kernel_constraint,
                      recurrent_constraint=vad_gru_src.recurrent_constraint,
                      bias_constraint=vad_gru_src.bias_constraint)
    vad_seq, vad_state_out = vad_gru_exp(tmp_export, initial_state=vad_state_in)
    vad_gru_exp.set_weights(vad_gru_src.get_weights())

    # 3) vad_output
    vad_output_src = training_model.get_layer('vad_output')
    vad_output_exp_layer = Dense(1, activation='sigmoid', name='vad_output_export',
                                 kernel_constraint=vad_output_src.kernel_constraint,
                                 bias_constraint=vad_output_src.bias_constraint)
    vad_output_exp = vad_output_exp_layer(vad_seq)
    vad_output_exp_layer.set_weights(vad_output_src.get_weights())

    # 4) noise_gru 输入：concat([tmp_export, vad_seq, features_in])
    noise_in = concatenate([tmp_export, vad_seq, features_in], name='noise_concat_export')
    noise_gru_src = training_model.get_layer('noise_gru')
    noise_gru_exp = GRU(48, activation='relu', recurrent_activation='sigmoid',
                        return_sequences=True, return_state=True, name='noise_gru_export',
                        kernel_regularizer=noise_gru_src.kernel_regularizer,
                        recurrent_regularizer=noise_gru_src.recurrent_regularizer,
                        kernel_constraint=noise_gru_src.kernel_constraint,
                        recurrent_constraint=noise_gru_src.recurrent_constraint,
                        bias_constraint=noise_gru_src.bias_constraint)
    noise_seq, noise_state_out = noise_gru_exp(noise_in, initial_state=noise_state_in)
    noise_gru_exp.set_weights(noise_gru_src.get_weights())

    # 5) denoise_gru 输入：concat([vad_seq, noise_seq, features_in])
    denoise_in = concatenate([vad_seq, noise_seq, features_in], name='denoise_concat_export')
    denoise_gru_src = training_model.get_layer('denoise_gru')
    denoise_gru_exp = GRU(96, activation='tanh', recurrent_activation='sigmoid',
                          return_sequences=True, return_state=True, name='denoise_gru_export',
                          kernel_regularizer=denoise_gru_src.kernel_regularizer,
                          recurrent_regularizer=denoise_gru_src.recurrent_regularizer,
                          kernel_constraint=denoise_gru_src.kernel_constraint,
                          recurrent_constraint=denoise_gru_src.recurrent_constraint,
                          bias_constraint=denoise_gru_src.bias_constraint)
    denoise_seq, denoise_state_out = denoise_gru_exp(denoise_in, initial_state=denoise_state_in)
    denoise_gru_exp.set_weights(denoise_gru_src.get_weights())

    # 6) denoise_output
    denoise_output_src = training_model.get_layer('denoise_output')
    denoise_output_exp_layer = Dense(22, activation='sigmoid', name='denoise_output_export',
                                     kernel_constraint=denoise_output_src.kernel_constraint,
                                     bias_constraint=denoise_output_src.bias_constraint)
    denoise_output_exp = denoise_output_exp_layer(denoise_seq)
    denoise_output_exp_layer.set_weights(denoise_output_src.get_weights())

    export_model = Model(
        inputs=[features_in, vad_state_in, noise_state_in, denoise_state_in],
        outputs=[denoise_output_exp, vad_output_exp, vad_state_out, noise_state_out, denoise_state_out],
        name='rnnoise_export_with_states'
    )
    
    print("  ✓ Model rebuilt successfully with GRU state ports")
    return export_model


def convert(hdf5_path: str, onnx_path: str, opset: int = 13, auto_rebuild: bool = False) -> None:
    if not os.path.isfile(hdf5_path):
        raise FileNotFoundError(f"HDF5 model not found: {hdf5_path}")

    print(f"Loading Keras model from: {hdf5_path}")
    # Load with custom objects registered for deserialization
    model = keras.models.load_model(hdf5_path, custom_objects=CUSTOM_OBJECTS)
    
    # Auto-rebuild model with GRU states if needed
    if auto_rebuild:
        print("\n=== Auto-Rebuild Mode ===")
        print("  Checking if model needs GRU state ports...")
        model = rebuild_model_with_states(model)
        print("  Model ready for conversion with GRU state ports\n")

    # Check if the model has GRU state inputs/outputs
    num_inputs = len(model.inputs)
    num_outputs = len(model.outputs)
    
    print(f"Model has {num_inputs} input(s) and {num_outputs} output(s)")
    
    # Print input information
    for i, inp in enumerate(model.inputs):
        print(f"  Input {i}: {inp.name}, shape: {inp.shape}")
    
    # Print output information
    for i, out in enumerate(model.outputs):
        print(f"  Output {i}: {out.name}, shape: {out.shape}")
    
    # Check if this is a model with GRU states (4 inputs and 5 outputs)
    if num_inputs == 4 and num_outputs == 5:
        print("Detected model with GRU state inputs/outputs")
        # Build input signature for model with efficient state management
        input_specs = []
        for inp in model.inputs:
            inp_name = inp.name.split(':')[0]
            inp_shape = inp.shape.as_list()
            
            # Handle different input shapes
            if len(inp_shape) == 3:  # features: (None, None, 42)
                spec = tf.TensorSpec([None, None, inp_shape[2]], tf.float32, name=inp_name)
            elif len(inp_shape) == 2:  # GRU states: (None, hidden_size)
                spec = tf.TensorSpec([None, inp_shape[1]], tf.float32, name=inp_name)
            else:
                # Fallback: use dynamic shape
                spec = tf.TensorSpec([None] * len(inp_shape), tf.float32, name=inp_name)
            
            input_specs.append(spec)
        
        print(f"Converting to ONNX (opset {opset}) with GRU state inputs/outputs...")
        # Convert with all input signatures
        tf2onnx.convert.from_keras(model, input_signature=input_specs, output_path=onnx_path, opset=opset)
        
    elif num_inputs == 1:
        print("Detected standard model without GRU state ports")
        # Use a dynamic input signature (None, None, 42) to preserve time dimension flexibility
        input_name = model.inputs[0].name.split(':')[0]
        spec = (tf.TensorSpec([None, None, 42], tf.float32, name=input_name),)
        
        print(f"Converting to ONNX (opset {opset})...")
        # Convert directly from the Keras model
        tf2onnx.convert.from_keras(model, input_signature=spec, output_path=onnx_path, opset=opset)
    else:
        # Generic conversion for models with multiple inputs but unknown structure
        print(f"Converting to ONNX (opset {opset}) with {num_inputs} inputs...")
        input_specs = []
        for inp in model.inputs:
            inp_name = inp.name.split(':')[0]
            inp_shape = inp.shape.as_list()
            # Use dynamic shapes for flexibility
            spec = tf.TensorSpec([None] * len(inp_shape), tf.float32, name=inp_name)
            input_specs.append(spec)
        tf2onnx.convert.from_keras(model, input_signature=input_specs, output_path=onnx_path, opset=opset)

    print(f"Saved ONNX model to: {onnx_path}")
    

def main():
    parser = argparse.ArgumentParser(description='Convert Keras HDF5 model to ONNX for RNNoise.')
    parser.add_argument('--input', '-i', required=True, help='Path to Keras HDF5 model file')
    parser.add_argument('--output', '-o', required=False, help='Path to output ONNX file')
    parser.add_argument('--opset', type=int, default=13, help='ONNX opset version (default: 13)')
    parser.add_argument('--auto-rebuild', action='store_true', 
                        help='Automatically rebuild model with GRU state ports if missing')
    args = parser.parse_args()

    input_path = os.path.abspath(args.input)
    output_path = args.output
    if not output_path:
        base, _ = os.path.splitext(input_path)
        output_path = base + '.onnx'
    output_path = os.path.abspath(output_path)

    os.makedirs(os.path.dirname(output_path), exist_ok=True)

    convert(input_path, output_path, opset=args.opset, auto_rebuild=args.auto_rebuild)

if __name__ == '__main__':
    main()