Camera and Microphone Capture with FFmpeg-iOS

English translation of the Chinese original, translated on 2026-10-23. The article body uses automated translation; app descriptions, historical reading notes and video subtitles receive separate editorial checks. Original publication dates, attribution, code and version notes are retained.

This historical illustration is currently unavailable. The original image URL is retained for reference. Building FFmpeg-iOS on a Mac H.264 Decoding with FFmpeg A Simple Streaming Wrapper for FFmpeg-iOS

Today we will use FFmpeg to access cameras and microphones on iOS. Very little code is needed; afterward, I will add an example using iOS's native camera access.

Accessing the Camera and Microphone with FFmpeg

First, import the necessary headers.

#include <stdio.h>
#ifdef __cplusplus
extern "C"
{
#endif
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>
#include <libavdevice/avdevice.h>
    
#ifdef __cplusplus
};
#endif

The code is wrapped simply as follows:

- (void)showDevice{
    avdevice_register_all();
    AVFormatContext *pFormatCtx = avformat_alloc_context();
    AVDictionary* options = NULL;
    av_dict_set(&options,"list_devices","true",0);
    AVInputFormat *iformat = av_find_input_format("avfoundation");
    printf("==AVFoundation Device Info===\n");
    avformat_open_input(&pFormatCtx,"",iformat,&options);
    printf("=============================\n");
    if(avformat_open_input(&pFormatCtx,"0",iformat,NULL)!=0){
        printf("Couldn't open input stream.\n");
        return ;
    }
    
}

Run it to see the following information in the log area:

==AVFoundation Device Info===
2017-07-20 16:59:36.325150+0800 LBffmpegDemo[2040:821433] [MC] System group container for systemgroup.com.apple.configurationprofiles path is /private/var/containers/Shared/SystemGroup/systemgroup.com.apple.configurationprofiles
2017-07-20 16:59:36.326529+0800 LBffmpegDemo[2040:821433] [MC] Reading from public effective user settings.
[AVFoundation input device @ 0x145d0100] AVFoundation video devices:
[AVFoundation input device @ 0x145d0100] [0] Back Camera
[AVFoundation input device @ 0x145d0100] [1] Front Camera
[AVFoundation input device @ 0x145d0100] AVFoundation audio devices:
[AVFoundation input device @ 0x145d0100] [0] iPhone 麦克风
=============================
[avfoundation @ 0x153ef800] Selected framerate (29.970030) is not supported by the device
[avfoundation @ 0x153ef800] Supported modes:
[avfoundation @ 0x153ef800]   192x144@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   192x144@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   352x288@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   352x288@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   480x360@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   480x360@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   640x480@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   640x480@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   960x540@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   960x540@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   1280x720@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   1280x720@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   1280x720@[1.000000 60.000000]fps
[avfoundation @ 0x153ef800]   1280x720@[1.000000 60.000000]fps
[avfoundation @ 0x153ef800]   1920x1080@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   1920x1080@[1.000000 30.000000]fps
[avfoundation @ 0x153ef800]   2592x1936@[1.000000 20.000000]fps
[avfoundation @ 0x153ef800]   2592x1936@[1.000000 20.000000]fps
[avfoundation @ 0x153ef800]   3264x2448@[1.000000 20.000000]fps
[avfoundation @ 0x153ef800]   3264x2448@[1.000000 20.000000]fps
Couldn't open input stream.

We have found the devices: the front and rear cameras and the microphone. Next, look at an example using the system's native camera APIs.

Native iOS camera access

First, import the required headers.

#import <AVFoundation/AVFoundation.h>
#import <UIKit/UIKit.h>

Next, some global properties:

@property(nonatomic, strong) AVCaptureSession                *captureSession;
@property(nonatomic, strong) AVCaptureDevice                 *captureDevice;
@property(nonatomic, strong) AVCaptureDeviceInput            *captureDeviceInput;
@property(nonatomic, strong) AVCaptureVideoDataOutput        *captureVideoDataOutput;
@property(nonatomic, assign) CGSize                          videoSize;
@property(nonatomic, strong) AVCaptureConnection             *videoCaptureConnection;
@property(nonatomic, strong) AVCaptureVideoPreviewLayer      *previewLayer;

Finally, here is the simple wrapper:

- (void)getMovieDevice:(UIView *)view{
    self.captureSession = [[AVCaptureSession alloc] init];
    //    captureSession.sessionPreset = AVCaptureSessionPresetMedium;
    self.captureSession.sessionPreset = AVCaptureSessionPreset1920x1080;
    
    self.videoSize = [self getVideoSize:self.captureSession.sessionPreset];
    
    self.captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
    
    NSError *error = nil;
    self.captureDeviceInput = [AVCaptureDeviceInput deviceInputWithDevice:self.captureDevice error:&error];
    
    if([self.captureSession canAddInput:self.captureDeviceInput])
        [self.captureSession addInput:self.captureDeviceInput];
    else
        NSLog(@"Error: %@", error);
    
    dispatch_queue_t queue = dispatch_queue_create("myEncoderH264Queue", NULL);
    
    self.captureVideoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
    [self.captureVideoDataOutput setSampleBufferDelegate:self queue:queue];
    
#if encodeModel
    // nv12
    NSDictionary *settings = [[NSDictionary alloc] initWithObjectsAndKeys:
                              [NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange],
                              kCVPixelBufferPixelFormatTypeKey,
                              nil];
#else
    // 32bgra
    NSDictionary *settings = [[NSDictionary alloc] initWithObjectsAndKeys:
                              [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA],
                              kCVPixelBufferPixelFormatTypeKey,
                              nil];
#endif
    
    self.captureVideoDataOutput.videoSettings = settings;
    self.captureVideoDataOutput.alwaysDiscardsLateVideoFrames = YES;
    
    if ([self.captureSession canAddOutput:self.captureVideoDataOutput]) {
        [self.captureSession addOutput:self.captureVideoDataOutput];
    }
    
    // 保存Connection，用于在SampleBufferDelegate中判断数据来源（是Video/Audio？）
    self.videoCaptureConnection = [self.captureVideoDataOutput connectionWithMediaType:AVMediaTypeVideo];
    
#pragma mark -- AVCaptureVideoPreviewLayer init
    self.previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:self.captureSession];
    self.previewLayer.frame = view.layer.bounds;
    self.previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill; // 设置预览时的视频缩放方式
    [[self.previewLayer connection] setVideoOrientation:AVCaptureVideoOrientationPortrait]; // 设置视频的朝向
    [self.captureSession startRunning];
    [view.layer addSublayer:self.previewLayer];
}

- (CGSize)getVideoSize:(NSString *)sessionPreset {
    CGSize size = CGSizeZero;
    if ([sessionPreset isEqualToString:AVCaptureSessionPresetMedium]) {
        size = CGSizeMake(480, 360);
    } else if ([sessionPreset isEqualToString:AVCaptureSessionPreset1920x1080]) {
        size = CGSizeMake(1920, 1080);
    } else if ([sessionPreset isEqualToString:AVCaptureSessionPreset1280x720]) {
        size = CGSizeMake(1280, 720);
    } else if ([sessionPreset isEqualToString:AVCaptureSessionPreset640x480]) {
        size = CGSizeMake(640, 480);
    }
    
    return size;
}

#pragma mark --  AVCaptureVideo(Audio)DataOutputSampleBufferDelegate method
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    
    // 这里的sampleBuffer就是采集到的数据了，但它是Video还是Audio的数据，得根据connection来判断
    if (connection == self.videoCaptureConnection) {
        
        // Video
        //        NSLog(@"在这里获得video sampleBuffer，做进一步处理（编码H.264）");
        
        
#if encodeModel
        // encode
        
#else
        CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
        
        //        int pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);
        //        switch (pixelFormat) {
        //            case kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange:
        //                NSLog(@"Capture pixel format=NV12");
        //                break;
        //            case kCVPixelFormatType_422YpCbCr8:
        //                NSLog(@"Capture pixel format=UYUY422");
        //                break;
        //            default:
        //                NSLog(@"Capture pixel format=RGB32");
        //                break;
        //        }
        
        CVPixelBufferLockBaseAddress(pixelBuffer, 0);
        
        // render
        [openglView render:pixelBuffer];
        
        CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
#endif
    }
    //    else if (connection == _audioConnection) {
    //        
    //        // Audio
    //        NSLog(@"这里获得audio sampleBuffer，做进一步处理（编码AAC）");
    //    }
    
}

Download LBffmpegDemo

That concludes this look at camera access on iOS. When I have time, I will write about other uses of FFmpeg on iOS. Follow along if FFmpeg interests you! 😄

Recommended reading: The FFmpeg-iOS Streaming Wrapper