Introduction to FFmpeg Kit API
FFmpeg Kit is a toolkit to use FFmpeg in applications. It provides different tools to developers, and one of them is the FFmpeg Kit library.
What does this library do? Well, although running an FFmpeg command is not very difficult. Executing commands inside an application can be tricky. There are some challenges. Especially on platforms where calling native libraries needs extra effort. FFmpeg Kit library fills this gap. It makes it easier for applications to run FFmpeg commands.
It encapsulates the features of the original FFmpeg. But, it doesn’t stay there. It also does things that FFmpeg cannot do. There are plenty of handy features coming with it that are essential for applications. You can access all of them through the FFmpeg Kit API. It is written in Java for Android, in Objective-C for Apple platforms, in Dart for Flutter and in JavaScript with TypeScript definitions for React Native.
Unified API
The first thing that must be underlined about FFmpeg Kit API is; it is unified across all platforms. Which means, the core API methods are the same on all platforms. They have the same classes, the same methods, the same input types and the same output types. And, they all work in exactly the same way. When you need to use FFmpeg Kit on a new platform, there won’t be big surprises for you.
Though, there is a minor exception to this: synchronous execute() methods. These methods are available only on native platforms (Android, iOS, macOS, tvOS). On hybrid platforms (Flutter and React Native) they are not implemented. Because, both of these platforms have an asynchronous, non-blocking nature. And, applications will become unresponsive if we run commands synchronously on them. That’s why we have only asynchronous execute() methods there.
Data Types
FFmpeg Kit API generally uses the same data types across all platforms. There are only minor differences. Just a few types are implemented in a different way, since the features they rely on are not supported on programming language the API is written in.
Callbacks are one of those types where some platforms have a different interpretation.
While Java and Dart APIs define them as Functions, Objective-C API defines a Callback as a Block type. On React Native API we don’t have a type for Callbacks. TypeScript definitions fill that gap by defining them as Function type expressions. Despite that, all platforms support and make use of them in exactly the same way.
Platform Specific Methods
These methods are built to support a particular platform feature.
The most popular methods under this category are the Storage Access Framework (SAF) methods. We have them in the FFmpegKitConfig class for Android. Flutter and React Native APIs necessarily define them too. Since they come with Android support. These methods convert content URIs (content://) into SAF protocol URLs (saf:).
Why do we need them? Because, Android introduced SAF at API Level 19. And made it a requirement to access files at API Level 29. When a file is opened using SAF, a content URI is returned for that file. Unfortunately, content URIs are not supported by FFmpeg. This is where these methods come into play. They convert a content URI into an SAF protocol URL, which FFmpeg Kit can process. In this way, we can use files with FFmpeg Kit API on API Level 29 and above.
SAF protocol is a custom FFmpegKit protocol we created for Android to process files opened with Storage Access Framework.
Flutter and React Native APIs go one step further with regard to SAF support. They expose two extra Android specific methods to select files natively. Which is not directly an SAF functionality. But something we had to implement to supplement SAF methods. They are responsible of selecting a file and getting the content URI for it. We tried to use the existing plugins for that. But React Native plugins didn’t work for us on some devices. And the Flutter plugin we tried was more clever than we needed it to be. So, we ended up adding them to the API.
Classes
There are two main classes in the API: FFmpegKit and FFprobeKit. Former is used to run FFmpeg commands and the latter is for FFprobe commands. FFprobeKit also supports extracting media information for a file using the getMediaInformation() method. Which is quite useful if you’re not good at running a command and parsing the result. This method does that for you internally. And, creates a MediaInformation object for the file given.
Both FFmpegKit and FFprobeKit are designed to be simple and easy to understand. Although there are overloaded versions of the execute() and getMediaInformation() methods in these classes. They both serve the same purpose. They run a single command. If their default behaviour doesn’t exactly do what you expect from them, there is another class you can look into: FFmpegKitConfig. It is the configuration class of the library. You can access, change or disable library features using this class. Or run platform specific methods and execute commands by sessions the belong to.
Redirection
Redirection is one of the internal solutions we implemented to overcome the limitations of the platforms. Also to improve the overall performance. It is heavily used by different parts of the API.
As you may already know, FFmpeg prints its output to stderr (standard error) by default. And, not all platforms have a stderr or allow you to view it. Which becomes an issue, because understanding what FFmpeg does without seeing the output is nearly impossible. Moreover, FFprobe is all about the output. What would be the purpose of running FFprobe if you can’t see what it prints, right?
To overcome this problem, FFmpegKit redirects FFmpeg output to a platform specific output tool.
On Android, this output tool is Logcat. FFmpeg output is printed to Logcat. On Apple platforms (iOS, macOS, tvOS) it is Apple System Log facility. We use NSLog statements to redirect the output there. On Flutter and React Native we print the output to console.
We call this behaviour redirection and enable it on all platforms by default. If you don’t want to use it, you can disable it via the FFmpegKitConfig’s disableRedirection() method. Though, I don’t recommend it because a lot of features on the library rely on it.
Print behaviour of redirection is controlled by a configuration parameter named LogRedirectionStrategy. It defines when to print logs. It supports 5 different strategies. And, can be configured globally via FFmpegKitConfig or per session when creating a new session. The default strategy configured on all platforms is printLogsWhenNoCallbacksDefined. Which means, print logs if a global or session log callback is not defined.
Callbacks
Another concept FFmpeg Kit API heavily relies on is Callbacks. Callbacks are functions that are called when an asynchronous operation completes. Or when you want to be notified about the logs or statistics generated during an execution. They can be set globally in FFmpegKitConfig class, or defined per session when creating a new session.
On native platforms, FFmpegKit uses a dedicated thread called Callback Thread to invoke these callbacks. This is a precaution to protect the library and keep the performance high for cases where callbacks are called too many times or too frequently.
On hybrid platforms, Flutter and React Native, platforms themselves call the callbacks. We don’t define a special component to invoke them.
Sessions
FFmpeg Kit API is designed around the session concept. Each command execution in the library creates a new session. And, all resources about that run are saved under the session object. You can check if the execution is completed; if it was successful; the output (logs), statistic, everything from the session object.
Sessions have a unique id and they are stored in the session history. You can access previously run sessions from the session history.
We have three session types and threes session classes that represent each type. FFmpegSession created by FFmpegKit, FFprobeSession created by execute() methods of FFprobeKit and MediaInformationSession created by getMediaInformation() methods of FFprobeKit. All session classes are almost identical with regard to fields and methods. There are only two differences compared to each other. FFmpegSession is capable of collecting the statistics. And, MediaInformationSession has a MediaInformation field. That’s all.
Conclusion
This is the overview of FFmpeg Kit API. I hope it gives you an idea about how it was designed. What are the key classes, methods and internal components.
To learn more about FFmpeg Kit API, please have a look at the test applications under the ffmpeg-kit-test project. You can find a test application for each platform and version there.
Update: As of September 14, 2022, I moved both FFmpegKit and FFmpegKitTest repositories under the ARTHENICA organisation on GitHub. It is just a cosmetic change to separate FFmpegKit from my other projects.
