File Formats and Codecs

Sharmistha Chakrabartty
May 3, 2024
7 min read

File Format

A standardized method of encoding and organizing data with in a file. How a data is structured (organized and formatted), stored and represented (What type of data it is - audio, video, image etc) is decided by the format.

The format makes it possible for the computer Programs to interpret (read) and manipulate (write, re-write) the information contained in the file. File formats are often associated with file extensions, which are appended to the file name to indicate the format of the file. File extensions help users and software applications identify the type of file and determine how to handle it.

Video File Format

Used for storage of digital video data. It is the format, the specifies how a data would be encoded and stored within a file. One file format may support various audio and video codecs. It defines the structure, codec, metadata and other parameters necessary for storage and playback. Eg: AVI, MP4, MOV etc.

Container Format (Container)

A type of file format that encapsulate multiple data streams, including video, audio, subtitles, and metadata, into a single file. It provides a standardized way to organize and synchronize different types of media content. Container formats do not specify how the video and audio streams are encoded but rather how they are packaged together within the file. Eg: AVI, MP4, MOV etc.

Now here is a conundrum, that video file format and container format sounds like one thing but video file format specifies the codec but a container file doesn't. How so?

Video File Format	Container Format
When we refer to "video file formats," we often mean formats that include both the container and the encoded video/audio streams. In this context, "video file format" typically refers to formats like MP4, AVI, MOV, etc., which encapsulate both the encoded video/audio streams and additional metadata within a container structure. It is basically Container format + Codec.	Container formats do not specify the encoding or compression methods used for the individual data streams. Instead, they define how these streams (video, Audio, Subtitle, Metadata) are packaged together - organization, synchronization, and encapsulated - within a single file.

Essence

The actual (RAW) audio and video data contained in the video file. The essence is encoded using specific video and audio codecs and is typically embedded within the container format along with other streams and metadata.

Codec

Software or hardware program that encodes (Compresses) and decodes (decompresses) digital media file. It compresses by eliminating redundant or unnecessary information from the original media stream. CODEC is short for Coder/Decoder; Where as Endec is Encoder/Decoder. Codec helps in the storage, transmission and playback of Audio, Video data. Eg: Mpeg 4, H.264, XVid, DIVX etc.

Digital Data can be:

Uncompressed
Compressed : Lossless - Data will be rewritten in the same manner as the original file. It reduces the file size without sacrificing any data (information) or quality. Original Data can be recovered on uncompressing the file. Lossless compression is typically used for archival purposes or when preserving the highest possible quality is essential. Lossy - Many of the redundant information (data) in the file are eliminated, reducing its size. Original data can't be recovered on decompressing. While lossy compression results in smaller file sizes, it may also lead to a loss of quality.

Tv monitors showcasing video of different formats and codecs

VIDEO FILE FORMATS AND CODECS :

Name	Extension	Container	Video Codec	Audio Codec	Remarks
3GPP (3GPP 2)	.3gp / .3g2	MPEG 4 -part 12	H.263, H.264, H.265, MPEG-4 Visual (Part 2), MPEG-4 AVC (Part 10)	AMR (Adaptive Multi-Rate) [both wideband and narrowband], AAC (Advanced Audio Coding), MP3 (MPEG Audio Layer III), EVRC (Enhanced Variable Rate Codec)	For cell phones
RAW video Format	.yuv , .raw, .dng: Digital Negative (DNG), .bayer, .r3d, .arw, .cr2, .nef (from different camera manufacturer)	-	-	-	Suport all resolution, frame rate and sampling structure.
Material exchange format (Container format)	.mxf	MXF	MPEG-2 Video, H.264 (AVC), H.265 (HEVC), DNxHD/DNxHR, ProRes	PCM, AAC, MP2, MP3 (MPEG-1 Audio Layer II), Dolby Digital (AC-3)	High-quality video and audio content, commonly used in broadcasting, film production, post-production, and digital archiving workflows.
MP4 (Container Format)	.mp4, .m4a	MPEG-4 Part 14 (MP4)	H.264, H.265, MPEG-4 Visual (Part 2), AV1, VP9	AAC, MP3, Dolby Digital (AC-3), FLAC, PCM	MP4 is widely used in video editing workflows due to its compatibility with popular video editing software, such as Adobe Premiere Pro, Final Cut Pro, and DaVinci Resolve.
M4V	.m4v	MPEG-4 Part 14	H.264	AAC	M4V files are commonly used for video content purchased or rented from the iTunes Store.
Quick Time File Format (Container Format)	.mov	Quicktime	H.264, Apple ProRes, Animation, MPEG-4 Visual (Part 2)	AAC, PCM, Linear PCM	MOV files are commonly used in professional video editing and post-production workflows.
Windows Media Video	.wmv	ASF	windows media video, Video screen, Video image	Window media audio	ASF - Advanced system Format. Microsoft's format for streaming media
AVI (Audio Video interleave)	.avi	AVI	Many	Many	Uses RIFF (Resource Interchange File Format) - generic file container format to store data in tagged chunks.
Matroska	.mkv	Matroska	Any	Any	Support nearly all video codec. Open Source.
Flash Video	.flv, .f4v	FLV (audio, video, text, data)	H.264	AAC	FLV files can be easily embedded into web pages using HTML code or other web technologies. FLV files can contain interactive elements such as clickable buttons, links, and overlays, which can enhance the user experience and engagement with the video content.
VOB (Video Object)	.vob	VOB	MPEG-2, H.264	AC-3 (Dolby Digital), PCM,	For storing DVD-Video media
Gif	.gif	-	-	-	Inefficient Compression, no sound, widely supported, Simple Animation

MPEG-2 and MPEG-4 are not file formats, codecs, or containers themselves, but rather standards developed by the Moving Picture Experts Group (MPEG) for video compression and encoding. These standards define the methods and algorithms used to compress video data efficiently while maintaining acceptable quality.

H.264 - Advanced Video Coding (AVC). Widely used for various applications, including video streaming, digital television, Blu-ray discs, video conferencing, and surveillance systems.

H.265 - High Efficiency Video Coding (HEVC). Commonly used for ultra-high-definition (UHD) video streaming, digital television, video surveillance, and other applications where efficient compression is crucial.

MUSIC FORMATS:

MP3 (MPEG-1 Audio Layer III):

Developer: The MP3 format was developed by the Moving Picture Experts Group (MPEG).
Codec: MP3 uses lossy compression to reduce the file size. It achieves compression by removing frequencies that are less audible to the human ear.
Compression: Lossy compression.
Bit Rate Range: Typically ranges from 32 kbps to 320 kbps.
Sampling Frequency: Commonly 44.1 kHz, but can vary.
Channels: Stereo (2 channels).
Notes: MP3 is one of the most widely used audio formats for digital music distribution and playback due to its small file size and compatibility with a wide range of devices.

WAV (Waveform Audio File Format):

Developer: Developed by Microsoft and IBM.
Codec: WAV files typically use PCM (Pulse-Code Modulation) codec, which is uncompressed.
Compression: Uncompressed (lossless).
Bit Rate Range: Depends on the sampling frequency and bit depth.
Sampling Frequency: Commonly 44.1 kHz or 48 kHz, but can vary.
Channels: Mono or Stereo, supports multichannel audio.
Notes: WAV is a widely supported uncompressed audio format commonly used for high-quality audio storage and editing.

AAC (Advanced Audio Coding):

Developer: Developed by the MPEG (Moving Picture Experts Group) and part of the MPEG-2 and MPEG-4 standards.
Codec: AAC uses lossy compression.
Compression: Lossy compression.
Bit Rate Range: Typically ranges from 16 kbps to 320 kbps.
Sampling Frequency: Commonly 44.1 kHz or 48 kHz, but can vary.
Channels: Mono or Stereo, supports multichannel audio.
Notes: AAC is known for its improved sound quality compared to MP3 at similar bitrates and is widely used for online streaming and digital music distribution.

PCM (Pulse-Code Modulation):

Codec: PCM is an uncompressed audio format, not a specific file format. It represents the raw audio waveform without any compression.
Compression: Uncompressed (lossless).
Sampling Frequency: Commonly 44.1 kHz or 48 kHz, but can vary.
Channels: Mono or Stereo, supports multichannel audio.
Notes: PCM is the standard digital representation of audio used in CDs and WAV files. It provides high-quality audio without any loss of fidelity.

AIFF (Audio Interchange File Format):

Developer: Developed by Apple Inc.
Codec: AIFF files typically use PCM codec, which is uncompressed.
Compression: Uncompressed (lossless).
Sampling Frequency: Commonly 44.1 kHz or 48 kHz, but can vary.
Channels: Mono or Stereo, supports multichannel audio.
Notes: AIFF is a high-quality uncompressed audio format commonly used in Apple's ecosystem.

FLAC (Free Lossless Audio Codec):

Developer: Developed by Josh Coalson.
Codec: FLAC uses lossless compression, meaning it reduces file size without any loss in audio quality.
Compression: Lossless compression.
Bit Rate Range: Typically ranges from 300 kbps to 1,200 kbps (variable).
Sampling Frequency: Commonly 44.1 kHz or 48 kHz, but can vary.
Channels: Mono or Stereo, supports multichannel audio.
Notes: FLAC is a popular open-source lossless audio codec known for its ability to compress audio files without sacrificing quality. It is widely used for archiving and digital audio distribution.

ALAC (Apple Lossless Audio Codec):

Developer: Developed by Apple Inc.
Codec: ALAC uses lossless compression, similar to FLAC, to reduce file size without sacrificing audio quality.
Compression: Lossless compression.
Bit Rate Range: Similar to FLAC, typically ranges from 300 kbps to 1,200 kbps (variable).
Sampling Frequency: Commonly 44.1 kHz or 48 kHz, but can vary.
Channels: Mono or Stereo, supports multichannel audio.
Notes: ALAC is Apple's proprietary lossless audio codec, primarily used within the Apple ecosystem for storing and playing back high-quality audio files.

WMA (Windows Media Audio):

Developer: Developed by Microsoft.
Codec: WMA supports both lossy and lossless compression methods, depending on the variant.
Compression: Can be lossy or lossless depending on the variant.
Bit Rate Range: Varies widely depending on the compression variant and settings.
Sampling Frequency: Commonly 44.1 kHz or 48 kHz, but can vary.
Channels: Mono or Stereo, supports multichannel audio.
Notes: WMA is a proprietary audio codec developed by Microsoft. It offers various compression options for different use cases, including streaming, digital music distribution, and archival purposes.

IMAGE FORMATS:

RAW (Raw Image Format):

Full Form: RAW stands for "Raw Image Format."
Pros:
Retains maximum image quality and detail as it captures raw sensor data without any compression.
Provides flexibility in post-processing, allowing for adjustments to exposure, white balance, and other parameters without loss of quality.
Cons:
Large file sizes due to uncompressed or minimally compressed data.
Limited compatibility with software and devices, as RAW files may require proprietary codecs or specialized software for viewing and editing.
Not suitable for direct sharing or distribution without conversion to more widely supported formats like JPEG.

JPEG (Joint Photographic Experts Group):

Full Form: JPEG stands for "Joint Photographic Experts Group."
Pros:
High compression efficiency, resulting in smaller file sizes while maintaining good image quality.
Widely supported by software, devices, and web browsers, making it ideal for sharing and distribution.
Cons:
Lossy compression can lead to degradation in image quality, especially with high compression ratios.
Limited support for transparency; JPEG does not support transparent layers.
Not suitable for applications requiring lossless compression or preservation of fine details, such as professional photography or archival purposes.

GIF (Graphics Interchange Format):

Full Form: GIF stands for "Graphics Interchange Format."
Pros:
Supports animation and short video clips through multiple frames.
Limited support for transparency; GIFs can have a single transparent color, allowing for simple transparency effects.
Widely supported by web browsers and social media platforms for sharing simple animations and graphics.
Cons:
Limited color depth (256 colors), leading to reduced image quality, especially for photographs or complex graphics.
Relatively large file sizes compared to modern formats like JPEG and PNG, especially for animated GIFs.
Not suitable for high-quality photography or graphics due to its limited color palette and lossy compression.

BMP (Bitmap Image):

Full Form: BMP stands for "Bitmap Image."
Pros:
Simple and widely supported uncompressed image format.
Suitable for basic graphics and iconography, especially in applications where file size is not a concern.
Cons:
Large file sizes due to uncompressed data, making BMP files inefficient for storage and distribution, especially for large images.
Limited support for transparency; BMP does not support transparent layers.
Lack of compression leads to larger file sizes compared to formats like JPEG and PNG, without significant benefits in image quality.

TIFF (Tagged Image File Format):

Full Form: TIFF stands for "Tagged Image File Format."
Pros:
Supports lossless compression, preserving image quality and detail without degradation.
High bit depth support, allowing for images with a wide range of colors and tones, suitable for professional photography and printing.
Supports transparency and multiple layers, making it versatile for various graphics and imaging applications.
Cons:
Large file sizes, especially for uncompressed or minimally compressed TIFF files, limiting their suitability for web distribution or sharing.
Limited compatibility with web browsers and certain software applications compared to more widely supported formats like JPEG and PNG.
Complexity and variability in TIFF specifications can lead to compatibility issues between different software implementations.

PNG (Portable Network Graphics):

Full Form: PNG stands for "Portable Network Graphics."
Pros:
Lossless compression preserves image quality and detail without degradation.
Supports transparency and alpha channels, allowing for complex transparency effects and layering.
Widely supported by web browsers, software applications, and operating systems, making it ideal for web graphics, digital art, and archival purposes.
Cons:
Larger file sizes compared to JPEG for similar image quality, especially for images with complex transparency or high color depth.
Limited support for animation compared to formats like GIF.
Compression efficiency may vary depending on the image content, leading to unpredictable file sizes in some cases.