Models, inputs and measured pipelines

Five CLAPv2 audio encoders

CLAP aligns audio with text descriptions. Its predictions use only the soundtrack; an ImageNet object or Kinetics action is therefore not shown for these models. We evaluate the supplied CLAPNaFlex-v3 XXS, XS, S, M and L checkpoints at epoch 320, after the run labelled 1,024 million samples seen. The L checkpoint is the requested 32-node run. We publish embeddings and scores, without distributing these model weights.

The first 30 seconds are decoded, downmixed to mono and resampled to 32 kHz. The model's frontend computes 128-band mel features before the audio tower produces a normalized vector. XXS through L use different embedding dimensions, so vectors from different model sizes must not be mixed in a shared nearest-neighbour index. The text tower encodes the 527 AudioSet names with three templates; those label vectors are reused for every clip.

Checkpoint sizes and output vectors

Parameter counts below describe the complete loaded checkpoint, including text and other model components. They are not counts of parameters exercised by every individual audio or video call.

Six PEAV multi-second clip encoders

Perception Encoder Audio-Visual supports audio-only, video-only and learned joint audio-video embeddings. The three sizes are Small, Base and Large, each with a native-frame and a fixed-16-frame checkpoint. All produce 1,024-dimensional vectors in the modality-specific contrastive spaces. Use the matching text head for each input mode; vectors from unrelated checkpoints are not automatically interchangeable.

16 frames versus native video

The fixed-16 checkpoint receives 16 frames selected at evenly spaced frame indices across the analyzed interval. A 30-second clip at 30 FPS therefore contributes roughly one image every two seconds. The native-frame checkpoint receives all approximately 900 frames from that same interval. Both can summarize multiple seconds; “16 frames” does not mean a clip lasting only 16 consecutive frames. “Native” refers to frame coverage, not image resolution: every selected image is resized to 336 × 336 RGB and normalized to [−1, 1].

Denser frame coverage supplies more observations of brief events, while sparse sampling can miss events between sampled frames. The relative accuracy of these choices on the gallery has not been measured with human labels. These are distinct released checkpoints of the same size class, rather than a single checkpoint with a display setting changed.

Audio, video and joint predictions

Audio input is mono at 48 kHz. Each clip is reflect-padded to a multiple of 1,920 samples, matching the codec stride. The video path extracts visual features per frame and aggregates them through the temporal encoder. The joint path aligns video features with the audio timeline and combines the modalities through the learned fusion encoder. Its embedding can use visual context to change a sound label, and sound to change an activity label; it is not the arithmetic mean of two independent embedding vectors.

Select Audio to isolate soundtrack predictions. Select Video to inspect object, activity and exploratory visual-format labels. Audio + Video supplies all these banks using the joint embedding. The three PE-A-Frame localization checkpoints are excluded from this study.

From embeddings to the top three labels

ImageNet contributes 1,000 descriptive object classes, Kinetics contributes 400 activities, and AudioSet contributes 527 sound classes. Each name is inserted into three prompts. We normalize the individual text embeddings, average the three vectors for a class, and normalize that average. For example, the audio templates are “This is a sound of {label}.”, “The sound of {label}.” and “An audio recording of {label}.” The full label files and templates are downloadable.

A raw cosine measures alignment with a candidate description. The percentage is softmax(100 × cosine) over the entire corresponding label bank, including labels not displayed. The top-three percentages consequently need not add to 100%. These are relative, uncalibrated scores, not probabilities that an event occurred. A high percentage can simply mean that one description fits better than the other candidates. AudioSet is multi-label, so speech, music and other sounds may coexist.

The five visual-format prompts distinguish static photos, captions over a static image, slideshows, animation and moving live action. They are an exploratory vocabulary and are not a validated static-video classifier. Original captions are source metadata; they are not generated descriptions or speech transcripts.

ImageNet names and templates · Kinetics names and templates · AudioSet names and templates · Visual-format prompts

What a throughput measurement includes

Every result names its pipeline version. The initial prototype provides a reproducible baseline. Optimized results are added only after full-clip timing and agreement checks against the original single-clip embeddings. Hardware is one NVIDIA GH200 120GB GPU per pipeline, with 102.0 GB visible to PyTorch. Other GPUs on the same node can be active; CPU and storage contention may affect elapsed time.

Initial prototype

CLAP uses FP32, batches of eight and eight CPU preprocessing workers. We run two warmup passes and take the median of three timed full-sample passes. Its pipeline includes decoding, resampling, the mel frontend, transfer, audio encoding and AudioSet scoring. PEAV uses BF16 and one clip at a time. The original video decoder resizes every native frame, including frames later discarded by the 16-frame sampler. Its pipeline figure sums the measured preprocessing, transfer and encoder stages and excludes final label scoring. These implementations are different configurations and do not establish an optimized architecture speed ranking.

Optimized batching and data loading

The optimized implementation reads the original compressed files on every timed pass. For fixed-16 models it resizes only selected frames. Container-derived frame counts are checked while decoding; if a count is wrong, the exact extra decoding pass is included in the timer. Clips of similar duration are grouped into batches, and validity masks exclude padding from temporal aggregation. CPU workers prepare upcoming batches while the GPU processes the current batch. Pinned host buffers and non-blocking transfers reduce transfer overhead; the visual backbone processes larger chunks of frames.

Batch sizes and loader-worker counts are tested explicitly. A configuration is eligible only if its normalized embeddings agree with the original single-clip vectors within the reported tolerance. The final pipeline timer covers file reads, decoding, preprocessing, batch construction, transfer, encoding, all required label-bank scores, top-three selection and buffered result-file writes. Model loading, precomputed text-bank construction and network downloads are outside the timer. Repeated reads can benefit from the operating system's file cache; decoded images are regenerated rather than cached between passes.

Encoder-only speed isolates the GPU model call. Pipeline speed measures completed clips per second through the timed path. A million-video estimate is 1,000,000 ÷ clips_per_second ÷ 3,600 hours for one GPU pipeline. This is a linear extrapolation for the same duration and format distribution, without network failures, cold-storage guarantees or an exhaustive search for the fastest possible deployment.

CLAP measurement details · Initial PEAV measurement details · Initial benchmark CSV · Optimized benchmark CSV · Optimized trials, repetitions and agreement checks

Cohorts, previews and visual-change diagnostics

The AI page contains 50 random successes from the earlier 29,377-video download run, with sampling seed 20261025. Their publication window begins on 24 August 2026 and ends at the snapshot on 3 October. The comparison page selects 50 of the 100 recovered September 2026 posts with explicit AI flag zero, using seed 20261026. Forty-nine of those 50 comparison posts have captions. AI flag zero is not an independent authenticity assessment. Download availability and the original replacement procedure can bias both samples.

Inference receives original downloaded media, capped at 30 seconds. Browser previews use H.264/AAC and fit within 720 pixels. Five frames spanning each analyzed clip are compared with DINOv2, SSIM and pixel-change measures. These can help find static imagery or localized changes, but cuts, captions and camera movement also affect the metrics. The AI-page regression uses held-out video folds to predict the DINO change proxy; it does not measure verified motion-detection accuracy.

Videos retain their original rights. A dataset's metadata licence does not establish a permissive reuse licence for the underlying media. The public files contain no model checkpoints or private cluster paths.