Universal Short-Time Fourier Transform

Overview

The short-time Fourier transform (STFT) characterizes the local frequency content of time-varying signals by evaluating the discrete Fourier transform over a sliding analysis window. It is governed by a fundamental resolution trade-off: a short window localizes rapid transients in time at the cost of frequency resolution, whereas a long window resolves closely spaced harmonics at the expense of temporal resolution.

The Universal Short-Time Fourier Transform (U-STFT) resolves this trade-off sequentially by modeling the analysis window duration as a data-adaptive state variable. At each frame, a bank of candidate window durations is scored using normalized order-three Rényi entropy to quantify joint time-frequency concentration. A window-length trellis aggregates the resulting O(3^M) candidate window schedules with linear complexity per frame, with a cumulative mix-loss regret bound of O((k+1) log M) relative to any hindsight window schedule with k resolution changes over M frames. By linearity of the Fourier transform, the posterior expectation across the candidate bank yields a single spectrum equivalent to an STFT with a data-adaptive effective window, while an envelope-normalized synthesis stream guarantees exact overlap-add reconstruction. Evaluations on synthetic signals, speech, and marine bioacoustic recordings demonstrate adaptive resolution tracking, enhanced spectral concentration, and exact time-domain invertibility.

NumPy and PyTorch implementations share the same API and output names.

Get started in 5 steps

Each step takes a minute or two. Steps 2–4 continue in the same Python session.

1. Install

   You need Python 3.10 or newer. One line installs the package straight from GitHub, no clone needed:

   pip install git+https://github.com/mananmittal/Universal-Short-Time-Fourier-Transform.git

   Or clone the repository to get the examples, tests and paper scripts too:

   git clone https://github.com/mananmittal/Universal-Short-Time-Fourier-Transform.git
   cd Universal-Short-Time-Fourier-Transform
   python -m pip install -e ".[examples]"         # package + matplotlib
   python -m pip install -e ".[examples,torch]"   # ...or with the PyTorch backend

   You will see: python -c "import universal_stft" runs without an error. Prefer not to install anything? Run the quickstart notebook in Colab.

2. Your first spectrogram

   Make a test signal with a steady part and a sudden part, run the default mode="analysis", and plot the result in decibels.

   import numpy as np
   import matplotlib.pyplot as plt
   from universal_stft import universal_short_time_fourier_transform
   
   # 1 s of a steady 440 Hz tone, then 1 s of short clicks.
   fs = 8000                                    # sample rate in Hz
   t = np.arange(2 * fs) / fs
   x = np.where(t < 1.0, np.sin(2 * np.pi * 440 * t), 0.0)
   x[fs::800] += 1.0                            # a click every 0.1 s
   x += 1e-3 * np.random.default_rng(0).standard_normal(len(x))
   
   n_fft, hop_size = 512, 32
   result = universal_short_time_fourier_transform(x, n_fft=n_fft, hop_size=hop_size)
   S = result.analysis_spectrogram              # complex, shape (frames, bins) = (500, 257)
   
   frame_times = (np.arange(S.shape[0]) + 1) * hop_size / fs   # seconds
   freqs = np.arange(S.shape[1]) * fs / n_fft                    # Hz
   S_db = 20 * np.log10(np.abs(S.T) + 1e-12)
   plt.imshow(S_db, origin="lower", aspect="auto", cmap="magma",
              extent=[frame_times[0], frame_times[-1], freqs[0], freqs[-1]],
              vmin=S_db.max() - 80, vmax=S_db.max())
   plt.xlabel("Time (s)")
   plt.ylabel("Frequency (Hz)")
   plt.show()

   You will see: a thin horizontal line at 440 Hz for the first second, then thin vertical lines for the clicks.

3. Look at what it chose

   dominant_lengths is the most likely window length (in samples) at each frame.

   print(result.blend_info[0][0])               # the window bank: [ 32  64 128 256]
   
   plt.figure()
   plt.step(frame_times, result.dominant_lengths, where="post")
   plt.yscale("log", base=2)
   plt.xlabel("Time (s)")
   plt.ylabel("Most likely window (samples)")
   plt.show()
   # result.pmf_history has the full probability per window, shape (frames, windows)

   You will see: the window climbs to 256 samples during the tone and drops back to 32–64 at the clicks.

4. Filter and resynthesize

   Run with mode="filtering" to also get the invertible companion stream. Multiply every frame by the same gain, then invert.

   from universal_stft import inverse_universal_spectrogram
   
   result = universal_short_time_fourier_transform(x, n_fft=n_fft, hop_size=hop_size,
                                                   mode="filtering")
   F = result.filtering_spectrogram
   
   x_hat = inverse_universal_spectrogram(F, len(x), n_fft=n_fft, hop_size=hop_size)
   print(np.max(np.abs(x - x_hat)))             # below 1e-15
   
   gain = (np.arange(F.shape[1]) * fs / n_fft < 1000).astype(float)   # low-pass at 1 kHz
   x_low = inverse_universal_spectrogram(F * gain, len(x), n_fft=n_fft, hop_size=hop_size)

   You will see: a reconstruction error near machine precision, and x_low keeps the tone but loses the clicks above 1 kHz.

5. PyTorch: batches, GPU, gradients

   Same names and outputs. Input is (samples,) or (batch, samples); device and precision are kept (float32 → complex64, float64 → complex128).

   import torch
   from universal_stft.torch import universal_short_time_fourier_transform
   
   device = "cuda" if torch.cuda.is_available() else "cpu"
   x = torch.randn(8, 16_000, device=device, requires_grad=True)   # (batch, samples)
   
   result = universal_short_time_fourier_transform(x, n_fft=512, hop_size=32)
   print(result.analysis_spectrogram.shape)     # torch.Size([8, 500, 257]), complex64
   
   loss = result.analysis_spectrogram.abs().pow(2).mean()
   loss.backward()                              # gradients flow back to x

   You will see: one spectrogram per batch row, and x.grad filled in after backward().

Several channels?

multichannel_stft chooses the windows on one reference channel and uses them for every channel, so phase differences between channels stay meaningful. This runs in the NumPy session from steps 2–4.

from universal_stft import multichannel_stft

x_multi = np.stack([x, 0.5 * np.roll(x, 3)])            # (channels, samples)
result = multichannel_stft(x_multi, reference_sensor_id=0, n_fft=n_fft, hop_size=hop_size)
print(result.analysis_spectrogram.shape)                # (2, 500, 257)

The README has the longer version of each step, an API table and a troubleshooting FAQ.

Which mode do I need?

Choosing parameters

How it works

1. A dyadic window bank. Candidate windows have lengths H, 2H, 4H, …, each analysed at every frame.
2. An entropy score. Each candidate spectrum is scored by its normalized order-3 Rényi entropy, which rewards energy that is concentrated in both time and frequency.
3. A posterior over window lengths. The window length is treated as a hidden state on a Markov trellis. Between frames it can stay, grow to the next longer window, or reset to the shortest one. The scores update a posterior over these states one frame at a time.
4. A blended spectrum. The output is the posterior-weighted mixture of candidate spectra. Because the mixing is linear, this equals an STFT with one data-adaptive effective window per frame.
5. Exact inversion. Dividing the input by the overlap envelope of the effective windows gives a companion stream that reconstructs perfectly by overlap-add, so it can be filtered and resynthesized.

Results

Four figures for each of seven recordings: marine bioacoustics, English speech and a synthetic chirp. Pick a recording, then click any figure to enlarge it. Window labels H, 2H, … are positions in the dyadic bank.

Data sources

The field recordings are short excerpts. Please cite the original collection when you reuse them; NOAA's disclaimer and use constraints apply to those excerpts independently of this repository's license.

1. NOAA Office of National Marine Sanctuaries and U.S. Navy, SanctSound Raw Passive Acoustic Data. NOAA National Centers for Environmental Information, 2020. doi:10.25921/saca-sp25. Used for the humpback whale (CI04), humpback and dolphin (HI04), snapping shrimp (FK02) and fish chorus (GR01) recordings.
2. J. Yamagishi, C. Veaux and K. MacDonald, CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92). University of Edinburgh, Centre for Speech Technology Research, 2019. doi:10.7488/ds/2645. Used for the speech recordings of speakers p227 and p228.

The hyperbolic chirp is synthetic and was generated for this work.

Cite this work

@misc{mittal2026ustft,
  title={Universal Short-Time Fourier Transform: Data-Adaptive Window-Length Selection for Time-Frequency Analysis},
  author={Mittal, Manan and Sarkar, Kanad and Corey, Ryan M. and Buck, John R. and Singer, Andrew C.},
  year={2026},
  eprint={2610.37795},
  archivePrefix={arXiv},
  primaryClass={eess.SP},
  url={https://arxiv.org/abs/2610.37795}
}