Noncoherent Detection and Interference Nulling for Terrestrial-Satellite Downlinz Coexistence in the Upper Mid-Band

Abstract

Terrestrial--satellite coexistence in the upper mid-band is challenging when a terrestrial base station has limited prior information about non-terrestrial receivers and their uplink transmissions. This paper studies noncoherent victim sensing and interference nulling, where uplink sensing snapshots are observed while the transmitted waveform is treated as unknown. For a single victim, we show that the generalized likelihood-ratio test reduces to a principal-eigenvector estimator of the sample covariance. For multiple victims, we combine MDL-based model-order selection with MUSIC to recover anonymous direction and power information that is sufficient for beam design without pilot knowledge or user identities. These estimates are then used in a nulling beamformer that preserves the intended terrestrial link while reducing leakage toward detected victims. We further analyze the single-victim estimator in the large-matrix regime and show that estimation accuracy improves with sensing SNR, which reveals the observed interplay between path loss and estimation quality. Site-specific ray-tracing results show significant reduction of NTN INR with only modest degradation of TN SINR.