Publications

Recent advances in stereo matching have focused on accuracy, often at the cost of significantly increased model size. Traditionally, the community has regarded efficient models as incapable of zero-shot ability due to their limited capacity. In this paper, we introduce Lite Any Stereo, a stereo depth estimation framework that achieves strong zero-shot generalization while remaining highly efficient. To this end, we design a compact yet expressive backbone to ensure scalability, along with a carefully crafted hybrid cost aggregation module. We further propose a three-stage training strategy on million-scale data to effectively bridge the sim-to-real gap. Together, these components demonstrate that an ultra-light model can deliver strong generalization, ranking 1st across four widely used real-world benchmarks. Remarkably, our model attains accuracy comparable to or exceeding state-of-the-art non-prior-based accurate methods while requiring less than 1% of their computational cost, setting a new standard for efficient stereo matching.

In this paper, we introduce POMA-3D, the first self-supervised 3D representation model learned from point maps. Point maps encode explicit 3D coordinates on a structured 2D grid, preserving global 3D geometry while remaining compatible with the input format of 2D foundation models. To transfer rich 2D priors into POMA-3D, a view-to-scene alignment strategy is designed. Moreover, as point maps are view-dependent with respect to a canonical space, we introduce POMA-JEPA, a joint embedding-predictive architecture that enforces geometrically consistent point map features across multiple views. Additionally, we introduce ScenePoint, a point map dataset constructed from 6.5K room-level RGB-D scenes and 1M 2D image scenes to facilitate large-scale POMA-3D pretraining. Experiments show that POMA-3D serves as a strong backbone for both specialist and generalist 3D understanding. It benefits diverse tasks, including 3D question answering, embodied navigation, scene retrieval, and embodied localization, all achieved using only geometric inputs (i.e., 3D coordinates). Overall, our POMA-3D explores a point map way to 3D scene understanding, addressing the scarcity of pretrained priors and limited data in 3D representation learning.

Video stereo matching is the task of estimating consistent disparity maps from rectified stereo videos. There is considerable scope for improvement in both datasets and methods within this area. Recent learning-based methods often focus on optimizing performance for independent stereo pairs, leading to temporal inconsistencies in videos. Existing video methods typically employ sliding window operation over time dimension, which can result in low-frequency oscillations corresponding to the window size. To address these challenges, we propose a bidirectional alignment mechanism for adjacent frames as a fundamental operation. Building on this, we introduce a novel video processing framework, BiDAStereo, and a plugin stabilizer network, BiDAStabilizer, compatible with general image-based methods. Regarding datasets, current synthetic object-based and indoor datasets are commonly used for training and benchmarking, with a lack of outdoor nature scenarios. To bridge this gap, we present a realistic synthetic dataset and benchmark focused on natural scenes, along with a real-world dataset captured by a stereo camera in diverse urban scenes for qualitative evaluation. Extensive experiments on in-domain, out-of-domain, and robustness evaluation demonstrate the contribution of our methods and datasets, showcasing improvements in prediction quality and achieving state-of-the-art results on various commonly used benchmarks.

We introduce SPFSplatV2, an efficient feed-forward framework for 3D Gaussian splatting from sparse multi-view images, requiring no ground-truth poses during training and inference. It employs a shared feature extraction backbone, enabling simultaneous prediction of 3D Gaussian primitives and camera poses in a canonical space from unposed inputs. A masked attention mechanism is introduced to efficiently estimate target poses during training, while a reprojection loss enforces pixel-aligned Gaussian primitives, providing stronger geometric constraints. We further demonstrate the compatibility of our training framework with different reconstruction architectures, resulting in two model variants. Remarkably, despite the absence of pose supervision, our method achieves state-of-the-art performance in both in-domain and out-of-domain novel view synthesis, even under extreme viewpoint changes and limited image overlap, and surpasses recent methods that rely on geometric supervision for relative pose estimation. By eliminating dependence on ground-truth poses, our method offers the scalability to leverage larger and more diverse datasets.

We introduce SPFSplat, an efficient framework for 3D Gaussian splatting from sparse multi-view images, requiring no ground-truth poses during training or inference. It employs a shared feature extraction backbone, enabling simultaneous prediction of 3D Gaussian primitives and camera poses in a canonical space from unposed inputs within a single feed-forward step. Alongside the rendering loss based on estimated novel-view poses, a reprojection loss is integrated to enforce the learning of pixel-aligned Gaussian primitives for enhanced geometric constraints. This pose-free training paradigm and efficient one-step feedforward design make SPFSplat well-suited for practical applications. Remarkably, despite the absence of pose supervision, SPFSplat achieves state-of-the-art performance in novel view synthesis even under significant viewpoint changes and limited image overlap. It also surpasses recent methods trained with geometry priors in relative pose estimation.

This paper introduces Stereo Any Video, a powerful framework for video stereo matching. It can estimate spatially accurate and temporally consistent disparities without relying on auxiliary information such as camera poses or optical flow. The strong capability is driven by rich priors from monocular video depth models, which are integrated with convolutional features to produce stable representations. To further enhance performance, key architectural innovations are introduced: all-to-all-pairs correlation, which constructs smooth and robust matching cost volumes, and temporal convex upsampling, which improves temporal coherence. These components collectively ensure robustness, accuracy, and temporal consistency, setting a new standard in video stereo matching. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple datasets both qualitatively and quantitatively in zero-shot settings, as well as strong generalization to real-world indoor and outdoor scenarios.

The rise of vision-language foundation models marks an advancement in bridging the gap between human and machine capabilities in 3D scene reasoning. Existing 3D reasoning benchmarks assume real-time scene accessibility, which is impractical due to the high cost of frequent scene updates. To this end, we introduce Hypothetical 3D Reasoning, namely Hypo3D, a benchmark designed to evaluate models’ ability to reason without access to real-time scene data. Models need to imagine the scene state based on a provided change description before reasoning. Hypo3D is formulated as a 3D Visual Question Answering (VQA) benchmark, comprising 7,727 context changes across 700 indoor scenes, resulting in 14,885 question-answer pairs. An anchor-based world frame is established for all scenes, ensuring consistent reference to a global frame for directional terms in context changes and QAs. Extensive experiments show that state-of-the-art foundation models struggle to reason in hypothetically changed scenes. This reveals a substantial performance gap compared to humans, particularly in scenarios involving movement changes and directional reasoning. Even when the context change is irrelevant to the question, models often incorrectly adjust their answers.

Embodied reasoning systems integrate robotic hardware and cognitive processes to perform complex tasks, typically in response to a natural language query about a specific physical environment. This usually involves changing the belief about the scene or physically interacting and changing the scene. We introduce a modular Closed Loop Interactive Embodied Reasoning (CLIER) approach that takes into account measurements of non-visual object properties, changes in the scene caused by external disturbances, and uncertain outcomes of robotic actions. CLIER performs multi-modal reasoning and action planning and generates a sequence of primitive actions that can be executed by a robot manipulator. The method operates in a closed loop, responding to changes in the environment. It is developed using the MuBle simulation environment and evaluated in 10 interactive benchmark scenarios, as well as in real-world manipulation tasks.

Current embodied reasoning agents struggle with long-horizon tasks that require physical interaction with the environment to acquire information. We introduce MuBlE, a simulation environment built on robosuite that combines the MuJoCo physics engine with Blender rendering to provide realistic visual observations consistent with the physical state of the scene. MuBlE supports multimodal data generation and closed-loop interaction at both visual-action and control-physics levels. Together with the simulator, we introduce SHOP-VRB2, a benchmark containing ten classes of multi-step reasoning scenarios requiring both visual and physical measurements.

Acquiring accurate channel state information is crucial for exploiting the benefits of massive multiple-input multiple-output systems. We formulate CSI compression using implicit neural representations, treating each CSI matrix as a neural function mapping spatial coordinates to channel gains. Instead of transmitting the parameters of individual neural functions, low-dimensional modulations derived through meta-learning are transmitted and applied to a shared base network at the receiver to reconstruct the CSI matrix. The proposed approach achieves state-of-the-art CSI compression performance while providing flexibility in feedback strategies.

Recent open-world 3D representation learning methods using Vision-Language Models (VLMs) to align 3D point cloud with image-text information have shown superior 3D zero-shot performance. However, CAD-rendered images for this alignment often lack realism and texture variation, compromising alignment robustness. Moreover, the volume discrepancy between 3D and 2D pretraining datasets highlights the need for effective strategies to transfer the representational abilities of VLMs to 3D learning. In this paper, we present OpenDlign, a novel open-world 3D model using depth-aligned images generated from a diffusion model for robust multimodal alignment. These images exhibit greater texture diversity than CAD renderings due to the stochastic nature of the diffusion model. By refining the depth map projection pipeline and designing depth-specific prompts, OpenDlign leverages rich knowledge in pre-trained VLM for 3D representation learning with streamlined fine-tuning. Our experiments show that OpenDlign achieves high zero-shot and few-shot performance on diverse 3D tasks, despite only fine-tuning 6 million parameters on a limited ShapeNet dataset. In zero-shot classification, OpenDlign surpasses previous models by 8.0% on ModelNet40 and 16.4% on OmniObject3D. Additionally, using depth-aligned images for multimodal alignment consistently enhances the performance of other state-of-the-art models.

We present a framework for automatically extracting physical object properties such as material composition, mass, volume, and stiffness through robotic manipulation. A Bayesian network represents dependencies between object properties and incorporates uncertainty associated with measurement actions. Exploratory actions are selected according to expected information gain and object properties are updated through Bayesian inference. Experiments demonstrate effective action selection, including cases where physical properties conflict with visual appearance. The system integrates with an online database containing more than 24,000 measurements of 63 objects collected using different grippers.

We introduce a vision transformer based deep joint source-channel coding scheme for wireless image transmission over multiple-input multiple-output channels. DeepJSCC-MIMO operates in both open-loop and closed-loop MIMO systems and outperforms classical separation-based approaches while remaining robust to channel estimation errors. Through the self-attention mechanism, the model learns feature mapping and power allocation strategies conditioned on both the source image and channel state. The same model adapts to varying channel conditions and antenna configurations without retraining, providing a flexible approach for semantic image communication.

We introduce the Drone Depth and Obstacle Segmentation dataset, a synthetic aerial dataset designed for semantic segmentation and depth estimation in autonomous drone applications. The dataset emphasizes challenging environmental conditions and the detection of thin structures such as wires, cables, and fences. It contains diverse weather conditions and provides accurate depth and segmentation annotations generated using photorealistic rendering. We also introduce drone-specific evaluation metrics for depth estimation that emphasize thin obstacle detection, providing a benchmark for developing safer and more robust autonomous navigation systems.

We study early exiting for collaborative inference systems operating over wireless channels. In these systems, part of a deep neural network runs on a resource-constrained edge device while the remaining computation is performed at an edge server. We introduce transmission-decision mechanisms that use information from an early exit together with wireless channel conditions to decide whether a local prediction should be accepted or intermediate features transmitted to the server. Experiments on image classification show that appropriately exploiting early exits can simultaneously improve inference performance and substantially reduce communication requirements.

We introduce AirNet, a family of methods for efficiently transmitting deep neural networks over wireless channels under power and latency constraints. Instead of attempting to reconstruct network parameters with maximum fidelity, AirNet directly maps network parameters to channel symbols and trains the network to tolerate channel noise. The approach combines noise-aware training, pruning, bandwidth expansion, and unequal error protection. AirNet achieves higher task accuracy than separation-based alternatives and can train a spectrum of networks targeting different channel conditions without requiring independent models for every operating point.

We present a transformer-based wireless image transmission method that exploits channel output feedback from the receiver. The proposed joint source-channel coding architecture uses a unified encoder across multiple transmission blocks and progressively refines the receiver’s reconstruction. Feedback provides information about the channel state and the decoder’s current estimate of the source, allowing subsequent coded symbols to be adapted accordingly. The method achieves strong transmission performance, remains robust to noisy feedback, and can adapt to channel conditions without explicit channel estimation. The approach is also extended to broadcast communication with feedback from multiple receivers.

We revisit knowledge distillation from the perspective of function matching and metric learning and investigate the role of normalization, soft maximum functions, and projection layers. We show theoretically that the projector can implicitly encode information about previous examples, enabling relational gradients for the student network. We further demonstrate that representation normalization interacts strongly with projector training dynamics and that a simple soft maximum formulation can alleviate capacity-gap problems. Experiments on image classification, object detection, and data-efficient transformer training show competitive or superior performance to existing distillation techniques while retaining high computational efficiency.

Dynamic stereo matching is the task of estimating consistent disparities from stereo videos with dynamic objects. Recent learning-based methods prioritize optimal performance on a single stereo pair, resulting in temporal inconsistencies. Existing video methods apply per-frame matching and window-based cost aggregation across the time dimension, leading to low-frequency oscillations at the scale of the window size. Towards this challenge, we develop a bidirectional alignment mechanism for adjacent frames as a fundamental operation. We further propose a novel framework, BiDAStereo, that achieves consistent dynamic stereo matching. Unlike the existing methods, we model this task as local matching and global aggregation. Locally, we consider correlation in a triple-frame manner to pool information from adjacent frames and improve the temporal consistency. Globally, to exploit the entire sequence’s consistency and extract dynamic scene cues for aggregation, we develop a motion-propagation recurrent unit. Extensive experiments demonstrate the performance of our method, showcasing improvements in prediction quality and achieving SoTA results on commonly used benchmarks.

In this paper we revisit the efficacy of knowledge distillation as a function matching and metric learning problem. In doing so we verify three important design decisions, namely the normalisation, soft maximum function, and projection layers as key ingredients. We theoretically show that the projector implicitly encodes information on past examples, enabling relational gradients for the student. We then show that the normalisation of representations is tightly coupled with the training dynamics of this projector, which can have a large impact on the students performance. Finally, we show that a simple soft maximum function can be used to address any significant capacity gap problems. Experimental results on various benchmark datasets demonstrate that using these insights can lead to superior or comparable performance to state-of-the-art knowledge distillation techniques, despite being much more computationally efficient. In particular, we obtain these results across image classification (CIFAR100 and ImageNet), object detection (COCO2017), and on more difficult distillation objectives, such as training data efficient transformers, whereby we attain a 77.2% top-1 accuracy with DeiT-Ti on ImageNet. Code and models are publicly available.

In the realm of fully autonomous drones, the accurate detection of obstacles is paramount to ensure safe navigation and prevent collisions. Among these challenges, the detection of wires stands out due to their slender profile, which poses a unique and intricate problem. To address this issue, we present an innovative solution in the form of a monocular end-to-end model for wire segmentation and depth estimation. Our approach leverages a temporal correlation layer trained on synthetic data, providing the model with the ability to effectively tackle the complex joint task of wire detection and depth estimation. We demonstrate the superiority of our proposed method over existing competitive approaches in the joint task of wire detection and depth estimation. Our results underscore the potential of our model to enhance the safety and precision of autonomous drones, shedding light on its promising applications in real-world scenarios.

Accurate obstacle detection in Unmanned Aerial Vehicles (UAVs) using fisheye lenses is challenged by image distortions. While advanced algorithms like Fast Region-Based Convolutional Neural Network (Fast R-CNN), Spatial Pyramid Pooling-Net (SPP-Net), and You Only Look Once (YOLO) are proficient with standard images, they underperform on fisheye images due to serious distortions. We introduce a real-time fisheye object detection system for UAVs, underpinned by specialized fisheye datasets. Our contributions encompass the creation of UAV-centric fisheye datasets, a distillation-based (also termed Teacher-Student) training method, and AirEyeSeg, a pioneering fisheye detector. AirEyeSeg achieved a Mask(mAP50) of 88.6% for cars on the combined Visdorone and UAVid datasets and 84.5% for people on the SEE dataset using the Box(P) metric. Our results demonstrate AirEyeSeg’s superiority over traditional detectors and validate our Teacher-Student training approach, setting a benchmark in fisheye-lensed UAV object detection.

The advancement of autonomous drones, essential for sectors such as remote sensing and emergency services, is hindered by the absence of training datasets that fully capture the environmental challenges present in real-world scenarios, particularly operations in non-optimal weather conditions and the detection of thin structures like wires. We present the Drone Depth and Obstacle Segmentation (DDOS) dataset to fill this critical gap with a collection of synthetic aerial images, created to provide comprehensive training samples for semantic segmentation and depth estimation. Specifically designed to enhance the identification of thin structures, DDOS allows drones to navigate a wide range of weather conditions, significantly elevating drone training and operational safety. Additionally, this work introduces innovative drone-specific metrics aimed at refining the evaluation of algorithms in depth estimation, with a focus on thin structure detection. These contributions not only pave the way for substantial improvements in autonomous drone technology but also set a new benchmark for future research, opening avenues for further advancements in drone navigation and safety.

Semantic segmentation from aerial views is a crucial task for autonomous drones, as they rely on precise and accurate segmentation to navigate safely and efficiently. However, aerial images present unique challenges such as diverse viewpoints, extreme scale variations, and high scene complexity. In this paper, we propose an end-to-end multi-class semantic segmentation diffusion model that addresses these challenges. We introduce recursive denoising to allow information to propagate through the denoising process, as well as a hierarchical multi-scale approach that complements the diffusion process. Our method achieves promising results on the UAVid dataset and state-of-the-art performance on the Vaihingen Building segmentation benchmark. Being the first iteration of this method, it shows great promise for future improvements.

Accurate real depth annotations are difficult to acquire, needing the use of special devices such as a LiDAR sensor. Self-supervised methods try to overcome this problem by processing video or stereo sequences, which may not always be available. Instead, in this paper, we propose a domain adaptation approach to train a monocular depth estimation model using a fully-annotated source dataset and a non-annotated target dataset. We bridge the domain gap by leveraging semantic predictions and low-level edge features to provide guidance for the target domain. We enforce consistency between the main model and a second model trained with semantic segmentation and edge maps, and introduce priors in the form of instance heights. Our approach is evaluated on standard domain adaptation benchmarks for monocular depth estimation and show consistent improvement upon the state-of-the-art.

CLIP is a significant vision-and-language training framework that has shown surprisingly general understanding of the world, with good performance in many open-ended tasks with little or no additional training. A recent technique has used CLIP to perform 0-shot Monocular Depth Estimation (MDE) by using depth-related prompts, but the use of human language in these prompts presents an unnecessary human bias. In this work, we use continuous learnable tokens in place of discrete human-language words to shed light on the problem. We achieve a significant boost in performance, and find that the learned tokens do not map neatly to depth-related human language, implying that CLIP’s concept of depth is not succinctly expressible in human language. We posit that this may extend to other CLIP concepts, and believe that this finding will spark further research into both the use and interpretation of non-linguistic tokens in all open-ended scene interpretation tasks.

We present an efficient alternative to the convolutional layer using cheap spatial transformations. This construction exploits an inherent spatial redundancy of the learned convolutional filters to enable a much greater parameter efficiency, while maintaining the top-end accuracy of their dense counter-parts. Training these networks is modelled as a generalised pruning problem, whereby the pruned filters are replaced with cheap transformations from the set of non-pruned filters. We provide an efficient implementation of the proposed layer, followed by two natural extensions to avoid excessive feature compression and to improve the expressivity of the transformed features. We show that these networks can achieve comparable or improved performance to state-of-the-art pruning models across both the CIFAR-10 and ImageNet-1K datasets.

CLIP has demonstrated strong general-purpose visual and linguistic representations and can be applied to zero-shot monocular depth estimation using depth-related text prompts. We investigate whether human-language prompts impose an unnecessary limitation by replacing discrete language prompts with continuous learnable tokens. Learned prompts substantially improve depth estimation performance and do not map cleanly onto depth-related human language. The results suggest that some concepts encoded by CLIP may not be succinctly expressible through natural-language prompts and motivate the use of non-linguistic learned tokens for open-ended visual understanding tasks.

We propose ViT-MIMO, a vision transformer based joint source-channel coding scheme for wireless image transmission over multiple-input multiple-output systems. The architecture outperforms conventional separation-based approaches while adapting to different channel conditions without retraining. By exploiting self-attention, ViT-MIMO learns feature mappings and power allocation jointly from the source image and current channel state. Numerical experiments demonstrate improved image transmission quality over a wide range of operating conditions, illustrating the potential of transformer architectures for adaptive semantic communication.

We study collaborative image retrieval at the wireless edge, where multiple devices observe the same object and jointly communicate visual features to an edge server over a shared multiple-access channel. We propose a semantic non-orthogonal multiple-access framework in which extracted features are mapped directly to channel inputs and combined over the air. A contrastive-learning based semantic communication method exploits correlations among observations by treating noisy correlated signals as different views of the same identity. Experiments demonstrate strong retrieval performance, particularly under low signal-to-noise ratios and constrained communication bandwidth.

We address fine-grained grasp-oriented segmentation of deformable clothes from depth images. Rather than detecting isolated grasping points, the proposed approach recognizes larger semantic regions useful for manipulation tasks such as folding and assisted dressing. For T-shirts, we define semantic regions corresponding to the neckline, sleeve cuffs, hem, and grasping points and train a U-Net based model to identify them. We further introduce a multilayer domain adaptation strategy that requires no manually annotated real training images. Experiments on real depth images show that training with synthetic annotations and the proposed domain adaptation achieves performance competitive with methods using real-data supervision.

We present an efficient alternative to conventional convolutional layers that exploits spatial redundancy among learned filters. Pruned filters are reconstructed using inexpensive spatial transformations of retained filters, providing greater parameter efficiency while preserving accuracy. Extensions reduce excessive feature compression and improve transformed-feature expressivity, producing competitive results on CIFAR-10 and ImageNet-1K.

We present a method for improving monocular depth estimation by explicitly incorporating semantic knowledge about objects and their relationships. The proposed ObjCAViT module extracts world knowledge from language models and combines it with image-object cross-attention and apparent object-size information. Experiments on NYUv2 and KITTI demonstrate that language information and cross-attention improve depth estimation performance.

We introduce a computer vision benchmark and approach for estimating water turbidity directly from photographs captured with a smartphone camera. A simple image acquisition protocol is proposed that requires no specialist measurement equipment. The resulting model estimates turbidity over the range of 0 to 40 NTU under controlled laboratory conditions, providing a low-cost alternative to conventional turbidimeters.

We investigate camera pose regression when models are trained using synthetic images that differ significantly from the real images encountered at test time. We introduce an approach designed to reduce this synthetic-to-real distribution gap and improve camera localisation from out-of-distribution synthetic views. Experiments across several localisation benchmarks demonstrate improved robustness compared with conventional camera pose regression methods.

We present a channel-adaptive joint source-channel coding scheme for wireless image transmission over multipath fading channels. The proposed autoencoder employs OFDM and a dual-attention mechanism that adapts feature mapping and transmission power according to estimated channel state information. The method achieves strong image reconstruction performance and remains robust across varying channel conditions.

We investigate whether explicitly providing monocular depth networks with cues inspired by biological vision can improve depth prediction. Semantic information, object-size priors, instance size, and relationships encoded using language embeddings are supplied alongside RGB inputs. Experiments on NYUv2 show that injecting these known useful cues improves depth estimation compared with relying entirely on features learned from images.

We introduce visual descriptors designed to preserve matching accuracy while concealing the appearance of the original image. An adversarial framework trains a feature encoder to prevent an image reconstruction network from recovering scene content from its descriptors. The resulting NinjaDesc representations provide competitive local-feature matching while significantly reducing the amount of visual information exposed by the descriptors.

We formulate relative scale estimation between two images as prediction of a probability distribution over scale factors. ScaleNet uses dilated convolutions together with self- and cross-correlation layers to infer the scale difference between image pairs. Correcting images using the predicted scale improves sparse feature matching, camera pose estimation, 3D reconstruction, and dense geometric matching.

We introduce SAMPLE-HD, a simulation environment for learning interactive reasoning and manipulation that combines visual understanding with behavioural planning. The environment supports simultaneous action and motion planning and provides multimodal observations for embodied agents. It is designed to facilitate learning systems that reason about scenes and subsequently execute physically feasible manipulation actions.

We address person re-identification from drone imagery, where viewpoint changes, unusual poses, low resolution, blur, and occlusion make matching particularly difficult. The proposed method combines part-based convolutional representations with adaptive sample weighting and introduces a re-ranking strategy based on expanded cross-neighbourhood and Jaccard distances. Experiments demonstrate significant improvements on aerial person re-identification benchmarks.

We establish an information-theoretic formulation of representation knowledge distillation using an entropy-like functional. The resulting framework introduces complementary objectives that maximise correlation and a quantity related to mutual information between teacher and student representations. The method achieves competitive performance on knowledge distillation and cross-model transfer while requiring substantially less computational overhead than many existing approaches.

We study the trade-off between matching accuracy, computational complexity, and model size for learned local descriptors. Starting from HardNet, we introduce depthwise separable operations and a Convolution-Depthwise-Pointwise decomposition that exploits redundancy in convolutional filters. The resulting models achieve large reductions in parameters and computation while retaining almost all of the original descriptor accuracy.

We present an integrated embodied reasoning system that combines visual and natural-language understanding, action planning, motion planning, robotic manipulation, and discovery of object properties. The system converts multimodal queries into executable actions and uses physical interaction to refine its knowledge about the environment. Experiments in simulation and on a real robotic manipulator demonstrate successful execution of challenging reasoning and manipulation queries.

We investigate automatic detection of spider monkey whinny vocalisations in challenging recordings collected in natural environments. A multi-attentive learning approach is used to identify relevant acoustic events despite background noise and large variability in real-world recordings. The work demonstrates how machine learning can support scalable monitoring of wildlife vocalisations in ecological studies.

We study image retrieval over bandwidth- and power-constrained wireless links. Rather than transmitting high-quality images, an edge device sends compact task-oriented representations to a server. We investigate both digital compression and analog joint source-channel coding approaches and show that direct transmission of learned visual features improves retrieval accuracy, encoding efficiency, and robustness to changing channel conditions.

We introduce Vision to Action, a task in which a robotic agent must convert visual observations and natural-language instructions into sequences of primitive manipulation actions. We extend the SHOP-VRB benchmark with task instructions and an execution engine for validating action sequences. A multimodal attention model is proposed and evaluated on the resulting reasoning and action-planning benchmark.

We propose Describe-to-Detect, an approach that reverses the conventional local-feature pipeline by deriving keypoint detections from informative descriptor responses. The method identifies image locations whose descriptors are discriminative and stable and can be applied to pretrained descriptor networks without requiring a dedicated detector. Experiments demonstrate improved matching and localisation performance across standard local-feature benchmarks.

We introduce a local feature architecture in which detector and descriptor extraction remain separate but interact through the learning objective. A reformulated hard-mining triplet loss refines candidate keypoint locations using the descriptor map, while a multi-scale hybrid descriptor combines handcrafted and learned features. The method achieves strong results on image matching, reconstruction, and camera localisation benchmarks.

We introduce second-order spatial attention and a second-order similarity loss for learning image descriptors. Spatial attention reweights feature maps to emphasise informative image regions, while the similarity objective regularises relationships between examples in descriptor space. Combining the two components improves both global image retrieval and local feature matching and achieves state-of-the-art performance on several benchmarks.

We propose filter-level pruning with hierarchical knowledge distillation using intermediate teaching-assistant networks. Each successively smaller model is pruned with guidance from an appropriately sized teacher, reducing the capacity gap encountered in direct teacher-student distillation. Experiments on CIFAR-10 and ImageNet with VGG16 and ResNet50 demonstrate improvements in the trade-off between compression and classification accuracy.

We introduce SHOP-VRB, a benchmark for visual reasoning in robotic grasping and manipulation scenarios. It contains household objects, their attributes and functions, natural-language descriptions, question-answer pairs, and semantic scene representations. A synthetic data-generation framework and a symbolic reasoning baseline are provided, exposing limitations of existing visual reasoning benchmarks and methods.

We propose a joint transmission and recognition framework for resource-constrained wireless edge devices. Deep joint source-channel coding is combined with neural-network pruning to reduce both transmission requirements and local computational cost. Experiments on image classification demonstrate that significant reductions in computation can be achieved while retaining reliable inference over noisy wireless channels.

We investigate task-oriented communication for image retrieval over wireless channels. Instead of transmitting a compressed reconstruction of the query image, a neural encoder maps retrieval features directly to channel symbols using joint source-channel coding. The method improves end-to-end retrieval accuracy and reduces encoding complexity, making it particularly suitable for latency- and power-constrained IoT devices.

We address person search using textual attributes by learning a shared representation between visual observations and structured language descriptions. The proposed model encodes attributes associated with different body regions and aligns them with image representations extracted using a convolutional network. The resulting multimodal descriptor supports retrieval of people from image collections using semantic attribute queries.

We propose an unsupervised domain adaptation approach for sparse-to-dense depth completion trained using synthetic data without real-domain ground truth. The method models realistic LiDAR noise, filters noisy real measurements for supervision, and adapts synthetic RGB images using CycleGAN. Experiments on the KITTI depth completion benchmark demonstrate substantial improvements over existing synthetic-to-real approaches.

We analyse the influence of L2 normalisation on descriptor learning and introduce HyNet, a local descriptor trained with a hybrid similarity measure and triplet loss. The architecture normalises intermediate feature maps and constrains descriptor norms during training. HyNet achieves state-of-the-art results on patch matching, verification, retrieval, and end-to-end 3D reconstruction benchmarks.

We introduce a shallow multi-scale keypoint detector that combines handcrafted filters with learned convolutional filters. Handcrafted operators provide stable structural information while learned filters localise and rank repeatable keypoints. A dedicated loss encourages repeatability across scale and geometric transformations. Key.Net improves repeatability and matching performance while retaining low computational complexity.

In this paper, a novel benchmark is introduced for evaluating local image descriptors. We demonstrate limitations of the commonly used datasets and evaluation protocols, that lead to ambiguities and contradictory results in the literature. Furthermore, these benchmarks are nearly saturated due to the recent improvements in local descriptors obtained by learning from large annotated datasets. To address these issues, we introduce a new large dataset suitable for training and testing modern descriptors, together with strictly defined evaluation protocols in several tasks such as matching, retrieval and verification. This allows for more realistic, thus more reliable comparisons in different application scenarios. We evaluate the performance of several state-of-the-art descriptors and analyse their properties. We show that a simple normalisation of traditional hand-crafted descriptors is able to boost their performance to the level of deep learning based descriptors once realistic benchmarks are considered. Additionally we specify a protocol for learning and evaluating using cross validation. We show that when training state-of-the-art descriptors on this dataset, the traditional verification task is almost entirely saturated.

We address person re-identification when only partial views of individuals are available. Partial Matching Net detects body joints, aligns visible body regions, and hallucinates missing parts using information learned from complete examples. The aligned and reconstructed representations are combined for matching. Experiments on multiple person re-identification datasets demonstrate significant improvements over existing partial matching approaches.

We investigate person re-identification using both images and natural-language descriptions. A joint vision-language representation enables matching across modalities and allows textual descriptions to enrich visual examples without language annotations. We additionally introduce natural-language descriptions for the CUHK03 and VIPeR benchmarks and demonstrate advantages over conventional attribute-based person representations.

We study multimodal analysis of news articles, where accompanying images and text are often only loosely and indirectly related. The work introduces the BreakingNews dataset and methods combining visual and textual representations for tasks including article illustration, geolocation, and semantic analysis. The benchmark supports research on multimodal understanding beyond datasets with directly aligned image captions.

In this paper, we propose a novel benchmark for evaluating local image descriptors. We demonstrate that the existing datasets and evaluation protocols do not specify unambiguously all aspects of evaluation, leading to ambiguities and inconsistencies in results reported in the literature. Furthermore, these datasets are nearly saturated due to the recent improvements in local descriptors obtained by learning them from large annotated datasets. Therefore, we introduce a new large dataset suitable for training and testing modern descriptors, together with strictly defined evaluation protocols in several tasks such as matching, retrieval and classification. This allows for more realistic, and thus more reliable comparisons in different application scenarios. We evaluate the performance of several state-of-the-art descriptors and analyse their properties. We show that a simple normalisation of traditional hand-crafted descriptors can boost their performance to the level of deep learning based descriptors within a realistic benchmarks evaluation.

We investigate learning compact local image descriptors using shallow convolutional networks and triplet-based training. The proposed approach jointly considers positive and negative patch examples and introduces effective sampling and loss strategies for learning discriminative descriptors. Despite its relatively shallow architecture, the resulting descriptor provides strong matching accuracy and became an influential baseline for subsequent learned local-feature methods.