The Signal and Image Processing Group in Little More than 3 Pages

download (PDF)

Group Description

Since its establishment, the Signal and Image Processing group has fostered a diverse and multidisciplinary community, defying categorization within a singular application area or discipline.

Emerging from its origins in communications and acoustic signal processing, SIPg has progressively directed its efforts towards tackling a series of complex problems that inherently involve optimization. This encompasses a wide spectrum of challenges, ranging from image analysis and computer vision to machine learning and the processing of large-scale, unstructured datasets.

Despite its inherent diversity, the group finds cohesion through the symbiotic relationship between the development of versatile and adaptable signal processing tools and the pursuit of domain-specific inquiries, driven by the interests of its members. Notably, the portfolio of methodologies encompasses prominent areas such as “low rank models,” “missing data/matrix completion,” and “large scale/distributed information processing,” which have yielded substantial breakthroughs in fields such as target and self-localization in sensor networks, visual matching, recognition, and 3D reconstruction. Complementing these advancements are SIPg’s foundational expertise in ocean acoustics and medical imaging, presenting a comprehensive snapshot of the group’s research endeavors. A special note for the fact that SIPg members include faculty both from IST and  the University of Algarve (UAlg) since ISR’s foundation. UAlg contributes mainly in areas related to HCI  and today are the leading and sole contributors of our oceanic (acoustic) science.

PhD Supervision

PhD supervision was, by far, the most relevant “mechanism” to fund and develop research at SIPg. This is to say that a significant part of our research was “detached” from constraints imposed by project deadlines or an otherwise intensive implementation and experimental work. This could be part of the reason for the 5 IMB (Portugal) Science Prize  thesis and 2 best PhD thesis in ECE at Carnegie Mellon University won in the past. Recently the scenario changed but this is still the most heartfelt job at SIPg as well as the main “tool” for international collaboration . Since 2012, SIPg members has graduated 12 dual PhD’s in the Carnegie Mellon|Portugal Program (4 in 2018-2023) with 3 more still ongoing.

As an active partner in the NetSys program (http://netsys.larsys.pt) SIPg completed 5 PhD’s in this period and 2 are in their final steps towards graduation. The BigMath Marie Curie PhD program (http://itn-bigmath.unimi.it) is another important program where two PhD’s also in final phase.

Of notice is also a special long-standing collaboration of UAlg with UFRJ (Rio de Janeiro, Brazil) in the area of ocean acoustics with 2 finished PhD’s and a special collaboration with the medical research center IMM in the supervision of an FCT funded PhD.

More recently SIPg researchers “crossed the aisle” and engaged in  Signal Processing/Machine Learning developments in “foreign” areas like Transportation and Structural Engineering  (Civil Eng). SIPg faculty co-advise two PhD students of the MIT-Portugal PhD Program in Transportation and one in Civil Engineering. Finally, SIPg was recipient of private funding by a Portuguese hi-tech company, an unusual feat in Portugal, that fully funds one of its employees to do the PhD at IST.  A list of 2018-2023 PhD thesis is included in annex and we will cite them when appropriate.

Research Highlights

Distributed Learning

In this work, we address signal processing (SP) challenges at a more canonical level in order to create unified solution methods that encompass a wide range of problems from real-life applications. Two canonical SP challenges that have been addressed in the past five years are as follows:

1. How can we design distributed algorithms that enable a collective of agents (such as robot teams, wireless sensor networks, or fleets of AUVs) to quickly learn from a large raw dataset? The dataset is dispersed across the agents, and the agents are loosely or asynchronously coupled through a communication network.
2. How can we learn parameters in extreme setups where the noise embedded in the available measurements is either of unknown shape or severely corrupts nearly half of the measurements?

Progress in these canonical SP challenges brings benefits that extend to multiple engineering contexts.

Focusing on the challenge of designing distributed learning algorithms for multi-agent systems, we have developed algorithms that can handle various physical constraints arising from real-life settings. These constraints determine how the dataset is distributed across the agents, either by examples (data matrix scattered horizontally) or by features (data matrix scattered vertically). The algorithms also consider the level of central coordination feasible, ranging from none (fully decentralized setups) to partial (federated learning setups).

The algorithms we have developed can be found in the following publications:

• In [A1], we exploit the mathematical framework of fixed-point operators to not only express broad classes of distributed processing problems, but also derive a template for an algorithm that can be deployed for inference in fully distributed settings. This template is then instantiated in applications of parameter estimation and dimensionality reduction (PCA) in [A2].
• In [A3], we propose a multi-token algorithm for addressing semi-decentralized federated learning setups where the data is scattered across the agents by features.
• In [A4] and [A5], we design simple and fast algorithms for parameter estimation in fully decentralized settings.

As the first authors in the references indicate, these works were developed in the PhD thesis of Francisco Lima, Ines Almeida and António Simões.

Distributed Self Localization

Methods developed in the distributed learning/optimization community have been instrumental in obtaining fast-converging collaborative localization algorithms for formations of agents. Here we highlight two methods that tackle typical scenarios in localization:

• In [A6] we propose a range-based convex localization method that is robust to the presence of outliers in pairwise measurements, attains fast convergence through the use of Nesterov’s optimal gradient method, and admits both synchronous and asynchronous communication schedules between agents. The use of convex relaxation affords significant advantages over more commonplace localization methods based on extended Kalman filtering, as convergence is guaranteed even if an initial estimated of the formation is not available.
• In [A7] a hybrid collaborative localization method that seamlessly fuses pairwise range and bearing measurements is proposed. Again, it is based on convex relaxation and optimization methods, inheriting its fast convergence from accelerated distributed gradient methods. The way that bearings are handled is markedly different from usual Kalman filtering approaches, where the equations for measurements are strongly nonlinear and non-convex, and may easily lead to misconvergence when the initial estimates are too inaccurate.

[A1]Distributed Banach–Picard Iteration for Locally Contractive Maps, F. Andrade, M. Figueiredo, and J. Xavier, IEEE Transactions on Automatic Control 68 (2), 1275-1280, 2022

[A2]Distributed Banach-Picard Iteration: Application to Distributed Parameter Estimation and PCA, F. Andrade, M. Figueiredo, J. Xavier, IEEE Transactions on Signal Processing 71, 17-30, 2023  

[A3]A Multi-Token Coordinate Descent Method for Vertical Federated Learning, Pedro Valdeira, Yuejie Chi, Claudia Soares, Joao Xavier, FL-NeurIPS 2022 

[A4]FADE: Fast and asymptotically efficient distributed estimator for dynamic networks, A. Simoes and J. Xavier, IEEE Transactions on Signal Processing, vol. 67, no. 8, pp. 2080—2092,2019

[A5]Distributed Jacobi asynchronous method for learning personal models, I. Almeida and J. Xavier, IEEE Signal Processing Letters, vol. 25, no. 9, pp. 1389—1392, 2018

[A6] C. Soares, J. Gomes, “STRONG: Synchronous and asynchronous RObust Network localization, under Non-Gaussian noise”, Signal Processing (Elsevier), vol. 185, pp. 108066, 2021.

[A7] B. Ferreira, J. Gomes, C. Soares, J Costeira, “FLORIS and CLORIS: Hybrid source and network localization based on ranges and video”, Signal Processing (Elsevier), vol. 153, pp. 355-367, Dec. 2018.

Ocean Acoustics

Visual Learning and Computer Vision

A significant number of SIPg researchers look at images and the processes of extracting information from them as their main object of study.  A necessarily small selection of highlights in this area is the following :

• Feedbot:  Starting as a short collaborative project with Carnegie Mellon University, the Feedbot project led by Manuel Marques developed a low-cost robot to feed people with motor disabilities[B4]. Today Feecbot became a line of research by itself with multiple goals from creating mobile versions to new interaction features. The central challenge is to be able to estimate head pose precisely from highly (self) occluded images of a face, a topic pursued by PhD student José Celestino.
• Fast Pose Graph Optimization: A 3D pose optimization framework that we claim to be (one of) the fastest and (one of) the most accurate available[B5] for a central problem in SLAM. This work became a central part in a relevant project we will mention ahead.
• Vehicle Counting in Camera Networks : We proposed a series of deep learning models to count vehicles in highly cluttered and low resolution cameras[B6]. These methods were applied to New York City’s  camera system (http://printart.isr.ist.utl.pt/citycam) and the research was done in the context of the dual PhD of Shanghang Zhang with Carnegie Mellon University. Beyond publications,  outcomes included a valuable dataset with 900,000 vehicle annotations, one US and one China patent.
• Mobility, Transportation and Walkability: This is an area of collaboration with the Civil Eng. Research Center (CERIS) where SIPg’s role is to develop technologies to assist surveying processes regarding cyclist safety and city infrastructure monitoring for pedestrians[B7]. With tight collaborations  since 2016, we highlight the PhD thesis of Miguel Costa running under the MIT-Portugal program in Transportation. Tackling the problem of forecasting availability in a city bike sharing system, Bárbara Tavares, currently a PhD student, won the 2nd prize of the Fraunhofer Portugal Challenge 2022  with her MSc thesis.
• Medical Imaging: Most relevant contributions in this key area deal with the segmentation of cardiac MRI as in Daniela Mendley’s PhD thesis and the  PAMI article[B1].  Breast cancer diagnosis is researched in Francisco Calisto’s ongoing  PhD under the umbrella of FCT project  MIA-Breast and described in [B2]. In the microscopy domain SIPg spun two collaborations with medical experts to find biological structures in images. One with CNRS-ISTI (Pisa,Italy)[B3] and another in the context of  Hemaxi Narotamo’s PhD thesis with co-supervision of a IMM researcher (https://imm.medicina.ulisboa.pt/).

[B1]“Cycoseg: a cyclic collaborative framework for automated medical image segmentation, Medley, Daniela O., Carlos Santiago, and Jacinto C. Nascimento, IEEE Transactions on Pattern Analysis and Machine Intelligence 44.11 (2021): 8167-8182.

[B2] Assertiveness-based Agent Communication for a Personalized Medicine on Medical Imaging Diagnosis. Calisto, F. M., Fernandes, J., Morais, M., Santiago, C., Abrantes, J. M., Nunes, N., & Nascimento, J. C. (2023, April), In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1-20).

[B3] Learning to count biological structures with raters’ uncertainty, Ciampi, L., Carrara, F., Totaro, V., Mazziotti, R., Lupori, L., Santiago, C. Gennaro, C. (2022).Medical Image Analysis, 80, 102500.

[B4]Vision Augmented Robot Feeding, Alexandre Candeias, Travers Rhodes, Manuel Marques, Jo ao P. Costeira, Manuela Veloso; Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0-0

[B5]Rotation Averaging in a Split Second: A Primal-Dual Method and a Closed-Form for Cycle Graphs, Gabriel Moreira, Manuel Marques, João Paulo Costeira; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 5452-5460

[B6] Adversarial multiple source domain adaptation, H Zhao, S Zhang, G Wu, JMF Moura, JP Costeira, GJ Gordon, Advances in neural information processing systems (NEURIPS 2019) 

[B7] Costa M, Cambra P, Moura F, Marques M. WalkBot: A Portable System to Scan Sidewalks. In2019 IEEE International Smart Cities Conference (ISC2) 2019 Oct 14 (pp. 167-172). IEEE.

“Special” Projects

From the list of projects we highlight here a small selection because of their somewhat singularity in SIPg’s past and because they bring a  different set of opportunities for the future.

PROJECT IFETCH

IFetch is a flagship project of the Carnegie Mellon University Program. Built with a consortium of 3 Universities (CMU, IST and Univ NOVA) and Farfetch, the first portuguese unicorn and a global player in the high-fashion online marketplace. The ambition was to develop a multimodal interactive conversational agent to help customers search and navigate in a huge catalog using  natural language and images(see example).

[C1] Explainable noisy label flipping for multi-label fashion image classification. B. Q. Ferreira, J. P. Costeira, J. P. Gomes. 2021. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3911–3915. 

[C2] Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of the Same Coin. G. Moreira, M. Marques, J. P. Costeira, A. Hauptmann. 2024., IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024.

[C3]Say yes to the fetch: Product retrieval on a structured multimodal catalog. J. B. Morais, C. Santiago, J. P. Costeira. 2022,  In 1st Annual AAAI Workshop on AI to Accelerate Science and Engineering. 

PROJECT AI4EU

The European Project AI4EU (htttp://ai4europe.eu) aimed at developing the European Platform for AI on demand. A 20M€ European extravaganza with 80 partners and a colossal ambition. It was an enlightening project more for its proposal than for the concrete outcomes. It introduced SIPg’s researchers to a world of containerization and orchestration of algorithms that we pursued after AI4EU finished and that we think it’s the right tool to bring the great theoretical developments to the “real world”.

PROJECT SMART RETAIL 

Due to the individual  initiative of one researcher (Manuel Marques), SIPg was invited by a “high profile¹” startup (http://sensei.tech) to join project  Smart Retail (http://smartretail.tecnico.ulisboa.pt). For Portugal this is a very large scale project (35M€) funded  under the National “Recovery and Resilience Program”(PRR). SIPg captured significant funds to develop technologies for large camera networks in the retail environment . Indeed we were coveted due to the performance of our pose-graph optimization algorithm that has already an evolution for this specific context https://github.com/gabmoreira/vican. 

Smart Retail involves the whole group and more, and is an enabler to other developments in a wide range of topics, including the outcome of AI4EU. We look at this project in a strategic and collective way since the volume of funding will impact SIPg beyond the project boundaries and “official” timeline.

PATENTS 

Deep Learning Methods For Estimating Density and/or Flow of Objects, and Related Methods and Software,  United States Patent No .: US 11,183,051 B2 ,  Nov. 23 , 2021 Inventor(s): José M. F. Moura; João Paulo Costeira; Shanghang Zhang; Evgeny Toropov

Deep Learning Methods for Estimating Density and/or Flow of Objects, and Related Methods and Software China Patent No.: ZL 2018800336678, September 8, 2023 Inventor(s): José M. F. Moura; João Paulo Costeira; Shanghang Zhang; Evgeny Toropov 

PRIZES

Hemaxi Narotamo was distinguished with the Maria de Lourdes Pintasilgo “Young Alumna” award to distinguish the crucial role of women in engineering .

Manuel Marques received an “honorable mention” in the  “Os Melhores do Portugal Tecnológico”(The best of  Portugal Tech), an award sponsored and organized by on of Portugal’s main media group (Impresa/Exame Informática)

Bárbara Tavares won 2nd Prize in the Fraunhofer Portugal Challenge 2022, awarded for bright technological ideas for MSc thesis

1. _{A startup that already lists as investors the largest retail group in the country and is frequently used by politicians as an example of national technological prowess} ↩︎