Publications

2025

1. ICLR

   Exploring the Impact of Activation Functions in Training Neural ODEs

   Tianxiang Gao, Siyuan Sun, Hailiang Liu, and 1 more author

   In the 13th International Conference on Learning Representations (ICLR), 2025

   Oral Presentation (1.8% Acceptance Rate)

   Abs PDF

   Neural Ordinary Differential Equations (ODEs) have been successful in various applications due to their continuous nature and parameter-sharing efficiency. However, these unique characteristics also introduce challenges in training, particularly with respect to gradient computation accuracy and convergence analysis. In this paper, we address these challenges by investigating the impact of activation functions. We demonstrate that the properties of activation functions—specifically smoothness and nonlinearity—are critical to the training dynamics. Smooth activation functions guarantee globally unique solutions for both forward and backward ODEs, while sufficient nonlinearity is essential for maintaining the spectral properties of the Neural Tangent Kernel (NTK) during training. Together, these properties enable us to establish the global convergence of Neural ODEs under gradient descent in overparameterized regimes. Our theoretical findings are validated by numerical experiments, which not only support our analysis but also provide practical guidelines for scaling Neural ODEs, potentially leading to faster training and improved performance in real-world applications.

2023

1. NeurIPS

   Wide neural networks as gaussian processes: Lessons from deep equilibrium models

   Tianxiang Gao, Xiaokai Huo, Hailiang Liu, and 1 more author

   In the 36th Advances in Neural Information Processing Systems (NeruIPS), 2023

   Abs PDF

   Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area.

2022

1. ICLR

   A global convergence theory for deep implicit networks via over-parameterization

   Tianxiang Gao, Hailiang Liu, Jia Liu, and 2 more authors

   In the 10th International Conference on Learning Representations (ICLR), 2022

   Abs PDF

   Implicit deep learning has received increasing attention recently, since it generalizes the recursive prediction rules of many commonly used neural network architectures. Its prediction rule is provided implicitly based on the solution of an equilibrium equation. Although many recent studies have experimentally demonstrates its superior performances, the theoretical understanding of implicit neural networks is limited. In general, the equilibrium equation may not be well-posed during the training. As a result, there is no guarantee that a vanilla (stochastic) gradient descent (SGD) training nonlinear implicit neural networks can converge. This paper fills the gap by analyzing the gradient flow of Rectified Linear Unit (ReLU) activated implicit neural networks. For an m-width implicit neural network with ReLU activation and n training samples, we show that a randomly initialized gradient descent converges to a global minimum at a linear rate for the square loss function if the implicit neural network is over-parameterized. It is worth noting that, unlike existing works on the convergence of (S)GD on finitelayer over-parameterized neural networks, our convergence results hold for implicit neural networks, where the number of layers is infinite.

2. arXiv

   On the optimization and generalization of overparameterized implicit neural networks

   Tianxiang Gao, and Hongyang Gao

   arXiv preprint arXiv:2209.15562, 2022

   Abs PDF

   Implicit neural networks have become increasingly attractive in the machine learning community since they can achieve competitive performance but use much less computational resources. Recently, a line of theoretical works established the global convergences for first-order methods such as gradient descent if the implicit networks are over-parameterized. However, as they train all layers together, their analyses are equivalent to only studying the evolution of the output layer. It is unclear how the implicit layer contributes to the training. Thus, in this paper, we restrict ourselves to only training the implicit layer. We show that global convergence is guaranteed, even if only the implicit layer is trained. On the other hand, the theoretical understanding of when and how the training performance of an implicit neural network can be generalized to unseen data is still under-explored. Although this problem has been studied in standard feed-forward networks, the case of implicit neural networks is still intriguing since implicit networks theoretically have infinitely many layers. Therefore, this paper investigates the generalization error for implicit neural networks. Specifically, we study the generalization of an implicit network activated by the ReLU function over random initialization. We provide a generalization bound that is initialization sensitive. As a result, we show that gradient flow with proper random initialization can train a sufficient over-parameterized implicit network to achieve arbitrarily small generalization errors.

2020

1. arXiv

   Randomized bregman coordinate descent methods for non-lipschitz optimization

   Tianxiang Gao, Songtao Lu, Jia Liu, and 1 more author

   arXiv preprint arXiv:2001.05202, 2020

   PDF

2018

1. AAAI

   DID: Distributed Incremental Block Coordinate Descent for Nonnegative Matrix Factorization

   Tianxiang Gao, and Chris Chu

   In the 32th AAAI Conference on Artificial Intelligence (AAAI), 2018

   Abs PDF

   Nonnegative matrix factorization (NMF) has attracted much attention in the last decade as a dimension reduction method in many applications. Due to the explosion in the size of data, naturally the samples are collected and stored distributively in local computational nodes. Thus, there is a growing need to develop algorithms in a distributed memory architecture. We propose a novel distributed algorithm, called distributed incremental block coordinate descent (DID), to solve the problem. By adapting the block coordinate descent framework, closed-form update rules are obtained in DID. Moreover, DID performs updates incrementally based on the most recently updated residual matrix. As a result, only one communication step per iteration is required. The correctness, efficiency, and scalability of the proposed algorithm are verified in a series of numerical experiments.

2016

1. GlobalSIP

   Minimum-volume-regularized weighted symmetric nonnegative matrix factorization for clustering

   Tianxiang Gao, Sigurdur Olofsson, and Songtao Lu

   In 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP), 2016

   Abs PDF

   In recent years, nonnegative matrix factorization (NMF) attracts much attention in machine learning and signal processing fields due to its interpretability of data in a low dimensional subspace. For clustering problems, symmetric nonnegative matrix factorization (SNMF) as an extension of NMF factorizes the similarity matrix of data points directly and outperforms NMF when dealing with nonlinear data structure. However, the clustering results of SNMF is very sensitive to noisy data. In this paper, we propose a minimum-volume-regularized weighted SNMF (MV-WSNMF) based on the relationship between robust NMF and SNMF. The proposed MV-WSNMF can approximate the similarity matrices flexibly such that the resulting performance is more robust against noise. A computationally efficient algorithm is also proposed with convergence guarantee. The numerical simulation results show the improvement of the proposed algorithm with respective to clustering accuracy in comparison with the state-of-the-art algorithms.

2015

1. Thesis

   Hybrid Classification Approach of SMOTE and Instance Selection for Imbalanced Datasets

   Tianxiang Gao

   Iowa State University, 2015

   Master’s Thesis

   Abs PDF

   The research area of imbalanced dataset has been attracted increasing attention from both academic and industrial areas, because it poses a serious issues for so many supervised learning problems. Since the number of majority class dominates the number of minority class are from minority class, if training dataset includes all data in order to fit a classic classifier, the classifier tends to classify all data to majority class by ignoring minority data as noise. Thus, it is very significant to select appropriate training dataset in the prepossessing stage for classification of imbalanced dataset. We propose an combination approach of SMOTE (Synthetic Minority Over-sampling Technique) and instance selection approaches. The numeric results show that the proposed combination approach can help classifiers to achieve better performance.