research | ARIA Research Lab at NJIT

The Algorithms and Architectures for Reasoning and Intelligent Automation (ARIA) Lab at NJIT develops learning-based methods for reasoning and decision-making in complex, structured domains. Our research lies at the intersection of machine learning, probabilistic modeling, symbolic reasoning, and mathematical optimization, with the goal of building AI systems that are reliable, interpretable, and scalable.

Our core research agenda focuses on neuro-symbolic reasoning and probabilistic inference, including both direct neural approximation of inference problems and neural methods that augment classical solvers, as well as neural combinatorial optimization for structured decision-making. Building on these foundations, we study structured and multimodal intelligence, including procedural video understanding and human-guided vision-language systems, and develop methods for AI for scientific discovery, particularly in computational biology.

🤝 We welcome collaborations with academic and industry researchers working in these or closely related areas. If you are interested in exploring a potential collaboration, please contact Dr. Arya.

Research directions

Neuro-Symbolic Reasoning and Probabilistic Inference
Neural Combinatorial Optimization
Structured and Multimodal Intelligence
- Structured Video Understanding and Activity Reasoning
- Human-Guided Vision-Language and Multimodal AI
AI for Scientific Discovery
- Computational Biology

Neuro-Symbolic Reasoning and Probabilistic Inference

We develop learning-based and optimization-based methods for reasoning under uncertainty in structured probabilistic models. Our research combines neural networks with probabilistic inference, symbolic structure, classical algorithms, and mathematical optimization to improve the efficiency and scalability of challenging inference tasks. Rather than committing to a single computational paradigm, we study how learned and algorithmic components can be combined at different levels of the inference pipeline.

Our work currently spans three complementary directions:

Neural Approximation: Neural networks directly learn computationally expensive inference mappings, producing high-quality solutions in one or a few forward passes, optionally followed by inference-time optimization.
Neural Augmentation: Learned components operate within or alongside classical inference algorithms, providing warm starts, conditioning strategies, branching policies, node-selection heuristics, or local-search guidance while retaining the underlying solver framework.
Optimization-Based Structured Inference: Optimization techniques, including integer linear programming and local search, are used to reason directly over structured probabilistic dependency representations.

Neural Approximation of Probabilistic Inference

Neural approximation treats probabilistic inference itself as a learnable mapping. Given a probabilistic model and evidence, a neural network predicts high-quality solutions to inference queries in one or a few forward passes. These predictions can also be refined through inference-time or test-time self-supervised optimization when additional accuracy is required.

SINE: Scalable MPE Inference for Probabilistic Graphical Models using Advanced Neural Embeddings (AISTATS 2025)
- Learns structural and parameter-aware embeddings of probabilistic graphical models together with advanced discretization schemes to predict near-optimal Most Probable Explanation (MPE) assignments in real time.
A Neural Network Approach for Efficiently Answering Most Probable Explanation Queries in Probabilistic Models (NeurIPS 2024 Spotlight; UAI TPM 2024 Best Paper) Best Paper Award Spotlight (Top 2%) Certificate
- Introduces the ITSELF engine: distills MPE queries for a probabilistic model into a neural network approximator and refines predicted configurations through inference-time self-supervised optimization for fast, high-accuracy query answering (demonstrated in the interactive benchmark below: 3.4× speedup with iterative test-time refinement).
Neural Network Approximators for Marginal MAP in Probabilistic Circuits (AAAI 2024 Oral) Oral Presentation (Top 3%)
- Introduces SSMP: solves challenging marginal MAP queries in probabilistic circuits by training neural network approximators over a continuous multilinear relaxation, enabling fast linear-time inference during evaluation (demonstrated in the interactive benchmark below: sub-10 µs inference, >273,000× speedup).

Interactive Demo Neural Inference Latency & Speedup Benchmark (SSMP, ITSELF vs. Baseline) ▶

Interactive Benchmark: Neural Inference Latency & Speedup

Empirical latency and throughput comparison of our methods (SSMP, ITSELF) against traditional combinatorial baselines.

AAAI Oral NeurIPS Spotlight Interactive Demo

Neural inference latency and speedup benchmark comparison: SSMP vs ITSELF vs Baseline

Benchmark Visualization: One Left→Right traverse represents 1 inference query. In the time the classical baseline takes to complete 1 inference query (2.703 s/inf), ITSELF completes an iterative test-time optimization pass (784 ms/inf, 3.4× faster, 3 traverses), while SSMP processes over 273,000 queries via amortized continuous relaxation (9.87 µs/inf, >273,000× faster, 273,860 traverses).

 SSMP AAAI Oral 
×273,860.2
 1 inference: 9.870 µs
 Throughput: 101,317.12 /s
 Traverses: 273,860 one-way (136,930 round-trips) 

 ITSELF NeurIPS Spotlight 
×3.4
 1 inference: 784.000 ms
 Throughput: 1.28 /s
 Traverses: 3 one-way (1 round-trip) 

 Branch and Bound Exact Solver 
×1.0
 1 inference: 2.703 s
 Throughput: 0.37 /s
 Traverses: 1 completed in window 

Interactive Workload Simulator

SSMP Execution Time

98.7 ms

ITSELF Execution Time

2.18 hours

Baseline Execution Time

7.51 hours

Key Takeaway: For 10,000 queries, SSMP finishes in 98.7 ms, whereas the traditional baseline stalls for 7.51 hours (over 273,000× speedup), unlocking real-time probabilistic reasoning.

Learning to Solve the Constrained Most Probable Explanation Task in Probabilistic Graphical Models (AISTATS 2024)
- Develops a self-supervised neural framework for probabilistic inference under explicit constraints, jointly optimizing solution quality and constraint satisfaction through specialized loss formulations.

Neural-Augmented Classical Probabilistic Inference

Learning to Condition (L2C) heuristic architecture

Neural augmentation preserves the structure of classical inference algorithms while introducing learned components that improve their computational behavior. Neural models can provide warm starts, conditioning decisions, branching and node-selection heuristics, or guidance for local search. This approach uses learning to reduce search and accelerate convergence while retaining the structure, guarantees, or certificates provided by the underlying solver when applicable.

Neural Dual Bounds: Valid-by-Construction JGLP Warm-Starts for MAP and Constrained MAP (NeurIPS 2026 Spotlight) Spotlight (Top 1%)
- Predicts valid-by-construction dual bounds to warm-start Join Graph Linear Programming (JGLP) for MAP and constrained MAP inference in graphical models, guaranteeing bound validity architecturally while accelerating solver convergence and providing rigorous bounding certificates.

Interactive Demo Valid-by-Construction Dual Bounds Simulation (Neural Dual Bounds) ▶

Dual Bounding Guarantees & JGLP Warm-Starts

Illustrative simulation demonstrating how antisymmetric neural messages guarantee 100% valid upper bounds to warm-start Join Graph Linear Programming (JGLP).

NeurIPS Spotlight Illustrative Simulation

Conceptual Bound Validity Simulation Illustrative • Synthetic Samples

Visualization Note: This interactive track is an illustrative conceptual simulation designed to demonstrate architectural bounding principles. The sample points generated below are dynamically simulated to contrast guarantees versus unconstrained heuristics: In branch-and-bound solvers, any predicted bound that falls below the true optimum (MAP*) prunes optimal branches and corrupts exactness. Antisymmetric messages architecturally guarantee every predicted bound lies above the optimum (100% valid upper bound), whereas unconstrained neural predictors are not guaranteed to satisfy the bounding condition and can violate it.

Antisymmetric Messages (Neural Dual Bounds) – Valid by Construction 25 of 25 simulated samples valid (100%)

Unconstrained Predictors – Simulated Heuristic Violations 17 of 25 simulated samples valid (68%)

True Optimum (MAP*) Valid Upper Bound Invalid (Below Optimum)

 Neural Dual Bounds NeurIPS Spotlight 
0 Violations
 Validity: 100% by construction
 Dual Gap: Up to 106× reduction in 100 iters
 Solvers: JGLP & Constrained MAP 

 Standard Exact Solvers Baseline 
×1.0
 Scaling: Exponential in treewidth w
 Search: Full combinatorial tree expansion
 Initialization: Zero or uniform warm-starts 

Learning to Condition: A Neural Heuristic for Scalable MPE Inference (NeurIPS 2025; Code)
- Learns a neural conditioning policy from solver search traces that serves both as a variable-conditioning strategy before exact inference and as a branching and node-selection heuristic within branch-and-bound, substantially reducing search spaces while maintaining solution quality.

Interactive Demo Neural Conditioning Mechanism Walkthrough (Learning to Condition – L2C) ▶

Learning to Condition (L2C): Neural Conditioning Mechanism

Interactive walkthrough demonstrating how L2C learns to select variable-value assignments based on optimality confidence and inference simplification to accelerate downstream MPE solvers.

NeurIPS 2025 Interactive Walkthrough

Stage 1 of 5: Original MPE Problem

Solving MPE requires finding the joint assignment maximizing P(Q=q, E=e) over all query variables Q in the graphical model.

Query Variable (Q) Observed Evidence (E) Conditioning Set (Q_d = q_d) Residual Query Variable (Q₀) Active Primal Dependency Restricted Incident Edge

 Learning to Condition (L2C) NeurIPS 2025 
 Conditioning: Learned variable-value selection
 Training signal: Solver runtime, search nodes, and solution quality
 Integration: Pre-solver conditioning or internal B&B guidance
 Evaluated with: SCIP and AOBB 

 Standard Exact Solvers Baseline 
 Methods: SCIP / AND-OR Branch-and-Bound
 Goal: Exact or anytime MPE inference
 Challenge: Search effort grows rapidly on difficult structured instances
 Benefit of conditioning: Smaller and less coupled residual problem 

BEACON: Learning to Guide Local Search for MPE Inference in Probabilistic Graphical Models (ArXiv)
- Amortizes repeated MPE inference in fixed-structure graphical models using an attention-based architecture that scores local-search moves according to estimated Hamming-distance reduction, guiding neighbor selection to improve convergence and solution quality in high-treewidth models.

Optimization-Based Structured Inference

Illustrating improvements from our new inference schemes for DDNs. The DDN learns relationships between labels, and the inference schemes reason over them to accurately identify concealed objects, such as sports ball.

Illustration of Dependency Network for multi-label video classification. The NN takes video clips (frames) as input and outputs the features $e_1,e_2,...,e_n$ (denoted by red colored nodes). These features are then used by the sigmoid output ($\sigma_1$, $\ldots$, $\sigma_n$) of the dependency layer to model the local conditional distributions.

We develop optimization-based structured inference methods that explicitly reason over structured dependencies and combinatorial constraints.

Deep Dependency Networks and Advanced Inference Schemes for Multi-Label Classification (AISTATS 2024)
- Formulates multi-label prediction in images and videos by coupling deep dependency networks with local search and integer linear programming (ILP) inference, capturing complex label dependencies without sacrificing training simplicity.

Interactive Demo Structured MPE Inference vs. Neural Baselines (DDN-ILP) ▸

Deep Dependency Networks & Structured MPE Inference (DDN-ILP)

Interactive demonstration showing how structured dependency reasoning recovers missed labels, followed by empirical benchmark results from AISTATS 2024 (Tables 1 & 2).

AISTATS 2024 Structured Inference

Inference Mechanism: Neural vs. Neurosymbolic

Paper Figure 1 Example

Visual Context (Figure 1): Pitcher throwing on the mound. The sports ball is small, fast-moving, or occluded. For the neural baseline, each label is selected independently using a threshold of τ = 0.50.

Input Image

Figure 1

Concealed/fast-moving sports ball during pitch motion

Label 1

person

Neural score: 0.99 (≥ 0.50)

Predicted (Active)

Label 2

baseball glove

Neural score: 0.99 (≥ 0.50)

Predicted (Active)

Label 3 (Concealed/Small)

sports ball

Neural score: 0.24 (< 0.50)

Missed by baseline

Independent Neural Baseline: For the neural baseline, each label is selected independently using a threshold of τ = 0.50 on individual neural outputs. Because the sports ball is partially occluded in the pitching motion, its neural score (0.24) falls below the threshold. Independent baseline prediction cannot use learned dependencies among labels to recover it.

 Subset Accuracy (SA) Gains Tables 1 & 2 
+23 percentage points SA
 Wetlab Actions: 0.35 → 0.65 (+30 percentage points)
 TACoS Cooking: 0.40 → 0.63 (+23 percentage points)
 PASCAL-VOC: 0.71 → 0.89 (+18 percentage points) 

 DDN-ILP Formulation Our Method 
 Structured MPE via ILP
 Inference Task: Most Probable Explanation (MPE)
 Solver: Integer Linear Programming (ILP)
 Approximation: Piecewise-linear approximation of nonlinear DDN terms 

 Jaccard Index & Error Hamming Loss 
Up to 0.95 JI
 PASCAL-VOC: 0.95 JI (0.006 Hamming loss)
 TACoS Actions: 0.72 JI (0.040 Hamming loss)
 MS-COCO: 0.83 JI (0.88 F1 score) 

Select Evaluated Benchmark Dataset (Tables 1 & 2, AISTATS 2024):

DDN-ILP (Ours, Structured MPE via ILP) 63.0% Subset Accuracy (0.72 JI, 0.040 HL)

DDN-Greedy (Ours, Greedy Local Search) 56.0% Subset Accuracy (0.69 JI, 0.040 HL)

DRF-ILP (Discriminative Random Field) 51.0% Subset Accuracy (0.65 JI, 0.030 HL)

InceptionV3 Feature Extractor Baseline 40.0% Subset Accuracy (0.61 JI, 0.082 HL)

Paper Finding (TACoS Cooking Actions, Table 1): DDN-ILP achieves a +23 percentage point increase in Subset Accuracy (from 0.40 to 0.63) over the InceptionV3 baseline and reduces Hamming error by more than half (0.082 → 0.040). Joint MPE inference reasons over learned dependencies among concurrent actions rather than making label decisions independently.

Neural Combinatorial Optimization

We develop learning-based methods for solving large-scale combinatorial optimization problems over structured domains. Our research combines deep reinforcement learning, representation learning, and classical optimization to learn effective decision policies for problems with large discrete action spaces, complex dependencies, and domain-specific constraints. A particular focus is graph-structured optimization, where learned representations can exploit topology and problem structure to make efficient sequential decisions.

RELINK deep reinforcement learning framework for edge activation

Our current work investigates neural combinatorial optimization for decision-making over complex networks, including problems in which solutions require sequentially selecting or modifying nodes, edges, or other discrete structures. These methods aim to amortize expensive optimization across problem instances by learning policies that capture reusable structural patterns while accommodating operational, privacy, and other application-specific constraints.

RELINK: Edge Activation for Closed Network Influence Maximization via Deep Reinforcement Learning (CIKM 2025 Oral) Oral Presentation
- Formulates edge-level influence maximization in privacy-constrained closed networks (IM-CSN) as a Markov Decision Process, introducing an edge-centric Double DQN framework with SVD node embeddings and edge-aware aggregation trained on true marginal influence gain rewards.
- Evaluated across 20 real-world social networks (including FilmTrust, Email-Eu-core, Wiki-Vote, MUSAE Facebook, Math Overflow, Deezer, and Epinions) using 100,000 Monte Carlo simulations under the Independent Cascade model.
- Consistently outperforms existing methods in over 90% of experimental settings, achieving up to 15% higher influence spread, a 95.0% win rate (456/480) against the specialized IM-CSN baseline PSNA, and superior scalability via GPU-accelerated sequential inference on large-scale graphs.

Interactive Demo RELINK Edge Activation Problem & Benchmark Explorer ▸

RELINK: Edge Activation for Closed-Network Influence Maximization

Conceptual walk-through of the edge-activation problem formulation under budget k, followed by empirical benchmark results from CIKM 2025.

CIKM 2025 Oral Combinatorial Optimization over Graphs

1. What is the Problem? Influence Maximization in Closed Social Networks (IM-CSN)

Conceptual Problem Formulation • Paper Figure 1 (k = 2)

Problem Formulation: In closed social networks, content propagation is restricted across privacy boundaries. Starting from a seed node, the goal is to choose closed candidate edges to activate subject to an edge activation budget k (where k is the maximum number of candidate edges that may be activated) to maximize total influence spread under the Independent Cascade (IC) diffusion model.

Clarification: This example illustrates the sequential optimization problem (Example 1, Figure 1 from the CIKM 2025 paper). It does not represent RELINK’s learned edge-selection policy.

Budget Used (Node a): 0 / 2

Seed Node (a) Influenced Node Inactive Node Candidate Edge Activated Edge Pruned Edge (Budget Exhausted) Network Link

Current Step

Step 0 of 3

Walkthrough progress

Activation Budget

k = 2 edges

Max out-degree limit

Influenced Spread

1 / 10 nodes

Seed active

Budget Status

0 / 2 used

Remaining budget: 2

Step 0: Initial Closed Network & Seed Node
In a closed social network, content propagation is constrained. The process begins with seed node a (red). Its outgoing links (a→b, a→h, a→d, a→c) form the initial set of candidate edges (red arrows). Dormant links deeper in the network remain inactive (dark arrows). The edge activation budget k = 2 allows at most 2 candidate edges to be activated from node a.

2. How Well Does RELINK Solve It? Empirical Benchmark Results from CIKM 2025

Empirical evaluation of RELINK (edge-centric n-step Double DQN) across 20 real-world social networks under the Independent Cascade (IC) model (100,000 Monte Carlo simulations per instance).

 RELINK (DRL Policy) Our Work 
Up to +15% Spread
 Formulation: Edge-centric n-step Double DQN with SVD embeddings
 Training: True marginal influence gain rewards (IC model)
 Win Rate: 456 / 480 wins (95.0%) across 20 real networks 

 PSNA (Huang et al.) IM-CSN Baseline 
95.0% RELINK Win
 Formulation: Practical Subnetwork Augmentation heuristic
 Complexity: Ο(I(|V| + |E|) log |V| + |V|dm log dm)
 Runtime: Deteriorates on large graphs; caught by budget saturation 

 Traditional Baselines Degree & Random 
>96% RELINK Win
 Formulation: Static degree centrality & uniform random selection
 Win Rate: 464 / 480 vs. Degree, 462 / 480 vs. Random & FoF
 Limitation: Static local connectivity metrics overlook downstream influence dynamics 

Empirical Results from CIKM 2025: Baseline Comparisons (Figure 3, 480 Experiments)

RELINK vs. Degree Centrality 96.7% (464 / 480 wins)

RELINK vs. Random Selection 96.2% (462 / 480 wins)

RELINK vs. Friend-of-Friend (FoF) 96.2% (462 / 480 wins)

RELINK vs. PSNA (Huang et al., IM-CSN Baseline) 95.0% (456 / 480 wins)

Paper Finding (Figure 3 Contingency Matrix & Section 6.2): Across 480 total experimental configurations spanning 20 real-world social networks (evaluated with 100,000 Monte Carlo runs under the IC model), RELINK outperforms all competing methods in at least 95% of experiments. In low edge budget regimes (k ≤ 10), Section 6.3 reports that RELINK frequently achieves influence spread improvements exceeding 15% over PSNA.

Structured and Multimodal Intelligence

We develop learning methods for reasoning over complex perceptual and multimodal data by combining high-dimensional representations with explicit structure. Our research studies how temporal dependencies, task structure, relational representations, and human feedback can improve the reliability, interpretability, and adaptability of models operating across visual, linguistic, and multimodal domains. More broadly, we investigate how structured reasoning can bridge low-level perception and higher-level understanding, prediction, and decision-making.

Structured Video Understanding and Activity Reasoning

CaptainCook4D egocentric 4D cooking dataset

Our work in video understanding focuses on modeling the temporal and procedural structure of complex activities. Rather than treating videos as collections of isolated frames or short clips, we study representations that capture multi-step workflows, dependencies among actions, deviations from expected procedures, and the context needed for explanation and prediction. These structured models support tasks such as activity recognition, procedural error detection, temporal localization, explanation, and predictive task guidance.

CaptainCook4D: a dataset for understanding errors in procedural activities (NeurIPS D&B Track 2024; DMLR 2023)
- Introduces a 94.5-hour egocentric 4D dataset of recipe execution containing both normal and errorful trials, with fine-grained annotations supporting error recognition, multi-step temporal localization, and procedure learning.

Interactive Demo CaptainCook4D: Normal vs. Error Execution Explorer ▸

CaptainCook4D: Normal vs. Error Procedural Execution & Taxonomy Explorer

Compare genuine normal executions against recorded procedural errors, and inspect egocentric 4D sensory capture and task graphs.

NeurIPS 2024 D&B Egocentric 4D Benchmark

 Dataset Scale Benchmark 
94.5 Hours
 Recordings: 384 sessions across 10 real kitchens
 Tasks: 24 complex WikiHow recipes (≤30 min)
 Annotations: 5.3K steps, 10K fine-grained actions 

 Error Taxonomy 7 Categories 
Real Deviations
 Categories: Technique, Measurement, Order, Prep, Timing, Temp, Missing
 Protocols: Adherent normal steps vs. induced error deviations
 Ground truth: Annotated step start/end times 

 Sensory Modalities Hardware 
Egocentric 4D
 Dual-view: GoPro Hero 11 (4K @ 30 fps), HoloLens 2 RGB
 Depth & Poses: HoloLens 2 AHAT depth, 3D hand/head poses
 IMU & Spatial: 3-stream IMU, audio, 3D spatial mesh 

1. Select Procedural Demonstration from Benchmark:

Authentic CaptainCook4D Benchmark Recordings

Recipe: Butter Corn Cup Instruction: "Mix the contents of the bowl well"

Error Type: Technique Recorded CaptainCook4D Session

Execution Comparison:

Normal Execution (Adherent Step) Recorded Dataset Session

▶

Expected Execution: Contents of the bowl are mixed thoroughly without spillage.

Error Execution (Procedural Deviation) Technique Error

▶

Recorded Deviation: Participant mixes too aggressively, spilling corn kernels out of the bowl while mixing.

0:00 / --:--

Linked Normal/Error Playback

Recorded CaptainCook4D session: Butter corn cup. First 2 snippets exhibit correct execution without spillage; next 3 exhibit induced corn spillage while mixing.

Recipe: Butter Corn Cup

Instruction: "Mix the contents of the bowl well"

Technique Error

Normal Execution: Contents of the bowl are mixed thoroughly without spillage.

Recorded Deviation: Participant mixes too aggressively, spilling corn kernels out of the bowl while mixing.

Sensory Capture: GoPro Hero 11 (4K @ 30 fps), HoloLens 2 AHAT Depth, 3D Hand/Head Poses, IMU & Audio

Benchmark Tasks: Error Recognition (Supervised & Zero-Shot), Multi-Step Localization (MSL), Procedure Learning

Official Resources: Project Website • NeurIPS 2024 Paper • Downloader • Annotations IRB Approved (UF) • Apache 2.0 License

Explainable Activity Recognition in Videos Using Deep Learning and Tractable Probabilistic Models (ACM TiiS 2023)
- Integrates deep video representations with dynamic cutset networks to construct tractable temporal models that support probabilistic explanation queries while maintaining competitive activity-recognition performance.
Predictive Task Guidance with Artificial Intelligence in Augmented Reality (IEEE VR 2024 Workshop / Poster)
- Investigates structured predictive models for augmented-reality task guidance that anticipate user actions and provide proactive assistance during complex procedural activities.

Human-Guided Vision-Language and Multimodal AI

Vision-language model feedback interface

We study multimodal systems that integrate visual and linguistic representations with structured human feedback. This work investigates how different forms of human guidance, from detailed natural-language feedback to lightweight corrective signals, can improve model reliability, task adaptation, and downstream multimodal understanding. More broadly, we are interested in interactive learning frameworks in which human feedback becomes an explicit component of model reasoning and adaptation.

Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models (ACM TiiS 2026)
- Studies different forms of human-in-the-loop feedback for video understanding, comparing detailed natural-language commentary, word-level corrections, and lower-cost scalar judgments for improving model reliability.

Interactive Mechanism & Benchmark: Human-in-the-Loop Feedback for VLMs ACM TiiS 2026 ▸

VLM Accuracy Across Human-in-the-Loop Feedback Modalities

Empirical findings from ACM TiiS 2026 evaluating video-description verification accuracy across feedback granularities, post-processing strategies, and fine-tuned VLM architectures

ACM TiiS 2026 Multimodal Benchmark

 Peak VLM Accuracy Free-Text 
90.0% Accuracy
 No Feedback Model: 40.0% → 90.0% (+50 percentage points)
 Architecture: LLaVA-NeXT-Video
 Condition: Free-text human feedback 

 Targeted Error Correction LLM Conversion 
81.7% – 84.6%
 Video-LLaVA Peak: 81.7% (+13.3 percentage points over raw 68.4%)
 LLaVA-NeXT-Video: 84.6% accuracy (vs. 83.6% raw)
 Strategy: Human word-level correction + LLM conversion to text 

 Lightweight Feedback Low-Cost 
Up to 80.0%
 Yes/No Binary: 80.0% (LLaVA-NeXT-Video raw)
 Error Selection: 80.0% (LLaVA-NeXT-Video raw)
 No Feedback Model: 40.0% (fine-tuned without corrective human feedback) 

1. Human Feedback Modalities & Semantic Expressiveness (Paper Figure 1):

Increasing Spectrum of Detail & Expressiveness

Evaluated Video Demonstration Paper Figure 1 Video

Participant tying running shoes on stone steps (Paper Figure 1)

Actual Video Activity: The person is sitting on outdoor steps putting on and tying his athletic shoes.

Human Interface: Yes / No Binary 1-Bit Verification

Model-Generated Statement:

The person is cleaning his shoes.

Participant Study Input:

Interface Protocol: Respond by Yes / No.

Semantic Information: Tells the system only that something is wrong. Provides only a correctness judgment without localizing the error or supplying a fix.

Semantic Granularity: 1/4: Binary Judgment (Low Effort, Sparse)

Fine-Tuning Experimental Pipeline Supervised QLoRA Fine-Tuning (Offline Training, Not Test-Time Feedback)

1. Human Feedback

Study interface input

2. Prompt Formatting

Training template (Fig. 5)

3. QLoRA Fine-Tuning

VLM parameter updates

4. Verification Acc.

Yes/No accuracy test

▾ How the feedback is formatted for VLM training (Paper Figure 5)

 Video Description: "The person is cleaning his shoes."
User Feedback: No
 System Question: "Does the above information accurately describe the video? Answer Yes or No."

Important distinction: Participants in the user study interface did NOT submit corrections using arrow notation (cleaning -> wearing); that syntax was introduced during VLM training prompt construction to condition the model.

2. Select Evaluation View (ACM TiiS 2026):

Free-Text (Natural Language Feedback) 90.0% Accuracy

Error Correction – LLM Conversion 84.6% Accuracy

Error Correction – Raw Human Feedback 83.6% Accuracy

Yes/No Binary – Raw Human Feedback 80.0% Accuracy

Error Selection – Raw Human Feedback 80.0% Accuracy

Yes/No Binary – LLM Conversion 77.9% Accuracy

Error Selection – LLM Conversion 75.8% Accuracy

No Feedback Model 40.0% Accuracy

Paper Finding (LLaVA-NeXT-Video): Human feedback substantially improves verification accuracy over the No Feedback Model, surging from 40.0% up to 90.0% (+50 percentage points) with Free-Text feedback. Even lightweight binary feedback elevates model accuracy to 80.0% (+40 percentage points), demonstrating that modern video-language models effectively ground corrective human signals across interaction granularities.

AI for Scientific Discovery

We develop machine learning methods that incorporate scientific structure, prior knowledge, and domain constraints into data-driven models for scientific discovery. Our research focuses on problems where observations are high-dimensional, interactions are structured, and purely black-box learning may fail to capture the mechanisms underlying the data. By combining representation learning, generative modeling, graph-based reasoning, and optimization with domain knowledge, we aim to build models that support both accurate prediction and scientifically meaningful interpretation.

Computational Biology

CoLa-VAE cell-cell communication-aware variational autoencoder

Our work in computational biology develops structured representation-learning methods for complex biological systems, with a current emphasis on single-cell genomics and transcriptomics. We incorporate biological knowledge, including molecular interactions and cell-cell communication structure, directly into deep generative models to learn representations that better reflect cellular organization, heterogeneity, and intercellular relationships.

CoLa-VAE: Cell-Cell Communication-aware Variational Autoencoder with Dynamic Graph Laplacian Constraints (bioRxiv 2026; Code)
- Integrates cell-cell communication structure into a variational autoencoder through dynamic graph Laplacian regularization derived from ligand-receptor interactions, improving learned representations of single-cell transcriptomic data.