Description: Asynchronous reinforcement learning speeds up LLM post-training by running rollout and training as parallel engines, but the resulting staleness makes standard PPO unstable. Decoupled PPO restores stability by using a separate proximal policy as the trust-region anchor, yet computing that policy requires an extra forward pass through the model at every training step (~10 seconds for large models). A-3PO removes this cost by recognizing that the proximal policy does not need exact neural-network evaluation - it only needs to lie between the behavior and target policies. We approximate it via a staleness-aware log-linear interpolation, a few lines of element-wise tensor arithmetic on quantities already present in the training loop. This yields an 8,500x speedup in proximal-policy computation and up to 1.8x faster end-to-end training at 8B scale, while provably contracting importance weights toward 1 as staleness grows. On GSM8K, DAPO-Math, AIME24, and MATH500, A-3PO matches or exceeds explicit recomputation - the approximation is not only faster but more stable, clipping the fewest tokens of any method.

Description: MXFP4 promises 4x throughput and 4x memory savings for LLM training on Blackwell/MI350-class accelerators, but naively applying it to RL post-training destroys accuracy (-21.9 pp on dense GSM8K). Prior work treats the quantization error as one monolithic noise term, which cannot explain why the same format hurts dense and MoE models so differently. We prove that MXFP4 error is exactly the sum of three components with disjoint structure - scale bias from the E8M0 power-of-two scale (which corrupts gradients through backward-pass amplification), deadzone truncation from zeroing small values (which corrupts rollouts through forward-pass information loss), and grid noise from rounding to the E2M1 grid (which raises policy entropy like an effective temperature). Two formal properties - exact orthogonality of the deadzone term and scale-invariance of grid noise - imply an irreducible error floor set by the grid itself. Each mechanism gets a matched correction - MBS (macro-block scaling) removes the scale bias, OF (outlier fallback with residual blend alpha=0.5) recovers the deadzone, and AQN (adaptive quantization noise) anneals the residual grid noise. On Qwen2.5-3B the recipe recovers BF16 within 0.7 pp; on Qwen3-30B-A3B-Base MoE it exceeds BF16 by +1.0 pp.

Description: Coordination in traffic signal control is crucial for managing congestion in urban networks. Existing pressure-based control methods focus only on immediate upstream links, leading to suboptimal green time allocation and increased network delays. However, effective signal control inherently requires coordination across a broader spatial scope, as the effect of upstream traffic should influence signal control decisions at downstream intersections, impacting a large area in the traffic network. Although agent communication using neural network-based feature extraction can implicitly enhance spatial awareness, it significantly increases the learning complexity, adding an additional layer of difficulty to the challenging task of control in deep reinforcement learning. To address the issue of learning complexity and myopic traffic pressure definition, our work introduces a novel concept based on Markov chain theory, namely "multi-hop upstream pressure", which generalizes the conventional pressure to account for traffic conditions beyond the immediate upstream links. This farsighted and compact metric informs the deep reinforcement learning agent to preemptively clear the multi-hop upstream queues, guiding the agent to optimize signal timings with a broader spatial awareness. Simulations on synthetic and realistic (Toronto) scenarios demonstrate controllers utilizing multi-hop upstream pressure significantly reduce overall network delay by prioritizing traffic movements based on a broader understanding of upstream congestion.

Description: Perimeter control prevents loss of traffic network capacity caused by congestion in urban areas. The conventional homogeneous perimeter control often fails in urban areas with spatially heterogeneous congestion because such control does not consider location-specific traffic conditions around the perimeter. Our research introduces a multi-hop generalization of traffic pressure that extends the spatial consideration beyond immediate intersections, to modulate inflows more effectively according to actual congestion around each access point of the perimeter. This generalization allows us to adjust how far we can reach downstream links, providing a customizable spatial granularity of metrics that bridges the gap between the overly extensive scope of MFDs and the very limited scope of the traditional traffic pressure metric.

Description: The objective Traffic Perimeter Control is to maintain high traffic efficiency within protected regions by controlling transfer flows among regions. This project explores an innovative model-free perimeter control strategy through deep reinforcement learning to optimize traffic flow rates. Breaking away from traditional model-based methods, which often suffer from inaccuracies due to model bias, this approach leverages a microscopic simulation perspective. It incorporates detailed spatial characteristics and vehicle-level dynamics without relying on network transmission models or macroscopic fundamental diagrams. The results showcase the potential of our deep reinforcement learning method to match, and in some cases surpass, the performance of model-based approaches, highlighting its scalability and generalizability in managing traffic densities efficiently.

Description: This project undertakes a thorough comparison between state-of-the-art graph convolutional neural networks (GCNNs) and the established random forest regression in the realm of traffic prediction. By dissecting the components of GCNN models, including matrix factorization, attention mechanisms, and weight sharing, our study evaluates their impact on traffic prediction accuracy. Utilizing both simulated and real-world traffic data from Toronto and California, the analysis reveals that while GCNNs benefit from these sophisticated components, random forests remain competitive, challenging the notion that GCNNs are the superior method for capturing spatiotemporal traffic patterns. These insights not only highlight the robustness of random forests but also underscore the potential for further advancements in traffic prediction models.

Description: This project extends instance-level 6D pose estimation from RGB images to category level using a Denoising Autoencoder to learn implicit 3D rotation representations. By utilizing synthetic CAD models or 3D point cloud models as category representatives, the Denoising Autoencoder is trained on synthetic 3D views to extract geometry-shared features, yielding a latent representation that is robust to variations in texture, color, illumination, and pose ambiguities due to symmetry. In my master's thesis, I additionally implemented contrastive learning to enforce the similarities between rotation representations to be consistent with the rotation distance. This work paves the way for more generalized and efficient pose estimation in complex visual environments.

Description: In robotic manipulation, the slip detection of objects is crucial. A Visual-Tactile Transformer is designed to enhance robotic grasping by accurately detecting slips through combined visual and tactile data. This method surpasses traditional tactile-based techniques, adeptly handling unaligned and diverse sensory inputs. Tested on multiple datasets, our approach demonstrates superior performance and versatility in slip detection tasks, marking a significant advance in robotic manipulation.

Description: In order to avoid complicated programming difficulties in robot control, we propose an automatic robot learning system which can learn skills from real-world demonstrations by robot. The system utilizes RGB-D camera to record one robot’s demonstrations and then the demonstration data are processed and transferred into robot simulation environment. The policy model is trained entirely in simulation with the advantage of avoiding safety problem which is the key difficulty of real-world training. Then the learned policy is automatically transferred to another robot to reproduce the demonstrated skills. The experiments show that the system could automatically finish entire learning process from recording the robot demonstrations to applying the learned policy to another robot. And with the selected policy learning method, the robot could not only acquire skills but outperform the demonstrator.