On Feature Space Reduction and State Space Discretization: A Pipeline for Vision-Based Reinforcement Learning

This thesis investigates feature space reduction and state space discretization in reinforcement learning for vision-based navigation tasks. The focus lies on comparing the performance of Slow Feature Analysis (SFA) and Convolutional Neural Networks (CNN) as feature extractors in the context of reinforcement learning. A Unity-based 3D environment is used to evaluate these two feature extraction methods, which reduce the egocentric input observations of an RL agent in both discrete and continuous action spaces. These experiments investigate whether SFA can be effectively applied in discrete action spaces using Deep Q-Networks (DQN) and in continuous action spaces with Proximal Policy Optimization (PPO). The results show that both SFA and CNNs can reach similar performance in RL tasks. SFA potentially offers advantages in terms of generalizing to different target positions and being more computationally efficient. A drawback of the SFA approaches is the need for pretraining and the absence of an end-to-end trainable alternative comparable to CNNs. In addition, a variation of the SFA-PPO is presented that can interpret allocentric Cartesian environment coordinates and navigate to them using the extracted SFA features. This variation could enable self-localization of an agent in an environment, and due to its small model capacity, it could be beneficial for mobile robotics. Future work could explore integrating SFA into hybrid architectures, reducing the reliance on pretraining, and testing the approach in real-world robotic systems.