Smartphones are capable of passively capturing persons’ social interactions, movement patterns, physiological activation, and physical environment. Nevertheless, little research has examined whether momentary anxiety symptoms can be accurately assessed using these methodologies. In this research, we utilize smartphone sensors and personalized deep learning models to predict future anxiety symptoms among a sample reporting clinical anxiety disorder symptoms. Participants (N = 32) with generalized anxiety disorder and/or social anxiety disorder (based on self-report) installed a smartphone application and completed ecological momentary assessment symptoms assessing their anxiety and avoidance symptoms hourly for the course of one week (T = 2007 assessments). During the same period, the smartphone app collected information about physiological activation (heart rate and heart rate variability), exposure to light, social contact, and GPS location. GPS locations were coded to reveal the type of location and the weather information. Personalized deep learning models using the smartphone sensor data were capable of predicting the majority of total variation in anxiety symptoms (R2 = 0.748) and predicting a large proportion of within-person variation at the hour-by-hour level (mean R2 = 0.385). These results suggest that personalized deep learning models using smartphone sensor data are capable of accurately predicting future anxiety disorder symptom changes.