Background and objectives: Cognitive bias theories posit that generalized anxiety disorder (GAD) and social anxiety disorder (SAD) are entwined with attention bias toward threats, commonly indexed by faster response time (RT) on threat-congruent (vs. threat-incongruent) trials on the visual dot probe. Moreover, although smartphone ecological momentary assessment (EMA) of the visual dot probe has been developed, their psychometric properties are understudied. This study thus aimed to assess the reliability of 8 smartphone-delivered visual dot probe attention bias and related indices in persons with and without GAD and SAD. Methods: Community-dwelling adults (n = 819; GAD: 64%; SAD: 49%; Mixed GAD and SAD: 37%; Non-GAD/SAD Controls: 24%) completed a five-trial smartphone-delivered visual dot probe for a median of 60 trials (12 sessions x 5 trials/session) and an average of 100 trials (20 sessions x 5 trials/session). Results: As hypothesized, Global Attention Bias Index, Disengagement Effect, and Facilitation Bias had low-reliability estimates. However, retest-reliability and internal reliability were good for Trial-Level Bias Scores (TLBS) (Bias Toward Treat: intra-class correlation coefficients (ICCs) = 0.626–0.644; split-half r = 0.640–0.670; Attention Bias Variability: ICCs = 0.507–0.567; split-half r = 0.520–0.580) and (In)congruent RTs. Poor retest-reliability and internal reliability estimates were consistently observed for all traditional attention bias and related indices but not TLBS. Limitations: Our visual dot probe EMA should have administered ≥320 trials to match best-practice guidelines based on similar laboratory studies. Conclusions: Future research should strive to examine attention bias paradigms beyond the dot-probe task that evidenced meaningful test-retest reliability properties in laboratory and real-world naturalistic settings.