This research explored whether negative events experienced at age 1 (AE-1) influence both the levels and the trajectories of cognitive abilities throughout childhood, drawing on integrated datasets from four low- and middle-income nations. The sample comprised cohorts born around 2001–2002, who were tracked over time during 2006–2007 and 2009–2010 via the Young Lives project in Ethiopia, India, Peru, and Vietnam. Cognitive performance among children was assessed through the Peabody Picture Vocabulary Test (PPVT) when they were 5 years old (PPVT-5) and 8 years old (PPVT-8). Additionally, impacts on cognitive growth from ages 5 to 8 (PPVT-Change) were analyzed. The AE-1 index was derived from questionnaire responses collected at age 1. OLS regression models were applied for the analysis. Findings showed that youngsters facing greater hardship exposure in infancy displayed lower cognitive scores at both ages 5 and 8. Cognitive progression between these ages was typically reduced in cases of substantial early hardships. This detrimental link was most evident in India. The outcomes supply valuable insights for policymaking aimed at easing the harmful impacts of early-life hardships through prompt measures.