Prompt recognition and continuous monitoring of health issues in newborns are vital to enable rapid therapeutic responses, reduce potential complications, improve clinical resource allocation, and provide essential support to families. While established methods that rely on structured tabular information and laboratory markers have proven useful, they remain limited in assessing neonatal morbidity risk. This study presents NeonatalBERT, a pre-trained large language model (LLM) specifically designed to assess the likelihood of neonatal morbidities from clinical documentation. This prognostic analysis examined retrospective primary and external cohorts sourced from two major quaternary academic medical institutions in the United States: Stanford Health Care and Beth Israel Deaconess Medical Center. NeonatalBERT underwent initial pre-training using clinical notes from the primary cohort, followed by separate fine-tuning procedures tailored to each cohort. We compared the model’s performance with other established LLMs, including BioBERT and Bio-ClinicalBERT, as well as standard machine learning techniques and tabular-variable logistic regression models. NeonatalBERT was tested on 19 neonatal morbidities (respiratory distress syndrome, bronchopulmonary dysplasia, pulmonary haemorrhage, pulmonary hypertension, atelectasis, aspiration syndrome, intraventricular haemorrhage, periventricular leukomalacia, neonatal seizures, other CNS disorders, patent ductus arteriosus, cardiovascular instability, sepsis, candidiasis, anaemia, jaundice, necrotising enterocolitis, retinopathy of prematurity, and death) within the primary cohort, and on ten morbidities in the external cohort (respiratory distress syndrome, bronchopulmonary dysplasia, pulmonary haemorrhage, intraventricular haemorrhage, patent ductus arteriosus, sepsis, jaundice, necrotising enterocolitis, retinopathy of prematurity, and death). For every outcome, evaluation metrics included the area under the receiver operating characteristic curve, area under the precision-recall curve (AUPRC), and F1 scores. A total of 32 321 newborns formed the primary cohort, consisting of 27 411 in the primary training set (mean gestational age 38.64 weeks [SD: 2.30]; 13 056 [47.6%] female and 14 355 [52.4%] male) and 4910 in the primary testing set (mean gestational age 38.64 [2.13] weeks; 2336 [47.6%] female and 2574 [52.4%] male). The external cohort comprised 7061 newborns, including 5653 in the external training set (1567 [27.7%] premature and 4086 [72.3%] term births; 2614 [46.2%] female and 3039 [53.8%] male) and 1408 in the external testing set (383 [27.2%] premature and 1025 [72.8%] term births; 624 [44.3%] female and 784 [55.7%] male). In the primary cohort, NeonatalBERT attained a mean AUPRC of 0.291 (95% CI: 0.268–0.314) across the 19 outcomes, compared with 0.238 (0.217–0.259) for Bio-ClinicalBERT, 0.217 (0.197–0.236) for BioBERT, and 0.194 (0.177–0.211) for the conventional tabular model. In the external validation cohort, NeonatalBERT achieved a mean AUPRC of 0.360 (0.328–0.393), exceeding those of all other models (range 0.224–0.333). In summary, the present work demonstrates that a neonatal-focused large language model can harness routinely recorded clinical notes to generate risk estimates for a wide range of neonatal morbidities across two distinct cohorts. These results support adopting automated risk assessment to enable earlier therapeutic action, optimize clinical processes, and strengthen interactions with families. At the same time, they underscore the necessity for careful, responsible deployment, ongoing validation, and appropriate oversight of artificial intelligence tools in neonatal medicine.
