01 / THE QUESTION
Can early interactions predict a thread’s later trajectory?
Online mental health forums contain long-running conversations, with the original poster and community members returning over time. I investigated whether early linguistic and interaction features could predict a later trajectory labelled Improved, Stable or Worsened.
The project used Beyond Blue forum data: 4,735 original posts and 306,334 comments across nine categories. My focus was the design and evaluation of an NLP research pipeline, with potential moderation use as the motivation.
The finished work was a research study. It did not establish a clinically validated prediction system.
02 / THE ENGINEERING
Making the data usable came first.
The first substantial problem was relational. Post IDs restarted within each forum category, so joining all files on a post ID caused rows to multiply. I constructed composite keys from the category and post identifier before joining the records.
- Clean and reconstructRemoved HTML and formatting artefacts, normalised timestamps and rebuilt chronological threads using category-based composite IDs.
- Separate early interactions from later outcomesUsed the first five comments for predictors. Later comments from the original author formed the basis for the target label. The final modelling set contained 4,428 eligible threads.
- Build behavioural and semantic featuresExtracted five features covering pronoun use, engagement velocity, sentiment volatility and sentiment slope, alongside 150 TF-IDF features. Fitted vectorisers and scalers on training data only.
- Make evaluation repeatableUsed a stratified 80/20 split and a shared evaluation function returning macro-F1, balanced accuracy, per-class metrics and confusion matrices.
Tools used
Python, pandas, NumPy, scikit-learn, NLTK, Hugging Face Transformers, Matplotlib and Jupyter notebooks.
03 / COMPARING MODELS
Overall accuracy hid the difficult cases.
Across the two project phases, I explored Logistic Regression, Random Forest, Linear SVM, XGBoost and DistilBERT-based pipelines. The final comparison used 886 held-out threads.
Random Forest reached 66.9% accuracy on the original automated labels, but recalled only 1% of Stable threads and 12% of Worsened threads. Macro-F1 and minority-class recall made that weakness visible.
Source: final project report, Table 3. Test set: 886 threads. All four rows use the original VADER labels. Bars use a common 0–0.50 scale. These scores measure agreement with automated labels, not clinical recovery.
These results made label validation the next priority. A classifier could reproduce the labeller’s decisions while still failing to capture the change a human reader saw in the conversation.
04 / VALIDATING THE LABELS
The target needed its own test.
Early versions scored the whole conversation, allowing supportive replies from other users to inflate the Improved class. I isolated the original author’s later comments and applied asymmetric VADER thresholds. The proportion labelled Improved fell from approximately 92% to 68%.
I then built an HTML annotation tool that hid the automated label and manually annotated 60 threads. I measured human–VADER agreement using Cohen’s kappa, with 2,000 bootstrap resamples for a 95% confidence interval.
95% confidence interval: 0.01–0.37
The agreement was weak. Even after separating the author’s replies, sentiment polarity remained an unreliable stand-in for their trajectory.
I compared five alternatives: threshold-tuned VADER, Twitter-RoBERTa, Flan-T5, BART-MNLI and a neural majority vote. None provided convincing evidence of a reliable solution on this small annotated set.
One useful correction came from the evaluation itself: tuned VADER appeared stronger when tested on the same data used to choose its thresholds. Evaluating out-of-fold predictions reduced its kappa from 0.242 to 0.064.
05 / WHAT I LEARNED
A useful result can be finding the limit.
The project strengthened my work in data engineering, NLP feature design and model evaluation. It also changed how I approach an ML task: I want evidence that the target variable reflects the thing we actually care about before treating a better model score as progress.
- Validate the measurement. The labels are part of the system and need to be evaluated.
- Look past the headline metric. Class-sensitive measures exposed errors that accuracy concealed.
- Keep tuning and evaluation separate. Out-of-fold testing changed the conclusion of the threshold experiment.
- Be precise about the limits. A single platform, retained duplicate records and a 60-thread, single-annotator validation set constrain the findings.
The next research step would be a larger, multi-annotator dataset with clear definitions and measured agreement, before making claims about practical moderation use.