Sentiment Data Analysis
Started December 2025
The effect of the invention of the internet cannot be understated. Its effects on language, politics and society is significant beyond reasonable doubt. In this project, I aim to hypothesise, quantify and analyse language used on the internet, and attempt to understand the relationship between language and the events that are occurring in the world in both short and long timeframes.
Update journal
AI-aided analytics
As per the last entry, I could have either begun analysing the visualisation that I had had, or improve my script with the help of Claude AI. I decided to go with the latter - I wrote a thorough prompt to Claude, outlining what I wanted (and what could be improved) for each step of this project form the import and cleaning of data to the visualisation and further analytics methods. Afterwards, Claude provided its changes, which look good at the outset but I am yet to understand what it has produced.
One particular idea I had was understanding outliers in the data. There are two ways to find these outliers: firstly, I could have eyeballed the data and made human judgements on what data points I wanted to call outliers (ineffective; extremely prone to human error and bias). Alternatively, I could find the standard deviation of each point, at that point in time, to quantify outliers instead of eyeballing it; this was ultimately what I explicitly suggested Claude to help me out with.
Next step: Scrutinise the code that Claude has produced, and comment the code accordingly (for myself) to ensure that I understand what was written and produced.
Initialisation and data accessibility
I’d finally picked this project back up, after working on other ones in the last few months. I had always wanted to do some sort of data analysis outside of work, but the problem was always accessibility of data.
In my corporate role, I became obsessed with what data analytics could reveal, and using trends to guide insights and decision making. But the one thing that was different from the corporate world and leisure data analytics is that in corporations, the data already exists and might already be actively managed by a data team, making the data easily accessible sans any direct retrieval or major cleaning. Outside of the workplace, I had to find the data myself.
Last year, I had tried to find data that I could use for analysis, without much success. For example, I tried to have a look at the free datasets that were available on Kaggle. I don’t particularly remember the datasets being interesting enough - as I say this, I realise this statement might be more reflective of a lack of hypotheses or ideas for the data that was available on Kaggle. In any case, I dropped the endeavour quite quickly.
Until finally, I revisited this project again, and found that I could extract Reddit data. From there, ideas flowed in: how has language on Reddit changed; might this be sufficiently representative of the internet; what were people saying around the time of significant political events? The availability of Reddit data as well as these questions lit a fire back into me to pursue this project.
I had actually spent a significant amount of time earlier in the year learning the ropes about Reddit data, and it came with its own problems before I could even get to analysis. For example, after learning how to torrent and download the compressed data, I then had to learn about decompression algorithms and associated syntax. Since the file was large (relative to my laptop’s capabilities), I ran into several processing problems, and had to find solutions to decompress and analyse on a smaller scale to debug before I would run analysis on the whole dataset. And to top it all off, there were (as always) data quality issues that were causing my script to throw errors that I had to troubleshoot.
But after all was said and done, I finally produced a visualisation:
Next step: There are a few things I can do at this point - I could perform some high-level analysis on this data and publish it, or alternatively I could bring in Claude to improve my script and develop a more thorough analytics program.