It seldom comes as a surprise when a fascinating vision of deriving insights starts with tons of bytes of texts and numbers looking like anything but structured to start with. An engagement gets all the more exciting while engaging with a pool of bright data individuals, and a great pleasure to be able to interact with a thoughtful team discussing the subtle intricacies and popular industry challenges and solutions. The tougher the path became, the more exciting the discussions became, often for hours.
If we push a data long enough, it will take a shape or form that we might be looking for it to get to. This is a scenario we would like to stay away from, stay unbiased, and prep the data to the fullest form of its usability. The analysis of the raw data is what can be called the most painful and time-consuming of all. Something that seems a mere non-classifier is an excellent piece when combined with another, or maybe not just one more, a few more. There is no guarantee that the data will learn to talk to us easily, hence we went down the path for weeks, implementing all our concepts and techniques along the way of prepping data. With quite a few tools at our disposal and picking up a few more along the way, things slowly turned to take the form which finally seemed insightful enough to all of us.
Read more: The Virtual Web 3 Paradise »
However, did we force the data to project this insight? Or is the insight a natural outcome of the way we processed the data? Would we still be getting this same answer from our data had we processed it differently? Was our question biased in some way unknown to us even after so much analysis? Are traditional obvious methodologies so imbibed in us that we tend to apply them erroneously where we shouldn’t be? Did we wrap up cleaning the data too early before we learned more? This was one of the main challenges that we faced, leading to iterative cycles of workshops and giving rise to deeper questions in every iteration. Often it seemed so dark without any possibility of light around. What works in one place, will very likely not work in another, even when things look almost identical, we learned this the hard way.
Keep reading: Evernote and how it transformed my digital life »
A situation like this is extremely satisfying and yet challenging after weeks of brainstorming, and enables more learning and nerve-wracking workshops. Hours when anomalies seem valuable, and omitting outliers feels like a sin, feeling the data blend into ourselves all around us as we crave towards the perfect insight with every tougher question presented to us, finally helped arrive at data analysis in front of us rich enough to satisfy most of us. While enough items remain unexplored still, strategic solutions such as these are quite a step in the vast expanse of the data world.
