Posts by silentwater

measuring things, rabbit holes and Demming

Great article! Thanks!

A few semi-random thoughts this triggered for me:

From 1930s to 1980s Edwards Demming helped make statistical process control mainstream and in the process made huge contribution to the World War II effort and rebuilding Japan after. In my opinion, successful business intelligence, data warehouse, big data, data science, .... are built on ideas he pioneered 60 years ago. Demming is often mis-quoted as saying:

"You can't manage what you can' measure."

When he actually believed that:

"The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them."

So arguably, big data is just a rabbit hole for most businesses to try and know the unknowable ('how do customers feel about my product?')

But... rabbit holes can be mind expanding (just ask Alice...) and maybe help make tangible intuition and the unknowable (or be a road to ruin and madness). Steve Jobs comes to mind ("have you ever taken acid?" interview question.)

The view from my work (similar to yours I think), Big Data (Map Reduce et. al.) is a cheaper and more scalable way to measure process, and that is a useful thing for complex processes problems (biology, large manufacturing, large supply chain) and large volume analytics (power generation, telcos). Agree that for smaller businesses it is a rabbit hole for now. Later they will benefit from resulting reduction in the price for data manipulation and management tools.

Cheers