Question

How can you ensure that you don’t analyse something that ends up producing meaningless results?

Answer 1

The reality is that you can’t be sure you’re not analyzing something that will end up being meaningless.

In fact, it is important that you do analyze things that end up being meaningless. That is how you determine whether or not they are relevant to your problem. Determining that a certain variable is not relevant to your problem is sometimes just as valuable as knowing what is relevant, particularly when that determination is counter-intuitive.

Suppose you are selling an expensive consumer product, for example, a flagship smartphone that costs $900. You want to know what best predicts a customers likelihood of purchasing. Your conclusions might inform advertising content, retail partnerships, ad placement & targeting, etc.

Given that your product costs $900, it is very likely that people within your organization have already made assumptions about their target customer. They might assume that he is higher income, has a higher credit score, lives in certain affluent areas or major metropolitan cities, and so on. [They probably also assume he is a he.]

If you spend a bunch of time analyzing these factors and find little evidence that they are meaningful predictors of purchasing, that is a huge a win. You have stumbled upon a counter-intuitive conclusion that contradicts assumptions that people within the organization may already be acting upon.

You aren’t doing analysis in a vacuum…or at least you shouldn’t be. You are always doing analysis within a particular context. Determining what isn’t relevant, and why, within your particular context is a critical to framing the question correctly.

I would also point out that exploratory data analysis doesn’t help you avoid analyzing things that are meaningless. Exploratory data analysis is analysis. Some of the other answers seem to imply that exploratory data analysis is like an appetizer that comes before the main course (presumably modeling, hypothesis testing, etc). However, in many real-world scenarios exploring the data and using the understanding you gain to frame the problem correctly will be the most important part of the analysis.