In the absence of preconditions, temporal order information or any other information that can help an analyst infer causal direction before-hand, it is necessary to look at the relationship between $X$ and $Y$ from both directions i.e.
\[X \rightarrow Y\]
\[Y \rightarrow X\]
The best illustration of why we need to take care when trying to establish the causal structure is Hempel's "flagpole problem". It states:
So it can be seen that the height of the pole and angle of the sun with the ground has a causal association to the length of the shadow but the the converse is not causal but can be calculated. This is evidence that mere correlations will not suffice in determining the causal structure of the system.
Like any mathematical equation, making one of the variables subject of formula with respect to the two given quantities, one can determine the value of the third, thus failing to detect any assymetries need to define a causal structure.
There is no objective difference between the the two directional associations but the shadow causing the height of the tree is pragmatically nonsensical.
James Woodward uses the idea of independence and initial conditions to determine the causal direction between the two directions.
Images source: Flagpoles, Anyone? Independence, Invariance and the Direction of Causation by James Woodward