“So are you saying the program did nothing?” That is often the question behind a methods discussion. A community project reports that participants felt more confident after six months. A researcher asks whether the project caused the change. The project team hears doubt about the work itself.
That is not what the question means. Causal caution is not a sneer. It is the difference between “something changed” and “we know why it changed.” Civil-society work deserves that distinction because overclaiming can backfire, and underclaiming can hide real learning.
Myth: Correlation Means Two Things Are Unrelated
Reality: Correlation means two things vary together. It does not mean the pattern is meaningless.
If people who attend a neighbourhood language cafe also report stronger local belonging, that correlation is worth noticing. It may suggest the cafe is helping. It may suggest that people who already feel ready to connect are more likely to attend. It may suggest that both attendance and belonging are shaped by another factor, such as childcare, health, housing stability, or having a friend who invited them.
The German question “Was ist der Unterschied zwischen Korrelation und Kausalität?” can be answered plainly: correlation describes an observed association; causality claims that one thing produces a change in another. The first can be measured with many ordinary datasets. The second needs a stronger design or a more careful argument.
Correlation is a clue. It is not a verdict.
Myth: If We See Improvement After a Program, the Program Caused It
Reality: Before-and-after change is useful, but it is not automatically causal.
Consider a youth mentoring program where participants report clearer education goals at the end than at the beginning. That is valuable information. It may show that the program is reaching the intended issue. It may also reflect the school year, exam timing, family conversations, maturation, or the fact that young people who stayed until the final session were already more motivated than those who left.
The missing question is the counterfactual: what would likely have happened without the program? We can never observe the same person both receiving and not receiving the same support at the same time. Causal research is the craft of building a fair comparison to approximate that missing world.
For small nonprofits, this does not mean every evaluation must become a laboratory study. It means the claim should fit the evidence. “Participants reported improvement during the program” is a different claim from “the program caused the improvement.” The first may be enough for learning. The second needs more support.
Myth: Confounding Is Just a Technical Excuse
Reality: Confounding is one of the most common reasons public claims go wrong.
A confounder is a third factor that influences both the supposed cause and the outcome. Suppose municipalities with more civic associations also have higher trust in local government. Associations might build trust. But wealthier districts may have more stable housing, more time for volunteering, better-resourced schools, and less administrative stress. Those conditions could support both associations and trust.
In health access, people who use preventive services may later have better outcomes. The services may help. But users may also have better insurance navigation, flexible work, German-language confidence, or previous good experiences with doctors. Those factors matter.
Confounding does not prove the original claim false. It asks whether the comparison is fair. A good study tries to measure and adjust for plausible confounders, or uses a design that reduces their influence. A weak public argument pretends they do not exist.
Myth: Reverse Causality Is Rare
Reality: Sometimes the arrow points the other way.
Reverse causality means the outcome may affect the supposed cause. If people with stronger belonging are more likely to volunteer, then volunteering may not be the original source of belonging. If residents who trust institutions are more likely to attend consultations, then consultation attendance alone does not prove the consultation built trust.
This issue appears often in civil-society settings because participation is rarely random. People choose into programs, groups, meetings, and surveys for reasons connected to the outcomes we care about. Confidence affects attendance. Time affects attendance. Previous harm affects attendance. Hope affects attendance.
Longitudinal data, including panel data where the same people are followed over time, can help with timing. If a change in participation comes before a change in belonging, the causal story becomes more plausible. It is still not automatically proven, but the arrow is less mysterious.
Myth: Selection Effects Mean Program Data Is Useless
Reality: Selection effects limit causal claims, but they can teach a great deal.
Selection effects occur when the people who receive an intervention differ from those who do not. A tenants’ rights workshop may attract people who already have the energy to seek help. A digital safety guide may be read by people with better internet access. A community survey may be completed by residents who are angry, engaged, or comfortable with the language used.
That does not make the data worthless. It tells the organization who it is reaching and who may be missing. It can reveal barriers: shift schedules, inaccessible rooms, lack of childcare, fear of authorities, or formats that assume high literacy. It can improve practice even when it cannot prove impact.
The mistake is to treat participants as if they represent everyone affected by the issue. A careful report says, “among those reached,” “among respondents,” or “among participants who completed the follow-up.” Those phrases are not weakness. They are accuracy.
Myth: Only Randomized Experiments Count
Reality: Randomized experiments are powerful for some questions and inappropriate for others.
In a randomized experiment, eligible participants are assigned by chance to receive an intervention or not, or to receive different versions. Randomization can create groups that are similar on average, which makes causal claims stronger. It is often useful for testing information letters, appointment reminders, application prompts, or program variants where random assignment is ethical and practical.
But many civil-society questions cannot be randomized. We cannot randomly expose people to discrimination, eviction pressure, inaccessible transport, or harmful school climates. We may not want to deny a scarce support service purely for research if there is another fair allocation method. Community trust can also be damaged if people experience evaluation as experimentation on their lives.
Good causal reasoning therefore includes ethics. The strongest design is not always the most appropriate design.
Myth: Quasi-Experiments Are Just Second Best
Reality: Quasi-experiments can be very useful when real-world rules create comparison opportunities.
A quasi-experiment uses a policy change, threshold, rollout, waiting list, eligibility rule, or timing difference to compare groups that are more similar than a simple participant/nonparticipant split. For example, if a municipality introduces an advice service in some districts before others because of administrative rollout, researchers may compare changes across districts while checking whether they were already on different paths.
Other designs compare people just above and below an eligibility threshold, or examine whether trends changed after a policy started. Each design has assumptions. A threshold design depends on people near the threshold being genuinely comparable. A rollout comparison depends on early and later areas not differing in ways that also affect outcomes.
Quasi-experiments do not remove judgement. They make the judgement visible.
Myth: If Studies Disagree, Someone Must Be Manipulating the Evidence
Reality: Studies can differ because they ask different questions, measure different outcomes, cover different groups, or observe different time periods.
One study may ask whether a mentoring program improves school attendance. Another may ask whether it improves self-confidence. A third may examine who remains in the program long enough to benefit. These are not identical findings competing for one crown.
Context also matters. A housing advice model may work differently in a tight rental market than in a less pressured one. A participation format may build trust in a municipality with a credible feedback loop and fail where consultation is symbolic. Civic Futures Lab covers campaign and participation evaluation from the organizing side; Institute for Social Insight focuses on how the evidence claim should be framed.
Methods matter too. A cross-sectional survey can map association. A panel study can show sequence. A randomized or quasi-experimental design can support stronger causal inference. Interviews can explain mechanisms and unintended effects. Different methods may appear to disagree because they illuminate different parts of the same process.
Myth: Causal Language Is Either Allowed or Forbidden
Reality: Causal claims come in strengths.
Weak evidence can support cautious language: “participants reported,” “the pattern is consistent with,” “the data suggests,” “one plausible explanation is.” Stronger designs may support stronger language: “the program increased,” “the policy reduced,” or “the intervention caused.” The wording should rise only as high as the design permits.
For a nonprofit report, this discipline is practical. Funders, journalists, and public institutions may prefer the clean sentence. But if a claim is too strong, critics can dismiss the whole project when they find the method gap. Careful wording protects both the truth and the work.
Myth: Mechanisms Are Optional
Reality: A causal claim is more convincing when we can explain how the change happened.
If a tenant advice program appears to reduce acute housing loss, what is the mechanism? Faster response to letters, better understanding of deadlines, referral to a Mieterverein, support contacting the Jobcenter, reduced panic, or stronger negotiation? If a youth project appears to increase civic participation, is the mechanism knowledge, confidence, transport access, peer invitation, adult recognition, or a concrete role?
Mechanisms help studies travel responsibly. A program that works because staff have deep local trust may not work when copied as a leaflet. A campaign tactic that works because officials fear public embarrassment may fail in a different institution. Knowing the mechanism prevents the false lesson.
What We Can Say
Research can show that a program caused an outcome when the design creates a credible comparison, the timing fits the causal story, plausible confounders are addressed, the outcome is measured well, and the mechanism makes sense. That is a high bar, but it is not unreachable.
Research can also be useful below that bar. It can show reach, participant experience, implementation problems, early signals, unintended effects, and plausible pathways. Those are not consolation prizes. They are often exactly what a civil-society team needs to improve.
The practical rule is proportion. Use correlation to ask better questions. Use evaluation to learn before proving. Use causal language when the evidence can carry it. When studies differ, inspect the design, population, outcome, time period, and context before choosing the finding that sounds most convenient.