| Attribute family | Levels | Profiles | Power at 3 pp | Type M | Type S |
|---|---|---|---|---|---|
| K = 2 (Need, Control, Reciprocity, Attitude, Sex) | 2 | 18,000 | 98.1% | 1.012 | 1.1e-09 |
| K = 3 (Effort, Identity) | 3 | 18,000 | 90.8% | 1.057 | 8.5e-08 |
| Note: | |||||
| Type M is the exaggeration ratio: how much a statistically significant estimate overstates the true effect. Type S is the probability that a significant estimate has the wrong sign. Both are negligible here, which is what adequate power looks like. |
1 Power analysis re-estimation for conjoint
1.1 Scope
This document asks one question—is a sample of 1,500 respondents large enough to test the hypotheses of the distributive conjoint?—and answers it using the analytic approach of Schuessler and Freitag (2020), implemented in their cjpowR package. It is the companion piece to the conjoint design chapter, which specifies the attributes, the randomization and the estimators; here we take that design as given and ask only what it can detect.
The document proceeds in five steps. It opens with the three properties of a conjoint that govern power—why the number of attributes is irrelevant, why what counts is profile evaluations rather than respondents, and how the target effect size was chosen—since each of them is needed to read the numbers that follow. It then evaluates the main effects: the power the planned sample delivers, how that changes if tasks per respondent are cut, and the smallest effect the design can resolve. The third step turns to interactions and heterogeneity, where the requirements are several times larger, and measures how sensitive they are to the distribution of the moderator. A fourth, shorter step reports the Type M error, which asks not whether effects would be detected but whether the ones declared significant would be inflated. The document closes with a design check that sets sample size aside and asks a different question—how efficiently the randomization itself extracts information from each observation—verified against two established packages, followed by a summary table of what is and is not covered at the planned sample.
ImportantHow to read every number in this document
All figures below are conservative upper bounds, not point estimates. Three assumptions drive that, and they do not all push the same way.
- Binary outcome (violated, in our favour). The formula was derived for forced-choice conjoints, where the outcome is a binary pick and its variance is therefore Bernoulli. Our outcome is a continuous share of a fixed fund. A binary outcome carries the maximum variance a bounded 0–100 variable with mean 50 can have; any realistic spread of allocated shares is lower, and lower outcome variance means more power at the same sample size. The formula charges us the worst case regardless.
- Balanced moderator (assumed, against us if false). The interaction rows assume the moderator splits 50/50. Schuessler and Freitag (2020) are explicit that conditional AMCEs require the covariate’s real marginal distribution; Table 5 quantifies the cost of getting this wrong.
- Dichotomised moderator (a different estimand). The interaction rows treat the moderator as binary, whereas the confirmatory hypotheses interact the attribute with a standardised continuous moderator. Dichotomising discards information, so this too is conservative.
The formulae themselves were verified against equations (4) and (6) of Schuessler and Freitag (2020) and against cjpowR directly; they agree to within 0.25%, always on the conservative side.
1.2 What drives power in a conjoint
Three facts from Schuessler and Freitag (2020) shape everything that follows, and the first two are counter-intuitive.
The number of attributes does not matter. Because attributes are randomised independently, each AMCE is a simple comparison of means that averages over all the others. Adding attributes does not dilute power. What matters is the number of levels of the attribute being tested: with \(K\) levels, each level’s mean is estimated from roughly \(N/K\) profiles, so more levels means larger standard errors. Studies should therefore be powered for the attribute with the most levels, not merely the one with the smallest expected effect.
What counts is profile evaluations, not respondents. The effective sample size is \(N_{\text{resp}} \times \text{tasks} \times 2\). Six tasks per respondent yield 18,000 profile evaluations from 1,500 people. This is why the number of tasks trades off directly against the number of respondents.
Effect sizes here are in percentage points of the allocated share. A 3 pp AMCE moves an applicant from 50% to 53% of the fund—about CLP 60,000 of the CLP 2,000,000 at stake. This is not the same scale as the published binary AMCEs Schuessler and Freitag (2020) survey, which measure changes in choice probability, so their benchmark distribution (median ≈ 5 pp) orients but does not translate directly.
1.2.1 Which effect size, and why 3 pp
This document sizes everything for a 3 pp effect. That is not a placeholder: it is the binding case of this design. Of the two attributes with three levels, the assumed true effects are Effort at 6 pp and Identity at 3 pp, so Identity—most levels, smallest expected effect—is what the design must clear.
| Target | Attribute | Levels | Assumed effect |
|---|---|---|---|
| This document | Identity | 3 | 3 pp |
| Simulation study | Effort | 3 | 6 pp |
The simulation in processing/power_declaredesign.R sizes for Effort instead, because Effort is the attribute the moderation hypotheses interact with the moderator. The two documents therefore answer different questions and their tables are not directly comparable: this one asks what the most demanding attribute requires, the simulation asks what power we have for the hypothesis we will actually test. Neither figure is wrong, and neither should be changed to match the other.
The choice of 3 pp follows the paper’s own advice on effect sizes. Schuessler and Freitag (2020) survey 258 AMCEs from 15 highly cited conjoints and find a median near 5 pp, but explicitly caution against sizing from that distribution: different research questions are not comparable, and publication bias inflates it. They recommend subject-specific priors instead, which is what the effect vector here does—it is anchored on Gilgen (2022), a comparable scholarship-allocation experiment. The same section warns that Cohen’s \(d\) misleads in this setting: a “small” \(d = 0.2\) corresponds to a 10 pp AMCE, which fewer than a quarter of published estimates exceed.
1.3 Main effects
1.3.1 Power at the study’s sample size
Under equation (4) of Schuessler and Freitag (2020), the minimum effective sample size to detect an AMCE \(\delta_1\) is
\[N = \frac{K}{2} \cdot \frac{(z_{1-\alpha/2} + z_{\kappa})^2}{\delta_1^2} \left( \frac{(\delta_0 + \delta_1)(1 - (\delta_0 + \delta_1))}{0.5} + \frac{\delta_0(1 - \delta_0)}{0.5} \right),\]
where \(\delta_0\) is set to 0.5, the value that maximises the variance and so yields the conservative bound. Table 1 evaluates this at the study’s 18,000 profile evaluations.
Both families clear the conventional 0.80 threshold with margin: 98.1% for two-level attributes and 90.8% for the two three-level attributes. Turning the formula around, detecting a 3 pp effect on a three-level attribute at 80% power requires about 1,090 respondents, so the planned 1,500 leaves roughly a third of the sample as headroom.
The Type M and Type S columns are worth a moment. An underpowered study is dangerous not only because it misses real effects, but because the effects it does declare significant are inflated—the point Schuessler and Freitag (2020) import from the Type S/M literature. A Type M ratio near 1 means no such inflation. Figure 4 shows how quickly that guarantee erodes at smaller samples.
1.3.2 How many tasks?
Because power depends on profile evaluations rather than respondents, the number of tasks per respondent and the number of respondents are substitutes. Figure 1 traces this directly.
At six tasks the design clears 80% power for a 3 pp effect on either attribute type well before 1,500 respondents. Cutting to four tasks shifts every curve right but does not breach the threshold at the planned sample; only at three tasks does the three-level, 2 pp combination become uncomfortable.
1.3.3 Minimum detectable effect
The complementary view fixes power at 0.80 and asks what effect size the design can resolve.
| N tasks | N profiles | MDE, K=2 attribute | MDE, K=3 attribute |
|---|---|---|---|
| 6 | 18,000 | 2.1 pp | 2.6 pp |
| 4 | 12,000 | 2.6 pp | 3.1 pp |
At the planned sample the design resolves effects of about 2.1 pp on two-level attributes and 2.6 pp on three-level ones. Dropping to four tasks costs roughly half a percentage point of sensitivity, taking the three-level figure to 3.1 pp—still at the 3 pp target, but without margin.
| Target | Minimum AMCE | Most demanding attribute | Min N (6 tasks) | Min N (4 tasks) |
|---|---|---|---|---|
| Main effects, K=2 | 3 pp | Binary | 727 | 1,091 |
| Main effects, K=3 | 3 pp | Effort / Identity | 1,091 | 1,636 |
| Main effects, K=3, power = 0.90 | 3 pp | Effort / Identity | 1,460 | 2,190 |
1.4 Interactions and heterogeneity
According to Schuessler and Freitag (2020): “It is known that testing for interaction effects often requires much larger sample sizes than testing for average effects” (2020, 8).
This is where the design is genuinely constrained. Equation (6) of Schuessler and Freitag (2020) gives the minimum sample for an interaction coefficient \(\delta_3\):
\[n = \frac{K_l K_m}{4} \cdot \frac{(z_{1-\alpha/2} + z_{\kappa})^2}{\delta_3^2} \left( \frac{A}{p_{00}} + \frac{B}{p_{10}} + \frac{C}{p_{01}} + \frac{D}{p_{11}} \right),\]
where \(A\) through \(D\) are cumulative response probabilities and \(p_{00} \dots p_{11}\) the joint treatment probabilities of the two interacting elements. The inflation relative to a main effect comes from the \(K_l K_m / 4\) term—it is not a fixed multiplier, and in particular it is not the factor of two that a naive shortcut would suggest.
| N tasks | Main effect, K=2 | Main effect, K=3 | Heterogeneity: attribute × moderator (N2, binary 50/50) | Attribute × attribute interaction (reference) |
|---|---|---|---|---|
| 3 | 1454 | 2181 | 8714 | 13070 |
| 4 | 1091 | 1636 | 6535 | 9803 |
| 5 | 873 | 1309 | 5228 | 7842 |
| 6 | 727 | 1091 | 4357 | 6535 |
At six tasks, a three-level main effect needs about 1,091 respondents while the same attribute interacted with a binary moderator needs about 4,357—a factor of 4. Taken at face value, the planned sample is ample for every main effect and short for every moderation effect.
Two qualifications matter before that is read as a verdict. First, the attribute-by-attribute column is a sizing reference only: no hypothesis of this study interacts two attributes, since all moderation hypotheses involve a respondent characteristic. Second, and more importantly, the whole table inherits the binary-outcome assumption, which is precisely the assumption our continuous fixed-sum outcome violates in our favour. The simulation study exists to remove it.
1.4.1 Sensitivity to the moderator’s distribution
The interaction rows assume a moderator that splits evenly. That assumption is doing real work, and it is the one most likely to fail in practice.
| Moderator split | Min N (6 tasks) | vs balanced |
|---|---|---|
| 50/50 | 4,357 | — |
| 40/60 | 4,539 | +4% |
| 30/70 | 5,189 | +19% |
| 20/80 | 6,811 | +56% |
A 30/70 split raises the requirement by about a fifth; a 20/80 split by more than half. Once the realised distribution of each moderator is known, these numbers should be recomputed with the observed marginals rather than the balanced default—n_min_interaction() accepts the joint probabilities directly for this purpose.
1.5 Type M error
A final diagnostic, and the reason power analysis matters even when a result is “significant”.
The exaggeration ratio is the factor by which a statistically significant estimate overstates the true effect. At the planned sample it sits essentially at 1 for every effect size considered, meaning estimates that clear significance are not systematically inflated. Below roughly 500 respondents the ratio climbs steeply for small true effects: a study that size would not merely miss real 2 pp effects, it would report the ones it caught as two or three times larger than they are. This is the concrete cost of underpowering, and the reason the moderation analyses are framed as exploratory unless the simulation shows otherwise.
1.6 Design check: D-efficiency
Everything up to this point concerns sample size. This section asks a different question about the same design: given the randomization actually implemented, how much information does each observation carry relative to the best possible design? The answer bears directly on a practical question—whether the factorial space could be reduced—so it is worth settling explicitly.
1.6.1 What is being measured
Each applicant profile is one cell of a factorial with 288 combinations: two levels each for Need, Control, Reciprocity, Attitude and Sex, three each for Effort and Identity. D-efficiency compares the information matrix of the design we actually field against that of the balanced, orthogonal ideal—the full factorial, each cell once:
\[D_{\text{rel}} = \left( \frac{\det(\mathbf{M}_{\text{realized}})}{\det(\mathbf{M}_{\text{ideal}})} \right)^{1/p}, \qquad \mathbf{M} = \frac{\mathbf{X}'\mathbf{X}}{n},\]
with effects coding and \(p\) the number of parameters. A value of 1.000 means the realized design extracts as much information per observation as the optimal design can; there is no headroom above it.
The design matrix is reconstructed from the randomizer as implemented (build_default_conjoint_design() in surveys/pre-piloto/sin_comentarios/app.R), including the restriction that the two profiles in a task differ on at least two of the six CARIN/NICER attributes. Sex is drawn separately, since it is signalled through the applicant’s first name and does not enter that check.
| Model | Parameters | Identifiable in the 288-cell factorial | D-efficiency | Without pair restriction | Obs. per parameter |
|---|---|---|---|---|---|
| Main effects | 10 | Yes | 0.9998 | 0.9998 | 1,800 |
| All two-way interactions | 44 | Yes | 0.9988 | 0.9990 | 409 |
1.6.2 Reading the result
The realized design sits at 0.9998 of the optimum for the main-effects model and 0.9988 for a model saturated with every two-way interaction. This is what theory predicts: complete independent randomization delivers orthogonality in expectation, and at 18,000 profile evaluations the realization has converged onto it. The largest absolute correlation between indicators of different attributes is 0.018.
The pair restriction is free. It operates on the joint distribution of the two profiles within a task and leaves each attribute’s marginal distribution untouched, so it costs nothing in efficiency—the two rightmost columns of Table 6 are identical to three decimals. It buys cognitive realism at no statistical price.
1.6.3 Verification with AlgDesign
The calculation above is ours, so it should be checked against an established implementation. AlgDesign (Wheeler 2025) is the natural choice, because it implements optimal design for linear models: its D criterion is \(|Z'Z/n|^{1/k}\), which is both the quantity computed above and the one implied by this study’s estimator. It requires no assumed parameter values, since linear D-efficiency does not depend on the coefficients.
Running eval.design() on the realized design and dividing by the same quantity for the full factorial gives 0.9998, against 0.9998 from the hand calculation—agreement to four decimal places. A second check asks whether the balanced factorial is itself D-optimal: optFederov() searches the 288-cell candidate set for anything better and returns a design with a D criterion 1.0000 times the factorial’s, i.e. it cannot be improved. The reference used above is therefore the right one.
1.6.4 Cross-check with idefix, and why the two disagree
A second package, idefix (Traets, Sanchez, and Vandebroek 2020), returns a very different number. The exercise turns out to be more informative than a confirmation: understanding the disagreement settles the design question more firmly than agreement would have.
idefix evaluates designs for discrete choice experiments under a multinomial logit model. For a two-alternative task with null priors, its information matrix reduces to
\[\mathcal{I} \;\propto\; \sum_{\text{tasks}} (\mathbf{x}_1 - \mathbf{x}_2)(\mathbf{x}_1 - \mathbf{x}_2)',\]
so all information lives in the contrast between the two profiles. An attribute taking the same level in both profiles—“level overlap”—contributes nothing to that task. Table 7 compares our randomized design against the D-optimal design idefix constructs with the modified Fedorov algorithm, at 72 choice sets.
| Design | DB-error | Orthogonality | Overlapping attributes per task | D-efficiency vs optimal |
|---|---|---|---|---|
| Randomized (as fielded) | 0.036 | 0.878 | 3.014 | 0.567 |
| D-optimal (idefix Modfed) | 0.020 | 0.932 | 0.000 | 1.000 |
By this criterion the fielded design scores 0.567, not the 1.000 of Table 6. The reason is visible in the overlap column: because each profile is drawn independently, an attribute happens to match across the two profiles with probability 1/2 (binary) or 1/3 (three-level), giving 3.0 overlapping attributes per task on average. A D-optimal choice design drives that to zero by forcing the alternatives to differ everywhere.
1.6.5 Which criterion applies
The two numbers are not a contradiction; they answer the question for two different estimators, and the design does not decide between them—the analysis plan does. Our estimator is a profile-level linear model of the allocated share with respondent fixed effects, not a conditional model on the within-task contrast. Under it, the competing profile’s attributes land in the error term, where randomization keeps them orthogonal to the regressor, so overlap costs almost nothing.
Table 8 states the three positions side by side. The spread across rows is not disagreement about arithmetic: the two linear implementations agree to four decimals because they compute the same criterion, and idefix differs because it computes a different one.
| Approach | Criterion | Model assumed | Priors required | D-efficiency |
|---|---|---|---|---|
| Hand calculation | Linear: |Z'Z/n|^(1/k) | Profile-level linear | No | 0.9998 |
| AlgDesign `eval.design` | Linear: |Z'Z/n|^(1/k) | Profile-level linear | No | 0.9998 |
| idefix `EvaluateDesign` | MNL: contrast information | Forced choice (conditional logit) | Yes | 0.5672 |
Rather than argue the point analytically, Table 9 settles it by simulating the real outcome—continuous, fixed-sum—and refitting the AMCE of Effort under both designs and both estimators. Standard errors are on a common AMCE scale, and relative D-efficiency is their squared ratio, which is the sample-size equivalent.
| Estimator | SE, as fielded | SE, minimum overlap | Relative D-efficiency |
|---|---|---|---|
| Profile-level linear model (analysis plan) | 0.889 | 0.886 | 0.991 |
| Contrast model (what idefix assumes) | 0.866 | 0.721 | 0.692 |
The result is clear for the question at hand. Under the profile-level estimator this study will use, moving to a minimum-overlap design buys a relative efficiency of 0.99—nothing worth having. Under the contrast estimator it buys 0.69, close to what idefix reports. AlgDesign is the pertinent tool here and idefix is not, because the estimator is linear and profile-level, not a conditional logit on the within-task contrast.
One honest qualification, visible in the same table. The two estimators are nearly equivalent on the design as fielded (0.889 against 0.866), but the combination of a minimum-overlap design with the contrast estimator reaches 0.721—about 19% smaller than the design and estimator actually chosen. That gain was available at the design stage and is not recoverable now: forcing minimum overlap is a property of how pairs are generated, not of how they are analysed. It is worth recording for a future wave, and it does not change any conclusion about the present one, where the relevant comparison is 0.99.
idefix is therefore reported here as a design diagnostic and as the answer to a question a reader may reasonably raise—“would a choice-based D-optimal design have been better?”—rather than as a correction to Table 6. Its verdict would bind if the outcome were a forced choice, or if the analysis moved to a conditional logit or a within-task difference model. Neither is the case, and this is one more consequence of the fact that this is a distributive conjoint rather than a choice-based one.
1.7 Summary
| Estimand | Requirement (6 tasks) | Covered at N = 1,500? |
|---|---|---|
| Main effect, K = 2 | 727 | Yes |
| Main effect, K = 3 (binding) | 1,091 | Yes |
| Heterogeneity, attribute × moderator | 4,357 | No, under the binary bound |
The design is well powered for all confirmatory main effects, with an MDE of 2.6 pp on the binding attribute and enough margin to absorb a reduction to four tasks if fieldwork requires it. For the moderation hypotheses the analytic bound is not met, but that bound assumes an outcome this study does not have. Whether the moderation hypotheses are adequately powered is therefore a question this document cannot settle on its own; it is settled, as far as it can be, by the simulation of the real continuous outcome reported alongside it.
Separately, the design is efficient as well as large enough. Under the profile-level estimator the analysis plan specifies, it sits at 0.9998 of the balanced-orthogonal optimum (Section 1.6), leaving nothing for a D-optimal alternative to recover. A choice-based criterion such as idefix’s rates the same design at 0.57, but that criterion presumes a forced-choice outcome and a contrast estimator, neither of which applies here—Table 9 shows the overlap it penalises costs this study nothing.
References
Gilgen, Sandra. 2022. Disentangling Justice: Needs, Equality or Merit? On the Situation-Dependency of Distributive Justice. 1. Auflage. Gesellschaftsanalyse 2. Baden-Baden: Nomos Verlagsgesellschaft mbH & Co. KG. https://doi.org/10.5771/9783748926955.
Schuessler, Julian, and Markus Freitag. 2020. “Power Analysis for Conjoint Experiments.” SocArXiv. https://doi.org/10.31235/osf.io/9yuhp.
Traets, Frits, Daniel Gil Sanchez, and Martina Vandebroek. 2020. “Generating Optimal Designs for Discrete Choice Experiments in R: The Idefix Package.” Journal of Statistical Software 96 (3): 1–41. https://doi.org/10.18637/jss.v096.i03.
Wheeler, Bob. 2025. AlgDesign: Algorithmic Experimental Design. https://CRAN.R-project.org/package=AlgDesign.