AI replication, directed by a co-author
This replication was carried out by Claude, an AI model made by Anthropic (Claude Opus 5.5, running in Claude Code), and guided by David Reinstein, one of the paper's two authors. Claude found the original files in David's archive, ported the authors' Stata code to Python, ran it, compared the output with the published tables, and wrote this page. David chose what to check and asked for the results to be published.
- Not independent. A co-author directed it, and it uses the authors' own data and code. It shows the published numbers follow from them; it does not re-collect the data or test other specifications.
- Not hand-checked. Every number here comes from running the posted code. No person has re-derived each figure by hand. If you find an error, please comment (the Hypothes.is sidebar on the right works on this page).
In brief
- The main results reproduce exactly. Every coefficient, standard error, R² and N in Tables IV, V and VI matches the paper (one constant differs by 0.01). That includes the headline estimates: one thumb up raises opening-weekend revenue by about 11% (not significant) and two thumbs up by about 25% (significant at 10%), with larger effects for dramas and narrow releases.
- Tables I–III reproduce apart from small print slips: one mean, one apparently transposed pair of correlations, a few standard errors off by 0.01, and one log likelihood.
- Table VII, on competitors' revenue, reproduces the authors' own Stata log to four decimals. But the one estimate the paper calls significant at 10% (two thumbs up stealing business from competing movies) has p = 0.137 in that log. The star is not supported. Its significance also depends on an arbitrary sort order in the authors' code: 8 of the 80 possible orderings give p < 0.10.
1. Table-by-table status
| Table | Content | Status | Notes |
|---|---|---|---|
| I | Descriptive statistics | Matches | EBERT UP mean is 0.4154; the paper prints 0.41 (0.42 when rounded). |
| II | Correlations | Matches | Except ln(TOTREV) with SISKEL UP and EBERT UP: 0.17 and 0.16 here, 0.16 and 0.17 in print. Looks transposed. |
| III | Probits for review timing (selection check) | Matches | Coefficients, pseudo R² and log likelihoods for columns 1–4 match; a few SEs differ by 0.01. Column 5: see section 4. |
| IV | Main difference-in-differences, by subsample | Exact | All coefficients, SEs, R² and N. |
| V | Dramas, robustness | Exact | All coefficients, SEs, R² and N. |
| VI | Comparison with the "traditional" method | Exact | Only difference: constant in column 2 is 12.15 here, 12.14 in print. |
| VII | Competitors' weekend revenue | Matches log; star unsupported | See section 3. |
Not replicated: the drama-only versions of Table VII that the authors ran but did not report (they are in original_code/mktbox2.log), and footnote 16 (weekends with only one movie in the data).
2. Main results (Tables IV–VI)
The paper's design compares movies Siskel and Ebert reviewed during their opening weekend with movies reviewed after it. For the second group a good review cannot have affected opening-weekend sales, so any correlation between their reviews and revenue reflects underlying quality. The difference between the two groups estimates the reviews' influence. The outcome is ln(opening-weekend revenue, 1999 dollars), with controls for Leonard Maltin's rating, screens, four-day weekends, and year, month, genre and distributor dummies. Standard errors are clustered by opening weekend.
Table IV: influence effects (identical to the paper)
| Sample | N | DURING × ONE UP | DURING × TWO UP | R² |
|---|---|---|---|---|
| All movies | 609 | 0.11 (0.12) | 0.25 (0.15)* | 0.78 |
| Wide release (>1,533 screens) | 304 | −0.06 (0.14) | −0.04 (0.20) | 0.69 |
| Narrow release | 305 | 0.13 (0.18) | 0.37 (0.21)* | 0.70 |
| Dramas | 198 | 0.51 (0.24)** | 0.65 (0.34)* | 0.79 |
| Action | 148 | 0.07 (0.22) | 0.08 (0.24) | 0.82 |
| Comedy | 267 | −0.01 (0.14) | −0.37 (0.30) | 0.75 |
Standard errors in parentheses. * p<0.10, ** p<0.05, *** p<0.01.
Table V: dramas, alternative specifications (identical to the paper)
| Specification | DURING × ONE UP | DURING × TWO UP | R² |
|---|---|---|---|
| (1) ln(OPENREV) | 0.51 (0.24)** | 0.65 (0.34)* | 0.79 |
| (2) Without MALTIN | 0.51 (0.25)** | 0.56 (0.33)* | 0.78 |
| (3) ln(revenue per screen) | 0.32 (0.24) | 0.58 (0.42) | 0.34 |
| (4) ln(TOTREV) | 0.70 (0.34)** | 0.85 (0.56) | 0.72 |
N = 198 in every column.
Table VI: the "traditional" method, which ignores timing (identical to the paper)
Regressing revenue on reviews without the timing comparison gives much larger "effects", which is the paper's point: most of the raw correlation reflects quality, not influence.
| (1) All | (2) All + MALTIN | (3) Reviewed during | (4) Reviewed during + MALTIN | |
|---|---|---|---|---|
| ONE UP | 0.22 (0.05)*** | 0.15 (0.05)*** | 0.25 (0.05)*** | 0.18 (0.05)*** |
| TWO UP | 0.37 (0.06)*** | 0.25 (0.06)*** | 0.40 (0.06)*** | 0.28 (0.07)*** |
| MALTIN | 0.26 (0.04)*** | 0.27 (0.05)*** | ||
| N | 609 | 609 | 492 | 492 |
| R² | 0.76 | 0.78 | 0.76 | 0.77 |
3. Table VII: does a good review steal business from competing movies?
Here the unit is a weekend (346 of them). The outcome is ln(total US weekend box office minus the opening revenue of the movies in the data opening that weekend), i.e. roughly competitors' revenue. Review variables are summed (column 1) or averaged (column 2) over the movies opening that weekend. Standard errors are heteroskedasticity-robust (White).
The key estimate
| f(DURING × TWO UP), column 1 (sum) | Estimate (SE) | p-value |
|---|---|---|
| Paper, as printed | −0.08 (0.05)* | < 0.10 (star) |
Authors' own Stata log, mktbox2.log | −0.0830 (0.0556) | 0.137 |
| This replication, ordering that matches the log | −0.0830 (0.0556) | 0.137 |
| This replication, all 80 possible orderings | −0.097 to −0.083 | 0.095 to 0.137 |
- The star isn't supported by the authors' run. The printed SE of 0.05 looks like 0.0556 truncated rather than rounded (other Table VII entries are truncated too, e.g. R² 0.6772 printed as 0.67). With t = −1.49 the estimate is not significant at 10%.
- The result depends on an arbitrary sort. To get one row per weekend,
MKTBOX2.DOsorts by weekend and keeps the first movie, taking FOURDAY, the month dummies and the inflation factor from that movie. In 4 weekends those differ between the movies opening together (FOURDAY in 1, month in 3), and Stata does not fix the order of tied rows. Trying all 80 combinations, 8 give p < 0.10 and 72 do not. - What it means for the paper. The paper calls this "suggestive evidence" of business stealing and says it cannot reject that total market revenue is unchanged by a positive review. The second statement stands. The first now rests on an estimate with p between 0.10 and 0.14 in most orderings.
Full Table VII: paper vs this replication (ordering that matches the authors' log)
| Paper (1) sum | Replication (1) | Paper (2) mean | Replication (2) | |
|---|---|---|---|---|
| f(DURING) | 0.01 (0.03) | 0.0097 (0.0252) | −0.03 (0.05) | −0.0262 (0.0486) |
| f(DURING × ONE UP) | 0.03 (0.04) | 0.0265 (0.0422) | 0.09 (0.07) | 0.0918 (0.0698) |
| f(DURING × TWO UP) | −0.08 (0.05)* | −0.0830 (0.0556) | −0.04 (0.08) | −0.0462 (0.0795) |
| f(ONE UP) | −0.03 (0.04) | −0.0331 (0.0392) | −0.08 (0.06) | −0.0769 (0.0637) |
| f(TWO UP) | 0.07 (0.05) | 0.0704 (0.0546) | 0.09 (0.07) | 0.0884 (0.0725) |
| f(MALTIN) | −0.01 (0.01) | −0.0073 (0.0132) | −0.06 (0.04) | −0.0585 (0.0440) |
| f(SCREENS) | −0.04 (0.01)*** | −0.0404 (0.0140) | −0.04 (0.03) | −0.0419 (0.0279) |
| FOURDAY | 0.31 (0.05)*** | 0.3119 (0.0523) | 0.27 (0.06)*** | 0.2739 (0.0610) |
| Constant | 17.36 (0.06)*** | 17.3578 (0.0585) | 17.55 (0.13)*** | 17.5523 (0.1267) |
| R² | 0.67 | 0.6772 | 0.54 | 0.5392 |
| N | 346 | 346 | 346 (stated) | 347 |
The replication column equals the authors' log to four decimals. SCREENS is in thousands.
Two smaller Table VII issues
- Column 2 uses 347 weekends in the authors' log, but the table note says 346.
- In column 2 the code subtracts the mean opening revenue of the movies in the data (
egen samgros = mean(wk1cpi)), not the sum, so its outcome is not exactly competitors' revenue as the note describes. Column 1 subtracts the sum. - Average competitors' weekend revenue is $48.1m here; the paper says $48m.
4. Other discrepancies
- Table III, column 5 log likelihood. Printed as −233; the code gives −247.0. With a null log likelihood of −290.5, −247 implies the printed pseudo R² of 0.15, while −233 would imply 0.20. So −233 is probably a typo.
- Table III sample size. The paper gives N = 573. Columns 1–4 have 573 because the documentary and film-noir dummies perfectly predict review timing, so Stata drops them with their 2 movies. Column 5 has no genre dummies and uses 575.
- Table I and II. See section 1. These look like rounding and transcription slips and change nothing.
5. Data and code
Everything needed to reproduce the tables. Download the whole package (zip), or see the GitHub repository.
Data
- movie.dta – the authors' movie-level file (Stata)
- movie.csv – same, as CSV, with ISO dates added
- codebook.csv – variables, labels, summary statistics
- mktbox.txt / mktbox.csv – weekly US box office, all movies
- original_variable_notes.txt – the authors' early variable notes
Authors' original code (JIE revision)
- revision4.do + log – Tables IV, V
- SELECT2.DO + log – Table III
- TRADIT2.DO + log – Table VI
- mktbox2.DO + log – Table VII
Replication code (2026)
- replicate.py – Tables I–V
- tables6_7.py – Tables VI, VII
- check_row_choice.py – Table VII, all 80 orderings
- check_t3col5.py, check_weekend_vars.py
- make_csv.py, run_all.sh, requirements.txt
- replicate.do – Stata version of Tables I–V, untested
To re-run
pip install -r replication/requirements.txt bash replication/run_all.sh # from the unzipped package; outputs go to replication/output/
Takes under a minute. Tested with Python 3.12, pandas 2.2, statsmodels 0.14.
How the Stata code was ported
- Regressions follow the authors' do-files line by line.
reg ..., robust cluster(weekend)becomes statsmodels OLS with errors clustered on opening weekend;probit ..., robust cluster(weekend)becomes a statsmodels probit with clustered errors;reg ..., robust(Table VII) uses HC1. - Opening weekend is the first Friday of the run:
weekend = opendate + 6 − openwday. - As Stata does, constant or collinear regressors are dropped, and in probits a dummy that perfectly predicts the outcome is dropped with its observations.
- SCREENS is divided by 1,000 to match the paper's units.
- Table VII merges each weekend with the market box-office date falling on its Friday or the three days before, deflates using the kept movie's CPI factor, and drops weekends where the movies in the data exceed the market total, as
MKTBOX2.DOdoes.
6. About the data
- Coverage. 648 US movies that opened on 50 or more screens between 1989 and 1997 and that Siskel and Ebert reviewed on a Saturday within 40 days of opening. The paper drops the 39 reviewed before opening weekend, leaving 609 (variable
pre). - Sources (collected by the authors for the paper): box office, release dates and screens from the Box Office Guru website; Siskel and Ebert's reviews and review dates from their website; ratings from Leonard Maltin's 1999 movie guide; genres, votes and average user rating from IMDb; weekly total US box office from Variety's website. Revenue in 1999 dollars uses the urban CPI.
- Dates in the Stata file are days since 1 January 1960 (the Stata/SAS convention); the CSVs add ISO dates.
- Use. The authors share these files for research and replication. Please cite the paper. Some variables were compiled from the third-party sources above, which keep their own terms.