# Dokime Quality Audit: 3 Popular HuggingFace Text Datasets

**Date:** 2026-03-19
**Tool:** `dokime score` v0.1.1
**Sample size:** 1,000 documents per dataset (streamed)

---

## Summary

| Dataset | Grade | Score | Docs | Duplicates | Short (<10 words) | Low Quality (<0.3) | High Special Chars |
|---------|-------|-------|------|------------|--------------------|--------------------|-------------------|
| allenai/c4 (en, validation) | **A** | 99.7/100 | 1,000 | 0 | 1 (0.1%) | 0 | 0 |
| EleutherAI/the_pile_deduplicated (train) | **A** | 96.7/100 | 1,000 | 0 | 8 (0.8%) | 0 | 3 (0.3%) |
| Skylion007/openwebtext (train) | **A** | 99.8/100 | 1,000 | 0 | 0 | 0 | 0 |

---

## 1. allenai/c4 (en, validation split)

- **Grade:** A (99.7/100)
- **Scanned in:** 0.44s
- **Duplicates:** 0
- **Short documents:** 1 (0.1%)
- **Low entropy:** 0
- **High special chars:** 0

### Quality Distribution

| Range | Count |
|-------|-------|
| Excellent (>0.8) | 998 |
| Good (0.6-0.8) | 2 |
| Medium (0.3-0.6) | 0 |
| Poor (<0.3) | 0 |

### Worst 5 Documents

| # | Score | Preview |
|---|-------|---------|
| 1 | 0.7690 | 1343 cover-me 98/01/01 Re: ... (CJK forum content) |
| 2 | 0.7880 | ^ 1.0 1.1 1.2 1.3 1.4 1.5 Annual Report 2018 (PDF). Infineon Technologies... |
| 3 | 0.8360 | John Pavlick (*) 860-354-2827 jpavlick@idseng.com 2 Mt. Tom Rd... |
| 4 | 0.8490 | CJK community content about political views... |
| 5 | 0.8540 | GC_THREADS discussion for Solaris configuration... |

**Notes:** C4 is heavily filtered at the source. Even the "worst" documents score 0.77+. The lowest-scoring entries are multilingual or contain structured reference data rather than prose.

---

## 2. EleutherAI/the_pile_deduplicated (train split)

- **Grade:** A (96.7/100)
- **Scanned in:** 1.56s
- **Duplicates:** 0
- **Short documents:** 8 (0.8%)
- **Low entropy:** 0
- **High special chars:** 3 (0.3%)

### Quality Distribution

| Range | Count |
|-------|-------|
| Excellent (>0.8) | 931 |
| Good (0.6-0.8) | 62 |
| Medium (0.3-0.6) | 7 |
| Poor (<0.3) | 0 |

### Worst 5 Documents

| # | Score | Preview |
|---|-------|---------|
| 1 | 0.4180 | `<HTML><HEAD> <TITLE>Invalid URL</TITLE> </HEAD><BODY>...` (HTML error page) |
| 2 | 0.5000 | `#ifndef CREATE_EMPTY_DIRECTED_GRAPH_WITH_GRAPH_NAME_H...` (C++ header guard) |
| 3 | 0.5260 | `File.open(${1:path_to_yml}, "w")...` (code snippet/template) |
| 4 | 0.5710 | `version https://git-lfs.github.com/spec/v1 oid sha256:...` (git-lfs pointer) |
| 5 | 0.5930 | `id: dsq-747531936 date: 2010-04-05T22:49:24...` (Disqus metadata) |

**Notes:** The Pile is intentionally diverse, pulling from code repos, academic papers, forums, and web scrapes. The lower-scoring documents are not "bad" per se -- they are code, metadata, and structured data that score lower on prose-oriented quality metrics. The 7 medium-quality documents and 8 short docs reflect this diversity. No documents scored below 0.3.

---

## 3. Skylion007/openwebtext (train split)

- **Grade:** A (99.8/100)
- **Scanned in:** 1.04s
- **Duplicates:** 0
- **Short documents:** 0
- **Low entropy:** 0
- **High special chars:** 0

### Quality Distribution

| Range | Count |
|-------|-------|
| Excellent (>0.8) | 1,000 |
| Good (0.6-0.8) | 0 |
| Medium (0.3-0.6) | 0 |
| Poor (<0.3) | 0 |

### Worst 5 Documents

| # | Score | Preview |
|---|-------|---------|
| 1 | 0.8140 | No. 1 - Retired for Billy Martin Earle Combs 1929-1935... (list of names/dates) |
| 2 | 0.8740 | This article is about Russian and Soviet chess composer... |
| 3 | 0.8750 | PHOTOS: Tornado damage in Tupelo, Mississippi... |
| 4 | 0.8750 | The connection between race and intelligence has been a subject of debate... |
| 5 | 0.8750 | Caution: JavaScript execution is disabled in your browser... |

**Notes:** OpenWebText is the cleanest dataset in this audit. Every single document scores in the "excellent" range (>0.8). This is expected: OpenWebText is filtered to Reddit-upvoted content, which naturally selects for coherent, well-formed English prose.

---

## Key Takeaways

1. **All three datasets earn an A grade**, confirming they are well-curated for general text quality.
2. **OpenWebText is the cleanest** (99.8/100) -- its Reddit-upvote filter acts as a natural quality gate.
3. **C4 is nearly as clean** (99.7/100) -- its heuristic filtering pipeline is effective.
4. **The Pile has the most variance** (96.7/100) -- by design, since it includes code, metadata, and structured data alongside prose. Its "lower quality" documents are domain-specific content, not noise.
5. **Zero exact duplicates** were found in any 1,000-sample slice, suggesting all three datasets have effective deduplication.
6. **No documents scored below 0.3** (poor) in any dataset.

---

*Generated by `dokime score` -- open-source data curation for ML.*
*Datasets streamed from HuggingFace Hub via the `datasets` library.*
