The Pile: Dream Training Dataset
Tech Talks
Mar 28, 2021•3 mins
The Pile is an amazing new dataset for training language models. It is like an all-you-can-eat buffet for those of us creating models in the GPT-3 space.

The Pile is an amazing new dataset for training language models. It is like an all-you-can-eat buffet for those of us creating models in the GPT-3 space. 

 

ComponentRaw SizeWeightEpochsEffective SizeMean Document Size
Pile-CC227.12 GiB18.11%1.0227.12 GiB4.33 KiB
PubMed Central90.27 GiB14.40%2.0180.55 GiB30.55 KiB
Books3100.96 GiB12.07%1.5151.44 GiB538.36 KiB
OpenWebText262.77 GiB10.01%2.0125.54 GiB3.85 KiB
ArXiv56.21 GiB8.96%2.0112.42 GiB46.61 KiB
Github95.16 GiB7.59%1.095.16 GiB5.25 KiB
FreeLaw51.15 GiB6.12%1.576.73 GiB15.06 KiB
StackExchange32.20 GiB5.13%2.064.39 GiB2.16 KiB
USPTO Backgrounds22.90 GiB3.65%2.045.81 GiB4.08 KiB
PubMed Abstracts19.26 GiB3.07%2.038.53 GiB1.30 KiB
Gutenberg (PG-19)10.88 GiB2.17%2.527.19 GiB398.73 KiB
OpenSubtitles12.98 GiB1.55%1.519.47 GiB30.48 KiB
Wikipedia (en)6.38 GiB1.53%3.019.13 GiB1.11 KiB
DM Mathematics7.75 GiB1.24%2.015.49 GiB8.00 KiB
Ubuntu IRC5.52 GiB0.88%2.011.03 GiB545.48 KiB
BookCorpus26.30 GiB0.75%1.59.45 GiB369.87 KiB
EuroParl4.59 GiB0.73%2.09.17 GiB68.87 KiB
HackerNews3.90 GiB0.62%2.07.80 GiB4.92 KiB
YoutubeSubtitles3.73 GiB0.60%2.07.47 GiB22.55 KiB
PhilPapers2.38 GiB0.38%2.04.76 GiB73.37 KiB
NIH ExPorter1.89 GiB0.30%2.03.79 GiB2.11 KiB
Enron Emails0.88 GiB0.14%2.01.76 GiB1.78 KiB
Total   1254.20 GiB5.91 KiB

 

8941
Share this
Leave a comment
There are no comments about this article, let us know what you think?

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Related Posts
09 Oct•5 mins
Share to FacebookShare to Twitter
Optimizing Vector Storage in PostgreSQL with pgvector Halfvec

Efficient storage and retrieval of high-dimensional vectors are critical for maintaining performance and scalability. The halfvec data type...

686
14 Jul•1 min
Share to FacebookShare to Twitter
Our Git-based deployment methodology.

East Agile Git-based deployment methodology.

4573
10 Apr
Share to FacebookShare to Twitter
CPUs may replace GPUs for training Deep Learning Models

CPUs can be four to fifteen times faster than GPUs at training Deep Learning models using new methods...

11183
23 Nov•4 mins
Share to FacebookShare to Twitter
Generative Pre-Training: GPT-2 vs. GPT-3

OpenAI’s language models GPT-2 and GPT-3 have revolutionized natural language processing (NLP) technology through their Transformer-based language models....

14505
17 Nov
Share to FacebookShare to Twitter
Half-baked AI might be better than you think.

Some folks have observed that language models like GPT-3, GPT-2, Megatron-LM, Turing-NLG, continue to improve in terms of...

11475
04 Nov•4 mins
Share to FacebookShare to Twitter
Monitoring Spanish Language Misinformation in U.S. using AI

Monitoring misinformation and disinformation in U.S. Spanish language radio broadcasts using AI.

258
30 Oct•2 mins
Share to FacebookShare to Twitter
Anthropic Computer Use will not Act as an Agent

Computer use refuses to act as an agent representing people.

“sign up for the next available workshop. At...

306
27 Sep
Share to FacebookShare to Twitter
Client: Cyber Security Startup

We built customer facing user interfaces for a leading cyber security venture.

558
27 Sep
Share to FacebookShare to Twitter
Client: An online dance studio

A dynamic business that provides online and offline dance classes.

420
27 Sep
Share to FacebookShare to Twitter
Client: A Web3 address name service for a major Blockchain

Web3 project from concept to acquisition in 6 months.

429