I've been fooling around with some natural language data from OPUS, the “open parallel corpus.” This contains many gigabytes of movie subtitles, UN documents and other text, much of it tagged by part-of-speech and aligned across multiple languages. In total, there's over 50 GB of data, compressed.
“50 GB, compressed” is an awkward quantity of data:
- It's large enough so that Pandas can't suck it all into memory.
- It's large enough that PostgreSQL stops being fun, and starts feeling like work. (Although cstore_fdw might help.)
- It's too small to justify cloud-based tools like Hadoop. As the saying goes, “If it fits on your laptop’s SSD, it’s not big data.” I have USB sticks large enough to hold 50 GB!
Let's look at various ways to tackle this.