Large-Scale Analysis of Zipf's Law in English Texts
Moreno-Sánchez, Isabel (Universitat Autònoma de Barcelona. Departament de Matemàtiques)
Font-Clos, Francesc (Centre de Recerca Matemàtica)
Corral, Álvaro (Universitat Autònoma de Barcelona. Departament de Matemàtiques)

Date:	2016
Abstract:	Despite being a paradigm of quantitative linguistics, Zipf's law for words suffers from three main problems: its formulation is ambiguous, its validity has not been tested rigorously from a statistical point of view, and it has not been confronted to a representatively large number of texts. So, we can summarize the current support of Zipf's law in texts as anecdotic. We try to solve these issues by studying three different versions of Zipf's law and fitting them to all available English texts in the Project Gutenberg database (consisting of more than 30 000 texts). To do so we use state-of-the art tools in fitting and goodness-of-fit tests, carefully tailored to the peculiarities of text statistics. Remarkably, one of the three versions of Zipf's law, consisting of a pure power-law form in the complementary cumulative distribution function of word frequencies, is able to fit more than 40% of the texts in the database (at the 0. 05 significance level), for the whole domain of frequencies (from 1 to the maximum value), and with only one free parameter (the exponent).
Grants:	Ministerio de Economía y Competitividad FIS2012-31324 Agència de Gestió d'Ajuts Universitaris i de Recerca 2014/SGR-1307
Rights:	Aquest document està subjecte a una llicència d'ús Creative Commons. Es permet la reproducció total o parcial, la distribució, la comunicació pública de l'obra i la creació d'obres derivades, fins i tot amb finalitats comercials, sempre i quan es reconegui l'autoria de l'obra original.
Language:	Anglès
Document:	Article ; recerca ; Versió publicada
Published in:	PloS one, Vol. 11, Issue 1 (January 2016) , art. e0147073, ISSN 1932-6203

DOI: 10.1371/journal.pone.0147073
PMID: 26800025

19 p, 1.9 MB

The record appears in these collections:
Articles > Research articles
Articles > Published articles