This is the readme for the k5 corpus (created around the start of december 2012). 6k diaries downloaded from kuro5hin.org in nested form, but not logged in (so downrated comments were invisible). The overall stats for the corpus are: total-words unique-words 5984912 93621 all diaries+comments 4751968 78020 all comments 1232945 50037 all diaries (for a very crude definition of word. Everything was cast to lowercase, then anything not in a-z or ' was considered a word boundary.) The stats broken down per user: (see report-total_words-total_unique-total_distinctive.txt) First column is total words, second is total unique words, third is number of words distinctive to that user. ---------- 202 118 3 A Bore 2783 662 9 Abominable Abitur 531 199 1 Altus 77 50 2 AlwaysAnonyminated 56 44 1 Ayn Rand the Objectivist 222 130 2 BJH 327 193 11 BadDoggie 1676 653 28 BirthControlSoothsayer 65102 8307 656 Blarney 229 123 1 Booji Boy 106 74 7 BoredWitless 7322 1453 44 BottleRocket 7032 1273 28 Brogdel 26 26 1 C Montgomery Burns 506 249 6 Calalily 701 149 1 Charles F Wilson 123 97 3 CheeseburgerBrown 169 100 2 ChemicalCastrator 15985 1640 93 Chino Ginelli 13 11 1 ClaytonBigsby 303 100 2 Clippy 58 47 0 Clock Nightstand Joke 54160 6769 304 Corey Haim 183 62 1 Corwin06 233 134 1 Creative Hate 3773 1245 45 DK4 Bloo 110 87 3 Defect 362785 17688 3094 Del Griffith 9100 2320 85 Delirium 33934 4792 262 Edmund Blackadder 19 18 1 Elija 9006 1611 46 Empedocles 56961 2418 80 Enlarged to Show Texture 30 26 2 Estanislao Martnez 14875 2658 128 Ezra Loomis Pound 239 86 1 FEKLAAR 70 55 0 France 924 379 2 Fujitsu Corp 2893 744 18 GhostOfTiber 970 391 1 Glock To The Head 16960 3184 96 GreyGhost 1668 671 32 GrubbyBeardedHermit 6202 1573 62 HackerCracker 167448 12420 1165 Harry B Otch 2051 648 15 Have A Nice Day 35 30 0 High Wire Act 170215 10710 872 HollyHopDrive 6526 1139 59 I Did It All For The Horse Cock 116 51 7 I am Spartacus 30171 4638 260 Ignorant Motherfuckers 101520 10072 847 Irish Baby Bacon 131 94 0 Jack Wagner 501 208 5 Jesus Christ Himself 5805 999 37 Jobst of Moravia 397 176 3 JosiahGibbs 177 128 1 JulianConrad 246 140 6 Kingmaker 17905 3115 171 Korean Loller Blader 52421 4031 216 Lady 3Jane 194934 12238 1364 Lexx Core 162436 12810 1157 LilDebbie 4119 1159 31 Linux or FreeBSD 140 76 1 LocalH 25807 1765 47 MDC Protector Protector 76713 9588 1452 Marvin Suggs 12779 336 15 Masquitos Davis 16 16 1 MazdaBongo 94206 11123 2218 McNugent 26906 4843 193 MichaelCrotchford 8238 2333 1735 Michael Crawfords Third Leg 219 142 3 Mindtwist 240 140 1 Mocknbird2 159638 10018 1031 Morally Inflexible 134 73 2 Morvael 130 101 1 MotorMachineMercenary 12788 2622 79 MotorMachineMercenery 76 62 4 Mr. Penguin 104 80 1 My Dupe Account 269 155 4 Mylakovich 176931 12207 1343 N0574 19 19 0 Naysayer 16419 3280 106 New Bottle Old Wine 112 51 5 New User Signups Open Again 167359 8663 722 Nimey 15 15 1 Niqht In White Satin11 280 136 1 Novelty Account No 59671 216 91 4 Nursie 247 133 1 Orion Blastar Reborn 44 25 0 Ouija Board 37 35 2 Owa Tajer Kyam 52112 6744 385 Pentashagon 409 112 6 Phil Elmore The Martialist 2446 668 28 Phssthpok 208584 10216 1861 Pnarp 316 156 9 Private Seal Club 252 161 0 Psycho Dave 20788 3273 71 QuantumFoam 37 36 2 QuantumG 207 118 2 R Mutt 13616 2244 109 Redcatblack 5006 1357 25 Repeatible Hairstyle 3212 971 43 Robot Buys A Pop 877 154 2 Ruston Rustov 157 78 1 SaintPort 1291 560 18 Scrymarch 39 36 0 Sgt York 7457 1047 37 ShiftyStoner 20 20 0 Smiley K 4592 1159 24 SoupIsGoodFood 73 61 1 Soviet Russian 3087 1012 35 Special K 21479 3992 343 Stick Apart 9691 1868 74 Strom Thurmond 631 255 11 Sturmbannfuehrer von Braun 76199 9554 1300 TDS 633 277 11 TRASG0 26 25 2 TSol 26753 3394 169 TaxiCabJesus 687 285 8 The Belief Doctor 964 438 15 The Pope Himself 83 67 0 The Real Orion Blastar 125 45 0 The Vast Right Wing Conspiracy 905 379 24 They Came from Planet 9 118 83 9 Tjalfi 25 24 2 Torka 51 39 0 Trollaxor 15318 2074 63 United Fools 1051 468 13 Untied Fools 8960 985 728 V 200 96 0 V1m 55420 6585 711 Vampire Zombie Abu Musab al Zarqawi 474 246 5 Wain 50307 7342 663 Wen Jian 573 249 1 Wexel Pixel 7371 1765 62 Wise Cracker 51 47 0 Wolf At Large 30 30 1 Yori 308 197 7 Zombie Buddha 37673 4644 205 Zombie Gautama Buddha 524253 21169 3906 Zombie Jesus Christ 17443 3494 141 Zombie Kitten 13407 3056 158 Zombie Osama bin Laden 37960 4509 89 Zombie Schrodingers Cat 104195 3351 236 Zombie Tyler Durden 23 23 0 achievingfluidity 370 196 11 alba 1823 669 12 alevin 10252 2000 96 anaesthetica 388 150 9 anonimouse 30 19 1 anticlimax 34 33 0 aphasia 144446 11559 1170 balsamic vinigga 32 26 1 bankebiff 700 290 3 barrensolitude 160 92 1 basj 180 95 6 billt 39 33 2 blaaf 938 397 4 blackbart 5396 1534 42 boomi 2791 795 13 boxed 77 57 13 buford 51 42 1 chakmol 73319 8544 773 channel 306 107 1 circletimessquare 441 174 1 circus29 101738 9960 788 claes 179483 10731 1313 cockskin horsesuit 264 160 2 coq 119 72 1 craw4ord 199646 13377 1875 donnalee 492 265 2 drwav 751 295 3 dtylercade 215 105 4 dubya 5659 1454 33 eavier 1198 541 13 ensignyu 24 24 0 errgh 1878 771 20 exolete 491 263 11 fenugeek 42 38 3 fleece 161 82 2 givemegmail111 140 85 2 glimmung 13671 2576 57 greengrass 672 383 19 headlesshorseman 6025 1933 99 horny smurf 854 391 27 horsecock sodomy 47405 5727 489 horseskin spacesuit 8990 1533 107 hugin 16 16 1 imrdkl 1111 418 17 infernalmajesty 479 242 5 it certainly is 824 388 42 jd 12 11 29 jew 222 151 5 jmzero 2923 742 30 jolt rush soon 68 63 2 jrincayc 127 73 2 jvcoleman 10576 2404 59 jxg 21194 3866 280 k31 47059 4672 138 king of fools 242 108 4 kpaul 11708 2859 137 krkrbt 20541 4027 206 ksandstr 46 41 1 kuro5hinatportkardotnet 15216 2547 32 lednerg 575 257 7 lemonjuicefake 31 29 3 levesque 231 144 10 lilburne 48784 6981 448 limpdawg 11357 1441 47 livus 101 72 3 ljt 56941 6406 344 localroger 76566 6646 348 lostincali 14 14 0 magic curl 30 17 1 make u friend 992 166 3 me0w 136727 10491 784 messybuu 430 159 1 mike4troll 30793 3870 539 mirko 4715 1258 54 mirleid 4921 1237 43 modus 1172 419 7 mr strange 12359 2684 178 mrbastard 129055 10341 1070 mumble 137 83 2 muyuubyou 28 28 0 my gold bling shines 505 259 8 mybostinks 88 61 3 nairobiny 16422 2759 71 nate s 74704 6907 652 nateo 17 17 0 noscript 34021 5046 331 nostalgiphile 1823 624 14 ogfomk 26598 4138 244 osm 24 15 0 paxman 1574 537 9 pharaoh 362 197 4 power guido 17031 2524 110 procrasti 475 230 36 psycho 2751 857 36 r rnq 220 122 3 regeya 46 30 0 rrmq 9425 1823 58 rusty 35200 5072 725 schlouse 5094 1453 45 sharewareguy 40033 4893 236 sholden 930 334 7 sick puppy 19 18 0 slice2 25815 5047 308 some nerd 53 33 0 spooky wookie 301 174 6 spunk 8097 1502 16 squirrelmaster 165 71 4 stuaart 83 66 2 sudogeek 1811 580 20 superdiva 53830 8838 1327 sye 394445 19602 3073 tdillo 1584 563 10 th0m 10 10 4 toki 33691 5537 395 trane 3204 937 30 tweet 61 53 4 twocents 1059 382 14 twopumpchump 34 32 20 ubu 33 31 0 undermyne 13849 2712 155 unknownlamer 29 24 1 unmovic 29 25 23 walk 187 121 0 walwyn 115 86 2 wiredog 4309 1155 38 xC0000005 40 38 0 zenofchai 388 210 0 zombie Fred Phelps 90 37 4 zombie HollyHopDrive 392 198 3 zombie lonelyhobo 261 127 3 zombie steve jobs The number of diaries and comments per user: (see diary-comment-tallies.txt) ---------- total diaries: 6000 ---------- 477 7.9% Del_Griffith 325 5.4% McNugent 308 5.1% United_Fools 266 4.4% messybuu 197 3.2% N0574 182 3% Zombie_Jesus_Christ 151 2.5% donnalee 148 2.4% cockskin_horsesuit 144 2.4% Nimey 144 2.4% sye 139 2.3% Lady_3Jane 139 2.3% Pnarp 133 2.2% Vampire_Zombie_Abu_Musab_al_Zarqawi 127 2.1% LilDebbie 122 2% Marvin_Suggs 114 1.9% king_of_fools 113 1.8% Harry_B_Otch 111 1.8% nateo 105 1.7% Lexx_Core 94 1.5% tdillo 91 1.5% balsamic_vinigga 86 1.4% TDS 85 1.4% claes 80 1.3% trane 79 1.3% channel 69 1.1% Zombie_Tyler_Durden 67 1.1% mirko 61 1% horseskin_spacesuit 57 0.9% New_Bottle_Old_Wine 55 0.9% HollyHopDrive 55 0.9% greengrass 55 0.9% k31 54 0.9% nostalgiphile 53 0.8% ShiftyStoner 51 0.8% limpdawg 50 0.8% MDC_Protector_Protector 48 0.8% Redcatblack 45 0.7% lostincali 45 0.7% mumble 42 0.7% osm 40 0.6% MichaelCrotchford 39 0.6% Edmund_Blackadder 38 0.6% Ezra_Loomis_Pound 38 0.6% Korean_Loller_Blader 38 0.6% Zombie_Gautama_Buddha 34 0.5% Blarney 32 0.5% Irish_Baby_Bacon 31 0.5% mirleid 29 0.4% GreyGhost 29 0.4% Wen_Jian 28 0.4% Brogdel 28 0.4% Morally_Inflexible 27 0.4% localroger 26 0.4% modus 25 0.4% Special_K 25 0.4% TaxiCabJesus 25 0.4% horny_smurf 23 0.3% jxg 23 0.3% lednerg 22 0.3% Ignorant_Motherfuckers 22 0.3% hugin 21 0.3% Pentashagon 21 0.3% QuantumFoam 20 0.3% Stick_Apart 19 0.3% Zombie_Osama_bin_Laden 16 0.2% MotorMachineMercenery 16 0.2% Strom_Thurmond 16 0.2% r_rnq 16 0.2% squirrelmaster 16 0.2% unknownlamer 15 0.2% Zombie_Kitten 15 0.2% schlouse 14 0.2% Michael_Crawfords_Third_Leg 14 0.2% anaesthetica 13 0.2% I_Did_It_All_For_The_Horse_Cock 13 0.2% sharewareguy 12 0.2% DK4_Bloo 12 0.2% Enlarged_to_Show_Texture 12 0.2% krkrbt 12 0.2% ksandstr 12 0.2% rusty 11 0.1% V 10 0.1% Jobst_of_Moravia 10 0.1% nate_s 9 0.1% Delirium 9 0.1% GhostOfTiber 8 0.1% Have_A_Nice_Day 8 0.1% procrasti 8 0.1% sick_puppy 8 0.1% some_nerd 7 0.1% Chino_Ginelli 7 0.1% Masquitos_Davis 7 0.1% SoupIsGoodFood 7 0.1% They_Came_from_Planet_9 6 0.1% GrubbyBeardedHermit 6 0.1% eavier 5 0% Sturmbannfuehrer_von_Braun 5 0% TRASG0 5 0% Zombie_Schrodingers_Cat 5 0% lemonjuicefake 5 0% zombie_lonelyhobo 4 0% Abominable_Abitur 4 0% HackerCracker 4 0% Jesus_Christ_Himself 4 0% Nursie 4 0% Untied_Fools 4 0% mr_strange 4 0% mrbastard 4 0% xC0000005 3 0% Charles_F_Wilson 3 0% Corey_Haim 3 0% Empedocles 3 0% blackbart 3 0% boomi 3 0% circus29 3 0% mybostinks 3 0% twopumpchump 2 0% BirthControlSoothsayer 2 0% Booji_Boy 2 0% BottleRocket 2 0% Creative_Hate 2 0% FEKLAAR 2 0% Glock_To_The_Head 2 0% I_am_Spartacus 2 0% Kingmaker 2 0% Linux_or_FreeBSD 2 0% Novelty_Account_No_59671 2 0% Phil_Elmore_The_Martialist 2 0% Repeatible_Hairstyle 2 0% Robot_Buys_A_Pop 2 0% Ruston_Rustov 2 0% The_Belief_Doctor 2 0% The_Pope_Himself 2 0% Wexel_Pixel 2 0% Zombie_Buddha 2 0% billt 2 0% dubya 2 0% kpaul 2 0% lilburne 2 0% livus 2 0% nairobiny 2 0% ogfomk 2 0% pharaoh 2 0% regeya 2 0% superdiva 1 0% A_Bore 1 0% Ayn_Rand_the_Objectivist 1 0% BJH 1 0% BadDoggie 1 0% BoredWitless 1 0% Calalily 1 0% ChemicalCastrator 1 0% Defect 1 0% Estanislao_Martnez 1 0% France 1 0% Fujitsu_Corp 1 0% High_Wire_Act 1 0% Jack_Wagner 1 0% JosiahGibbs 1 0% Mindtwist 1 0% Morvael 1 0% MotorMachineMercenary 1 0% Mylakovich 1 0% Psycho_Dave 1 0% QuantumG 1 0% SaintPort 1 0% TSol 1 0% Tjalfi 1 0% Torka 1 0% Wise_Cracker 1 0% anticlimax 1 0% barrensolitude 1 0% buford 1 0% craw4ord 1 0% exolete 1 0% fenugeek 1 0% headlesshorseman 1 0% horsecock_sodomy 1 0% infernalmajesty 1 0% it_certainly_is 1 0% jd 1 0% jmzero 1 0% jolt_rush_soon 1 0% jrincayc 1 0% jvcoleman 1 0% muyuubyou 1 0% power_guido 1 0% psycho 1 0% spunk 1 0% stuaart 1 0% ubu 1 0% wiredog 1 0% zombie_steve_jobs ---------- total comments: 65300 ---------- 4077 6.2% Del_Griffith 3184 4.8% tdillo 2984 4.5% LilDebbie 2542 3.8% donnalee 2378 3.6% Harry_B_Otch 2303 3.5% Zombie_Jesus_Christ 2244 3.4% HollyHopDrive 2157 3.3% Nimey 2065 3.1% N0574 1546 2.3% McNugent 1475 2.2% balsamic_vinigga 1449 2.2% Enlarged_to_Show_Texture 1421 2.1% Lexx_Core 1332 2% nateo 1294 1.9% mumble 1271 1.9% king_of_fools 1253 1.9% Pnarp 1251 1.9% lostincali 1233 1.8% channel 1199 1.8% claes 1185 1.8% Vampire_Zombie_Abu_Musab_al_Zarqawi 1049 1.6% Marvin_Suggs 1019 1.5% Morally_Inflexible 945 1.4% mirko 910 1.3% Pentashagon 891 1.3% Zombie_Gautama_Buddha 829 1.2% cockskin_horsesuit 822 1.2% messybuu 780 1.1% horseskin_spacesuit 771 1.1% TDS 741 1.1% Edmund_Blackadder 687 1% Blarney 662 1% Wen_Jian 652 0.9% schlouse 564 0.8% sye 536 0.8% limpdawg 493 0.7% nostalgiphile 480 0.7% ksandstr 480 0.7% localroger 475 0.7% sholden 465 0.7% Lady_3Jane 434 0.6% osm 426 0.6% Ignorant_Motherfuckers 410 0.6% lednerg 396 0.6% Corey_Haim 396 0.6% Zombie_Schrodingers_Cat 347 0.5% QuantumFoam 337 0.5% Irish_Baby_Bacon 317 0.4% Redcatblack 315 0.4% GreyGhost 313 0.4% some_nerd 312 0.4% nate_s 287 0.4% greengrass 269 0.4% MotorMachineMercenery 265 0.4% Zombie_Tyler_Durden 237 0.3% Stick_Apart 235 0.3% New_Bottle_Old_Wine 223 0.3% k31 200 0.3% squirrelmaster 192 0.2% rusty 189 0.2% Delirium 188 0.2% MDC_Protector_Protector 185 0.2% mrbastard 167 0.2% Korean_Loller_Blader 160 0.2% MichaelCrotchford 159 0.2% procrasti 157 0.2% Ezra_Loomis_Pound 150 0.2% SoupIsGoodFood 141 0.2% Chino_Ginelli 141 0.2% I_Did_It_All_For_The_Horse_Cock 139 0.2% modus 128 0.1% BottleRocket 126 0.1% Jobst_of_Moravia 121 0.1% HackerCracker 121 0.1% livus 120 0.1% ShiftyStoner 112 0.1% jxg 111 0.1% krkrbt 109 0.1% V 105 0.1% Empedocles 99 0.1% hugin 96 0.1% Wise_Cracker 92 0.1% United_Fools 87 0.1% sharewareguy 83 0.1% Brogdel 83 0.1% jolt_rush_soon 82 0.1% unknownlamer 77 0.1% mirleid 75 0.1% GhostOfTiber 68 0.1% Strom_Thurmond 67 0.1% DK4_Bloo 65 0% tweet 63 0% Phssthpok 61 0% boomi 61 0% r_rnq 60 0% horny_smurf 51 0% Linux_or_FreeBSD 47 0% TaxiCabJesus 43 0% superdiva 40 0% boxed 40 0% eavier 36 0% anaesthetica 36 0% th0m 35 0% xC0000005 34 0% Scrymarch 34 0% Zombie_Kitten 30 0% Michael_Crawfords_Third_Leg 27 0% Repeatible_Hairstyle 27 0% alevin 26 0% pharaoh 24 0% Zombie_Osama_bin_Laden 23 0% Abominable_Abitur 21 0% Masquitos_Davis 20 0% BirthControlSoothsayer 20 0% me0w 20 0% mr_strange 19 0% Have_A_Nice_Day 18 0% GrubbyBeardedHermit 18 0% ensignyu 17 0% circus29 16 0% exolete 15 0% They_Came_from_Planet_9 15 0% sick_puppy 14 0% Glock_To_The_Head 13 0% Special_K 12 0% alba 11 0% Private_Seal_Club 11 0% twopumpchump 10 0% Sturmbannfuehrer_von_Braun 10 0% Wain 10 0% lemonjuicefake 9 0% Altus 9 0% Ruston_Rustov 9 0% The_Pope_Himself 9 0% anonimouse 9 0% circletimessquare 9 0% dtylercade 9 0% power_guido 8 0% Phil_Elmore_The_Martialist 8 0% horsecock_sodomy 8 0% kpaul 8 0% mike4troll 7 0% Booji_Boy 7 0% Novelty_Account_No_59671 7 0% Robot_Buys_A_Pop 7 0% V1m 7 0% blackbart 7 0% dubya 7 0% infernalmajesty 7 0% psycho 6 0% Clippy 6 0% Fujitsu_Corp 6 0% LocalH 6 0% Mocknbird2 6 0% Morvael 6 0% New_User_Signups_Open_Again 6 0% The_Belief_Doctor 6 0% drwav 6 0% ogfomk 6 0% zombie_steve_jobs 5 0% Calalily 5 0% Jesus_Christ_Himself 5 0% Kingmaker 5 0% mybostinks 5 0% regeya 5 0% stuaart 4 0% Corwin06 4 0% Orion_Blastar_Reborn 4 0% R_Mutt 4 0% SaintPort 4 0% coq 4 0% craw4ord 3 0% AlwaysAnonyminated 3 0% BJH 3 0% BadDoggie 3 0% BoredWitless 3 0% CheeseburgerBrown 3 0% ChemicalCastrator 3 0% Creative_Hate 3 0% FEKLAAR 3 0% I_am_Spartacus 3 0% JulianConrad 3 0% Psycho_Dave 3 0% The_Vast_Right_Wing_Conspiracy 3 0% Wexel_Pixel 3 0% billt 3 0% givemegmail111 3 0% glimmung 3 0% it_certainly_is 3 0% lilburne 3 0% rrmq 3 0% spooky_wookie 3 0% spunk 3 0% walwyn 3 0% zombie_Fred_Phelps 3 0% zombie_HollyHopDrive 2 0% Clock_Nightstand_Joke 2 0% Mr._Penguin 2 0% Nursie 2 0% Trollaxor 2 0% bankebiff 2 0% blaaf 2 0% chakmol 2 0% fenugeek 2 0% make_u_friend 2 0% muyuubyou 2 0% paxman 2 0% zombie_lonelyhobo 1 0% Ayn_Rand_the_Objectivist 1 0% C_Montgomery_Burns 1 0% ClaytonBigsby 1 0% Elija 1 0% MazdaBongo 1 0% My_Dupe_Account 1 0% Naysayer 1 0% Niqht_In_White_Satin11 1 0% Ouija_Board 1 0% Owa_Tajer_Kyam 1 0% Sgt_York 1 0% Smiley_K 1 0% Soviet_Russian 1 0% The_Real_Orion_Blastar 1 0% Tjalfi 1 0% Wolf_At_Large 1 0% Yori 1 0% Zombie_Buddha 1 0% achievingfluidity 1 0% aphasia 1 0% basj 1 0% buford 1 0% errgh 1 0% fleece 1 0% imrdkl 1 0% jd 1 0% jew 1 0% jmzero 1 0% kuro5hinatportkardotnet 1 0% levesque 1 0% ljt 1 0% magic_curl 1 0% my_gold_bling_shines 1 0% noscript 1 0% slice2 1 0% sudogeek 1 0% toki 1 0% trane 1 0% twocents 1 0% undermyne 1 0% unmovic 1 0% walk 1 0% zenofchai ---------- For the full user list see: the-k5-user-list.txt For the frequency list of all words used on k5, see: the-k5-frequency-list.txt For the "official k5 dictionary" (which contains many non-words) see: the-official-k5-dictionary.txt (it contains 93,620 words) For the list of words distinctive to each user see: distinctive-words-per-user.txt (the columns are: frequency, word, user) For the full k5 corpus, see: all.txt For a quick explanation of the format of "all.txt" see: quick-guide-to-all.txt For the full corpus, but split into one file per user, see: the-k5-users-archive.tar.gz For the 6k raw diaries still in their original html format see: the-6k-diary-archive.tar.gz For the index file of those 6k diaries, see: index-diaries.txt