Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
A Hacking Toolset
for Big Tabular Files
2016-12-05 Toshiyuki Shimono
1IEEE Bigdata 2016, Washington DC
Uhuru Corp.
Tokyo, ...
A Hacking Toolset
for Big Tabular Files
2016-12-05 Toshiyuki Shimono
2IEEE Bigdata 2016, Washington DC
• Nowadays, so much data.
• Then, how do we
decipher/analyze data?
3IEEE Bigdata 2016, Washington DC
The Difficulties :
1. Deciphering / Understanding
given data.
2. Pre-Processing before
main-analysis, modeling.
3. Verifyi...
The reasons
Though Excel, R, SQL, Python, ..
are widely used,
1. Many common routines are not implemented
for data-deciphe...
IEEE Bigdata 2016, Washington DC 6
Almost no one can make a
computer program from scratch
if it combines more than 5 or 6
...
Thus, I created a hacking toolset:
1. Focusing on
tabular data (TSV, CSV as well),
with a heading line (optionally).
2. Ba...
Function Examples
IEEE Bigdata 2016, Washington DC 8
Coloring ; colorplus
• A CUI named “colorplus” puts color,
with various options :
-3 for number legibility, -4 for Asian n...
colsummary : grasping what each column is.
1st (white) : column number
2nd (green) : how many distinct values in the colum...
“colsummary" command
11
olympic.tsv :
The numbers of
medals Japan got in
the past Olympic
Games (1912-2016).
The biggest h...
“colsummary" command
12
number of
different values
Value
range
Numeric
average
Frequent
values
olympic.tsv :
The numbers o...
crosstable : 2-way contingency tables
• Almost impossible by a SQL query.
• Excel-pivot-table is an cumbersome alternative...
crosstable 2-way contingency table
Provides the
cross-table from 2
columned table
Add blue
color on “0”
Extract 3rd and 4t...
freq : How frequent each string appears?
Functionally equivalent to Unix/Linux “ sort | uniq -c ” .
Much more speedy in co...
freq : with options
For freq,
-f for sorting in frequency number,
-r for reversing the order,
-% for showing the ratio amo...
• Knowing the cardinalities
helps your data analysis task well. 
• Thus, determine them before
your main analysis work.
1...
How many elements overlap, and
how the inclusion-relations occur
are beneficial to understand data files.
Venn Diagrams
18...
venn2 (Cardinalities for overlapping 2 sets)
n.g19 u.g19
646,408
19
92
506
IEEE Bigdata 2016, Washington DC
venn4 (Cardinalities for complex conditions)
A B
C
D
20
Compare with the
15 numbers below.
This 10 x 10 table
also helpful...
cols : extracting columns
• Easier than AWK and Unix-cut .
• Can avoid the geo-locale problems unlike Unix-cut.
cols –t 2...
Useful CUI functions, more.
22
NAME FUNCTION DETAIL
xcol Table looking-up Similar to SQL-join, Unix-join, Excel-vlookup
sa...
CUI already developed.
As of 2016-08-18.
23IEEE Bigdata 2016, Washington DC
How each program created?
1. One meets a routine
that may need to be combined
from many existing functions.
2. Generalize ...
Comparisons
25IEEE Bigdata 2016, Washington DC
Comparison with existing software.
• Excel does not handle more than 1 million lines.
• R takes long time to load big data...
Comparison in detail.
Excel R Pandas SQL Hadoop Unix Our soft
Primary usage
Spread
Sheet
Statistics Matrix etc.
DB
manipul...
Application Examples
28IEEE Bigdata 2016, Washington DC
Ex-1) crosstable
1. About 3 million Twitter remarks are collected.
2. 2 columns of date and time are extracted and by usin...
Ex-2) silhouette (integrated histogram)
1. Suppose you are interested in the distribution
of the ratio of the follower num...
Further perspective :
Stage I
1. Core engine for tabular data hacking tools as CUIs.
2. Sophisticated policy/philosophy
ca...
Features and summaries
Useful functions for analyses in TSV.
Especially in deciphering and pre-processing.
Handles a hea...
Nächste SlideShare
Wird geladen in …5
×

A Hacking Toolset for Big Tabular Files (3)

Despite the existence of data analysis tools such as R, SQL, Excel and others, it is still insufficient to cope with today's big data analysis needs.
The author proposes a CUI (Character User Interface) toolset with dozens of functions to neatly handle tabular data in TSV (Tab Separated Values) files.
It implements many basic and useful functions that have not been implemented in existing software with each function borrowing the ideas of Unix philosophy and covering the most frequent pre-analysis tasks during the initial exploratory stage of data analysis projects.
Also, it greatly speeds up basic analysis tasks, such as drawing cross tables, Venn diagrams, etc., while existing software inevitably requires rather complicated programming and debugging processes for even these basic tasks.
Here, tabular data mainly means TSV (Tab-Separated Values) files as well as other CSV (Comma Separated Value)-type files which are all widely used for storing data and suitable for data analysis.

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

A Hacking Toolset for Big Tabular Files (3)

  1. 1. A Hacking Toolset for Big Tabular Files 2016-12-05 Toshiyuki Shimono 1IEEE Bigdata 2016, Washington DC Uhuru Corp. Tokyo, Japan https://github.com/ tulamili/bin4tsv
  2. 2. A Hacking Toolset for Big Tabular Files 2016-12-05 Toshiyuki Shimono 2IEEE Bigdata 2016, Washington DC
  3. 3. • Nowadays, so much data. • Then, how do we decipher/analyze data? 3IEEE Bigdata 2016, Washington DC
  4. 4. The Difficulties : 1. Deciphering / Understanding given data. 2. Pre-Processing before main-analysis, modeling. 3. Verifying the data. Those are tedious. But, why? 4IEEE Bigdata 2016, Washington DC
  5. 5. The reasons Though Excel, R, SQL, Python, .. are widely used, 1. Many common routines are not implemented for data-deciphering / pre-processing. 2. So, one needs to combine many functions. 3. Thus, testing/debugging takes a ridiculously large amount of time  5IEEE Bigdata 2016, Washington DC
  6. 6. IEEE Bigdata 2016, Washington DC 6 Almost no one can make a computer program from scratch if it combines more than 5 or 6 components. You need 10 or 20 times test processes usually! Thus, providing useful common functions should be developed in a bunch!
  7. 7. Thus, I created a hacking toolset: 1. Focusing on tabular data (TSV, CSV as well), with a heading line (optionally). 2. Based on UNIX-philosophy. 3. Covering many functions carefully. 4. Imposing swiftness in many aspects. 7IEEE Bigdata 2016, Washington DC
  8. 8. Function Examples IEEE Bigdata 2016, Washington DC 8
  9. 9. Coloring ; colorplus • A CUI named “colorplus” puts color, with various options : -3 for number legibility, -4 for Asian number system, -t number for column-wise coloring, -s regex for specifying character string to put color, -b color_name for specifying the color 9IEEE Bigdata 2016, Washington DC
  10. 10. colsummary : grasping what each column is. 1st (white) : column number 2nd (green) : how many distinct values in the column 3rd (blue) : numerical average (optionally can be omitted) 4th (yellow) : column name 5th (white) : value range ( min .. max ) 6th (gray) : most frequent values ( quantity can be specified. ) 7th (green) : frequency ( highest .. lowest, “x” means multiplicity ) When new data come, one should understand them. Enough information to go on to next step is provided by colsummary command. 10IEEE Bigdata 2016, Washington DC
  11. 11. “colsummary" command 11 olympic.tsv : The numbers of medals Japan got in the past Olympic Games (1912-2016). The biggest hurdle at data deciphering quickly goes away by colsummary command. IEEE Bigdata 2016, Washington DC
  12. 12. “colsummary" command 12 number of different values Value range Numeric average Frequent values olympic.tsv : The numbers of medals Japan got in the past Olympic Games (1912-2016). The biggest hurdle at data deciphering quickly goes away by colsummary command. IEEE Bigdata 2016, Washington DC Column name Frequencies highest .. lowest (with multiplicity)
  13. 13. crosstable : 2-way contingency tables • Almost impossible by a SQL query. • Excel-pivot-table is an cumbersome alternative. 13IEEE Bigdata 2016, Washington DC
  14. 14. crosstable 2-way contingency table Provides the cross-table from 2 columned table Add blue color on “0” Extract 3rd and 4th columns One may draw many cross-tables from a tabular data. The crosstable commands provides cross-tables very quickly. 14IEEE Bigdata 2016, Washington DC
  15. 15. freq : How frequent each string appears? Functionally equivalent to Unix/Linux “ sort | uniq -c ” . Much more speedy in computation. Note that many useful sub-functions are implemented on “freq”. So you can easily analyze in many ways related to counting numbers. 15IEEE Bigdata 2016, Washington DC “HTTP status code”
  16. 16. freq : with options For freq, -f for sorting in frequency number, -r for reversing the order, -% for showing the ratio among the whole, -s for cumulative counting, -= for recognizing the heading line (appeared in the previous page.) 16IEEE Bigdata 2016, Washington DC
  17. 17. • Knowing the cardinalities helps your data analysis task well.  • Thus, determine them before your main analysis work. 17 You can easily refocus your data analysis task not by the file names but by the numbers, after breaks. IEEE Bigdata 2016, Washington DC This is for a technique for data analysis people. The file sizes, the line numbers, column numbers carry you much imagination, especially Monday morning in the middle of data analysis projects.
  18. 18. How many elements overlap, and how the inclusion-relations occur are beneficial to understand data files. Venn Diagrams 18IEEE Bigdata 2016, Washington DC
  19. 19. venn2 (Cardinalities for overlapping 2 sets) n.g19 u.g19 646,408 19 92 506 IEEE Bigdata 2016, Washington DC
  20. 20. venn4 (Cardinalities for complex conditions) A B C D 20 Compare with the 15 numbers below. This 10 x 10 table also helpful. IEEE Bigdata 2016, Washington DC A B C D 2 163 929 79 4919 506 646164 4835
  21. 21. cols : extracting columns • Easier than AWK and Unix-cut . • Can avoid the geo-locale problems unlike Unix-cut. cols –t 2 ⇒ moves the 2nd column to rightmost.  cols –h -1 ⇒ moves the last column to leftmost. cols –p 5,9..7 ⇒ shows 5th,9th,8th,7th columns.  cols –d 6..9 ⇒ shows except 6th,.., 9th columns. -d stands for deleting, -p for printing, -h for head, -t for tail. 21IEEE Bigdata 2016, Washington DC
  22. 22. Useful CUI functions, more. 22 NAME FUNCTION DETAIL xcol Table looking-up Similar to SQL-join, Unix-join, Excel-vlookup sampler Probabilistic extraction Can specify random seeds, probability weight. keyvalues Key multiplicity Inspects many aspects of a file of “key + value”. kvcmp Check the KV relation When 2 KV files are given, check the sameness. colgrep Column-wise grep More useful than Unix-grep idmaker ID creation Second DB normalization denomfind Finding common denom.Guessing the whole number from the fractions. shuffler Shuffle lines Also an option with preserving the heading line. alluniq Check the uniqueness Also print out how many same lines. expskip Logarithmic extraction Useful for taking a glace at the first stage. eofcheck Check the end of file Important at the initial stage of data analysis task. dirhier Directory structure How many files on each depth. timeput Time stamp on each line When one line comes, time added on its head. IEEE Bigdata 2016, Washington DC
  23. 23. CUI already developed. As of 2016-08-18. 23IEEE Bigdata 2016, Washington DC
  24. 24. How each program created? 1. One meets a routine that may need to be combined from many existing functions. 2. Generalize the routine to be used widely that can be described within 2 words. 3. Design the user interface with option-switches which specifies additional parameters etc. 4. Try not to make too many programs, or necessary programs cannot be recalled. 24IEEE Bigdata 2016, Washington DC Although the produced programs are important for exhibition, how to make them is much important.
  25. 25. Comparisons 25IEEE Bigdata 2016, Washington DC
  26. 26. Comparison with existing software. • Excel does not handle more than 1 million lines. • R takes long time to load big data. • Python with pandas also takes long time to load. • SQL requires burdensome table designing. • Elaborate Unix/Linux usage is also insufficient. 26IEEE Bigdata 2016, Washington DC
  27. 27. Comparison in detail. Excel R Pandas SQL Hadoop Unix Our soft Primary usage Spread Sheet Statistics Matrix etc. DB manipulation Distributed DB File manipulations Initial deciphering and Pre-processing Ease of use ◎ ○ △ △ × ○ ○ Transparency × ○ ○ depends depends ○ ○ Forward compatibility △ △ Too new ○-◎ Too new ◎ Too new Processing speed × Skill required ○ △ ◎ in narrow scope ○ ○ Large size data handling × Skill required ○ specific ◎ specific ◎ High quantity file processing × △ -- -- ○ ◎ Small data handling ◎ ◎ ○ Need to know SQL grammar × ◎ ◎ Column selection (Alphabe tically) Name & Number Similar to R Column name Column name Column number (cut/AWK) Name/Number And also by range 27 ◎: very good ; ○ good ; △ moderate ; ×bad. IEEE Bigdata 2016, Washington DC
  28. 28. Application Examples 28IEEE Bigdata 2016, Washington DC
  29. 29. Ex-1) crosstable 1. About 3 million Twitter remarks are collected. 2. 2 columns of date and time are extracted and by using crosstable, they are counted. 3. Subsequently copied and pasted into Excel, they are also colored by “conditional formatting” by Excel. 4. Followings have become clear: differences between morning and afternoon, the downtime of data collecting server. 日付/時間 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 2013-12-10 32 157 153 156 139 161 170 182 315 356 276 221 256 2013-12-11 127 93 64 58 39 72 90 144 200 149 170 116 177 144 170 149 156 268 318 253 216 187 203 235 2013-12-12 136 86 68 63 46 61 64 115 177 134 130 121 463 582 423 442 703 1285 1394 1446 1444 1611 2023 1974 2013-12-13 1532 958 634 478 396 403 526 837 1122 1097 1457 1386 1988 1816 2037 3025 3662 4125 4551 5062 4655 4993 5576 2969 2013-12-14 2873 1626 1139 1018 529 682 807 1164 1655 2115 2536 2715 2868 3454 3720 3812 3611 3438 4202 5005 2557 3133 5031 4274 2013-12-15 3617 2232 1694 1330 1051 974 1173 1732 2590 3549 4942 5871 5306 3056 3571 5247 6666 6798 6964 2597 1710 3382 7879 6425 2013-12-16 4623 2741 1631 1304 927 1094 1185 2053 2597 2474 3415 3614 4580 4836 4187 4999 4115 4122 5957 6427 6328 6793 6545 5963 2013-12-17 4045 2405 1545 1189 1007 946 1237 2692 3532 3435 4178 4925 6158 5049 4756 4375 3890 4138 5090 6265 7628 7077 6639 6325 2013-12-18 4458 2807 1713 1404 1131 1051 1347 1954 2299 2624 3412 2707 1962 1773 1910 3851 5263 6359 7177 6583 6936 6644 6234 6119 2013-12-19 4637 3064 1739 1509 1141 1100 1394 2362 2994 3897 3999 4344 5260 6844 5092 4459 5706 6004 7407 3817 1672 4132 6727 5514 2013-12-20 5393 3082 1977 1973 1321 1462 1667 974 1222 1423 1818 2954 5261 4082 4889 6194 6355 6755 9276 6215 6275 6152 5849 5959 2013-12-21 4924 3247 1859 1637 1331 1414 1617 1249 1327 2072 2385 2784 2927 2630 2740 2916 2871 3139 6222 5042 4680 4828 4829 4633 2013-12-22 3838 2702 2073 1836 1417 1186 1427 1442 1633 2043 3376 4120 4346 4159 4473 4759 4450 4773 4962 6876 7718 7327 7472 6768 2013-12-23 5260 3316 2134 1776 1258 1231 1672 1299 894 1023 1195 2402 3516 3584 5172 7778 8700 8331 8030 6035 7712 7669 8084 7450 2013-12-24 7098 4150 2293 1774 1205 1189 1589 1476 1145 1733 2023 2678 4647 3835 4571 6430 6943 7864 8863 5181 6080 7125 8609 6496 2013-12-25 4905 2734 1695 1604 1377 1360 1580 1200 1289 1335 2341 4287 5532 4332 5077 6474 6952 7888 7702 3655 6388 9370 9115 8470 2013-12-26 7468 3877 2269 1959 1370 1398 1908 467 39 140 155 143 131 418 1081 2148 8863 8775 8934 8498 9086 8356 2013-12-27 6419 4015 1833 1627 1421 1361 1590 2511 3768 3996 5245 6196 7894 8098 7373 7497 7296 7324 8437 8120 8059 8384 8435 8277 2013-12-28 7678 4543 2619 2041 1587 1568 1726 2539 3578 4911 5945 7082 7563 7343 7916 7529 7863 8234 8165 8076 8528 8763 8670 7892 2013-12-29 6661 4372 2621 1823 1336 1224 1433 2290 3223 4477 5873 7296 7842 7614 7975 8060 8639 8402 8889 8398 8457 8320 9454 8700 2013-12-30 7607 4656 2765 2100 1528 1440 1635 2511 3997 5140 7044 7658 9435 9147 9147 10058 9956 10529 9852 9662 10644 11448 12848 12136 2013-12-31 10349 6155 3885 2703 1900 1803 2110 3586 5362 8051 10713 12249 13035 13001 13298 15048 15950 17235 17573 15644 16229 15365 14898 17461 2014-01-01 22849 15263 4149 3157 4218 4828 7216 15086 7594 12145 27222 2067 2511 2540 2478 4938 9088 14816 21263 7463 1861 4411 10241 9638 2014-01-02 8627 4964 2749 4744 3729 4168 6328 1915 1342 1889 2848 3233 3197 3612 4528 4449 7701 12565 20139 6596 4637 6998 8677 8949 2014-01-03 6857 5227 6151 3906 2678 2656 2899 2227 1378 2470 2706 3105 3573 3287 3754 7378 10992 11090 10936 5549 6432 8894 9887 8318 2014-01-04 6833 4113 2371 2687 2248 2016 2138 1689 1079 1437 1921 2448 2451 4279 7620 8104 8364 8326 7957 6164 5549 5233 5850 8926 2014-01-05 9757 6208 3823 2627 2073 1782 1854 695 419 511 661 1073 1976 2815 3173 3884 9464 11131 11139 9671 9997 10924 9716 10967 2014-01-06 7623 4818 2873 2159 1656 1480 1733 2948 3538 4550 5638 6605 9118 7973 7552 7691 7392 8170 7714 8308 8974 9368 10272 9641 2014-01-07 7025 4124 2604 1800 1488 1417 1647 1235 1434 2369 3146 3886 5686 6811 6426 6926 7317 8297 8453 6858 9441 9961 9958 8910 2014-01-08 7551 3997 2606 1837 1379 1248 1872 886 647 1022 1465 1828 2463 2138 1909 2163 2298 2832 2944 5583 7953 9266 10316 10737 2014-01-09 9161 4801 2999 2361 1871 1734 2094 3674 5135 5177 6758 7622 11370 10181 10548 12154 13235 13557 11017 4679 5650 6648 7858 7154 2014-01-10 5273 3243 1795 1349 1206 1020 1556 673 526 505 698 827 1039 996 1127 1232 2925 5981 10609 1481 29IEEE Bigdata 2016, Washington DC
  30. 30. Ex-2) silhouette (integrated histogram) 1. Suppose you are interested in the distribution of the ratio of the follower number and the following number of each Twitter account. 2. To properly understand the ratio distribution, stratifying according to the number of the follower of each account is also considered : ≧5000, ≧ 500 from the rest, ≧ 50 ditto, and the others. 3. The silhouette command given the data provides the PDF file as shown in right. 4. You can read that with 48% possibility, the account with ≧5000 followers has followers that are more than 2 times of the number of following. (On average, 95% twitter accounts does not.) 0% 25% 50% 75% 100% ◀ The command name “silhouette” comes from the image when various height people are aligned in the order of height. See the curve connecting the heads. 30IEEE Bigdata 2016, Washington DC The ratio of following number divided by followers number. This commands depends on R installation to draw graph. The cross grids are carefully designed so that one can easily read the numbers directly from the graph.
  31. 31. Further perspective : Stage I 1. Core engine for tabular data hacking tools as CUIs. 2. Sophisticated policy/philosophy carefully covering the functions necessary. 3. Simple GUI with Wx and Tk, possibly. Stage II 4. Parallel computation mechanism. 5. Full-fledged GUI. 6. API, with clouding services on the Internet. Stage III 6. Ubiquitous “analysis” deployment beyond spreadsheet and MapReduce model. 31IEEE Bigdata 2016, Washington DC
  32. 32. Features and summaries Useful functions for analyses in TSV. Especially in deciphering and pre-processing. Handles a heading line properly if it exists. Swiftness in computation, recalling, understanding. Well-designed interface, based on Unix philosophy. Can be used on any Perl5 installed machine.  The software partly appears on GITHUB repository. 32IEEE Bigdata 2016, Washington DC

×