HPCC Systems Query Language , ECL , is easy to learn. If you are a software developer you will have no problems learning it. Go through this slide to see a quick example.
HPCC Systems - ECL for Programmers - Big Data - Data Scientist
1. HPCC Systems - ECL Intro
Big Data Querying Made EZ
By Fujio Turner
Enterprise Control Language
explained for Programmers
@FujioTurner
2. Comparison
Block Based File Based
JAVA C++
Petabytes
1-80,000 Jobs/day
Since 2005
Exabytes
Non-Indexed 4X-13X
Indexed: 2K-3K Jobs/sec
Since 2000
? ? ? ? ? ?
Thor Roxie
3. What Is ECL?
ECL (Enterprise Control Language) is a C++ based query
language for use with HPCC Systems Big Data platform.
ECLs syntax and format is very simple and easy to learn.!
!
Note - ECL is very similar to Hadoop’s pig ,but!
more expressive and feature rich.
4. Comparing ECL to General Programming
In this presentation you will see how in ECL loading and
querying data is just like reading and finding data in a
plain text file.!
general programming (general common logic)!
vs.!
ECL
General Code HERE ECL Code HERE
General ECL
5. Example Text File
Name State Age
Kevin CA 45
Mark MI 27
Sara FL 64
Customer Data May 2010
~/cdata_2010.txt!
example file name
= ~/hpcc::cdata_2010.txt
ECL example file distributed in HPCC cluster
6. Opening File: general programming vs ECL
d = fopen(‘~/cdata_2010.txt’)
File Location
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
General ECL
7. Opening File: general programming vs ECL
d = fopen(‘~/cdata_2010.txt’)
File Location
Open File Function
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
General ECL
8. Organizing: general programming vs ECL
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data(d) by Row
Kevin CA 45
Mark MI 27
Sara FL 64
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
General ECL
9. Organizing: general programming vs ECL
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
Split Data(d) by Row
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
Use This Schema on this file!
to Give Structure to Data
Kevin CA 45
Mark MI 27
Sara FL 64
General ECL
10. Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
General ECL
11. Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
Filter Data By
General ECL
12. Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
Filter Data By
Output
General ECL
13. Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
sara := d(Name = ‘Sara’);
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
Filter Data By
Output
General ECL
14. Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
sara := d(Name = ’Sara’);
OUTPUT(sara);
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
Filter Data By
Output
General ECL
15. Find “Sara” & Older then 50: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = row.split(“ ”)!
! if(new row[0] == ‘Sara’ and row[2] >50){!
! ! print ”Found Sara”!
! }!
}
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
sara := d(Name = ‘Sara’ AND Age > 50);
OUTPUT(sara);
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
General ECL
16. ECL is EZ
•Make your own functions & libraries in ECL.!
•Modularize your code with “Import”: reuse old code
Machine Learning Built-in
http://hpccsystems.com/ml
17. ECL Plugin for Eclipse IDE
http://hpccsystems.com/products-and-services/products/plugins/eclipse-ide
18. ECL + Others Languages
ECL is C++ based so all your C/C++ code can be used in ECL.!
&!
Use other languages and methods like below to query too.
20. For More HPCC “How To’s” Go to
Query with
Plain SQL
http://www.slideshare.net/hpccsystems/jdbc-hpcc
or SQL TO ECL
http://www.slideshare.net/FujioTurner/meet-up-sqldemopp
21. Watch how to install
HPCC Systems
in 5 Minutes
Download HPCC Systems
Open Source
Community Edition
http://hpccsystems.com/download/
http://www.youtube.com/watch?v=8SV43DCUqJg
or
Source Code
https://github.com/hpcc-systems