2. What is Dataweave
• Any product to be powerful you need a built in language which can be used
to manipulate and transform data and this is more true for the Integration
platforms. Some of the languages like Java, Javascript, python, Ruby, Perl
and the list goes on give this power for various integration platform as a
external language or native language in their frameworks.
• Dataweave is the native language for Mule ESB platform giving the power
to transform data with huge array of operators and data types. Major
advantage of dataweave language is its processed by Dataweave
Transformation engine built into the Mule Runtime environment thus
making it inherently optimized to run the various build in functions and
operators taking away the complexity of optimization from the developer.
3. Dataweave structure
• Dataweave version dw1.0 is supported in mule runtime 3.8.5 with more powerful dw2.0
coming with mule runtime 4.0 (Currently beta version)
• Anypoint studio has 3 panes for DW transform left:incoming message, middle:
Transformation and Right: Expected output
• Transformation pane is where we apply various transformation rules with the help of
operators and functions. This is divided into 2 parts Header and Body separted by --- (3
dashes). Example given below:
%dw 1.0
%output application/xml
---
{
user: {
name: payload.user_name,
lastName: payload.user_lastName
}
}
4. Dataweave datatypes
• There are 3 data types which will be the result of dataweave expression
Simple Type: (Strings and Numbers): ‘Hello World’
Array Type: Represented as a sequence of value expressions [1,2,3]
Object Literal: Represented as a comma separated sequence of key: value pairs surrounded by curly brackets { }.
{
Company: ‘Mulesoft’,
Approach: ‘Api Led’,
Areas: [‘SAAS’,’ SOA’,’APIs’]
}
• Dataweave object is a sequence of key:value pairs with key a string without
quotes and value one of the 3 types. It does not matter if your incoming
data is JSON or XML or CSV or Java, as these will always be resolved to
DataWeave simple types, arrays or objects which has simple JSON like
syntax.
5. Dataweave header
• Dataweave header contains directives for the transformations in body giving example for
each directive type.
DataWeave version: %dw 1.0
Output type: %output application/xml
Input type: %input payload application/xml
Namespaces: %namespace ns0 http://mulesoft.org/tshirt-service/
Constants: %var conversionRate=13.15
Functions: %var toUser = (user) → {firstName: user.name, lastName: user.lastName}
• Currently supported output types:
application/java
application/csv
text/csv
application/json
text/json
application/xml
text/xml
text/plain
application/dw
6. Dataweave Expressions
• There are 7 basic expression types give below:
Literal : Literally from input
Variable reference: Variables are referenced (flowVars,session vars and record vars)
Semi-literal: Mixing the literal with transformation expressions.
Selector: Selectors are applied to objects or arrays.
Function call : We can declare functions that are reusable throughout the entire transformation using the %function <function-name>
<function-body> header
Flow Invocation: Done through lookup function call (a: lookup("mySecondFlow",{b:"Hello"})) and this can have a call to connector as well.
Compound: Real power of the language when you combine these expressions together using operators.
• Simple values can be of the following types:
String: Double quoted ("Hello") or Single quoted ('Hello')
Boolean: Literals true or false
Number: Decimal and Integer values are supported (ex: 2.0)
Date: IS0-8601 enclosed by "|" (ex:|2003-10-01T23:57:59Z|)
Regex: Regex expression enclosed by "/" (ex:/(d+)-(d+)/)
7. Dataweave continued
• Variables
Constants: We can define constants as directives
Scoped Variables: To declare a variable in the DataWeave body, the following syntax is supported:
using (<variable-name> = <expression>) and it must be written before defining the contents of the literal that it exists in.
using (x = 2) 3 + x
• Expressions
Operators : An operator applies a specific logic/transformation over a data-structure an example below
name: upper "mulesoft“
Selectors: A selector allows for the navigation and querying the multiple levels of a data-structure to reference
users: payload.users.*user
• Flow Control Expressions
{ currency: "USD“ } when payload.country == "USA“ otherwise { currency: "EUR“ }
{ currency: "EUR"} unless payload.country == "USA"otherwise { currency: "USD"}
Default: Assigns a default value in case no value is found in the input field.
Pattern Matching: DataWeave supports four different types of patterns as part of pattern matching
literal
type/traits
regex
expression
8. • Understand how DW will parse the inputs and present it to the transform engine helps navigate some of the
implicit type conversions going on as well as better understand the data structure being traversed in your map.
Dataweave Parsing
%dw 1.0
%output application/dw
---
(
payload
)
<ns0:invoiceResponse
xmlns:ns0="http://ns0.com.au/api/invoices">
<ns0:invoices>
<ns0:invoice>
<ns0:date>2016-01-
01T12:00:00</ns0:date>
<ns0:billingHours>240</ns0:billingHours>
</ns0:invoice>
<ns0:invoice>
<ns0:date>2016-01-
02T12:00:00</ns0:date>
<ns0:billingHours>540</ns0:billingHours>
</ns0:invoice>
</ns0:invoices>
</ns0:invoiceResponse>
%dw 1.0
%namespace ns0
http://ns0.com.au/api/invoices
---
{
ns0#invoiceResponse: {
ns0#invoices: {
ns0#invoice: {
ns0#date: "2016-01-01T12:00:00",
ns0#billingHours: "240"
},
ns0#invoice: {
ns0#date: "2016-01-02T12:00:00",
ns0#billingHours: "540"
}
}
}
}
9. DataWeave Memory Management
• Dataweave uses systems memory while processing transformation unless threshold
(default 1572864) is reached on which it uses the systems hard disk as buffer. This
value refers to memory usage of each individual Transform Message component, not
to an aggregate of all the ones in the project.
• Threshold can be changed adding system property com.mulesoft.dw.buffersize and
assign it the number (in bytes) of your new threshold. System properties may be
defined in several ways, for example by editing the mule-app.properties file located in
your project’s src/main/app folder, see system properties for more details and more
ways you can set these. This affects the entire Mule application.
• Immediate vs Deferred Execution: By default, DataWeave processes the
transformation of a message as soon as the component is called out in the flow, you
can change this behavior so that the DataWeave transformation returns a
WeaveOutputHandler which is only processed when read by another component. This
handler is capable of deferring writing the Mule Message’s payload until there is a
stream available to write it to. This allows for the DataWeave output to remain
outside of the heap as processing continues on other components in the flow.
10. Operators in Dataweave
• Map: Returns an array as a result of applying the transformation function on each of
the elements
• Map Object: Processes both keys and values as a tuple instead of only values of object
(Map)
• Pluck: Pluck pulls a value/object and maps into an array.
• Filter: Returns an array that contains those that pass the criteria specified in the
lambda.
• Remove: Removes a member specified by index and returns an array
• Remove by Matching Key and Value: Remove from array specified by key:value pair.
• AND: Boolean and to link multiple conditions.
• OR: Boolean or to link multiple conditions.
• IS: evaluates a condition and validates to true/false.
• Concat: Append operation acts on numbers and strings according to data types.
• Contains: Evaluates to true/false matching value in an array for a given condition.
• Type Coercion using as: simple types can be coerced to string/number/date/object.
11. Operators in Dataweave
• Type Of: Returns type of provided element.
• Flatten: passed in array of arrays flattened into single array
• Size Of: Returns number of elements in array.
• Array Push: pushes a new element into end of array.
• Remove from Array: Removes a matching element.
• Remove Matching from Array: Removes elements matching passed in array.
• Average of Array: Average of all values in array.
• Reduce: Reduces the array using just 2 parameters accumulator ($$), and the
value ($).
• Join By: Merges an array into single element.
• Split By: Opposite of Join By.
• Order By: Returns the array according to value returned by lambda.
• Group By: Partitions array into object containing arrays based on lambda invoked
with two parameters: index and the value.
12. • Distinct By: Returns unique values from array that may have duplicates.
• Zip Arrays: Two or more lists are merged into single list of consecutive tuples.
• Unzip Array: Opposite of zip.
• Replace: Raplaces string with another.
• Matches: Matched a string with regular expression and returns true/false.
• Starts With: Returns true/false if string starts with provided with substring.
• Ends With: Returns true/false if string ends with provided with substring
• Find: Given a string returns an array with index position.
• Match: Match returns an array that contains the entire matching expression, followed
by all of the capture groups that match the provided regex.
• Scan: Each match is returned as an array that contains the complete match, as well as
any capture groups there may be in your regular expression.
• Similar: Evaluates and returns true/false if 2 values are similar regardless of types.
• Upper: Returns upper case of passed string.
Operators in Dataweave
13. • Lower: Return lower case of passed in string.
• Camelize: Returns provided string in camel case.
• Capitalize: Returns with starting letter as capital.
• Dasherize: Replace space and _ with -.
• Underscore: Opposite of Dasherize.
• Pluralize: Returns plural of passed in string.
• Singularize: Returns singular form of passed in string.
• Trim: Remove spaces at start and end of string.
• Substring using .. or to: Extracts a set of characters out of a string, based on
the position. Passed in first and last character in [].
• Ordinalize: Returns provided numbers as ordinals.
Operators in Dataweave
14. • Sum: Addition of numbers (a+b)
• Minus: Subtraction of numbers (a-b)
• Multiply: Multiplication (a*b)
• Division: Division (a/b)
• Max: Returns highest number in an array/object.
• Min: Returns lowest number in an array/object.
• Round: Rounds value of number to nearest interger.
• Sqrt: square root of a provided number.
• Pow: power of a number: a pow b (a to the power of b)
• Ceil: Rounding of number upwards.
• Floor: Rounding of number downwards
• Abs: Absolute of a number (abs -2=2)
• Mod: Remainder after division
• Now: Returns datetime object with current date and time.
Operators in Dataweave (Math Operations)
15. • Several operators that deal with date related types which include date, time, localtime,
datetime, localdatetime, period, timezone.
• Extraction of particular time unit is possible from a date examples below:
• a: |2003-10-01|.day = 1
• b: |2003-10-01|.month=10
• c: |2003-10-01|.year =2003
• d: |2003-10-01T23:57:59Z|.hour=23
• e: |2003-10-01T23:57:59Z|.minutes=57
• f: |2003-10-01T23:57:59Z|.seconds=59
• g: |2003-10-01T23:57:59-03:00|.offsetSeconds=-10800
• h: |23:57:59Z|.hour=23
• i: |23:57:59.700|.nanoseconds=700000000
• j: |23:57:59.700|.milliseconds=700
• k: |2003-10-01T23:57:59Z|.dayOfWeek=3
• l: |2003-10-01T23:57:59Z|.dayOfYear=274
• m: |P3Y2M10D|.years=3
Operators in Dataweave (DateTime Operations)
16. • Shift Time Zone: Shift a date time to the specified timezone.
• Append Time: append a date to a time (or localtime) object so as to provide a
more precise value.
• Append Time Zone: Appends a time zone to a date type value.
• Adding a Period of Time: Add or subtract a period of time from a given date or
time type object.
• Subtracting a Period of Time: subtracting time periods from a date or time type
object.
• Subtracting two Dates: Subtracting one date or time type object from another
Operators in Dataweave (DateTime Operations)
17. • Since the variables and functions defined in transform component has a limited
scope confined that component only we run into limitations where we want to
define a function which can be reused across many transform components. This
can be overcome in 2 ways one is defining a global function in config xml and
other is create a wev file(needs to be in class path) add the functions and
variables to be reused. We will discuss the implementation details in next slides.
• Mulesoft provided components like set variable to declare a flow variable which
can be used in later parts of the mule flow however when we need to set many
variables at once, flow would look very long so a better way to do this is define
the variable/session variable or property in dataweave transform component
which increases the readability of the code.
• Dataweave transform component also gives the option creation multiple
payloads through variable declaration discussed above and we can generate any
number of payloads we desire to and later send the appropriate payload to any
of the destinations.
Global functions for Dataweave
