# DSL *** ## Forming Expressions In order to retrieve data from terrakio, an expression must be sent with the request. This expression is a single string that specifies which datasets to retrieve and allows an array of calculations to alter and combine different datasets. An expression can span multiple lines however only datasets specified in the last line will be returned. In these docs, we will be using the dataset MSWX and its products (air\_temperature, precipitation, etc...) as examples. *** ## Datasets Datasets available in terrak.io can be retrieved by specifying the dataset name and one of its products in the following format: 
MSWX.air_temperature
Simply giving the dataset and product by itself will return all associated temporal data associated with it. The data retrieved will correspond to the dataset value for each pixel over the area and resolution defined in the request. Multiple datasets can also be returned by using a comma. ``` // Return two datasets MSWX.air_temperature, MSWX.precipitation ``` *** ## Temporal Selection In most cases, you will only need data for a specific time period rather than the entire dataset. This will also be more efficient in terms of time as less data will need to be returned to you. There are several ways to specify which time period you require, however, they all use the '@' operator to define the time period. Terrak.io will only return data for timestamps that have have data associated with them (null values won't be returned for timestamps with no data even if specified in the temporal selection) ### Keyword Specifications Keywords: \[day, month, quarter, year, hour] Certain keywords can be used to define which timestamps of a dataset to return. The keywords can all be used with each other to narrow down the time period. ``` //Single keyword (get all timestamps where day = 1) MSWX.air_temperature@(day=1) //Multiple keywords (get all timestamps in 2021/01/01) MSWX.air_temperature@(day=1, month=1, year=2021) //List arguments (get all timestamps in 2021/01/01, 2021/02/01 and 2021/03/01) MSWX.air_temperature@(day=1, month=(1,2,3), year=2021) //Range arguments (get all timestamps from 2021-2023) MSWX.air_temperature@(year=(2021:2023)) ``` ### Delta Specification Keywords: \[delta\_day, delta\_month, delta\_year] In some cases, there may be situations where you need a range of dates that span multiple time periods. To meet this use case, you can use a delta specification. A delta specfication takes a list or range of numbers, where each number specifies the offset to the time period to include. ``` // Include all timestamps in 2021/11, 2021/12 and 2022/01 MSWX.air_temperature@(month=12, year=2021, delta_month=(-1,0,1)) // Include all months between 2021/06 and 2022/05 (inclusive) using range syntax MSWX.air_temperature@(month=6, year=2021, delta_month=(0:11)) ``` ### ISO Timestamps ISO timestamps conforming to [ISO standard 8601](https://en.wikipedia.org/wiki/ISO_8601) can be used in the following way: ``` MSWX.air_temperature@(20190101T000000Z) ``` As defined by the ISO standard, the timestamp corresponds to date January 1st, 2019 at 12am. Multiple dates in the dataset can be retrieved by using a list of timestamps or a range. ``` //List of dates separated by ',' MSWX.air_temperature@(20190101T000000Z, 20200101T000000Z, 20210101T000000Z) //Range of dates using ':' MSWX.air_temperature@(20190101T000000Z:20210101T000000Z) ``` ### Chaining The DSL can recognise chaining of any type of temporal selection and functions the same way as AND. ``` //These three expressions are equivalent MSWX.air_temperature@(year=2021, month=2) MSWX.air_temperature@(year=2021)@(month=2) MSWX.air_temperature@(20210101T000000Z:20220101T000000Z)@(month=2) ``` *** ## Reduction Method Keywords : \[mean, median, sum, std, max, min, perc] Dimension Keywords : \[time, space] By using the reduction operation, you can reduce data over an area to a single value in a certain dimension. To apply a reduction, combine a reduction method and a dimension with ':' and use it as a function. ``` //Mean air temperature of every pixel for each timestamp in 2021 (space dimension) mean:space(MSWX.air_temperature@(year=2021)) //Mean air temperature of each pixel for every timestamp from 2021-2023 (time dimension) mean:time(MSWX.air_temperature@(year=(2021:2023))) ``` *** ## Group-by Method Keywords : \[mean, median, sum, std, max, min] Time Period Keywords : \[day, month, year, quarter] The DSL also exposes a slightly more powerful reduction in the time dimension with the ability to group timestamps by a specific time period. To apply a group-by operation, the '#' operator can be used after a temporal selection or dataset in the form "#(time\_period, method)". ``` //Get the mean air temperature of each month in 2021 MSWX.air_temperature@(year=2021)#(month, mean) ``` ### Server-side Aggregation Doing reduction operations on datasets with many timestamps can often be an expensive procedure due to the sheer size of the data. To make this process more efficient, the DSL exposes the ability to do group-by reductions on the data-server side by moving the '#" operator to after the dataset name, instead of after the temporal selection. ``` //Get the mean air temperature of each month in 2021 (server side aggregation) MSWX#(month, mean).air_temperature@(year=2021) ``` *** ## Functions Keywords : \[ sin, cos, log, abs, clip, strip] A few predefined functions are provided natively, however, custom functions are continually added to the [function registry](https://github.com/HaizeaAnalytics/function-registry). Arguments of the functions can either be ordered or defined with a keyword. Argument types accepted are floats, strings, booleans and lists of floats/strings/booleans. ``` // Remove the time dimension in the final output (single func arg) strip(MSWX.air_temperature@(year=2021, month=3, day=1)) // Clip the values of the dataset outside of 0-10 (ordered func args) clip(MSWX.air_temperature@(year=2021), 0, 10) // Rolling mean with a window of 3 (keyword func args) rolling_mean(MSWX.air_temperature@(year=2021), window=3) ``` *** ## Variables Some expressions can be quite verbose if complex calculations in many steps are needed. To make expressions more clear, variables in multi-line expressions can be used to hide away complexity from the final expression in the final line. 
//Multiline expression using user defined variables
air_temp = MSWX.air_temperature@(year=2021)
r_mean = rolling_mean(air_temp, window=3)
r_mean
The following expression also has the added benefit of returning a Dataset that has the value column labelled as r\_mean rather than the default var\_0. New lines can be marked as '\n' if the application you are using to send a request does not support multi-line strings. ### Reserved Keywords Keywords : \[resolution, area] Some variables are reserved by the DSL to provide useful information to use within expressions. Avoid overriding these variables with your own definitions. *** ## Arithmetic Basic Operators: \[+, -, \*, /, ^] Logical Operators: \[<, > , <=, >=, ==, !=, |, &] Basic arithmetic operations can be used between datasets, variables and even regular floating numbers aswell. ``` // Add 1 to every value in the dataset. air_temp = MSWX.air_temperature@(year=2021) air_temp + 1 // 1 (True) if value is above 10, otherwise 0 (False) air_temp = MSWX.air_temperature@(year=2021) air_temp > 10 ``` *** ## Masking Masking can be used on datasets/variables to mask out values according to boolean conditions. Values that don't meet the condition will be set to NaN. ``` // Mask out dataset values below 20 air_temp = MSWX.air_temperature@(year=2021) air_temp[air_temp>=20] // Chaining conditions with '&' to mask out values below 20 and above 30 air_temp = MSWX.air_temperature@(year=2021) air_temp[air_temp>=20 & air_temp<=30] ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://test-341.gitbook.io/terrakio/terrak.io/dsl.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.