--- title: "Steps and Shaping Data" description: "How items flow from step to step, how to write expressions and text templates, how to use data from earlier steps, and what each shaping step does: Set fields, Filter, Sort, Group and summarise, Formula, Date, Regular expression and the rest." section: "Automations" canonical: "https://banksync.io/docs/automations/automations-steps-and-data" specs: ["automations"] operationBindings: [{"capability":"automations","operation":"automations.nodes.search"},{"capability":"automations","operation":"automations.nodes.describe"},{"capability":"automations","operation":"automations.nodes.options"},{"capability":"automations","operation":"automations.nodes.test"},{"capability":"automations","operation":"automations.packages.list"}] --- Every step is handed a list of **items** and hands on a list of its own. A trigger hands on one item (the event, the row, the request); a reading step hands on what it read (every bank, a page of transactions); a shaping step changes, keeps or combines items; an acting step does something with each. This page is about the shaping steps and the expressions they take. ## Once per item, or once for the list A step either runs **once per item** (a text template is filled in for each transaction) or **once for the list** (a sort orders the whole list). The library says which, and a step that can do either shows **Change mode** in its Setup. Inside a loop, "the list" is the one iteration's items. Ports matter too. Most steps have one output, **main**. Some have a second: **Filter** hands the items it drops to **rejected**; **Approval** hands on **approved** and **rejected**; **JSON** and **CSV** send text they cannot read to an **error** port; a loop or branch can hand a failure on the same way. A wire from a second port is how a run carries on with the items that did not fit. ## Data from earlier steps A step's **Setup** shows **Data from earlier steps**: every step visible from this one, nearest first, each with its fields and a sample value. Clicking a field puts it into the setting you last clicked, as a pill, or at the cursor in a formula. Only steps on the path to this one are visible: a step cannot read a step in another branch or another parallel path. ## Expressions Any setting can be written as an expression instead of a fixed value: press the **ƒx** mark on the field. The language is the one table formulas use, with a few additions for automations. It is committed when you leave the field or press **⌘Enter** (Ctrl+Enter); **Esc** puts the saved text back. **What an expression can read** | Name | What it is | | -------------- | ------------------------------------------------------------------------------------------------------ | | `item` | The item this step is handling (once per item), or the list (once for the list). | | `items` | The whole list handed to this step. | | `index` | The position of `item` in the list, from 0. | | `node('name')` | What an earlier step gave, by the name under its **Notes → Advanced**: `node('read_balance').balance`. | | `trigger` | What the trigger handed on. | | `inputs` | What the run was started with (by hand, by a call, or by an API). | | `config` | The automation's values from **Settings**: `config.threshold`. | | `locals` | Values a **Remember values** step set in this scope. | | `run` | About this run: its id and when it started. | | `automation` | About this automation. | | `error` | Inside a **Try**'s handler: the failure being handled. | | `current` | Inside `map`, `filter` and the other list functions: the element being looked at. | **Paths and values.** Read a field with a dot: `item.account.name`, `trigger.data.bankId`. Read a list's element with brackets: `items[0]`. Write a list as `[1, 2, 3]` and an object as `{ name: item.name, total: item.amount }`. Text is in double quotes; numbers and `true` / `false` are bare. **Operators.** `=` and `!=` compare; `<`, `<=`, `>`, `>=` order; `+`, `-`, `*`, `/`, `%` and `^` do arithmetic; `and`, `or` and `not` combine conditions; `a ? b : c` chooses. `not` binds looser than a comparison, so `not item.pending = true` means `not (item.pending = true)`. **Functions.** The same library tables have, called as `name(args)` or with a dot: `sum(items.amount)` and `items.map(current.amount).sum()` are the same. Among them: `if`, `ifs`, `switch`, `empty`, `coalesce`; text — `concat`, `lower`, `upper`, `trim`, `length`, `contains`, `find`, `left`, `right`, `substring`, `replace`, `replaceAll`, `split`, `regexMatch`, `regexExtract`, `regexReplace`, `format`, `toText`, `toNumber`; numbers — `abs`, `round`, `roundUp`, `roundDown`, `floor`, `ceil`, `mod`, `pow`, `sqrt`, `min`, `max`; dates — `today`, `dateAdd`, `dateSubtract`, `dateBetween`, `formatDate`, `parseDate`, `startOf`, `endOf`, `fiscalYear`, `period`; lists — `map`, `filter`, `some`, `every`, `findItem`, `first`, `last`, `at`, `slice`, `reverse`, `sort`, `unique`, `compact`, `flat`, `includes`, `join`, `count`, `countValues`, `sum`, `average`, `median`, `mode`, `range`, `stddev`, `variance`, `earliest`, `latest`. There is no `random()`: use the **Make a new id** step, which records what it made so a replay sees the same one. **Text templates.** Text between backticks is filled in: `` `Balance of ${item.account.name} is ${round(item.balance, 2)}` ``. Without backticks it is plain text and nothing is filled in. An object in a hole renders as JSON. **Examples** ```text item.amount < 0 and not item.pending ``` ```text item.balance < config.threshold ? "low" : "ok" ``` ```text items.filter(current.category = "Fees").map(current.amount).sum() ``` ```text formatDate(dateAdd(today(), -7, "days"), "YYYY-MM-DD") ``` **Conditions as rows.** A yes/no setting (a filter's condition, a branch's "when") can also be built as rows — **Amount · is over · 100** — without writing the expression. ## The shaping steps All of these are in the **Shape data** group. They are pure: they read and change items in memory and never touch anything outside the run. | Step | What it does | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Set fields | Adds or changes fields on each item from expressions or fixed values. **Keep all** adds to the item; **keep only the set fields** hands on just those. | | Filter | Keeps the items for which the condition is true and sends the rest to **rejected**. Evaluated per item, with `item` and `index`. | | Sort | Orders the items by one or more keys (up to eight), ascending or descending. Text sorts by code point, so it sorts the same every time. | | Limit | Keeps the first or last *n* items. | | Group and summarise | Groups items by up to four keys and works out a count, sum, minimum, maximum, average, first, last or list for each group — one item per group. | | Formula | Works out one expression. Once per item (a new value for each) or once for the list (one value from all). | | Text template | Builds text from a template, once per item: the words between backticks with `${ }` holes. | | Combine lists | Puts two lists together: **one after the other**, or **side by side** (pairs by position into `{ a, b }`). Runs once both inputs have arrived. | | JSON | Reads a text field as data, or turns a value into JSON text, per item. Text that cannot be read goes to the **error** port. | | CSV | Reads spreadsheet-style text into items (one per row, with or without a header; comma, semicolon, tab or pipe), or writes the items out as CSV text. | | Hash | Adds a SHA-256 fingerprint of a value to each item, for spotting repeats or changes. Not for passwords. | | Date | Formats a date, adds to it, rounds it to the start of a minute, hour, day, week, month or year, or measures between two dates, per item, in the automation's time zone unless you say otherwise. | | Regular expression | Pulls the first match (or every match) of a pattern out of a value into a field, per item. Patterns run in linear time, so a pattern cannot hang a run. | | Repeat on dates | Makes one item per date of a schedule (every day, week, two weeks, month, three months, year or few days) from a start, for a number of dates or until an end date, keyed by name and date. | | Find repeating payments | Finds the payments that repeat in a list of transactions: how often, the next date and the usual amount, one item per repeating payment. | | Pay down debts | Plans paying off debts month by month (avalanche, snowball or your order) until each is zero: one item per debt per month, and a summary with each debt's payoff date. | | Skip repeats | Keeps the first item of each key and drops the rest — **during this run**, or **across every run** of this automation (remembered in the automation's state). | | Make a new id | Makes a new random id or secret token for later steps to use. | | Remember values | Keeps values, by name, for the steps after it in the same scope (`locals.name`). Every expression is worked out before any value is set. | | Label the run | Adds labels to the run so you can find it in history later. | | Read a saved value | Reads a value an earlier run saved — the last date handled, a cursor. Saved **on this step**, **for this automation** or **for the whole workspace**. | | Save a value | Saves a value for later runs to read. Workspace-wide state is shared by every automation and needs its own permission in the grant. | **Remember values**, **Label the run** and the shaping steps spend no step of a run's budget: a run counts the calls, expression operations, bytes and credits it uses and stops with a budget error if it reaches a limit. Shaping is free; reading and acting are what count. ## Finding a step The library's search box finds steps by what you type ("balance", "row", "notify", "csv"); **Show more** pages through the rest. The **More → Settings** panel also lists the kinds of step your workspace offers and how many of each. A step the workspace cannot run yet — an AI step in a workspace without a model, a mailbox step where mailboxes are not on — is not offered, and an imported automation that uses one is marked in **To do** for you to replace. > **Ask AI steps:** Where your workspace runs on BankSync's AI, the Ask AI group offers steps > that classify, extract from, draft, summarise, validate or answer a question > about your items, each spending your plan's credits. They are not offered in a > workspace without a model. ## Trust What a webhook sent, what a bank or table holds, what a saved value read back, and what an AI step wrote are **untrusted**: they may have been written by a stranger. A step that cannot be undone will not take its target (which table, which address, which automation) from untrusted data — the target must be a fixed value, a picker, a bound resource, a config value or the run's inputs. Shaping steps carry trust through: a field copied from an untrusted item stays untrusted.