Dynamic Workflows
Isolate and reuse workflow logic with Dynamic Workflows
Last updated
Isolate and reuse workflow logic with Dynamic Workflows
Last updated
Building new Cascade workflows can be even easier with Dynamic Workflows. Sometimes workflow tool steps can be repetitive or convoluted within the Canvas. Dynamic Workflows allow you to clean up your canvas by isolating groups of tool steps and placing them in a separate, reusable workflow.
Creating a Dynamic workflow couldn't be easier. Simply scroll to the "Dynamic Workflows" section of your right navigation bar and drag the "Add Dynamic Input" tool onto the canvas. Then connect the dynamic input tool to a sample dataset that looks like the expected datasets for Dynamic workflow.
Pro tip: For your sample dataset, you can use the New Table tool to define the schema and data types that you want in your dynamic workflow.
Once your dynamic input is on the canvas, you can build out the workflow with as many tool steps as necessary. Use any tools available on the canvas after your dynamic input, including additional dynamic workflows. Dynamic workflows can be nested within each other to create multiple levels of dynamic logic.
Note: Be careful not to create circular dependencies within Dynamic Workflows, as they will create never ending loops and your workflow with fail.
When you're done building out the steps of the Dynamic Workflow, drag the "Add Dynamic Output" tool onto the canvas and connect it to your final step. Everything in between the square input and output tools will be run when this Dynamic Workflow is used.
Dynamic Workflows are extremely powerful, but can only be put to use if Deployed. When creating or updating workflows while working in "Draft" mode, don't forget to click the orange "Deploy" button in the top right corner of your canvas so that dependent workflows have the most up-to-date logic. If you are working in "Live" mode, your work will automatically be deployed.
Dynamic Workflows can be valuable in many different use cases, but are especially helpful when trying to cleanup and transform multiple datasets with the same schema.
Below we have an example Dynamic Workflow called "Crypto Avgs". We can see that the Dynamic Input takes in an example cryptocurrency_prices dataset, executes a Validate Schema tool against a template file, transforms the data with a Pivot tool, and finally outputs our new data with the Dynamic Output.
Then, in a New Workflow, we can use the "Crypto Avgs" Dynamic Workflow to transform multiple Crypto_Prices files in just one step each. To access the Dynamic Workflows, scroll down to the "Dynamic Workflows" section in the right navigation bar and click on "Link to Workflow". All deployed Dynamic Workflows will show up in that list. Simply drag out the Dynamic Workflow you want to use and connect it to tool nodes as if it were any other tool.
If you don't see your Dynamic Workflow in the "Link to Workflow" list, it likely has not been deployed yet. Jump back into your Dynamic Workflow and deploy the most recent version and then refresh your New Workflow to see it in the list.
