Visual Flow
GET STARTED. Build ETL Pipelines Today
An ETL tool powered by Apache Spark on Kubernetes
We developed Visual Flow as a low-code solution for developers looking to leverage the ETL benefits of Spark without learning a programming language.
Try Visual Flow on AWS or GitHub
Apply for online demo
SPARKING ETL
A Low-Code Option for Building and Scheduling Spark Jobs.
| We developed Visual Flow so you can jump-start Spark without writing a single line of code. |
Until now, you’ve relied on a single-node ETL system. Now, your data is growing faster than your current engine can manage it. Distributed ETL engines like Spark offer tremendous upside. But when faced with coding jobs and pipelines manually, you run into a dilemma:
 | You don’t have time to learn coding languages |
| You can’t afford to hire external programmers |
| You don’t want to master an ETL tool, a build tool, and an orchestration tool |
| Your data continues to multiply out of control |
The difficulty, expensiveness, and inefficiency of these options make Spark ETL seem impossible.
What You Will Get
By using Visual Flow, you will gain access to Spark ETL, transforming your data reserves into actionable wisdom.
Simplify ETL with a Drag-and-Drop Interface
Forget about Python, Scala, and Java, and embrace low- to no-code ETL on an easy-to-use interface.
Under-the-Hood Build and Orchestration
Visual Flow incorporates Argo Workflows, Kubernetes and the Spark processing engine. By mastering Visual Flow, you reap the benefits of all three software products.
Scale at the Speed of Data
Chain functions into jobs, then chain jobs into pipelines to transform raw data into Business Intelligence.
Technical Features
Visual Flow is ETL, simplified. It allows users to leverage the scaling and efficiency capabilities of Spark ETL through an intuitive drag-and-drop interface.
Visual Flow includes:
- Stage palette featuring nine discrete data transformation functions
- Slack integration for user notifications
- Support for data sources, including
● Relational DBs: IBM DB2, PostgreSQL, Oracle, MySQL, MSSQL, Amazon Redshift, ClickHouse
● NoSQL DBs: Elasticsearch, Cassandra, Redis, Mongo
● Cloud Object Storages: Amazon S3, IBM Cloud Object Storage
● Others: local files, dataframe
- Ability to externally invoke Visual Flow pipelines using POST requests
- GitHub and GitLab user directory support
Our Guiding Principles
Go Open Source
Visual Flow is an open source product that is based on other open source products.
Scale Optimally
Virtually unlimited horizontal scalability is provided by Spark.
Design Once — Run Anywhere
Once designed, deploy your ETL on any Kubernetes cluster anywhere.
Design Visually
Think of your data flows instead of the programming language syntax.
HOW TO USE VISUAL FLOW
Visual Flow is a low-cost, low-code, open-source product. Its portability, flexibility, and multi-cloud capability make it a powerful tool for a range of data projects.
- Load Data from transactional sources into the data lake
- Load application logs and structured files from a cloud object storage to the data lake
- Cleanse and transform information inside the data lake
- Load data from the data lake into the data warehouse
WHO VISUAL FLOW IS FOR
We developed Visual Flow for anyone looking for a convenient, intuitive ETL solution. Whether your data is multiplying faster than your single-server warehouse can process it, or you don’t have access to ETL coders for Spark, Visual Flow is for you.
- Set up Visual Flow in under 30 minutes
- Quickly design applications to transform large-scale, semi-structured datasets
- Turn a messy backlog of data into uniform data lakes and warehouses
- Intuitive, no-code interface
- Standalone solution for intuitive ETL
- Save money and time on training and/or new hires