Chapter 1: Introduction to Dataiku DSS Architecture

Welcome to the technical handbook for Dataiku Data Science Studio (DSS). If you are reading this, you likely already know that Dataiku is one of the most powerful "All-in-One" platforms for data science and machine learning. But as you move from being a beginner who clicks buttons to an intermediate user who builds scalable pipelines, you need to understand what is happening "under the hood."

Understanding the architecture of Dataiku DSS is the difference between building a project that works on your laptop and building a project that handles billions of rows in a corporate production environment.

1.1 The "Orchestrator" Philosophy

The most important thing to understand about Dataiku DSS is that it is primarily an orchestration layer.

Unlike older tools that required you to move all your data into their specific proprietary format, Dataiku is designed to be "data-resident." This means Dataiku prefers to leave the data where it is (in your SQL database, your Hadoop cluster, or your Cloud storage) and send the commands to the data, rather than pulling the data to the commands.

Key Insight: Dataiku DSS does not want to be your database. It wants to be the brain that tells your databases, clusters, and cloud services what to do.

1.2 The Three Main Nodes

In a professional enterprise setup, Dataiku isn't just one single application. it is divided into "Nodes" based on the stage of the data lifecycle.

1. The Design Node

This is where you will spend 90% of your time. It is the development environment where data scientists and analysts create "Flows," write code, and train models. It is optimized for interactivity.

2. The Automation Node

Once your project is finished, you don't want to run it manually in the Design node. You bundle your project and move it to the Automation Node. This is the production environment. It has no "Undo" button and is built for stability and scheduled execution.

3. The API Node

If you build a machine learning model to predict whether a customer will churn, you might need a website to ask that model for a prediction in real-time. The API Node hosts these models as high-availability web services.

1.3 The Dataiku Storage Model

How does Dataiku keep track of your data without "owning" it? It uses Connections and Datasets.

• Connections: These are defined at the administration level (e.g., a connection to a Snowflake warehouse or an AWS S3 bucket).
• Datasets: In the UI, a dataset looks like a table. In reality, it is just a pointer to a specific location in a connection.

Real-World Example: The Retail Bank

Imagine a bank with 50 million transactions stored in a PostgreSQL database.

1. The Admin creates a Connection called
   text

   Core_Banking_SQL

   .
2. The Data Scientist creates a Dataset in Dataiku called
   text

   Transactions

   .
3. When you open the
   text

   Transactions

   dataset, Dataiku sends a
   text

   SELECT * FROM table LIMIT 100

   query to PostgreSQL. Dataiku only stores a small sample in its memory to show you the preview, keeping the 50 million rows safely in the database.

1.4 The Compute Engines: Where the Work Happens

This is where the "Intermediate" knowledge kicks in. When you run a Recipe (a step in your data flow), Dataiku must decide which "engine" will do the heavy lifting.

1. The DSS Engine (Local)

The data is streamed through the Dataiku server itself. This is great for small datasets but becomes a bottleneck for millions of rows because it is limited by the RAM and CPU of the Dataiku server.

2. The SQL Engine (Pushdown)

If your input and output datasets are in the same SQL database, Dataiku doesn't pull the data out. Instead, it translates your visual recipe into a SQL script and sends it to the database.

Example of a generated SQL pushdown:

sql
/* Dataiku will generate something like this behind the scenes */
INSERT INTO "PROD_DATA"."CLEANED_TRANSACTIONS"
SELECT 
    CAST("transaction_id" AS STR),
    "amount",
    UPPER("merchant_name") as "merchant_name"
FROM "PROD_DATA"."RAW_TRANSACTIONS"
WHERE "amount" > 0;

3. Spark Engine

For massive datasets in HDFS or Cloud Storage, Dataiku leverages Apache Spark. It translates your logic into Scala or Python (PySpark) and distributes the work across a cluster.

1.5 Managing Code in the Architecture

While Dataiku is famous for its visual recipes, intermediate users often use the Python API. This allows you to interact with the architecture programmatically.

For example, if you want to programmatically list all datasets in your project to check if they have grown too large, you would use the

dataiku

python
import dataiku

# Get a handle on the current project
client = dataiku.api_client()
project = client.get_project(dataiku.default_project_key())

# List all datasets and their underlying type
datasets = project.list_datasets()

for ds in datasets:
    name = ds['name']
    connection_type = ds['type']
    print(f"Dataset: {name} | Type: {connection_type}")

This code doesn't interact with the data—it interacts with the Design Node's metadata. This is a crucial distinction in Dataiku architecture.

1.6 Security and Governance

Because Dataiku sits on top of your existing infrastructure, it acts as a security gateway.

• User Impersonation: In a secure setup, when User "Alice" runs a query, Dataiku doesn't run it as "Dataiku Service Account." It uses Alice's actual credentials (via Kerberos or LDAP) to run the query in the database.
• The Flow: The visual "Flow" isn't just for show. It acts as a Lineage Map. If a column name changes in your source database, you can look at the architecture of your Flow to see every single downstream recipe that will be affected.

1.7 Practical Tips for Intermediate Users

Tip 1: Always Check the Engine

Before running a recipe on a large dataset, look at the "Engine" selection (usually at the bottom of the recipe screen). If you see "DSS Engine" but your data is in Snowflake, try to switch it to "Snowflake" (SQL pushdown) to save time and resources.

Tip 2: Managed Folders vs. Datasets

Not everything is a table. If you are dealing with images, PDFs, or

.pkl

Tip 3: Keep the Design Node Clean

The Design Node is shared. If you write temporary files to the local disk of the Dataiku server, you might crash the node for everyone. Always use "Managed Datasets" stored in your designated database or cloud storage rather than writing files to

/tmp/

1.8 Summary of the Architecture Workflow

To visualize how this all fits together, follow the life of a data project:

1. Integration: The Admin connects the Design Node to a Cloud Data Warehouse.
2. Exploration: The Data Scientist explores data; Dataiku fetches samples for the UI.
3. Transformation: The Data Scientist builds a Flow. Dataiku "pushes down" the work to the Warehouse using SQL.
4. Deployment: The project is bundled and sent to the Automation Node.
5. Operation: The Automation Node runs the Flow every morning at 8:00 AM, updating the tables in the Warehouse.
6. Consumption: A BI tool like Tableau or PowerBI reads the final tables directly from the Warehouse—not from Dataiku.

Conclusion

Dataiku DSS Architecture is designed for flexibility and scale. By acting as a central orchestrator rather than a siloed data store, it allows teams to collaborate regardless of their technical background. As you move forward in this book, keep the "Orchestrator" mindset: always ask yourself, Where is my data actually sitting, and which engine is doing the work?

In the next chapter, we will dive into Project Governance and Version Control, where we will see how the architecture handles changes over time using Git.

Check Your Knowledge

1. What is the main difference between the Design Node and the Automation Node?
2. Why is "SQL Pushdown" generally faster than the "DSS Engine"?
3. True or False: Dataiku stores all your uploaded data in its internal database by default. (Answer: False—it points to external storage!)