Tutorials

These tutorials guide you through running Spark jobs on HPC clusters using sparkctl. Each tutorial covers a different workflow - choose the one that best fits your needs.

Which Tutorial Should I Use?

Tutorial	Best For	Client Install	Interface
Python Library	Programmatic control, automation scripts	sparkctl	Python
Interactive Development	Exploratory analysis, debugging	sparkctl	Python REPL
Ibis	Ibis users, portable dataframe code	sparkctl + ibis-framework[pyspark]	Python
Spark Connect CLI	Lightweight client, remote connectivity	sparkctl	CLI
spark-submit / pyspark	Traditional Spark users, production jobs	sparkctl[pyspark] (full)	CLI

Decision Guide

Start here if you’re new to sparkctl: Python Library - this uses sparkctl’s managed cluster to handle the Spark lifecycle automatically.

Choose by use case:

• “I want to control the cluster from Python code” → Python Library

• “I want to explore data interactively” → Interactive Development

• “I use Ibis and want portable dataframe code” → Ibis

• “I want a minimal client installation” → Spark Connect CLI

• “I want to submit batch jobs” → spark-submit / pyspark

Contents:

• Run jobs in a Python script
• Run jobs interactively with sparkctl
• Use Ibis with sparkctl
  • Prerequisites
  • Option A: Managed Cluster (Spark Connect)
  • Option B: CLI with Spark Connect
  • Option C: CLI with Traditional PySpark
  • Working with Data Files
  • Choosing Between Approaches
• Run jobs with Spark Connect
• Run jobs with spark-submit or pyspark