How Tos

Contents:

• Getting Started
  • Installation
  • Deploy sparkctl in an HPC environment
    • URLs
    • sparkctl configuration file
    • Environment module (recommended)
• Configuration
  • How to Select Compute Nodes
    • Storage Speed Requirements
    • Storage Options (Best to Worst)
    • Recommendations by Workload
    • Configuring Shuffle Storage Location
    • Capacity Planning
    • Checking Node Storage
    • Network Considerations
  • Heterogeneous Slurm jobs
    • Interactive job
    • Batch job
  • How to handle compute node failures
  • How to use a custom spark-defaults.conf file
  • How to access the Spark web UIs through a reverse proxy
    • Enable the reverse proxy
    • Reach the master from your laptop
    • When the master is itself behind another proxy
  • How to use GPU acceleration
    • RAPIDS Accelerator
    • Under the hood: GPU scheduling
    • Monitor GPU usage while a job runs
    • When are GPUs worth it?
  • How to Set a Custom Spark Log Level
    • Reducing Log Verbosity
    • Available Log Levels
    • Recommendations
    • Example Output Comparison
    • Per-Logger Configuration
• Execution
  • Start a Spark Cluster
  • How to run Spark jobs in Python
    • Interactive session with pyspark
    • Jupyter notebook
    • Script execution with spark-submit
  • How to to monitor Spark resource utilization
    • Managed execution
  • How to expose Spark metrics in Prometheus format
    • Enable Prometheus metrics
    • Scrape endpoints
    • Write metrics to CSV files
• Applications
  • How to configure a Hive metastore
  • Visualize Data with Tableau
    • Architecture
    • Concepts
    • Compute Node Instructions
    • Client-side Instructions
    • Persistent metastore and spark-warehouse
  • How to run a Jupyter notebook against the cluster
    • Prerequisites
    • Start Jupyter with the Connect server
    • Connect to the server
    • Working in the notebook
    • Troubleshooting
    • Reducing log noise
    • Stopping
• Debugging
  • Spark web UI
  • Log files
  • Jobs hang with “Initial job has not accepted any resources”
  • Spark shuffle partitions
  • Slow local storage
  • Executors are spilling to disk
  • Data skew
  • Performance monitoring
    • Automated resource monitoring
    • Spark tuning resources