ConfigurationΒΆ Contents: How to Select Compute Nodes Storage Speed Requirements Storage Options (Best to Worst) Recommendations by Workload Configuring Shuffle Storage Location Capacity Planning Checking Node Storage Network Considerations Heterogeneous Slurm jobs Interactive job Batch job How to handle compute node failures How to use a custom spark-defaults.conf file How to access the Spark web UIs through a reverse proxy Enable the reverse proxy Reach the master from your laptop When the master is itself behind another proxy How to use GPU acceleration RAPIDS Accelerator Under the hood: GPU scheduling Monitor GPU usage while a job runs When are GPUs worth it? How to Set a Custom Spark Log Level Reducing Log Verbosity Available Log Levels Recommendations Example Output Comparison Per-Logger Configuration