Package org.apache.spark.launcher

Class AbstractLauncher<T extends AbstractLauncher<T>>

Object
org.apache.spark.launcher.AbstractLauncher<T>
Direct Known Subclasses:
InProcessLauncher, SparkLauncher
public abstract class AbstractLauncher<T extends AbstractLauncher<T>> extends Object
Base class for launcher implementations.
Since:
2.3.0

  • Method Details

    • setPropertiesFile

      public T setPropertiesFile(String path)
      Set a custom properties file with Spark configuration for the application.
      Parameters:
      path - Path to custom properties file to use.
      Returns:
      This launcher.

    • setConf

      public T setConf(String key, String value)
      Set a single configuration value for the application.
      Parameters:
      key - Configuration key.
      value - The value to use.
      Returns:
      This launcher.

    • setAppName

      public T setAppName(String appName)
      Set the application name.
      Parameters:
      appName - Application name.
      Returns:
      This launcher.

    • setMaster

      public T setMaster(String master)
      Set the Spark master for the application.
      Parameters:
      master - Spark master.
      Returns:
      This launcher.

    • setRemote

      public T setRemote(String remote)
      Set the Spark master for the application.
      Parameters:
      remote - Spark remote url.
      Returns:
      This launcher.

    • setDeployMode

      public T setDeployMode(String mode)
      Set the deploy mode for the application.
      Parameters:
      mode - Deploy mode.
      Returns:
      This launcher.

    • setAppResource

      public T setAppResource(String resource)
      Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.
      Parameters:
      resource - Path to the main application resource.
      Returns:
      This launcher.

    • setMainClass

      public T setMainClass(String mainClass)
      Sets the application class name for Java/Scala applications.
      Parameters:
      mainClass - Application's main class.
      Returns:
      This launcher.

    • addSparkArg

      public T addSparkArg(String arg)
      Adds a no-value argument to the Spark invocation. If the argument is known, this method validates whether the argument is indeed a no-value argument, and throws an exception otherwise.

      Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

      Parameters:
      arg - Argument to add.
      Returns:
      This launcher.
      Since:
      1.5.0

    • addSparkArg

      public T addSparkArg(String name, String value)
      Adds an argument with a value to the Spark invocation. If the argument name corresponds to a known argument, the code validates that the argument actually expects a value, and throws an exception otherwise.

      It is safe to add arguments modified by other methods in this class (such as setMaster(String) - the last invocation will be the one to take effect.

      Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

      Parameters:
      name - Name of argument to add.
      value - Value of the argument.
      Returns:
      This launcher.
      Since:
      1.5.0

    • addAppArgs

      public T addAppArgs(String... args)
      Adds command line arguments for the application.
      Parameters:
      args - Arguments to pass to the application's main class.
      Returns:
      This launcher.

    • addJar

      public T addJar(String jar)
      Adds a jar file to be submitted with the application.
      Parameters:
      jar - Path to the jar file.
      Returns:
      This launcher.

    • addFile

      public T addFile(String file)
      Adds a file to be submitted with the application.
      Parameters:
      file - Path to the file.
      Returns:
      This launcher.

    • addPyFile

      public T addPyFile(String file)
      Adds a python file / zip / egg to be submitted with the application.
      Parameters:
      file - Path to the file.
      Returns:
      This launcher.

    • setVerbose

      public T setVerbose(boolean verbose)
      Enables verbose reporting for SparkSubmit.
      Parameters:
      verbose - Whether to enable verbose output.
      Returns:
      This launcher.

    • startApplication

      public abstract SparkAppHandle startApplication(SparkAppHandle.Listener... listeners) throws IOException
      Starts a Spark application.

      This method returns a handle that provides information about the running application and can be used to do basic interaction with it.

      The returned handle assumes that the application will instantiate a single SparkContext during its lifetime. Once that context reports a final state (one that indicates the SparkContext has stopped), the handle will not perform new state transitions, so anything that happens after that cannot be monitored. If the underlying application is launched as a child process, SparkAppHandle.kill() can still be used to kill the child process.

      Parameters:
      listeners - Listeners to add to the handle before the app is launched.
      Returns:
      A handle for the launched application.
      Throws:
      IOException
      Since:
      1.6.0