org.apache.spark.scheduler

Class JobLogger

java.lang.Object

org.apache.spark.scheduler.JobLogger

All Implemented Interfaces:

Logging, SparkListener

public class JobLogger
extends java.lang.Object
implements SparkListener, Logging

:: DeveloperApi :: A logger class to record runtime information for jobs in Spark. This class outputs one log file for each Spark job, containing tasks start/stop and shuffle information. JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext is created. Note that each JobLogger only works for one SparkContext

NOTE: The functionality of this class is heavily stripped down to accommodate for a general refactor of the SparkListener interface. In its place, the EventLoggingListener is introduced to log application information as SparkListenerEvents. To enable this functionality, set spark.eventLog.enabled to true.

Constructor Summary

Constructors

Constructor and Description
JobLogger()
JobLogger(java.lang.String user, java.lang.String logDirName)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	buildJobStageDependencies(int jobId, scala.collection.Seq<java.lang.Object> stageIds) Build up the maps that represent stage-job relationships
protected void	closeLogWriter(int jobId) Close log file, and clean the stage relationship in stageIdToJobId
protected void	createLogDir() Create a folder for log files, the folder's name is the creation time of jobLogger
protected void	createLogWriter(int jobId) Create a log file for one job
protected void	jobLogInfo(int jobId, java.lang.String info, boolean withTime) Write info into log file
java.lang.String	logDirName()
void	onJobEnd(SparkListenerJobEnd jobEnd) When job ends, recording job completion status and close log file
void	onJobStart(SparkListenerJobStart jobStart) When job starts, record job property and stage graph
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) When stage is completed, record stage completion status
void	onStageSubmitted(SparkListenerStageSubmitted stageSubmitted) When stage is submitted, record stage submit info
void	onTaskEnd(SparkListenerTaskEnd taskEnd) When task ends, record task completion status and metrics
protected void	recordJobProperties(int jobId, java.util.Properties properties) Record job properties into job log file
protected void	recordTaskMetrics(int stageId, java.lang.String status, TaskInfo taskInfo, org.apache.spark.executor.TaskMetrics taskMetrics) Record task metrics into job log files, including execution info and shuffle metrics
protected void	stageLogInfo(int stageId, java.lang.String info, boolean withTime) Write info into log file
java.lang.String	user()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.scheduler.SparkListener

onApplicationEnd, onApplicationStart, onBlockManagerAdded, onBlockManagerRemoved, onBlockUpdated, onEnvironmentUpdate, onExecutorAdded, onExecutorMetricsUpdate, onExecutorRemoved, onTaskGettingResult, onTaskStart, onUnpersistRDD

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JobLogger

public JobLogger(java.lang.String user,
         java.lang.String logDirName)

JobLogger

public JobLogger()

Method Detail

user

public java.lang.String user()

logDirName

public java.lang.String logDirName()

createLogDir

protected void createLogDir()

Create a folder for log files, the folder's name is the creation time of jobLogger

createLogWriter

protected void createLogWriter(int jobId)

Create a log file for one job

Parameters:

jobId - ID of the job

Throws:

FileNotFoundException - Fail to create log file

closeLogWriter

protected void closeLogWriter(int jobId)

Close log file, and clean the stage relationship in stageIdToJobId

Parameters:

jobId - ID of the job

buildJobStageDependencies

protected void buildJobStageDependencies(int jobId,
                             scala.collection.Seq<java.lang.Object> stageIds)

Build up the maps that represent stage-job relationships

Parameters:

jobId - ID of the job

stageIds - IDs of the associated stages

jobLogInfo

protected void jobLogInfo(int jobId,
              java.lang.String info,
              boolean withTime)

Write info into log file

Parameters:

jobId - ID of the job

info - Info to be recorded

withTime - Controls whether to record time stamp before the info, default is true

stageLogInfo

protected void stageLogInfo(int stageId,
                java.lang.String info,
                boolean withTime)

Write info into log file

Parameters:

stageId - ID of the stage

info - Info to be recorded

withTime - Controls whether to record time stamp before the info, default is true

recordTaskMetrics

protected void recordTaskMetrics(int stageId,
                     java.lang.String status,
                     TaskInfo taskInfo,
                     org.apache.spark.executor.TaskMetrics taskMetrics)

Record task metrics into job log files, including execution info and shuffle metrics

Parameters:

stageId - Stage ID of the task

status - Status info of the task

taskInfo - Task description info

taskMetrics - Task running metrics

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)

When stage is submitted, record stage submit info

Specified by:

onStageSubmitted in interface SparkListener

Parameters:

stageSubmitted - Stage submitted event

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

When stage is completed, record stage completion status

Specified by:

onStageCompleted in interface SparkListener

Parameters:

stageCompleted - Stage completed event

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

When task ends, record task completion status and metrics

Specified by:

onTaskEnd in interface SparkListener

Parameters:

taskEnd - Task end event

onJobEnd

public void onJobEnd(SparkListenerJobEnd jobEnd)

When job ends, recording job completion status and close log file

Specified by:

onJobEnd in interface SparkListener

Parameters:

jobEnd - Job end event

recordJobProperties

protected void recordJobProperties(int jobId,
                       java.util.Properties properties)

Record job properties into job log file

Parameters:

jobId - ID of the job

properties - Properties of the job

onJobStart

public void onJobStart(SparkListenerJobStart jobStart)

When job starts, record job property and stage graph

Specified by:

onJobStart in interface SparkListener

Parameters:

jobStart - Job start event