plaso.multi_processing package

Submodules

plaso.multi_processing.analysis_process module

The multi-process analysis process.

class plaso.multi_processing.analysis_process.AnalysisProcess(event_queue, storage_writer, knowledge_base, analysis_plugin, data_location=None, event_filter_expression=None, **kwargs)[source]

Bases: plaso.multi_processing.base_process.MultiProcessBaseProcess

Multi-processing analysis process.

SignalAbort()[source]

Signals the process to abort.

plaso.multi_processing.base_process module

Base class for a process used in multi-processing.

class plaso.multi_processing.base_process.MultiProcessBaseProcess(enable_sigsegv_handler=False, **kwargs)[source]

Bases: multiprocessing.process.Process

Class that defines the multi-processing process interface.

rpc_port

int – port number of the process status RPC server.

SignalAbort()[source]

Signals the process to abort.

name

str – process name.

run()[source]

Runs the process.

plaso.multi_processing.engine module

plaso.multi_processing.multi_process_queue module

A multiprocessing-backed queue.

class plaso.multi_processing.multi_process_queue.MultiProcessingQueue(maximum_number_of_queued_items=0, timeout=None)[source]

Bases: plaso.engine.plaso_queue.Queue

Multi-processing queue.

Close(abort=False)[source]

Closes the queue.

This needs to be called from any process or thread putting items onto the queue.

Parameters:abort (Optional[bool]) – True if the close was issued on abort.
Empty()[source]

Empties the queue.

IsEmpty()[source]

Determines if the queue is empty.

Open()[source]

Opens the queue.

PopItem()[source]

Pops an item off the queue.

Raises:
  • QueueClose – if the queue has already been closed.
  • QueueEmpty – if no item could be retrieved from the queue within the specified timeout.
PushItem(item, block=True)[source]

Pushes an item onto the queue.

Parameters:
  • item (object) – item to add.
  • block (Optional[bool]) – True to block the process when the queue is full.
Raises:

QueueFull – if the item could not be pushed the queue because it’s full.

plaso.multi_processing.plaso_xmlrpc module

XML RPC proxy server and client.

class plaso.multi_processing.plaso_xmlrpc.ThreadedXMLRPCServer(callback)[source]

Bases: plaso.multi_processing.rpc.RPCServer

Class that defines the threaded XML RPC server.

Start(hostname, port)[source]

Starts the process status RPC server.

Parameters:
  • hostname – the hostname or IP address to connect to for requests.
  • port – the port to connect to for requests.
Returns:

A boolean indicating if the RPC server was successfully started.
Stop()[source]

Stops the process status RPC server.

class plaso.multi_processing.plaso_xmlrpc.XMLProcessStatusRPCClient[source]

Bases: plaso.multi_processing.plaso_xmlrpc.XMLRPCClient

Class that defines a XML process status RPC client.

class plaso.multi_processing.plaso_xmlrpc.XMLProcessStatusRPCServer(callback)[source]

Bases: plaso.multi_processing.plaso_xmlrpc.ThreadedXMLRPCServer

Class that defines a XML process status RPC server.

class plaso.multi_processing.plaso_xmlrpc.XMLRPCClient[source]

Bases: plaso.multi_processing.rpc.RPCClient

Class that defines the XML RPC client.

CallFunction()[source]

Calls the function via RPC.

Close()[source]

Closes the RPC communication channel to the server.

Open(hostname, port)[source]

Opens a RPC communication channel to the server.

Parameters:
  • hostname – the hostname or IP address to connect to for requests.
  • port – the port to connect to for requests.
Returns:

A boolean indicating if the communication channel was established.

plaso.multi_processing.psort module

plaso.multi_processing.rpc module

The RPC client and server interface.

class plaso.multi_processing.rpc.RPCClient[source]

Bases: object

RPC client interface.

CallFunction()[source]

Calls the function via RPC.

Close()[source]

Closes the RPC communication channel to the server.

Open(hostname, port)[source]

Opens a RPC communication channel to the server.

Parameters:
  • hostname (str) – hostname or IP address to connect to for requests.
  • port (int) – port to connect to for requests.
Returns:

True if the communication channel was established.
Return type:

bool
class plaso.multi_processing.rpc.RPCServer(callback)[source]

Bases: object

RPC server interface.

Start(hostname, port)[source]

Starts the RPC server.

Parameters:
  • hostname (str) – hostname or IP address to connect to for requests.
  • port (int) – port to connect to for requests.
Returns:

True if the RPC server was successfully started.
Return type:

bool
Stop()[source]

Stops the RPC server.

plaso.multi_processing.task_engine module

plaso.multi_processing.task_manager module

The task manager.

class plaso.multi_processing.task_manager.TaskManager[source]

Bases: object

Manages tasks and tracks their completion and status.

A task being tracked by the manager must be in exactly one of the following states:

  • abandoned: no status information has been recently received from a worker
    about the task, and is assumed to be abandoned.
  • queued: the task is waiting for a worker to start processing it. It’s also
    possible that a worker has already completed the task, but no status update was collected from the worker while it processed the task.
  • processing: a worker is processing the task.
  • pending_merge: a worker has completed processing the task and the
    results are ready to be merged with the session storage.
  • merging: tasks that are being merged by the engine.

Once the engine reports that a task is completely merged, it is removed from the task manager.

Tasks that are not abandoned, or abandoned, but need to be retried are considered “pending”, as there is more work that needs to be done to complete them.

CompleteTask(task)[source]

Completes a task.

The task is complete and can be removed from the task manager.

Parameters:task (Task) – task.
CreateTask(session_identifier)[source]

Creates a task.

Parameters:session_identifier (str) – the identifier of the session the task is part of.
Returns:task attribute container.
Return type:Task
GetAbandonedTasks()[source]

Retrieves all abandoned tasks.

Returns:tasks.
Return type:list[Task]
GetRetryTask()[source]

Creates a task that is an attempt to retry an abandoned task.

Returns:
a task that is a retry of an existing task, or None if there are
no tasks that need to be retried.
Return type:Task
GetStatusInformation()[source]

Retrieves status information about the tasks.

Returns:tasks status information.
Return type:TasksStatus
GetTaskPendingMerge(current_task)[source]

Retrieves the first task that is pending merge or has a higher priority.

This function will check if there is a task with a higher merge priority than the current_task being merged. If so, that task with the higher priority is returned.

Parameters:current_task (Task) – current task being merged or None if no such task.
Returns:
the next task to merge or None if there is no task pending merge or
with a higher priority.
Return type:Task
GetTasksCheckMerge()[source]

Retrieves the tasks that need to be checked if they are ready for merge.

Returns:
tasks that are being processed by workers or that have been
abandoned.
Return type:list[Task]
HasPendingTasks()[source]

Determines if there are tasks running, or in need of retrying.

Returns:
True if there are tasks that are active, ready to be merged, or
need to be retried.
Return type:bool
UpdateTaskAsPendingMerge(task)[source]

Updates the task manager to reflect the task is ready to be merged.

Parameters:task (Task) – task.
Raises:KeyError – if the task was not processing or abandoned.
UpdateTaskAsProcessingByIdentifier(task_identifier)[source]

Updates the task manager to reflect the task is processing.

Parameters:task_identifier (str) – unique identifier of the task.
Raises:KeyError – if the task is not known to the task manager.

plaso.multi_processing.worker_process module

Module contents