ziggy.hdmc

Package ziggy :: Package hdmc |
[hide private]
[frames] | no frames]

# Package hdmc

source code

Created on Jul 28, 2010

Author: dwmclary

Submodules | [hide private]
ziggy.hdmc.checkpoint_frame ziggy.hdmc.code_generator: Created on Jul 28, 2010 ziggy.hdmc.frame ziggy.hdmc.hadoop_config ziggy.hdmc.hdfs: Created on Jul 28, 2010 Access functions for files and directories stored in HDFS (Hadoop File System). ziggy.hdmc.hdfs.hdfs_access: Created on May 17, 2010 ziggy.hdmc.hdfs.hdfs_config ziggy.hdmc.hdmc: Module for running monte carlo and other batch jobs on a Hadoop instance. ziggy.hdmc.hdmc_tests: Created on Jul 29, 2010

Functions | [hide private]
submit(script, output_data_name, iterations=1, supporting_file_list=None, reduction_script=None, arguments=`'````'`, debug=False) Submits script non-blocking job to a MapReduce cluster and collects output in output_data_name. | source code
make_frame(script, arguments=`'````'`, iterations=1, debug=False) Generates a basic python frame for running a batch job on a MapReduce cluster. | source code
build_generic_hadoop_call(mapper, reducer, input, output, supporting_file_list=None, num_mappers=None, num_reducers=None, key_comparator=None) Builds a call array suitable for subprocess.Popen which submits a streaming job to the configured MapReduce instance. | source code
execute(hadoop_call) Nonblocking execution of the given call array | source code
execute_and_wait(hadoop_call) Blocking execution of the given call array | source code
submit_checkpoint(script, output_data_name, file_list, reduction_script=None, arguments=`'````'`, debug=False) Submits a script to a MapReduce cluster for parallel operation on a number of files. | source code

Variables | [hide private]
| __author__ = ``'``D. McClary (dan.mcclary@northwestern.edu)``'``
| __package__ = ``'``ziggy.hdmc``'``

Function Details | [hide private]

### submit(script, output_data_name, iterations=1, supporting_file_list=None, reduction_script=None, arguments=`'````'`, debug=False) | source code
Submits script non-blocking job to a MapReduce cluster and collects output in output_data_name. Supporting filenames can be passed as a list, as can a reducing/filtering script. Arguments to the submitted script should be passed as a string.

### submit_checkpoint(script, output_data_name, file_list, reduction_script=None, arguments=`'````'`, debug=False) | source code
Submits a script to a MapReduce cluster for parallel operation on a number of files. An optional reducer script can be applied as well, but should filter the map results by splitting file output on ===HDMC_CHECKPOINT===. Arguments to the submitted script should be passed as a string. Non-blocking.