To set number of reducer task dynamically

To set number of reducer task dynamically.


The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps.
Thus, if you expect 10TB of input data and have 128MB DFS blocks, you'll end up with 82k maps. Ultimately the InputFormat of files determines the number of maps.

So below code excerpt let us to set number of reducer task dynamically to adjust with number of map task at runtime.

In Java code:
long defaultBlockSize = 0;
int NumOfReduce = 10; //default value you can give any number.
long inputFileLength=0; 
try {
    FileSystem fileSystem = FileSystem.get(this.getConf()); //hdfs file system
    inputFileLength =fileSystem.getContentSummary(new Path(PROP_HDFS_INPUT_LOCATION)).getLength();//getting size of input files stored in hdfs location
    defaultBlockSize = fileSystem.getDefaultBlockSize(new Path(hdfsFilePath.concat("PROP_HDFS_INPUT_LOCATION")));//getting default block size
   
      if (inputFileLength > 0 && defaultBlockSize > 0) {
        NumOfReduce = (int) (((inputFileLength / defaultBlockSize) + 1) * 2);//calculating number of tasks
      }
      System.out.println("NumOfReduce : " + NumOfReduce);
          }catch (Exception e) {
            LOGGER.error(" Exception{} ", e);
          }  

job.setNumReduceTasks(NumOfReduce);