How can data compression be implemented in Hadoop?

In Hadoop, data compression can be achieved by setting compression formats in MapReduce jobs. Hadoop supports various compression formats such as Gzip, Bzip2, Snappy, and LZO. The compression format to be used can be specified in Hadoop’s configuration file or set in the JobConf of MapReduce jobs.

Here is an example code using Gzip compression format:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.compress.GzipCodec;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class MyJob {

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "MyJob");

        // 设置压缩格式为Gzip
        FileOutputFormat.setCompressOutput(job, true);
        FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);

        job.setJarByClass(MyJob.class);
        job.setMapperClass(MyMapper.class);
        job.setReducerClass(MyReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

In the example code above, the output data compression format is set to Gzip by calling FileOutputFormat.setCompressOutput and FileOutputFormat.setOutputCompressorClass methods. Setting up other compression formats is similar, just replace GzipCodec.class with the corresponding compression format class.

It is important to note that the choice of compression format should be based on the characteristics and requirements of the data, as different compression formats have different compression rates and performance.