File segmentation and merging are a common requirement. For example: when uploading large files, they can be divided into small pieces first, and then transferred to the server before merging. In many high-end distributed file systems (such as Google's GFS and taobao's TFS), files are also divided or merged according to block units.
Take a look at the basic ideas:
If there is a large file, specify the split size (for example: cut by 1M)
step 1:
First, calculate the number of small files that are finally divided based on the original file size and segmentation size.
step 2:
Create these N small files on disk
step 3:
Open multiple threads (number of threads = number of split files). In each thread, use the seek function of RandomAccessFile to locate the reading pointer to the first position of each segment in the original file, and then read the specified size backward (i.e., the size of the segmented block), and finally write the corresponding split file. Because multi-threads process in parallel, each writes its own small file, which is relatively fast.
The following code is to split a file into multiple subfiles, each with a size of 100K
package testIO;import java.io.File;import java.io.FileInputStream;import java.io.FileOutputStream;import java.util.Arrays;public class subStream {public static void main(String[] args) {//Read the source file into memory first int eachSize=100*1024;File srcFile =new File("F:/test/test.txt");//Create a file object splitFile(srcFile,eachSize);}public static void splitFile(File srcFile,int eachSize){//Determine whether the file meets the split requirements if(srcFile.length()==0){throw new RuntimeException("The file does not meet the split requirements");}byte[] fileContent= new byte[(int) srcFile.length()];try {//Read the file content into memory FileInputStream fis=new FileInputStream(srcFile);fis.read(fileContent);fis.close();}catch (Exception e) {e.printStackTrace();}//Calculate how many copies of int to be split into fileNumber;if(fileContent.length%eachSize==0){fileNumber = fileContent.length/eachSize;} else{fileNumber = fileContent.length/eachSize+1;}for (int i=0;i<fileNumber;i++){String fileName = srcFile.getName()+"-"+i+".txt";File fi = new File(srcFile.getParent(), fileName);//Create a split file under the current file path byte[] eachContent;//Copy the source file contents into the split file if(i!=fileNumber-1){eachContent = Arrays.copyOfRange(fileContent, eachSize*i, eachSize*(i+1));} else{eachContent = Arrays.copyOfRange(fileContent, eachSize*i, fileContent.length);}try {FileOutputStream fos = new FileOutputStream(fi);fos.write(eachContent);fos.close();System.out.printf("Output subfile %s, whose size is %d, each size is %d/n",fi.getAbsoluteFile(),fi.length(), eachContent.length);}catch (Exception e) {// TODO: handle exception e.printStackTrace();}}}}Summarize
The above is the entire content of this article about splitting a file into multiple subfile code examples. I hope it will be helpful to everyone. Interested friends can continue to refer to other related topics on this site. If there are any shortcomings, please leave a message to point it out. Thank you friends for your support for this site!