Java

CentOS

VirtualBox

1、說明

有時候會對很大的文本文件進行處理，用流一次性讀入肯定是不可能的，內存吃不住，所以需要對文件進行分割、處理、合并，以下是寫的工具類。

2、磁碟空間使用率獲取

在處理文本之前，一定要備份一下，在備份之前要判斷一下磁碟空間是否足夠，用到的Linux命令是「df -hl -P」。命令詳情請自己查閱，以下是Java代碼：

public static int getDiskUsage (String filePath) throws Exception {

//filePath是要看的目錄

int diskUsage = 0;

ProcessBuilder builder = null;

Process ps = null;

try {

String[] cmd = { "/bin/sh", "-c", "df -hl -P" };

builder = new ProcessBuilder(cmd);

builder.redirectErrorStream(true);

ps = builder.start();

BufferedReader br = new BufferedReader(new InputStreamReader(ps.getInputStream(), "utf-8"));

while (true) {

String result = br.readLine();

if (result == null) {

break;

}

String[] r = result.split("
");

for (int i = 0; i < r.length; i++) {

if (r[i].startsWith("/")) {

String[] t = r[i].split(" ");

for (int j = 0; j < t.length; j++) {

if (t[j].endsWith(filePath)) {

if (t[j - 1].contains("%")) {

diskUsage = Integer.parseInt(t[j - 1].replace("%", ""));

}

}

}

}

}

}

ps.waitFor();

return diskUsage;

} catch (Exception e) {

e.printStackTrace();

throw e;

} finally {

if (ps != null) {

ps.destroy();

}

}

}

3、獲取文件行數

獲取總行數主要是為了能確認按行分割文件的話，能分割幾個文件。使用的Linux命令是：find /home/leo -name "java.txt"|xargs cat|wc -l

Java代碼如下：

public static long getLineNum(String filePath, String fileName) throws Exception {

long lineNums = 0l;

ProcessBuilder builder = null;

Process ps = null;

try {

String[] cmd = { "/bin/sh", "-c", "find " + filePath + " -name "" + fileName + ""|xargs cat|wc -l" };

builder = new ProcessBuilder(cmd);

builder.redirectErrorStream(true);

ps = builder.start();

BufferedReader stdoutReader = new BufferedReader(new InputStreamReader(ps.getInputStream(), "utf-8"));// linux終端的編碼為utf-8

while (true) {

String outLine = stdoutReader.readLine();

if (outLine == null) {

break;

}

if (outLine.contains("No such file or directory")) {

throw new Exception("查詢文件行數失敗,文件不存在!");

} else {

lineNums = Long.parseLong(outLine);

}

}

ps.waitFor();

return lineNums;

} catch (Exception e) {

e.printStackTrace();

throw e;

} finally {

if (ps != null) {

ps.destroy();

}

}

}

4、分割文件

這裡使用了分割、改名兩條命令，所以寫成了一個sh，Java代碼如下：

public static String splitFile(String filePath, String prefix) throws Exception {

String restr = "fail";

ProcessBuilder builder = null;

Process ps = null;

try {

String[] cmd = { "/bin/sh", "-c", "/home/leo/splitfile.sh " + filePath + " " + prefix };

builder = new ProcessBuilder(cmd);

builder.redirectErrorStream(true);

ps = builder.start();

BufferedReader stdoutReader = new BufferedReader(new InputStreamReader(ps.getInputStream(), "utf-8"));// linux終端的編碼為utf-8

while (true) {

String outLine = stdoutReader.readLine();

if (outLine == null) {

break;

}

if (outLine.contains("No such file or directory")) {

throw new Exception("查詢文件行數失敗,文件不存在!");

} else {

restr = outLine;

}

}

ps.waitFor();

return restr;

} catch (Exception e) {

e.printStackTrace();

throw e;

} finally {

if (ps != null) {

ps.destroy();

}

}

}

sh文件代碼如下：

#!/bin/sh

#要分割的文件

filename=$1

#分割後文件的前綴

sdpre=$2

#每個文件分割4W行

split -l 40000 ${filename} -d -a 3 ${sdpre}

#給分割後的文件加擴展名

ls|grep ${sdpre}|xargs -n1 -i{} mv {} {}.txt

echo "done"

注意不要忘了給sh賦權。

5、合并文件

使用的Linux命令是：cat /home/leo/sd_000.txt /home/leo/sd_001.txt /home/leo/sd_002.txt /home/leo/sd_003.txt /home/leo/sd_004.txt > /home/leo/java2.txt

Java代碼如下：

public static String catFiles(String fileList, String destFileName) throws Exception {

String restr = "done";

ProcessBuilder builder = null;

Process ps = null;

try {

// cat split00.txt split01.txt > split.txt

String[] cmd = { "/bin/sh", "-c", "cat " + fileList + " > " + destFileName };

builder = new ProcessBuilder(cmd);

builder.redirectErrorStream(true);

ps = builder.start();

BufferedReader stdoutReader = new BufferedReader(new InputStreamReader(ps.getInputStream(), "utf-8"));// linux終端的編碼為utf-8

while (true) {

String outLine = stdoutReader.readLine();

if (outLine == null) {

break;

}

if (outLine.contains("No such file or directory")) {

throw new Exception("查詢文件行數失敗,文件不存在!");

} else {

restr = outLine;

}

}

ps.waitFor();

return restr;

} catch (Exception e) {

e.printStackTrace();

throw e;

} finally {

if (ps != null) {

ps.destroy();

}

}

}

複製文件的代碼就不寫了。

喜歡這篇文章嗎？立刻分享出去讓更多人知道吧！