How to read or parse MHTML (.mht) files in java
I need to mine the content of most of known document files like:
- html
- doc/docx etc.
For most of these file formats I am planning to use:
http://tika.apache.org/
But as of now Tika does not support MHTML (*.mht) files.. ( http://en.wikipedia.org/wiki/MHTML )
There are few examples in C# ( http://www.codeproject.com/KB/files/MhtBuilder.aspx ) but I found none in Java.
I tried opening the *.mht file in 7Zip and it failed...Although the WinZip was able to decompress the file into images and text (CSS, HTML, Script) as text and binary files...
As per MSDN page ( http://msdn.microsoft.com/en-us/library/aa767785%28VS.85%29.aspx#compress_content ) and the code project page i mentioned earlier ... mht files use GZip compression ....
Attempting to decompress in java results in following exceptions:
With java.uti.zip.GZIPInputStream
java.io.IOException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(Unknown Source)
at java.util.zip.GZIPInputStream.<init>(Unknown Source)
at java.util.zip.GZIPInputStream.<init>(Unknown Source)
at GZipTest.main(GZipTest.java:16)
And with java.util.zip.ZipFile
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at GZipTest.main(GZipTest.java:21)
Kindly suggest how to decompress it....
Thanks....
Solution 1:
Frankly, I wasn't expecting a solution in near future and was about to give up, but some how I stumbled on this page:
http://en.wikipedia.org/wiki/MIME#Multipart_messages
http://msdn.microsoft.com/en-us/library/ms527355%28EXCHG.10%29.aspx
Although, not a very catchy in first look. But if you look carefully you will get clue. After reading this I fired up my IE and at random started saving pages as *.mht file. Let me go line by line...
But let me explain beforehand that my ultimate goal was to separate/extract out the html content and parse it... the solution is not complete in itself as it depends on the character set or encoding I choose while saving. But even though it will extract the individual files with minor hitches...
I hope this will be useful for anyone who is trying to parse/decompress *.mht/MHTML files :)
======= Explanation ======== ** Taken from a mht file **
From: "Saved by Windows Internet Explorer 7"
It is the software used for saving the file
Subject: Google
Date: Tue, 13 Jul 2010 21:23:03 +0530
MIME-Version: 1.0
Subject, date and mime-version … much like the mail format
Content-Type: multipart/related;
type="text/html";
This is the part which tells us that it is a multipart document. A multipart document has one or more different sets of data combined in a single body, a multipart Content-Type field must appear in the entity's header. Here, we can also see the type as "text/html".
boundary="----=_NextPart_000_0007_01CB22D1.93BBD1A0"
Out of all this is the most important part. This is the unique delimiter which divides two different parts (html,images,css,script etc). Once you get hold of this, everything gets easy... Now, I just have to iterate through the document and finding out different sections and saving them as per their Content-Transfer-Encoding (base64, quoted-printable etc) ...
.
.
.
SAMPLE
------=_NextPart_000_0007_01CB22D1.93BBD1A0
Content-Type: text/html;
charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Location: http://www.google.com/webhp?sourceid=navclient&ie=UTF-8
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" =
.
.
.
** JAVA CODE **
An interface for defining constants.
public interface IConstants
{
public String BOUNDARY = "boundary";
public String CHAR_SET = "charset";
public String CONTENT_TYPE = "Content-Type";
public String CONTENT_TRANSFER_ENCODING = "Content-Transfer-Encoding";
public String CONTENT_LOCATION = "Content-Location";
public String UTF8_BOM = "=EF=BB=BF";
public String UTF16_BOM1 = "=FF=FE";
public String UTF16_BOM2 = "=FE=FF";
}
The main parser class...
/**
* This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0
* which accompanies this distribution, and is available at
* http://www.eclipse.org/legal/epl-v10.html
*/
package com.test.mht.core;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.OutputStreamWriter;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import sun.misc.BASE64Decoder;
/**
* File to parse and decompose *.mts file in its constituting parts.
* @author Manish Shukla
*/
public class MHTParser implements IConstants
{
private File mhtFile;
private File outputFolder;
public MHTParser(File mhtFile, File outputFolder) {
this.mhtFile = mhtFile;
this.outputFolder = outputFolder;
}
/**
* @throws Exception
*/
public void decompress() throws Exception
{
BufferedReader reader = null;
String type = "";
String encoding = "";
String location = "";
String filename = "";
String charset = "utf-8";
StringBuilder buffer = null;
try
{
reader = new BufferedReader(new FileReader(mhtFile));
final String boundary = getBoundary(reader);
if(boundary == null)
throw new Exception("Failed to find document 'boundary'... Aborting");
String line = null;
int i = 1;
while((line = reader.readLine()) != null)
{
String temp = line.trim();
if(temp.contains(boundary))
{
if(buffer != null) {
writeBufferContentToFile(buffer,encoding,filename,charset);
buffer = null;
}
buffer = new StringBuilder();
}else if(temp.startsWith(CONTENT_TYPE)) {
type = getType(temp);
}else if(temp.startsWith(CHAR_SET)) {
charset = getCharSet(temp);
}else if(temp.startsWith(CONTENT_TRANSFER_ENCODING)) {
encoding = getEncoding(temp);
}else if(temp.startsWith(CONTENT_LOCATION)) {
location = temp.substring(temp.indexOf(":")+1).trim();
i++;
filename = getFileName(location,type);
}else {
if(buffer != null) {
buffer.append(line + "\n");
}
}
}
}finally
{
if(null != reader)
reader.close();
}
}
private String getCharSet(String temp)
{
String t = temp.split("=")[1].trim();
return t.substring(1, t.length()-1);
}
/**
* Save the file as per character set and encoding
*/
private void writeBufferContentToFile(StringBuilder buffer,String encoding, String filename, String charset)
throws Exception
{
if(!outputFolder.exists())
outputFolder.mkdirs();
byte[] content = null;
boolean text = true;
if(encoding.equalsIgnoreCase("base64")){
content = getBase64EncodedString(buffer);
text = false;
}else if(encoding.equalsIgnoreCase("quoted-printable")) {
content = getQuotedPrintableString(buffer);
}
else
content = buffer.toString().getBytes();
if(!text)
{
BufferedOutputStream bos = null;
try
{
bos = new BufferedOutputStream(new FileOutputStream(filename));
bos.write(content);
bos.flush();
}finally {
bos.close();
}
}else
{
BufferedWriter bw = null;
try
{
bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(filename), charset));
bw.write(new String(content));
bw.flush();
}finally {
bw.close();
}
}
}
/**
* When the save the *.mts file with 'utf-8' encoding then it appends '=EF=BB=BF'</br>
* @see http://en.wikipedia.org/wiki/Byte_order_mark
*/
private byte[] getQuotedPrintableString(StringBuilder buffer)
{
//Set<String> uniqueHex = new HashSet<String>();
//final Pattern p = Pattern.compile("(=\\p{XDigit}{2})*");
String temp = buffer.toString().replaceAll(UTF8_BOM, "").replaceAll("=\n", "");
//Matcher m = p.matcher(temp);
//while(m.find()) {
// uniqueHex.add(m.group());
//}
//System.out.println(uniqueHex);
//for (String hex : uniqueHex) {
//temp = temp.replaceAll(hex, getASCIIValue(hex.substring(1)));
//}
return temp.getBytes();
}
/*private String getASCIIValue(String hex) {
return ""+(char)Integer.parseInt(hex, 16);
}*/
/**
* Although system dependent..it works well
*/
private byte[] getBase64EncodedString(StringBuilder buffer) throws Exception {
return new BASE64Decoder().decodeBuffer(buffer.toString());
}
/**
* Tries to get a qualified file name. If the name is not apparent it tries to guess it from the URL.
* Otherwise it returns 'unknown.<type>'
*/
private String getFileName(String location, String type)
{
final Pattern p = Pattern.compile("(\\w|_|-)+\\.\\w+");
String ext = "";
String name = "";
if(type.toLowerCase().endsWith("jpeg"))
ext = "jpg";
else
ext = type.split("/")[1];
if(location.endsWith("/")) {
name = "main";
}else
{
name = location.substring(location.lastIndexOf("/") + 1);
Matcher m = p.matcher(name);
String fname = "";
while(m.find()) {
fname = m.group();
}
if(fname.trim().length() == 0)
name = "unknown";
else
return getUniqueName(fname.substring(0,fname.indexOf(".")), fname.substring(fname.indexOf(".") + 1, fname.length()));
}
return getUniqueName(name,ext);
}
/**
* Returns a qualified unique output file path for the parsed path.</br>
* In case the file already exist it appends a numarical value a continues
*/
private String getUniqueName(String name,String ext)
{
int i = 1;
File file = new File(outputFolder,name + "." + ext);
if(file.exists())
{
while(true)
{
file = new File(outputFolder, name + i + "." + ext);
if(!file.exists())
return file.getAbsolutePath();
i++;
}
}
return file.getAbsolutePath();
}
private String getType(String line) {
return splitUsingColonSpace(line);
}
private String getEncoding(String line){
return splitUsingColonSpace(line);
}
private String splitUsingColonSpace(String line) {
return line.split(":\\s*")[1].replaceAll(";", "");
}
/**
* Gives you the boundary string
*/
private String getBoundary(BufferedReader reader) throws Exception
{
String line = null;
while((line = reader.readLine()) != null)
{
line = line.trim();
if(line.startsWith(BOUNDARY)) {
return line.substring(line.indexOf("\"") + 1, line.lastIndexOf("\""));
}
}
return null;
}
}
Regards,
Solution 2:
You don't have to do it on you own.
With dependency
<dependency>
<groupId>org.apache.james</groupId>
<artifactId>apache-mime4j</artifactId>
<version>0.7.2</version>
</dependency>
Roll you mht file
public static void main(String[] args)
{
MessageTree.main(new String[]{"YOU MHT FILE PATH"});
}
MessageTree will
/**
* Displays a parsed Message in a window. The window will be divided into
* two panels. The left panel displays the Message tree. Clicking on a
* node in the tree shows information on that node in the right panel.
*
* Some of this code have been copied from the Java tutorial's JTree section.
*/
Then you can look into it.
;-)
Solution 3:
A more compact code using Java Mail APIs
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.net.URL;
import java.util.Properties;
import javax.mail.BodyPart;
import javax.mail.Session;
import javax.mail.internet.MimeMessage;
import javax.mail.internet.MimeMultipart;
import org.apache.commons.io.IOUtils;
public class MhtParser {
private File mhtFile;
private File outputFolder;
public MhtParser(File mhtFile, File outputFolder) {
this.mhtFile = mhtFile;
this.outputFolder = outputFolder;
}
public void decompress() throws Exception {
MimeMessage message =
new MimeMessage(
Session.getDefaultInstance(new Properties(), null),
new FileInputStream(mhtFile));
if (message.getContent() instanceof MimeMultipart) {
outputFolder.mkdir();
MimeMultipart mimeMultipart = (MimeMultipart) message.getContent();
for (int i = 0; i < mimeMultipart.getCount(); i++) {
BodyPart bodyPart = mimeMultipart.getBodyPart(i);
String fileName = bodyPart.getFileName();
if (fileName == null) {
String[] locationHeader = bodyPart.getHeader("Content-Location");
if (locationHeader != null && locationHeader.length > 0) {
fileName =
new File(new URL(locationHeader[0]).getFile()).getName();
}
}
if (fileName != null) {
FileOutputStream out =
new FileOutputStream(new File(outputFolder, fileName));
IOUtils.copy(bodyPart.getInputStream(), out);
out.flush();
out.close();
}
}
}
}
}