English document analysis and visualization

Abstract

This is a software project aims at developing a system which can be used to analyze and auto-categorize English documents, and study how different factors could affect system performance. The categorization system can be partitioned into 3 stages: data representation, classifier building and performance evaluation. As a document is originally raw text, system transforms document into N-grams which would be more suitable for later processing. And feature selection methods are used to filter out less important features (n-gram) and to reduce calculation complexity; and Chi Square Test is used in this project. After pre-processing of data, classifying approach is used to build a classifier for categorization. The system provides 2 classifying approaches; they are k-nearest neighbor (KNN) and Naive Bayes. By using the classifier, documents can be categorized automatically. To evaluate system performance, F1 score approach is used for measuring accuracy of classification. After testing of system, it is found that the classifying system is working with an acceptable accuracy with both KNN approach and Naive Bayes approach.