Operating System (OS) detection significantly impacts network management and security. Current OS classification systems used by administrators use human-expert generated network signatures for classification. In this study, we investigate an automated approach for classifying host OS by analyzing the network packets generated by them without relying on human experts. While earlier approaches look for certain packets such as SYN packets, our approach is able to use any TCP/IP packet to determine the host systems' OS. We use genetic algorithms for feature subset selection in three machine learning algorithms (i.e., OneR, Random Forest and Decision Trees) to classify host OS by analyzing network packets. With the help of feature subset selection and machine learning, we can automatically detect the difference in network behaviors of OSs and also adapt to new OSs. Results show that the genetic algorithm significantly reduces the number of packet features to be analyzed while increasing the classification performance.
