Unpacking Techniques and Tools in Malware Analysis

Abstract:

Nowadays most of malware samples are packed with runtime packers to complicate the task of reverse engineering and security analysis in order to evade detection of signature-based anti-virus engines. In the overall process of malware analysis, unpacking a packed malicious binary effectively is a necessary preliminary to extract the structure features from the binary for generation of its signature, and therefore several unpacking techniques have been proposed so far that attempt to deal with the packer problem. This brief survey article provides an overview of the currently published prevalent unpacking techniques and tools. It covers the operation process of packing and unpacking, packer detection methods, heuristic policies for spotting original entry point (OEP), environments for runtime unpacking, anti-unpacking techniques, and introduces several typical tools for unpacking.

• Periodical: Applied Mechanics and Materials (Volumes 198-199)
• Pages: 343-350
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.198-199.343
• Online since: September 2012
• Authors: Pei Dai Xie, Mei Jian Li, Yong Jun Wang, Jin Shu Su, Xi Cheng Lu
• Keywords: Malware Analysis, Packed Binaries, Runtime Packer, Spotting OEP, Unpacker, Unpacking

References

[1] Internet Security Threat Report, Vol. 16. Symantec Corporation, Jan. 2012. Available: http: /www. symantec. com/business/threatreport.

[2] T. S. Manuel Egele, Engin Kirda, and Chrstopher Kruegel, A Survey on Automated Dynamic Malware Analysis Techniques and Tools, ACM Computing Surveys, pp.1-49, (2010).

DOI: 10.1145/2089125.2089126

[3] C. Collberg, C. Thomborson, and D. Low, A Taxonomy of Obfuscating Transformations, Tech. Report, No. 48, Department of Computer Science, the University of Auckland, New Zealand, July (1997).

[4] M. Christodorescu, J. Kinder, S. Jha, S. Katzenbeisser, and H. Veith, Malware Normalization, Tech. Report, No. 1539, University of Wisconsin, Madison, Wisconsin, USA, Nov. (2005).

[5] T. Brosch and M. Morgenstern, Runtime Packers: The Hidden Problem, in Black Hat briefings USA, 2006, p.3.

[6] M. Sharif, A. Lanzi, J. Giffin, and W. Lee, Impeding Malware Analysis Using Conditional Code Obfuscation, " in Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS, 08), (2008).

[7] F. Guo, P. Ferrie, and T. Chiueh, A Study of the Packer Problem and Its Solutions, " in In proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection (RAID, 08), (2008).

DOI: 10.1007/978-3-540-87403-4_6

[8] R. Lyda and J. Hamrock, Using Entropy Analysis to Find Encrypted and Packed Malware, " in Proceedings of the IEEE Symposium on Security and Privacy (SSP, 07), March 2007, pp.40-45.

DOI: 10.1109/msp.2007.48

[9] A. Stepan. Improving Proactive Detection of Packed Malware. March 2006. Available: http: /www. virusbtn. com/virusbulletin/archive/2006/ 03/vb200603-packed. dkb.

[10] L. Bohne, Pandora's Bochs: Automatic Unpacking of Malware, Diploma Thesis, 28th January (2008).

[11] UPX. Available: http: /upx. sourceforge. net.

[12] H. C. Kim, D. Inoue, M. Eto, Y. Takagi, and K. Nakao, Toward Generic Unpacking Techniques for Malware Analysis with Quantification of Code Revelation, in Joint Workshop on Information Security, August. (2009).

[13] K. Babar and F. Khalid, Generic Unpacking Techniques, " in IEEE Proceedings of the 2nd International Conference on Computer, Control and Communication (IC4, 09), 2009, pp.1-6.

DOI: 10.1109/ic4.2009.4909168

[14] X. Ugarte-Pedrero, I. Santos, and P. G. Bringas, Structural Feature based Anomaly Detection for Packed Executable Identication, " in Proceedings of the 4th International Conference on Computational Intelligence in Security for Information Systems (CISIS, 11), 2011, pp.50-57.

DOI: 10.1007/978-3-642-21323-6_29

[15] K. Coogan, S. Debray, T. Kaochar, and G. Townsend, Automatic Static Unpacking of Malware Binaries, in Working Conference on Reverse Engineering, October (2009).

DOI: 10.1109/wcre.2009.24

[16] M. G. Kang, P. Poosankam, and H. Yin, Renovo: A Hidden Code Extractor for Packed Executables, in Proceedings of the ACM Workshop on Recurring Malcode, New York, NY, USA, 2007, pp.46-53.

DOI: 10.1145/1314389.1314399

[17] M. Sharif, A. Lanzi, J. Giffin, and W. Lee, Automatic Reverse Engineering of Malware Emulators, " in Proceedings of the IEEE Symposium of Security and Privacy (SSP, 09), (2009).

DOI: 10.1109/sp.2009.27

[18] R. Rolles, Unpacking Virtualization Obfuscators, " in Proceedings of 3rd USENIX Workshop on OensiveTechnologies (WOOT, 09), (2009).

[19] I. Santos, X. Ugarte-Pedrero, and B. Sanz, Collective Classification for Packed Executable Identification, " in Proceedings of the 8th Annual Collaboration, Electronic messaging, AntiAbuse and Spam Conference (CEAS, 11), 2011, pp.231-238.

DOI: 10.1145/2030376.2030379

[20] R. Perdisci, A. Lanzi, and W. Lee, Classification of Packed Executables for Accurate Computer Virus Detection, in Pattern Recognition Letters, 2008, p.1941-(1946).

DOI: 10.1016/j.patrec.2008.06.016

[21] P. Royal, M. Halpin, D. Dagon, R. Edmonds, and W. Lee, PolyUnpack: Automating the Hidden-Code Extraction of Unpack-Executing Malware, " in Proceedings of the 22nd Annual Computer Security Applications Conference (ACSAC, 06), Washington, DC, USA, 2006, pp.289-300.

DOI: 10.1109/acsac.2006.38

[22] PEiD. 2007. Available: http: /www. peid. info.

[23] P. Bania. Generic Unpacking of Self-modifying, Aggressive, Packed Binary Programs. March 2009. Available: http: /piotrbania. com/all/ articles/pbania-dbi-unpacking2009. pdf.

[24] L. Martignoni, M. Christodorescu, and S. Jha, OmniUnpack: Fast, Generic, and Safe Unpacking of Malware, " in Proceedings of the 23rd Annual Computer Security Applications Conference (ACSAC, 07), Miami Beach FL, USA, 2007, pp.1-4.

DOI: 10.1109/acsac.2007.15

[25] N. Nethercote and J. Seward, Valgrind: A Program Supervision Framework, " in Proceedings of the Third Workshop on Runtime Verification (RV, 03), Boulder, Colorado, USA, July (2003).

[26] N. Nethercote and J. Seward, Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation, " in Proceedings of the ACM Conference on Programming Language Design and Implementation (PLDI, 07), (2007).

DOI: 10.1145/1250734.1250746

[27] C. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood, Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation, " in Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI, 05), 2005, pp.190-200.

DOI: 10.1145/1065010.1065034

[28] M. Li, Y. Wang, P. Xie, Z. Huang, S. Jin, and S. Liu, Reverse Engineering of Security Protocol Format Based on Dynamic Binary Analysis, " in International Conference on Computer Convergence Technology (ICCCT, 11), October (2011).

[29] T. H. Carsten Willems, and Felix Freiling, Toward Automated Dynamic Malware Analysis Using CWSandbox, " in Proceedings of the IEEE Symposium on Security and Privacy (SSP, 07), (2007).

DOI: 10.1109/msp.2007.45

[30] R. P. Goldberg, Survey of Virtual Machine Research, vol. IEEE Computer Magazine, pp.34-45, June (1974).

[31] F. Bellard, Qemu: A Fast and Portable Dynamic Translator, presented at the Usenix Annual Technical Conference, (2005).

[32] Bochs. Bochs: The open source IA-32 emulation project.

[33] D. Song, D. Brumley, H. Yin, J. Caballero, I. Jager, M. G. Kang, Z. Liang, J. Newsome, P. Poosankam, and P. Saxena, BitBlaze: A New Approach to Computer Security via Binary Analysis, " in Proceedings of the 4th International Conference on Information Systems Security (ICISS, 08, keynote invited paper), Hyderabad, India, December (2008).

DOI: 10.1007/978-3-540-89862-7_1

[34] R. Paleari, L. Martignoni, G. F. Roglia, and D. Bruschi, A fistful of red-pills: How to Automatically Generate Procedures to Detect CPU Emulators, " in Proceedings of the USENIX Workshop on Offensive Technologies (WOOT, 09), (2009).

DOI: 10.1145/1572272.1572303

[35] P. Ferrie, Attacks on Virtual Machine Emulators, ed: Symantec Advanced Threat Research, (2006).

[36] D. Balzarotti, M. Cova, C. Karlberger, C. Kruegel, E. Kirda, and G. Vigna, Efficient Detection of Split Personalities in Malware, " in In Proceedings of the 17th Annual Network and Distributed System Security Symposium (NDSS, 10), San Diego, CA, USA, (2010).

[37] J. Stewart, OllyBonE: Semi-Automatic Unpacking on IA-32, in Defcon 14, Las Vegas, NV, (2006).

[38] D. Quist and Valsmith, Covert Debugging: Circumventing Software Armoring Techniques, in Black Hat Briengs, USA, August (2007).

[39] L. Boehne, Pandora's Bochs: Automated Unpacking of Malware, Diploma thesis, January, (2008).

[40] V. Yegneswaran, H. Saidi, P. Porras, and M. Sharif, Eureka: A Framework for Enabling Static Analysis on Malware, Tech. Report, No. SRI-CSL-08-01, SRI Project 17382, Computer Science Laboratory SRI International and the College of Computing Georgia Instititue of Technology, 12 April (2008).

DOI: 10.1007/978-3-540-88313-5_31