publications

2025

1. 

   ChatAssert: LLM-based Test Oracle Generation with External Tools Assistance

   Ishrak Hayet, Adam Scott, and Marcelo d’Amorim

   IEEE Transactions on Software Engineering, 2025

   Abs HTML PDF Code

   

   Test oracle generation is an important and challenging problem. Neural-based solutions have been recently proposed for oracle generation but they are still inaccurate. For example, the accuracy of the state-of-the-art technique TECO is only 27.5% on its dataset including 3,540 test cases. We propose CHATASSERT, a prompt engineering framework designed for oracle generation that uses dynamic and static information to iteratively refine prompts for querying large language models (LLMs). CHATASSERT uses code summaries and examples to assist an LLM in generating candidate test oracles, uses a lightweight static analysis to assist the LLM in repairing generated oracles that fail to compile, and uses dynamic information obtained from test runs to help the LLM in repairing oracles that compile but do not pass. Experimental results using an independent publicly-available dataset show that CHATASSERT improves the state-of-the-art technique, TECO, on key evaluation metrics. For example, it improves Acc@1 by 15%. Overall, results provide initial yet strong evidence that using external tools in the formulation of prompts is an important aid in LLM-based oracle generation.

2024

1. 

   Feedback-Directed Partial Execution

   Ishrak Hayet, Adam Scott, and Marcelo d’Amorim

   In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024

   Abs HTML PDF Code

   

   Partial code execution is the problem of executing code with missing definitions. The problem has gained recent traction as solutions to the problem could enable various downstream analyses. We propose feedback-directed partial execution, a technique supported by a tool, named Incompleter, that uses the error feedback from executions to enable partial code execution. Incompleter builds on the observation that errors observed during the execution of incomplete snippets often follow similar error patterns. Incompleter takes an incomplete snippet as input and applies rules (e.g., add class, add field, add file, etc.) to resolve the successive dynamic errors it encounters during execution of the snippet. Incompleter stops when the snippet successfully executes or when it reaches certain bounds. Our results indicate that Incompleter outperforms LExecutor, the state-of-the-art in partial execution. For example, considering a dataset of 4.7K incomplete StackOverflow snippets, Incompleter enables the execution of 10% more code snippets compared to LExecutor and covers 23% more statements. We also show that Incompleter’s type inference significantly improves over LExecutor’s type inference, with a 37% higher F1 score.

2022

1. 

   Invernet: An inversion attack framework to infer fine-tuning datasets through word embeddings

   Ishrak Hayet, Zijun Yao, and Bo Luo

   In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022

   Abs HTML PDF Code

   

   Word embedding aims to learn the dense representation of words and has become a regular input preparation in many NLP tasks. Due to the data and computation intensive nature of learning embeddings from scratch, a more affordable way is to borrow the pretrained embedding available in public and fine-tune the embedding through a domain specific downstream dataset. A privacy concern can arise if a malicious owner of the pretrained embedding gets access to the fine-tuned embedding and tries to infer the critical information from the downstream datasets. In this study, we propose a novel embedding inversion framework called Invernet that materializes the privacy concern by inferring the context distribution in the downstream dataset, which can lead to key information breach. With extensive experimental studies on two real-world news datasets: Antonio Gulli’s News and New York Times, we validate the feasibility of proposed privacy attack and demonstrate the effectiveness of Invernet on inferring downstream datasets based on multiple word embedding methods.

2019

1. 

   Designing a hierarchical keyboard layout for brain computer interface based text entry

   Ishrak Hayet, Tanveer Fahad Haq, Hasan Mahmud, and Md Kamrul Hasan

   In IEEE International Conference on Electrical, Computer and Communication Engineering (ECCE), 2019

   Abs HTML PDF

   

   Human beings are naturally blessed with sensory and motor functionalities. These abilities help us understand our surroundings and respond accordingly. However, sometimes, due to innate reasons or trauma, some people are deprived of these functionalities. In order to restore the natural motor functionalities to people with such disabilities, Brain Computer Interface (BCI) has been widely employed with other Human Computer Interaction (HCI) measures to find an optimum solution that ensures a high speed system with reasonable accuracy. In this paper, we have explored one of such systems for Text Entry mechanism using BCI. We have addressed the following three challenges for a BCI based Text Entry system-i)Four motor imagery signal based interaction ii)Processing and classification of noisy EEG data using the available methods iii) Language prediction model to increase input speed. We have tried to contribute in designing the Keyboard Layout for the Text Entry. Existing soft keyboard layouts for BCI systems have either a flat layout or a hierarchical static layout with very few functionalities in more than 2 layers. We propose to improve the hierarchical layout using a 2 layer multi-functional approach. We have prepared a prototype solution to substantiate our hypothesis. With a cognitive walkthrough based evaluation we have validated our hierarchical layout’s performance to be accurate and more functional.