AI TEXT CLASSIFICATION

Utilize Azure OpenAI to categorize incoming client tickets

I was tasked by my boss to create an AI model that would categorize incoming client tickets based on priority level, client, and number of tickets from that same client code asking for similar help. To do this I utilized Azure OpenAI to create a custom text classification model that includes name entity and language recognition.

I began by pulling around 2000 past tickets sent from various clients ranging in different requests and uploaded them to an Azure Blob storage. I went through each of these tickets labeling the data based on how they were categotized in our internal ticket database system.

Once labeled, I trained the model to understand and automatically prioritize ticket levels based on the ticket submitted. I trained around 5 different models, each time increasing the amount of data used. While doing the training, I encountered an issue where the tickets were all worded similarly and had no rhyme or reason as to being categorized the way that they were. Therefore, I went back to the drawing board and relabeled the 2000 tickets into priority levels based on keywords, client priority, and number of tickets by the same client with similar issues needing to be resolved. I continued the process of training, evaluating, and re-labeling until the model understood and learned what tickets should be categorized as what priority level.

Once the model was sufficiently trained, I got an overall precision, recall, and F1 score of 96% which is 6% greater than the standard. I reviewed with my team, deployed the model and tested in a live setting.

The deployed model classifies ticket priorities with an accuracy of 95%. The completion of this project standardized our ticketing system resulting in faster ticket closures and better customer engagement.