Laya: a zero-shot decision engine for real-world applications

Laya, an open-source decision engine from Convai Innovations, has been tested on real-world data to demonstrate its capabilities in zero-shot learning. Released as version 0.3.27, Laya processes multiple typed questions in a single forward pass without generating text, offering speed and calibrated probabilities. The model was evaluated on the CLINC150 dataset, achieving 0.878 accuracy on 15 banking intents without training data, surpassing a TF-IDF classifier with 30 examples per intent. Laya’s calibration module adjusts temperatures based on validation data, reducing calibration error from 0.102 to 0.059. It also implements an abstention gate, filtering out 35 of 450 test answers. The model’s confidence scores, however, remain biased, with yes/no questions showing a 0.5 threshold that misses 28.9% of banking queries. A pydantic schema integrates Laya into application code, allowing for explicit handling of uncertain intents. The study highlights the need for custom calibration and testing on real data, as default settings may not align with specific use cases.