AI that works despite limitations

Our Principal Engineer joined an international team developing an AI model build tool. The project was implemented within the R&D center of a global technology company, and its goal was to enable the launch of language models (LLM) on devices with limited computing resources.

‍

The team worked in a model of high autonomy — without a rigidly assigned backlog, which gave room for their own analyses and technical input.

Already at the beginning of the cooperation, the specialist noticed a potential area of risk — the operation of the Softmax operator, responsible for converting the results of the AI model into a probability distribution.

In the past, Softmax was mainly used at the end of classification models (e.g. image recognition). Today it also plays a key role in modern language models — among others. in transformer architectures, where it is responsible for analyzing the relationships between words.

On devices with limited computing power, the operator had to operate in a quantized version, that is, numerically simplified. This form of calculation was associated with a loss of precision, which affected the quality of the results generated by the models.

Instead of waiting for the assigned task, the Principal Engineer himself analyzed how Softmax works in the quantized version. He identified under what conditions the inaccuracy occurs and how it affects the final results of the models, especially for embedded devices.

Based on his own observations and tests, he developed an alternative version of the action of the Softmax component, which:

• increased the precision of the results in a quantized environment,
• did not exceed the available hardware resources,
• could be used in existing architecture without the need for remodeling.

For a long time there was no certainty that the component would be used - but the specialist continued to develop the solution, considering it potentially relevant. Finally, it was integrated into the main stream of the project development.

Streamlined Softmax has become a real support in the development of AI models on resource-limited devices.

‍

Thanks to the proactivity and technical inquisitiveness of the specialist, the project gained not only a ready-made solution, but also a direction of development based on anticipating challenges, and not just extinguishing them.

• Softmax is an operator used in language models (LLM) among others. to transform neural network results into a probability distribution.
• On low-power embedded devices, the models operate in INT8 quantization mode, i.e. they operate on integers in the range of, for example, -127 to 128, instead of floating point numbers.
• The result of quantization is the loss of precision — important especially in the calculations that determine the context and decisions of the model.
• Our specialist proposed and implemented a new implementation of the Softmax operator, taking into account optimization in terms of: accuracy (improvement of final values), operation in a limited environment, the possibility of further integration into the compiler architecture.
• The technologies used in the project included among others. C++, Python, Qt, Node.js and areas related to Deep Learning and neural network processing.