My Favorite DevTools to Build AI/ML Applications! - DEV Community
I come from the DevOps parts of the world and the AI/ML world was new for me until I joined my current company SingleStore. It's been 8 months since I joined and it is going amazingly well. Learning all the new cool things in AI/ML from various blogs, tutorials, tools, etc. Well, initially when I joined this company, I got to know that we are investing heavily in AI/ML and I started looking out for tools that could simplify the learning process and also do some hands-on. BTW, we had a lot of content and material to start with, but I wanted to see what else was happening outside the industry and do my own research. Like I said, it's been 8 months already. Throughout my journey here, from conducting webinars, and speaking at conferences, to writing blogs on emerging tech trends, I've stumbled upon a collection of indispensable tools. In this article, I'll share those tools—ranging from emerging programming languages to AI frameworks to vector databases to development tools that ease the creation of AI/ML applications. Let’s get started.
Wing Programming Language
I tried to play with many languages and what I found was that most of the languages are overhyped. Yes, and that is when I met one of the communities talking about this new language made for cloud and AI applications. That was the first time I tried the Wing programming language and found it pretty impressive. You might ask - why Wing? Wing offers a unified programming model that consolidates infrastructure and application code within a cohesive framework. This unique approach allows developers to streamline their workflow, eliminating the need for constant context switching and significantly enhancing productivity and creativity. This is what exactly you need while building AI/ML applications, focusing more on the core features rather than the underlying infrastructure. I encountered their Open AI Joker application, which generates and translates jokes into different languages. It was amazing to see how smooth the entire framework was. Just a note that Wing is still under active development. You can build any AI/ML application with minimal code. Let’s see how the Joker application works. The application generates jokes using OpenAI and translates them to different languages. There is a comedian, an Open AI assistant that generates jokes and whenever it generates jokes, the joke gets put/stored in a bucket. There are two translators, Spanish and Hebrew. These translators subscribe to a topic. Whenever a joke is generated, they receive it and translate it. Also, they put the translated joke into the bucket as well. Pretty simple. Below is a technical overview image of this example of how to use OpenAI’s API with Wing.
SingleStore Database
For AI/ML applications, you need a database to store the unstructured data. I joined SingleStore almost 8 months back and I was new to AI/ML and vector databases. With all the hype around vector databases, I started learning about vector databases and found out how SingleStore is an amazing addition to the industry as a vector database. It is not just used to store the vector data but companies use SingleStore for real-time analytics also. Now, see the power of vector data storage supporting real-time analytics, that's a superpower. Let me introduce SingleStore to you all; it is a cloud-based database management system (RDBMS) that is designed for data-intensive applications. It is known for its speed in data ingestion, transaction processing, and query processing. SingleStore started supporting vector storage long back in 2017. SingleStore's Notebook feature is based on the popular Jupyter Notebook, which is widely used in data science and machine learning communities. The SingleStore Notebook extends the capabilities of Jupyter Notebook to enable data professionals to easily work with SingleStore's distributed SQL database while providing great extensibility in language and data sources. Try SingleStore and get $600 worth of free credits.
Pandas and NumPy
At the heart of any AI/ML application is data. Tools like Pandas and NumPy are foundational for data manipulation and analysis in Python. Pandas provide high-level data structures and operations for manipulating numerical tables and time series, making it ideal for preprocessing and cleaning data before it's used for training models. NumPy adds support for large, multi-dimensional arrays and matrices, along with a large collection of mathematical functions to operate on these arrays, crucial for performance-heavy operations in data preprocessing and model training.
Frameworks for AI/ML
I have used TensorFlow, PyTorch and recently came across LangChain and LlamaIndex and was impressed by their ability to help AI/ML engineers with all the toolkits required such as APIs, vector storage functionalities, logic, reasoning, etc to build robust applications. Let’s go through them one by one to see their superpowers. TensorFlow, developed by Google, and PyTorch, developed by Facebook, are two of the most popular frameworks for building and training complex machine learning models. TensorFlow is known for its flexibility and robust scalability, making it suitable for both research prototypes and production deployments. PyTorch is praised for its ease of use, simplicity, and dynamic computational graph that allows for more intuitive coding of complex AI models. Both frameworks support a wide range of AI models, from simple linear regression to complex deep neural networks. Developed by Harrison Chase, and debuted in October 2022, LangChain serves as an open-source platform designed for constructing sturdy applications powered by Large Language Models, such as chatbots like ChatGPT and various tailor-made applications. Langchain seeks to equip data engineers with an all-encompassing toolkit for utilizing LLMs in diverse use cases, such as chatbots, automated question-answering, text summarization, and beyond. LlamaIndex is an advanced orchestration framework designed to amplify the capabilities of LLMs like GPT-4. While LLMs are inherently powerful, having been trained on vast public datasets, they often lack the means to interact with private or domain-specific data. LlamaIndex bridges this gap, offering a structured way to ingest, organize, and harness various data sources — including APIs, databases, and PDFs. By indexing this data into formats optimized for LLMs, LlamaIndex facilitates natural language querying, enabling users to seamlessly converse with their private data without the need to retrain the models. As a beginner, I was looking for something simple and flexible for developing deep learning models and that is when I found Keras. Many AI/ML professionals appreciate Keras for its simplicity and efficiency in prototyping and developing deep learning models, making it a preferred choice, especially for beginners and for projects requiring rapid development.
