As I built my agentic AI system, I tested numerous tools across the ecosystem, from autonomous agents to vector databases. For each of these, I cloned the repos (or signed up for their services) and attempted to use them to help build my system. Here’s a rundown of the ones I explored, my rating, and how they performed in practice.
Agentic AI and Autonomous Agents
AutoGPT ⭐️⭐️⭐️: Multi-agent system aimed at handling complex workflows, particularly useful for task automation.
Early on, I was drawn to AutoGPT’s vision of a multi-agent system that could handle complex workflows. At the time, it wasn’t ready for production-level tasks, but the project’s rapid growth and active community of over 54,000 users make it worth revisiting.
gpt-researcher ⭐️⭐️⭐️: Assistant designed for synthesizing knowledge from multiple sources, ideal for summarizing content.
A solid assistant for in-depth knowledge synthesis; great for summaries, but I found it limited for deeper applications. It really has one use case: input a topic and it'll research it for you, emitting a report created from various sources. Easy to set up and use but not a tool to integrate to build systems.
Web Scraping and Data Extraction
firecrawl ⭐️⭐️⭐️: Web scraping tool for structured data extraction, initially promising but later shifted focus towards commercial solutions.
Initially promising for structured web data extraction, but as it shifted to commercial focus, fewer open-source features were available for testing, so I moved on. With a little effort, I got it running locally but it was no better than playwright or unstructured.
octoparse ⭐️⭐️: Web data extraction tool, lacking recent updates, with an optional paid version for additional features.
This was one of the first tools I tested, but with no recent GitHub updates, it seemed outdated. I ultimately didn’t pursue its paid version.
Scrapegraph-ai ⭐️⭐️⭐️⭐️: Community-driven web scraper, effective for single-page extraction but struggles with more complex models.
Easy to set up and effective for simpler sites, with a helpful community. I even contributed a PR. The project has since evolved into a full-fledged company, which is great to see. Congrats to the team! If you have a simple data model and want to collect content from single pages this is a great tool. It was weaker on more complex models that needed hydration from multiple sources. Still a great tool.
jina.ai ⭐️⭐️⭐️: Lightweight content extraction tool for websites, includes a quick URL-based parsing feature.
A neat little tool similar to running unstructured on a website. There's a cool little trick where you can add your URL to the end of r.jina.ai to see the parsed content. Give it a shot. Here's the Agentic AI Page from Wikipedia: r.jina.ai/en.wikipedia.org/wiki/Multi-agent... For some reason LinkedIn won't let me make that a link. Running the reader locally isn't as easy as it should be.
Agentic and RAG Frameworks, Embedding Models, and Semantic Content Manipulation
crewai ⭐️⭐️⭐️⭐️: Versatile agentic AI framework with strong agent abstractions, ideal for orchestrating AI tasks.
I invested significant time here, thanks to its strong agent abstractions and integrated components. I contributed to the community and found it promising but ultimately needed finer agent control, which I found with llama-index. I’ll likely revisit CrewAI as my inability to gain exact control over the agentic orchestration could be user error. Also, they now claim over 100M of their agents run weekly. Quite the accomplishment.
embedchain ⭐️⭐️⭐️: Tool for managing vector embeddings across different workflows, generally applicable but didn’t align with my needs.
Useful for many applications, though its focus didn’t align with my specific needs. Use this for simple RAG pipelines.
llama-index ⭐️⭐️⭐️⭐️⭐️: Robust framework for embedding and search, allows model swapping and flexible integration.
My ultimate choice, with strong GitHub activity and community support. It gave me the control I needed for model swapping and flexible integrations, making it ideal for my system.
langroid ⭐️⭐️⭐️: Framework designed for orchestrating LLMs through multi-agent abstractions, grounded in academic research.
Designed for orchestrating LLMs through a multi-agent approach, langroid impressed me with its academic grounding and agent abstractions. The smaller community and limited support, however, led me to explore other options, though I expect its valuable concepts will find their way into other projects.
phi-data ⭐️⭐️⭐️⭐️: Full-stack agentic AI platform, notable for its use of Docker and pgvector for vector embeddings.
A compelling platform for full-stack agentic AI, and its Docker image for pgvector was particularly helpful. I used it until I needed more advanced support for PostGIS.
unstructured ⭐️⭐️⭐️⭐️: Tool for processing unstructured data, with strong configurability and ease of use.
This tool was exceptional for processing unstructured data, with strong configurability and ease of use. Hats off to the team behind it.
haystack ⭐️⭐️⭐️: Modular RAG framework designed for knowledge retrieval pipelines, more component-driven than cohesive.
I considered it as a RAG pipeline framework but found it more component-driven than cohesive, which led me to move in another direction. Overall, I liked their offering, community, and organization. But ultimately, I found llama-index better suited for my application.
langchain ⭐️⭐️⭐️: Pioneering RAG framework, popular but with complex abstractions and workflows.
Among the first RAG frameworks and widely popular, but I found its abstractions and workflows more complex than necessary. This could be due to the fact it was one of the first released and we were still figuring out our abstractions. Other options met my needs with less setup and cognitive load.
FastAPI and Related Frameworks
fastapi ⭐️⭐️⭐️⭐️⭐️: Modern Python web framework with async support, type hints, and integrated tools for API development.
As a long-time Flask user, I’m now fully converted to FastAPI. Its async support, type hints, and auto-generated docs make it a fantastic choice for API projects. The integration of Pydantic and SQLAlchemy through SQLModel is seamless, though for advanced use, direct SQLAlchemy interaction is still needed.
fastapi-users ⭐️⭐️: User management extension for FastAPI, primarily for OAuth workflows, but has usability limitations.
Easily my worst technical decision. I wanted to avoid the annoyances of implementing oAuth workflows and bought into FastAPI Users marketing hype. While I sought an easy-to-use user management and oAuth solution, I found myself working around it more than with it. I'll likely rip this out with prejudice later. It did handle some things well, like JWTs and parts of the oauth flows.
fastcrud ⭐️⭐️⭐️⭐️⭐️: CRUD utility for FastAPI that simplifies async database operations with advanced configurations.
An intuitive and agile tool, FastCRUD offers seamless async CRUD operations with powerful configuration options. It made complex operations (joins, multi-joins, nested returns) simple to implement. I love how well this is thought out. The author(s) did great work. Kudos!
Image Generation and Manipulation
image-gen ⭐️⭐️: Basic placeholder image generation tool that had stability issues during use.
I initially tried this tool for generating placeholder images. Unfortunately, it quickly caused my machine to crash. Though probably valuable for others, I couldn’t risk the interruptions to my workflow. Honestly, I don't even know why I'm giving it a rating as all it did was crash my machine. :-D But, alas, it seems to have promise.
Utilities
uv ⭐️⭐️⭐️⭐️⭐️: Streamlined alternative to Poetry, offering much faster and simpler Python project management.
A lightning fast replacement for Poetry, UV has transformed my workflow with Python project management. Its simplicity and performance make it a must-have. While I love me some Python Poetry, this is a real game changer. poetry lock on most of my projects take upwards of 2 minutes. With uv it takes ~3 seconds. Adding packages is also lightning fast. Once I saw major open source projects moving to this I decided to check it out. Once they finally added support for script running, I made the leap.
ngrok ⭐️⭐️⭐️⭐️⭐️: Local development tool for sharing web services via public domains, suitable for testing and collaboration.
If you're developing locally and want to share your service with some internal alpha/beta testers and don't have a public domain an CD setup yet, this is a great way to get a live public domain with SSL attached. You can even hook up a custom domain if you want. These networking gurus know exactly what they're doing and do a fantastic job of it. Highly recommended.
Google Cloud Run ⭐️⭐️⭐️⭐️: Serverless compute platform for hosting web apps, cost-effective with easy CI/CD integration.
This service hosts your app on a public domain. You can configure your custom domain and tie free SSL to it with just a few clicks. Previous GCP-based application hosting solutions could be somewhat costly. Cloud Run is reasonably priced and worked quite well out of the box. It hooks up to GitHub for easy CI/CD.
Semantic Search Services
serper ⭐️⭐️⭐️⭐️⭐️: Fast and effective semantic search service with a convenient testing playground.
Fast and easy to use, with a fantastic playground for query testing. Serper was so effective that I didn’t feel the need to explore further. Will definitely use on future projects.
exa ⭐️⭐️⭐️: Semantic search engine, though less needed due to Serper meeting most of my requirements.
A reliable semantic search engine, though ultimately unnecessary as Serper met my needs.
tavily ⭐️⭐️: Semantic search utility for RAG contexts, beyond my needs at this stage.
A versatile tool for RAG context, though beyond what I required at this stage.
Databases
pgvector ⭐️⭐️⭐️⭐️⭐️: PostgreSQL extension for vector storage and search, integrates smoothly with SQLAlchemy.
Seamlessly combines PostgreSQL with vector search, and integrating it with Supabase simplified development. This all-in-one database solution let me focus on building, not managing multiple services. SQLAlchemy worked well with this out of the box. pgvector offers a variety of distance metrics/search functions for vectors. I hear this doesn't scale well because it wasn't designed to be a vector DB; however, in my hands it's been perfect. I wish pgvector also shipped with the BGE embedding model. That would be perfect.
chromadb ⭐️⭐️⭐️: Strong vector database, though managing another separate service conflicted with my operational preferences.
A strong vector database but hosting another separate service didn’t fit my operational desires.
pinecone ⭐️⭐️⭐️: High-performing vector database with strong capabilities, but didn’t match my preference for simplicity.
Similar to chromadb, Pinecone was solid but didn’t suit my preference for a consolidated database solution.
supabase ⭐️⭐️⭐️⭐️⭐️: Cloud database hosting platform offering an effortless PostgreSQL setup.
This service provides effortless and super cost-effective cloud database hosting. With just a few clicks, you can get a free Postgres DB. I started with Google Cloud SQL and was surprised at how expensive it would be to run. Coming over to Supabase was a no-brainer.
IDEs and Code Editors
GitHub Copilot and VS Code ⭐️⭐️⭐️⭐️: Highly popular IDE stack combining efficient code editing and AI-powered assistance.
The go-to stack for many developers, and it’s easy to see why. Copilot and VS Code provide a streamlined workflow, though new challengers are emerging.
GitHub Workspaces ⭐️⭐️⭐️: Cloud-based development environment, promising but currently lacking in large-scale capability.
I tried Workspaces for large-scale operations, but it fell short each time. Still, the concept is fantastic, and with further development, it could be a game-changer for remote workflows.
Cursor ⭐️⭐️⭐️⭐️⭐️: IDE that integrates LLM feedback directly with the codebase, making complex coding tasks more efficient.
Cursor integrates directly with your codebase and terminal, tightening the LLM feedback loop for coding tasks. You can select your preferred language model and customize response speed, making it a great tool for both routine and complex tasks. Highly recommend it.