During a 48-hour hackathon, we developed an AI-powered mobile application designed to enhance the museum experience and address a significant challenge faced by small museums: providing engaging guided tours and audio guides.

With our multi-modal image processing architecture, any museum can upload their collection of artworks and provide as much contextual information as they wish. This allows for a rich, interactive experience for visitors.

Our novel architecture allows the app to detect multiple pieces of art within a single photograph, providing the correct contextual information for each one. We also offer an integrated chatbot that provides access to reliable sources, showing them as links so that users can deep dive into the artists or the art itself.

Despite this being a more UX-oriented hackathon, all of this was made possible by leveraging a powerful Large Language Model (LLM). To achieve maximum performance, we stored the images in Google Cloud for image processing. This approach allows our architecture to detect images in less than 2 seconds, a significant improvement compared to standard LLM integrations that can take more than 20 seconds. This optimization is crucial for delivering a fast and seamless user experience.

• Docker;
• Node (npm, npx);

docker compose up --build -d

cd frontend
npm install
npx expo start