A web scraper using Crawl4AI to find real estate developers who play golf, with contact information extraction (email, phone) and a Streamlit dashboard for data visualization. This project implements a FAISS-based vector knowledge graph for storing and querying entities and relationships.

1. Install the package:

   pip install -e .

2. Run the scraper:

   python -m construction_scraper scrape

3. Launch the dashboard:

   python -m construction_scraper dashboard

• construction_scraper/: Main package
  • core/: Core data models and structures
  • knowledge_graph/: FAISS-based vector knowledge graph implementation
  • scrapers/: Web scrapers for collecting data
  • utils/: Utility functions and helpers
  • web/: Web dashboard for data visualization
• tests/: Test files
• docs/: Documentation
• scripts/: Utility scripts
• data/: Data directory (created automatically)

• Advanced Data Collection:

  • Uses Crawl4AI for efficient, LLM-friendly web crawling
  • Automatically follows relevant links to discover more information
  • Identifies real estate developers with golf connections

• Contact Information Extraction:

  • Extracts email addresses using regex pattern matching
  • Identifies phone numbers in various formats
  • Links contacts to specific developer profiles

• FAISS Vector Knowledge Graph:

  • Stores entities and relationships in a structured knowledge graph
  • Enables semantic similarity search using FAISS vector embeddings
  • Maintains persistent data storage using SQLite
  • Performs network analysis to identify key influencers
  • Visualizes connections between developers, golf entities, and companies

See the docs/ directory for detailed documentation, including:

• DEVELOPMENT.md: Detailed architecture and implementation notes

MIT