Ayima Docs

Appearance

Mercury ​

Introduction to SERP Monitoring API Microservice ​

Welcome to the documentation for our API microservice designed for SERP (Search Engine Results Page) monitoring. This API empowers developers and data analysts to efficiently monitor and analyze search engine data for various keywords, domains, and search engine platforms.

Microservice Structure ​

The SERP monitoring microservice comprises several key components:

  1. ClickHouse Database: This database serves as the backbone, storing both raw data and derived data for efficient querying and analysis.
  2. Redis Instance: Used for maintaining state and tracking task progress, ensuring smooth and efficient operation of the microservice.
  3. Python HTTP REST Server: Acts as the interface for interacting with the services mentioned above, along with integration capabilities with S3 for seamless data import functionalities.

Table Structures in ClickHouse ​

  • Buckets: Stores identifiers for groups of keywords generated by the API, along with associated search volumes and search engine IDs.
  • Host Rankings: Cache tables that facilitate faster calculation of metrics such as share of traffic at the domain level.
  • Domain Rankings: Similar to host rankings, these cache tables expedite metric calculations but aggregate at the domain level.
  • Keyword Rankings: The primary table housing daily SERP results for all monitored keywords.
  • Keyword Stats: Tracks all keywords that have been monitored on any given day, providing valuable insights into keyword performance over time.

With its robust architecture and comprehensive data management capabilities, our SERP monitoring API microservice offers a powerful solution for businesses and developers seeking to harness the potential of search engine data analysis.

Installation ​

Development ​

To install for development mode run the following command in the

bash
pip install -r requirements.txt
pip install --editable .

You'll also need to set the following enviroment variables

  • MERCURY_REDIS_SERVER: default: "redis://0.0.0.0:6379/0", to add a password use this pattern redis://:password@0.0.0.0:6379/0
  • MERCURY_DATABASE_SERVER default: "0.0.0.0"
  • MERCURY_DATABASE_USER default: "default"
  • MERCURY_DATABASE_PASSWORD default: ""
  • MERCURY_DATABASE_PORT default: 9000

Running ​

Development ​

Simply run

bash
python3 ./mercury/__main__.py