News Analytics Pipeline with Elasticsearch & Kibana

A complete news monitoring and analytics system using newsdata.io API and serverless Elasticsearch with Kibana dashboards.

🎯 Features

• Fetch news articles from newsdata.io API
• Index articles into Elasticsearch
• Sentiment analysis on news content
• Kibana dashboards for visualization
• Deduplication to avoid indexing duplicate articles
• Error handling and logging
• Configurable topics and categories

📋 Prerequisites

• Python 3.8+
• Serverless Elasticsearch instance (Elastic Cloud)
• newsdata.io API key
• Kibana access (comes with Elastic Cloud)

🚀 Quick Start

1. Clone the Repository

cd news-elasticsearch-pipeline

2. Install Dependencies

pip install -r requirements.txt

3. Configure Environment Variables

Copy the example environment file and fill in your credentials:

cp .env.example .env

Edit .env file with your credentials:

# newsdata.io API
NEWSDATA_API_KEY=your_newsdata_api_key_here

# Elasticsearch Configuration
ELASTIC_CLOUD_ID=your_cloud_id_here
ELASTIC_API_KEY=your_api_key_here

# Or use endpoint + credentials
ELASTIC_ENDPOINT=https://your-deployment.es.us-central1.gcp.cloud.es.io
ELASTIC_USERNAME=elastic
ELASTIC_PASSWORD=your_password_here

# Application Settings
INDEX_NAME=news-articles
LOG_LEVEL=INFO

4. Run Initial Setup

This creates the Elasticsearch index with proper mapping:

python scripts/setup_index.py

5. Fetch and Index News

python scripts/fetch_news.py

6. Access Kibana Dashboard

1. Log into your Elastic Cloud Kibana instance
2. Go to Management → Stack Management → Kibana → Saved Objects
3. Import the dashboard: kibana/news_dashboard.ndjson
4. View your dashboard under Analytics → Dashboard

📁 Project Structure

news-elasticsearch-pipeline/
├── config/
│   ├── __init__.py
│   ├── settings.py          # Configuration management
│   └── elasticsearch.py     # ES connection setup
├── src/
│   ├── __init__.py
│   ├── news_fetcher.py      # newsdata.io API client
│   ├── indexer.py           # Elasticsearch indexing logic
│   ├── sentiment.py         # Sentiment analysis
│   └── utils.py             # Helper functions
├── scripts/
│   ├── setup_index.py       # Create ES index
│   ├── fetch_news.py        # Main script to fetch and index
│   └── delete_index.py      # Utility to delete index
├── kibana/
│   └── news_dashboard.ndjson # Pre-built Kibana dashboard
├── tests/
│   ├── __init__.py
│   ├── test_news_fetcher.py
│   └── test_indexer.py
├── logs/                    # Application logs (auto-created)
├── .env.example             # Environment variables template
├── .gitignore
├── requirements.txt
└── README.md

🔧 Configuration

Topics and Categories

Edit config/settings.py to customize what news to fetch:

SEARCH_QUERIES = [
    'artificial intelligence',
    'machine learning',
    'cryptocurrency',
    'climate change'
]

CATEGORIES = ['technology', 'business', 'science', 'politics']
COUNTRIES = ['us', 'gb', 'in']
LANGUAGES = ['en']

📊 Kibana Dashboards

The included dashboard provides:

• Article Timeline: Publications over time
• Category Distribution: Pie chart of news categories
• Sentiment Analysis: Positive/Negative/Neutral breakdown
• Top Sources: Most active news sources
• Trending Keywords: Word cloud of popular terms
• Recent Articles: Table of latest indexed news

🔄 Scheduling (Optional)

Using Cron (Linux/Mac)

# Edit crontab
crontab -e

# Add this line to run every 6 hours
0 */6 * * * cd /path/to/news-elasticsearch-pipeline && /usr/bin/python3 scripts/fetch_news.py >> logs/cron.log 2>&1

Using Task Scheduler (Windows)

1. Open Task Scheduler
2. Create Basic Task
3. Set trigger (daily, hourly, etc.)
4. Action: Start a program
5. Program: python
6. Arguments: scripts/fetch_news.py
7. Start in: C:\path\to\news-elasticsearch-pipeline

🧪 Testing

Run tests:

pytest tests/

📝 Usage Examples

Fetch Specific Topic

from src.news_fetcher import NewsFetcher

fetcher = NewsFetcher()
articles = fetcher.fetch_news(query='bitcoin', country='us')
print(f"Found {len(articles)} articles")

Index Custom Articles

from src.indexer import NewsIndexer

indexer = NewsIndexer()
indexer.index_articles(articles)

Analyze Sentiment

from src.sentiment import analyze_sentiment

text = "This is great news for the technology sector!"
sentiment = analyze_sentiment(text)
print(sentiment)  # 'positive'

🐛 Troubleshooting

Connection Issues

If you get connection errors:

• Verify your Elasticsearch credentials in .env
• Check if your IP is whitelisted in Elastic Cloud
• Ensure your Elastic deployment is running

API Rate Limits

newsdata.io free tier limits:

• 200 requests per day
• Implement caching or reduce fetch frequency

Index Already Exists

# Delete and recreate index
python scripts/delete_index.py
python scripts/setup_index.py

📚 API Documentation

• newsdata.io API Docs
• Elasticsearch Python Client
• Kibana Guide

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

⚠️ Disclaimer

This project is for educational purposes. Make sure to comply with:

• newsdata.io Terms of Service
• Elasticsearch Cloud usage policies
• News source copyright and attribution requirements

🙏 Acknowledgments

• newsdata.io for news API
• Elastic for Elasticsearch and Kibana
• Community contributors

📧 Contact

For questions or support, please open an issue on GitHub.

---

Happy News Monitoring! 📰✨