Job market analysis

An analysis of data science and data analyst job market trends

---

Project Overview

This project creates a pipeline that automatically scrapes job data from the websites remoteok.com, weworkremotely.com, and hackernews.com using GitHub actions. It cleans the data and presents it on a dashboard for insights.

Project structure

 job_scraper/
│
├── .github/
│   └── workflows/
│       ├── scrape.yml          # Daily RemoteOK & WWR automation
│       └── scrape_hn.yml       # Bi-monthly HN automation
│
├── analysis/
│   └── analyze.py              # Exploratory data analysis
│
├── dashboard/
│   └── app.py                  # Streamlit dashboard
│
├── config.py                   # Centralized configuration
├── create_db.py                # Database schema setup
├── remote_scraper.py           # RemoteOK scraper
├── wwr_scraper.py              # We Work Remotely scraper
├── hn_scraper.py               # Hacker News scraper
├── requirements.txt            # Python dependencies
├── .gitignore                  # Git exclusions
└── README.md                   # You are here

---

🚀 Quick Start

Prerequisites

• Python 3.10+

Installation

# Clone the repository
git clone https://github.com/boyeshenry-byte/job_market_analyzer.git
cd job_scraper

# Install dependencies
pip install -r requirements.txt

Running the Analysis

Execute in order:

1. create_db.py - Creates a database to house data
2. remote_scraper.py, wwr_scraper.py, and hn_scraper.py - Scrapes and cleans data for analysis, then appends to the database
3. app.py - Creates a dashboard for visualizing job trends

---

Dashboard Preview

The full dashboard can be found here: https://job-market-analyzer.streamlit.app

🛠️ Technologies Used

Core Libraries:

• pandas - Data manipulation
• numpy - Numerical computing
• sqlalchemy - Create databases
• streamlit - Build dashboard
• requests - Fetch urls
• BeautifulSoup - Scrape html data
• plotly - Visualize findings

👤 Author

Henry Boyes

• GitHub: @boyeshenry-byte
• LinkedIn: Henry Boyes
• Email: boyeshenry@gmail.com

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• Data Sources: RemoteOK, WeWorkRemotely & Hacker News
• Inspiration: Understanding the job market of my field

---

📞 Questions?

Feel free to open an issue or reach out if you have questions about the methodology or findings!

---

⭐ If you found this analysis interesting, please consider starring the repository!