Find Tech Jobs on twitter using Jobtweets
Millions of tweets are sent out everyday, I often wondered how possible it would be to search for Job postings on twitter amidst the Spam that the platform struggles with.
I began building Jobtweets some weeks back during my Job search, I got tired of searching manually on twitter and dealing with spam so i used twitter's api to make things easier for me. The project is open source and PR's are welcomed.
Technologies used
Fastapi
React(Mantine Ui). You would also need Nodejs(I used Version 16) and npm installed on your server.
It was built as a Hybrid monolithic application, to cut costs. Fastapi serves the react frontend and also acts as the backend api. Demo: https://jobtweets.xyz
Notes on Building the Job board.
Searching For tweets.
This was done using Tweepy V2, a Python module that acts as a wrapper around Twitter’s API V2
Filtering Tweets with a Spam list
A spam list was needed to filter tweets that contain Spam Keywords,as i see fit.
Filtering Duplicate tweets.
This was done with a for loop, A set is preferable for speed consideration.
Preprocessing Tweets data.
This was done with a Preprocessor Module to clean the tweets (remove Emoji’s,remove Hashtags and Urls).
API Restrictions
Twitter’s API V2 does not provide access to tweet archives(older tweets), so search results are limited to 7 days. Twitter provides access to Older tweets for only Educational/Academic purposes and for commercial use.
Improvements in comparison to API V1
Twitter's API V2 is a lot faster with less rate limiting in comparison to API V1.
Serving ReactJs on FastApi.
One of the hardest things i've had to do was to serve react with fastapi, i initially learnt this with django using this article
After 2 days of Trial and error because FastApi's CORS errors are completely insane. I was able to get it to work using this.
origins = [
"https://jobtweets.xyz",
"http://jobtweets.xyz",
"http://localhost:8000",
]
api_app = FastAPI(title="Job tweets api Doc")
app = FastAPI(title="my app root")
app.mount('/twitter', api_app)
app.mount(
"/",
StaticFiles(directory="../twittr/build", html=True, check_dir=False),
name="static",
)
app.add_middleware(CORSMiddleware,
allow_origins=origins,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],)
I had to mount the API on another route and the build folder on another fastapi instance and a separate URL route. As for the cors error issue i faced, the origins had to be at the top. Also, disable the firewall sudo ufw disable
Deployment
I deployed this on an Ubuntu 22.04 LTS server with 1 CPU Core,25 GB Storage and 1 GB RAM. You can follow this guide on deploying a Fastapi application on an ubuntu instance. Github actions was also used for continuous deployment.
I used Certbot to issue free ssl certificate from lets encrypt, learn how to do that here. You can also follow this guide from linode .
#Linode #LinodeHackathon
TODO LIST
Improve the filtering method.
Improve the Preprocessing of tweets, current method is a bit too aggressive as some postings begin with hashtags for example, “#Facebook is hiring” will be displayed as “is hiring”.
Proper error Handling on the frontend
Support for tweets without links specifically (”DM for more details”, “Send an Email to...”)
More Comments in the code.
Caching of tweets to reduce the requests on the API, because it's currently linked to my account. Personal accounts all have limits.
Conclusion
This was a great experience for me, I had not deployed on a VPS before. I learned a lot while building this.