Find Tech Jobs on twitter using Jobtweets

·

3 min read

Millions of tweets are sent out everyday, I often wondered how possible it would be to search for Job postings on twitter amidst the Spam that the platform struggles with.

I began building Jobtweets some weeks back during my Job search, I got tired of searching manually on twitter and dealing with spam so i used twitter's api to make things easier for me. The project is open source and PR's are welcomed.

Technologies used

  • Fastapi

  • React(Mantine Ui). You would also need Nodejs(I used Version 16) and npm installed on your server.

It was built as a Hybrid monolithic application, to cut costs. Fastapi serves the react frontend and also acts as the backend api. Demo: https://jobtweets.xyz

Jobtweets landing Page

Notes on Building the Job board.

Searching For tweets.

This was done using Tweepy V2, a Python module that acts as a wrapper around Twitter’s API V2

Filtering Tweets with a Spam list

A spam list was needed to filter tweets that contain Spam Keywords,as i see fit.

Filtering Duplicate tweets.

This was done with a for loop, A set is preferable for speed consideration.

Preprocessing Tweets data.

This was done with a Preprocessor Module to clean the tweets (remove Emoji’s,remove Hashtags and Urls).

API Restrictions

Twitter’s API V2 does not provide access to tweet archives(older tweets), so search results are limited to 7 days. Twitter provides access to Older tweets for only Educational/Academic purposes and for commercial use.

Improvements in comparison to API V1

Twitter's API V2 is a lot faster with less rate limiting in comparison to API V1.

Serving ReactJs on FastApi.

One of the hardest things i've had to do was to serve react with fastapi, i initially learnt this with django using this article

After 2 days of Trial and error because FastApi's CORS errors are completely insane. I was able to get it to work using this.

origins = [
    "https://jobtweets.xyz",
    "http://jobtweets.xyz",
    "http://localhost:8000",
]
api_app = FastAPI(title="Job tweets api Doc")

app = FastAPI(title="my app root")
app.mount('/twitter', api_app)
app.mount(
    "/",
    StaticFiles(directory="../twittr/build", html=True, check_dir=False),
    name="static",
)

app.add_middleware(CORSMiddleware,
allow_origins=origins,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],)

I had to mount the API on another route and the build folder on another fastapi instance and a separate URL route. As for the cors error issue i faced, the origins had to be at the top. Also, disable the firewall sudo ufw disable

Deployment

I deployed this on an Ubuntu 22.04 LTS server with 1 CPU Core,25 GB Storage and 1 GB RAM. You can follow this guide on deploying a Fastapi application on an ubuntu instance. Github actions was also used for continuous deployment.

I used Certbot to issue free ssl certificate from lets encrypt, learn how to do that here. You can also follow this guide from linode .

#Linode #LinodeHackathon

TODO LIST

  • Improve the filtering method.

  • Improve the Preprocessing of tweets, current method is a bit too aggressive as some postings begin with hashtags for example, “#Facebook is hiring” will be displayed as “is hiring”.

  • Proper error Handling on the frontend

  • Support for tweets without links specifically (”DM for more details”, “Send an Email to...”)

  • More Comments in the code.

  • Caching of tweets to reduce the requests on the API, because it's currently linked to my account. Personal accounts all have limits.

Conclusion

This was a great experience for me, I had not deployed on a VPS before. I learned a lot while building this.