We are looking for a PySpark Expert to help optimise our PySpark pipeline. We are using Kafka for ingestion, PySpark for Stream processing and Elasticsearch for storage. Our pipeline has a couple of bottlenecks, which we want to unclog… (Budget: $25 – $50 USD, Jobs: Elasticsearch, Python, Spark, Yarn)
