My recent searches
Filter by:
    Job State
    4,904 spark jobs found, pricing in USD
    Java/Golang/Rust Developer 6 days left

    Exceptional proficiency in Java/Golang/Rust or any other strongly typed language. Working knowledge of big data technologies like apache spark or something similar is a big plus. Strong understanding of Sql/NoSql databases (Postgres/Cassandra) Well versed with operating and enhancing scalable, fault-tolerant, distributed systems. Working knowledge of AWS or any other cloud infrastructure provider. Comfortable operating in an Agile environment and dealing with uncertainty and ambiguity. Understanding of system design, architecture and distributed systems (Messaging systems, Eventual Consistency, CAP, ACID) Keen interest in web 3.0 ecosystem (blockchain, DeFi, NFTs etc)

    $8 / hr (Avg Bid)
    $8 / hr Avg Bid
    5 bids
    Big Data Developer 6 days left

    I am looking someone who has good experience in Python, spark, shell scripting , Hadoop, hive , sql, git, sqoop, oracle to develop the data pipeline. Also, any cloud experience- GCP/AWS

    $19 / hr (Avg Bid)
    $19 / hr Avg Bid
    13 bids

    ...(Oracle/Postgresql/MySQL/MariaDB) • 2+ years of experience in Spring Framework including Spring Boot, Spring Data JPA • 1+ years of experience in Microservices, Docker and Spring Cloud • A good understanding of real-time messaging systems, like WebSockets • Experience in one or more visualization frameworks/libraries like AngularTS, Leaflet, ECharts, and YFiles is a huge plus • Experience in Apache Spark is a huge plus • Experience in building an asynchronous Event Driven System using Apache Kafka or RabbitMQ is a plus • Knowledge of JUnit or other unit testing tools, preferably with hands-on experience • Experience in CI/CD pipelines (Gitlab CI-CD/Jenkins/CircleCI) and branching strategies • Experience in code review and code quality ...

    $30 / hr (Avg Bid)
    $30 / hr Avg Bid
    13 bids

    Hi Tapasi K., I noticed your profile and would like to offer you my project. Write a spark submit job that accesses a data in a hive table in one hadoop/spark cluster , accesses data in an hbase table in another hadoop cluster , combine( do some aggregation) this data and save result in both hive and hbase. P.S. Hive is in a different hadoop cluster than hbase ( both in same network / VPC subnet ) .

    $100 (Avg Bid)
    $100 Avg Bid
    1 bids

    Background: Autotrading project that allows users to connect their metamask account to an API so they can trade and earn passive income without the project having access to their funds. Fully built on Metamask plugin to connect to the users wallet is already ...Crypto-Enviro App that plants a tree when you trade (we donate to a partner charity) - ? 17%-20% monthly passive income. Bot runs 24/7 - ? 2% fees - ?We do not hold or have access to your money - ? account is secured and encrypted - ? Trading is done on the Binance chain for safety - ? No Setup fee - ? No minimum deposit - ? Have an Offical Discord Community Thank you very much! We look forward to spark a conversation learn API work you've done and send you our application we've had great complimen...

    $145 (Avg Bid)
    $145 Avg Bid
    4 bids

    I'm currently final year student doing a project which is based on iot on network intrusion detection systems. I have implemented our project in python and have the code ready as well. I want to implement the same in Apache spark framework, so kindly let me know if you can implement it. Respond asap as it’s urgent :)

    $90 (Avg Bid)
    $90 Avg Bid
    1 bids
    Spark AR expert 3 days left

    - You will be required to support the development of Spark AR certification courses - Participate in live meetings with the Learning Design team - Conduct reviews of course assets and course materials - Write and / or source Spark AR content to be utilized in edX courses

    $46 / hr (Avg Bid)
    $46 / hr Avg Bid
    3 bids

    If anybody have that theme i will pay immediately. thanks

    $30 (Avg Bid)
    $30 Avg Bid
    15 bids

    ❖ Roles and Responsibilities ⮚ Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Python & SQL. ⮚ Work with Data Scientists/BI teams to assist with data-related technical issues and support their data processing needs. ⮚ Build analytical tools that utilize the data pipeline to provide actio...Good understanding of Relational databases, Big data environments ⮚ Proficient in building end-to-end data pipelines. ⮚ Experience Handling end to end development of projects without any additional support ⮚ Excellent communication skills and proven ability to handle customers. ❖ Good to Have ⮚ Any experience with SAS/R ⮚ Experience using Rest/Soap apis for data movement. ⮚ Any experience with Spark/Scala

    $1454 (Avg Bid)
    $1454 Avg Bid
    16 bids

    Hello, My requirement is described below 1) Read two flat files(.dat files with ^ delimiter) on Hadoop Edge Node into Spark data frames 2) Drop unwanted columns from those data frames 3) Generate a md5 hash key for each record in the data frames after dropping columns 4) Join two data frames on hash key and identify mismatched records and write those records to a file on Hadoop Edge node 5) This code should be in Java and Apache Spark

    $126 (Avg Bid)
    $126 Avg Bid
    5 bids

    Hello There Urgently need small development project that has to be developed via Spark - PySpark, Python. Please apply ASAP if you can get it done urgently today ie 10th May Thanks for reading

    $132 (Avg Bid)
    $132 Avg Bid
    5 bids

    Hello There Urgently need small development project that has to be developed via Spark - PySpark, Python. Please apply ASAP if you can get it done urgently today ie 10th May Thanks for reading

    $91 (Avg Bid)
    $91 Avg Bid
    6 bids

    Sourcing an artist who understands the Chinese culture of gift giving, and the symbiology of Chinese New Year. keeping these elements in mind, we want an artist to design / illustrate a theme to hero this time of year. The art will be wrapped around a box and elements will be used on a label or used as a wrap for a 700ml spirits bottle. Attached are some references for inspirat...symbiology of Chinese New Year. keeping these elements in mind, we want an artist to design / illustrate a theme to hero this time of year. The art will be wrapped around a box and elements will be used on a label or used as a wrap for a 700ml spirits bottle. Attached are some references for inspiration only. We do not represent the brands listed in the references. And are only to be used to spark creat...

    $3026 (Avg Bid)
    $3026 Avg Bid
    43 bids

    GlobalPeople is a Job agency based in Faro, Portugal. For our partner an international IT company we are looking for a Java Developer Role Definition We’re proud to offer innovative products that meet the...functional, bug-free deliveries (review and suggest to enhance interaction between teams) Essential Skills & Requirements • Passion for building great software • Java8+ • Spring Boot (REST, Hibernate) • Project Reactor, Spring Web Flux, Reactive Kafka (in general reactive programming) • MongoDB and/or MySQL and Redis as cache • Kafka Streams • Big Data stack experience is a plus (e.g. kafka, Apache Spark, hadoop) • Experience with Agile or Scrum software development methodologies • Experience with version Control (GIT / SV...

    $16 / hr (Avg Bid)
    $16 / hr Avg Bid
    10 bids

    Urgent! Your task is to prepare the batch layer (of-line processing pipeline) of the lambda architecture that will enable us to perform some analytics on a dataset. You are required to use Apache Spark’s DataFrame to perform the required computation.

    $75 (Avg Bid)
    $75 Avg Bid
    4 bids

    Hi Mahira K., I noticed your profile and would like to offer you my project. We can discuss any details over chat.

    $20 / hr (Avg Bid)
    $20 / hr Avg Bid
    1 bids

    BIGDATA spark assignment with hive and spark dataframes

    $14 / hr (Avg Bid)
    $14 / hr Avg Bid
    7 bids

    Hi Ahmed Galal Mohamed M., I noticed your profile and would like to offer you my project. We can discuss any details over chat.

    $10 / hr (Avg Bid)
    $10 / hr Avg Bid
    1 bids

    Virtual Dataset syncing through Spark. Spark job to written in Scala/Java which will retrieve the data from multiple tables with complex join conditions. There will be multiple stages before the final outcome and each stage can transformed and will be input to next stage for joining with other table/previous output. Request will have all the metadata in JSON format for the stages involved in the virtual dataset and there detailed joins as well as transformation conditions. Screenshots attached for more understanding.

    $230 (Avg Bid)
    $230 Avg Bid
    6 bids

    To amend PySpark codes so as to write Spark Dataframes into MongoDB but with GridFS necessarily incorporated, so as to save files larger than 16MB into Mongo DB, Note: done with write Streams in PySpark

    $183 (Avg Bid)
    $183 Avg Bid
    5 bids

    I want some help in spark, scala, EMR, data pipeline, airflow

    $172 (Avg Bid)
    $172 Avg Bid
    8 bids

    ...required. Есть 2 задания: Задание 1: Разработать приложение для обработки сведений о пользователях. Входные данные: запись, содержащая имя, страну и время. Выходными данными являются структуры со значениями счётчика уникальных стран и счётчика уникальных имён за определённый интервал времени. Интервал времени определяется статически. Разработать потоковое приложение для Apache Flink, Beam, Spark, Storm или Samza (любого из них) в три этапа: написать конвейер для расчёта количества уникальных строк за интервал для одного канала (имя или страна) написать аналогичный второй конвейер и обеспечить выдачу результатов в разные очереди написать параллельный вариант с объединением двух значений в одну структуру При реализации приложений использовать модульные тесты для отладки, спец...

    $565 (Avg Bid)
    $565 Avg Bid
    1 bids
    Trophy icon Band Logo - Moist Miracle 10 days left

    We need a logo for our band. The band's name is Moist Miracle. A few details about us: We are a hardcore punk rock band, we love playing some heavy bad-ass guitar riffs and bass licks. Our approach is pretty experimental and we don't really stick to any particular genre, as various tricks and techniques are ...justice and politics to mental crisis. But whatever the message is, it is often delivered in a satirical and humorous manner. We want our audience to have fun, not to feel depressed. However, some songs are quite serious. There are only four of us: vocalist, guitarist, bassist and drummer. A classic punk band, you know. I attached a few recordings, maybe those will be somehow helpful and will spark some creativity… Here is a music video:

    $46 (Avg Bid)
    161 entries

    The objetive is to transform the iterative and classic COP-K-MEANS algorithm (with must-link and cannot-link constraints) using Spark in Python with the library PYSPARK to make it accesible to big data.

    $157 (Avg Bid)
    $157 Avg Bid
    12 bids

    I have created all functions in using pandas data frame. First I just loaded the data into Pyspark data frame and convert it back to the pandas data frame and created all the functions. This solution is ad-hoc. I do not know pyspark. So I need a helper to convert all the functions into pure Pyspark functions... I have also added the dataset attached along with the main python files. There are only 7 - 8 functions you need to convert it to pyspark function. You need to only use PySpark Dataframe and do all transformations and finally save the processed data- frame to csv and parquet as I did in I have tested in spark version - 3.2.1 An expert can do this within 1 hour or max 2 hours.

    $13 (Avg Bid)
    $13 Avg Bid
    1 bids

    We need to hire a Hadoop and Spark expert. Tasks to be done: - Configure properly Hadoop cluster in HA mode - Configure properly Spark cluster in HA mode - Install and configure HBase - Install and configure Oozie - Install and configure SSL for all the tools mentioned above. - Configure authentication for all the tools mentioned above. Installation will be done in an on-premise environment. Linux based OS (centos 9) will be used. All the Hadoop and Spark software will be the full open source version. We are not using Cloudera, Hortonworks, MapR or similars. Project will be payed by an hourly rate for the amount of time it takes to finish the tasks mentioned above. Only tech folks with experience will be considered! :)

    $57 / hr (Avg Bid)
    $57 / hr Avg Bid
    2 bids

    I am looking for help on spark scala aws projects Monthly support.

    $1241 (Avg Bid)
    $1241 Avg Bid
    24 bids

    I need a recruiter who can find me a smart data scientist. The data scientist should be comfortable with surveys, Python, SQL, tableau, machine learning, Hadoop platform or apache spark.

    $97 (Avg Bid)
    $97 Avg Bid
    8 bids

    I have 13 questions from bigdata

    $28 (Avg Bid)
    $28 Avg Bid
    10 bids an equal opportunity workplace and is an affirmative action employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, or Veteran status. We also consider qualified applicants regardless of experience in any technology if you demonstrate the spark in you to learn, adapt and produce results. ...

    $11 / hr (Avg Bid)
    $11 / hr Avg Bid
    14 bids

    ...Engineering. We add value by building distributed and highlyparallelized data processing pipelines which process a massive amount of data. We leverage Scala and Spark to enrich and transform corporate data to enable searching, data visualization andadvanced analytics. As the (Senior) Data Engineer, you will play a valuable role in creating high-performing, flexible, robust, scalable and easily maintainable global reporting solution for the Bank. What you’ll be doing: • Building distributed and highly parallelized Big data processing pipeline which process massiveamount of data (both structured and unstructured data) in near real-time • Leveraging Spark to enrich and transform corporate data to enable searching, data visualization,and advanced analytics &bu...

    $38 / hr (Avg Bid)
    $38 / hr Avg Bid
    1 bids

    The project consists of completing the code of a project to develop an anomaly detector with KMeans or BisectionKMeans algorithms. The project is semi-completed and both the code and the data sets are provided. The project specifications and details are also attached. Example project for inspiration:

    $144 (Avg Bid)
    $144 Avg Bid
    4 bids

    This opportunity will perfectly fit those candidates, who feel confident working with Big Data. What would make you a strong fit: - 3+ years of experience working with large datasets using technologies such as Apache Flink, Beam, and Spark - Expert programmer in Java or other object-oriented programming languages - Experience with python, docker container, k8s, terraform, etc. - You are open to learning and patient to teach when working with our diverse team members with various backgrounds in engineering and data science - You have a strong interest to work with real-world machine learning applications and technologies - Advanced English and experience in Scrum Methodology would be a plus.

    $26 (Avg Bid)
    $26 Avg Bid
    3 bids

    ...Object-Oriented Programming (OOP) Patterns and Concepts Familiarity with different design and architectural patterns Skill for writing reusable Java libraries Knowhow of Java concurrency patterns Basic Understanding of the concepts of MVC (Model-View-Controller) Pattern, JDBC (Java Database Connectivity), and RESTful web services Experience in working with popular web application frameworks like Play and Spark Relevant Knowledge of Java GUI frameworks like Swing, SWT, AWT according to project requirements Ability to write clean, readable Java code Basic knowhow of class loading mechanism in Java Experience in handling external and embedded databases Understanding basic design principles behind a scalable application Skilled at creating database schemas that characterize and supp...

    $1810 (Avg Bid)
    $1810 Avg Bid
    11 bids

    We are looking for Date Engineer/Scientist with 2 Year Experience JOB Description Must Have skills Spark, Python, Scripting, PySpark, Good to have Apache hive Apache Nifi/ AWS

    $15 / hr (Avg Bid)
    $15 / hr Avg Bid
    13 bids

    I have a project that is having spark sql performance issues after upgrading it from spark 2.x running java 1.8 and scala 2.11 to spark 3.1.2 running java 11 along with scala 2.12 version.

    $220 (Avg Bid)
    $220 Avg Bid
    6 bids

    we are looking for experienced working professionals for supporting bigdata Hadoop/ Pyspark short term projects online live sessions. this is urgent requirement need to be filled in with reasonable budget .

    $115 (Avg Bid)
    $115 Avg Bid
    3 bids

    I need to build an online python app this is the full detail of the exercixe. The company's desire is to preserve fruit biodiversity by allowing specific treatments for each fruit species by developing intelligent pickin...matrix written to a CSV or other file) available in cloud storage. A presentation support for the defense, presenting: the different architectural bricks chosen on the cloud; their role in Big Data architecture; the stages of the processing chain. USEFUL LINKS: link of others peoples work and the data

    $12 / hr (Avg Bid)
    $12 / hr Avg Bid
    7 bids

    The epub file must upload to ingram spark. This is not a kdp epub.

    $124 (Avg Bid)
    $124 Avg Bid
    23 bids

    Hi, I need a good understanding of 3 concepts. Kafka, Spark and Avro schema. These three are being used in my upcoming project, I need someone to help with below POC. Kafka: Install Kafka related software (producer/consumer/message-viewer) in my system. Sample program to post and retrieve messages from Kafka. Apache spark: Local setup in my system, and a good understanding of how it works Sample program to read Kafka messages in stream through Spark. Avro schema: How to generate a schema and use it. Local setup, and sample programs to read and write Avro schema messages from Kafka.

    $272 (Avg Bid)
    $272 Avg Bid
    4 bids

    Building data pipelines using spark and scala You will be provided mapping document in which all the information is mentioned like source and target tables , based on the business logic you need to create dataframes and write the code Based on your expertise amount can be increased and its a urgent requirement Ide - intellij , sbt Files format used mostly parquet Pease feel free to ping me on whats app on (Removed by Admin).

    $776 (Avg Bid)
    $776 Avg Bid
    4 bids

    parse a dat file which has headers and perform aggregation operation and add couple of columns with in the end of the existing with default data and write it back into file with the same file name.

    $21 (Avg Bid)
    $21 Avg Bid
    5 bids

    Hi, I need apache spark developer with pyspark experience. If you are interested please start your cover letter by saying " i am spark developer" Thanks

    $6 / hr (Avg Bid)
    $6 / hr Avg Bid
    8 bids

    Make spark more parallel our goal is to propose a new parallel performance model for different workloads of Spark Big Data applications running on HPC clusters. We need to add parallel technique on the rdd, to be parallelized. Then the execution will be faster. After we add parallel technique on RDDs, then we need to call openAcc and MPI pragmas to execute these parallel RDDs on GPU, we will call MPI and openAcc pragmas by a wrapper. For now we need to create a new parallel algorithm (foreach,for loops …etc) and apply it on the RDDs, but we need to make sure these new techniques are equivalent or appropriate to parallelization of MPI and OpenACC programming model. using Scala Language

    $313 (Avg Bid)
    $313 Avg Bid
    3 bids 30% theory and 70% practical. Fluency in English is must. Below are the Training topics of Interest-  Explore compute and storage options for data engineering workloads  Run interactive queries using Azure Synapse Analytics serverless SQL pools  Lessons  Data exploration and transformation in Azure Databricks  Explore, transform, and load data into the Data Warehouse using Apache Spark  Ingest and load data into the data warehouse  Transform data with Azure Data Factory or Azure Synapse Pipelines  Orchestrate data movement and transformation in Azure Synapse Pipelines  End-to-end security with Azure Synapse Analytics  Support Hybrid Transactional Analytical Processing (HTAP) with Azure Synapse Link  Real-time Stream Processing with Stream Analytics  C...

    $261 (Avg Bid)
    $261 Avg Bid
    4 bids

    I need a logo for my Educational company website: ; I want the logo invoke the concept of "Creativity & Innovation". I've just randomly put up a logo.

    $19 (Avg Bid)
    $19 Avg Bid
    82 bids

    Experience developing high-performing, secure, production quality code •data-...DI frameworks (like Guice), SOLID and DRY principles, REST frameworks, ORMs (like Hibernate), CI/CD solutions using Maven, Jenkins, TeamCity etc. • Deep expertise in object-oriented programming, preferably in Java and Java based frameworks • Nice to have: Experience with AWS platform, specifically Lambda, API Gateway, IAM •UI Knowledge • Nice to have: Experience with Big Data, specifically EMR, Spark, and Scala Additionally, we seek individuals with the following skills: • Strong analytic, problem solving, and programming ability • Strong understanding of algorithms and data structures • Initiative and willingness to work in fast paced, agile teams Excellent organ...

    $23 / hr (Avg Bid)
    $23 / hr Avg Bid
    27 bids

    deadline 16th april Project : Hadoop and Spark The purpose of this project is to support your in-class understanding of how data analytics stacks work and get some hands-on experience in using them. You will need to deploy Apache Hadoop as the underlying file system and Apache Spark as the execution engine. You will then develop several small applications based on them. Task 1: Launch a cluster of virtual machines in a cloud environment (e.g., AWS, Azure, or GCP). You will need to have one node as the master and at least two nodes as workers (slaves). Task 2: Deploy the HDFS service on the cluster. Task 3: Download the text version of Pride and Prejudice from Project Gutenberg, and save it to the HDFS cluster. Task 4: Deploy the Spark service on the cluster. Task 5: ...

    $162 (Avg Bid)
    $162 Avg Bid
    5 bids

    ...Laurea in Informatica, Information Technology o equivalente esperienza tecnica. - Almeno 3 anni di esperienza professionale. - Profonda conoscenza ed esperienza in statistica. - Previa esperienza in programmazione, preferibilmente in Python, Kafka o Java e volontà di apprende nuovi linguaggi. - Competenze su Hadoop v2, MapReduce, HDFS. - Buona conoscenza dei Big Data querying tools. - Esperienza con Spark. -Esperienza nel processare grandi quantità di dati, sia strutturati che non, inclusa l’integrazione di dati che provengono da fonti diverse. - Esperienza con NoSQL databases, come Cassandra o MongoDB. - Esperienza con vari sistemi di messagistica, come Kafka o RabbitMQ Durata attività: contratto di 12 mesi Impegno: full time Modalità di lavoro...

    $22 / hr (Avg Bid)
    $22 / hr Avg Bid
    6 bids

    Looking for someone who is really good at coding in Java, Spark and knows latest technologies. Don't apply if you're from an agency.

    $21 / hr (Avg Bid)
    $21 / hr Avg Bid
    16 bids

    Top spark Community Articles