Darshil Parmar
Darshil Parmar
  • 152
  • 6 429 216
My Data Engineering Journey in 5 Minutes | Projects | Tools
Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479
USE CODE: COMBO50 for 50% off
Learn more about courses here - datavidhya.com/courses
In this video, I have shared my experience of working as a Data Engineering for the last 5 years, from the projects to the tech stack that I worked on.
👦🏻 My Linkedin - www.linkedin.com/in/darshil-parmar/
📷 Instagram - darshilparmarr
🎯Twitter - parmardarshil07
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: ua-cam.com/play/PLBJe2dFI4sgtlK_zaqaIBdJFgieYPnQ07.html
👉Data Engineering Project Series: ua-cam.com/play/PLBJe2dFI4sgukOW6O0B-OVyX9c6fQKJ2N.html
👉Become Full-Time Freelancer: ua-cam.com/play/PLBJe2dFI4sgtza0sAnNFwo8KPG0GcO9Il.html
👉Data With Darshil Podcast: ua-cam.com/play/PLBJe2dFI4sgv_XmEDaXF3z1MNib7R3KUY.html
✨ Tags ✨
✨ Hashtags ✨
Переглядів: 11 547

Відео

What Does a Data Engineer Do? Explained Simply
Переглядів 14 тис.Місяць тому
Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Learn more about courses here - datavidhya.com/courses The Data Engineering market is going to cross the $100 Billion mark by 2028, this is the best time to learn about data engineering and start your career, in this video we will understand wh...
The State of Data Engineering in 2024 - Jobs, Salary, Layoff, AI and Future
Переглядів 40 тис.Місяць тому
Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Learn more about courses here - datavidhya.com/courses The Data Engineering market is going to cross the $100 Billion mark by 2028, this is the best time to learn about data engineering and start your career, in this video we will talk about sa...
IPL Data Analysis | Apache Spark End-To-End Data Engineering Project
Переглядів 52 тис.2 місяці тому
Enroll in the Apache Spark Course Here - datavidhya.com/courses/apache USE CODE: EARLYSPARK for 50% off ➡️ Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off In this video, we are going to analyze IPL data by building a data pipeline, main focus of this video is to focus on writing Apache Spark...
You're not stupid: How to learn difficult things | Complete Guide
Переглядів 21 тис.2 місяці тому
Enroll in the Apache Spark Course Here - datavidhya.com/courses/apache USE CODE: EARLYSPARK for 50% off ➡️ Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off How To Take Technical Notes? One of the best ways to retain whatever you learn is by preserving the information that you can access as pe...
Best Apache Spark Course with Databricks for Data Engineering | 2 End-To-End Projects
Переглядів 19 тис.2 місяці тому
Enroll in the Apache Spark Course Here - datavidhya.com/courses/apache USE CODE: EARLYSPARK for 50% off ➡️ Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Best course on Apache Spark with Databricks, you will learn everything from basics fundamentals, internal working, hands-on practice, an ...
How I Got 60LPA Remote Job as Data Engineer
Переглядів 131 тис.3 місяці тому
Join My Data Engineering Course Here: datavidhya.com/courses ➡️ Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Purchase Individual Courses: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT...
How To Read Tech Books? (Things I learned after reading 21 books)
Переглядів 12 тис.3 місяці тому
Join My Data Engineering Course Here: datavidhya.com/courses ➡️ Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Purchase Individual Courses: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT...
Data Engineering Certifications To Get Ahead Of 99% People
Переглядів 42 тис.4 місяці тому
Join My Data Engineering Course Here: datavidhya.com/courses You can directly purchase here: ➡️ Python SQL Data Warehouse Combo Plan - courses.datavidhya.com/checkout/zwbzl8e2Cx?code=COMBO50 Purchase Individual Courses: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT50 3. Data Warehouse for D...
Learn Snowflake in 10 Minutes| High Paying Skills | Step by Step Hands-On Guide
Переглядів 185 тис.4 місяці тому
Join My Snowflake Data Engineering Course Here: datavidhya.com/courses/datawarehouse ➡️ Combo Package Python SQL Data warehouse (Snowflake) Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Purchase Individual Courses: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/...
AWS Data Engineering Complete Roadmap 2024 (Top 10 Services To Focus)
Переглядів 40 тис.5 місяців тому
Join My Data Engineering Course Here: datavidhya.com/courses In this video, we will talk about how to become an AWS Data Engineer, the complete roadmap, top services to learn, and how to prepare for a certificate, I hope you find it useful. Resources mentioned in video: datawithdarshil.notion.site/AWS-Data-Engineering-Roadmap-cc3689031faf4e11b591df1a9139326f?pvs=4 🌟 Please leave a LIKE ❤️ and S...
Best Way To Land Data Analytics/Engineering/Science Job in 2024
Переглядів 21 тис.5 місяців тому
Join My Data Engineering Course Here: datavidhya.com/courses USE CODE: MERRYCHRISTMAS to get 50% off on courses Find Resources Mentioned In Video: datawithdarshil.notion.site/Cracking-DE-Interview-Guide-30725b58ae2b43f391c31d0b64b003fa?pvs=4 Timestamps 0:00 Why Data Engineering is the best career & Job openings 1:29 Getting Your Basics Clear 2:23 How To Build Data Portfolio Projects 5:51 How To...
God Tier Data Engineering Roadmap 2024 with End-To-End Projects
Переглядів 132 тис.6 місяців тому
Join My Data Engineering Course Here: datavidhya.com/courses You can directly purchase here: ➡️ Python SQL Data Warehouse Combo Plan - courses.datavidhya.com/checkout/zwbzl8e2Cx?code=COMBO50 Purchase Individual Courses: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT50 3. Data Warehouse for D...
My High Paying Data Engineering Projects Explained
Переглядів 20 тис.6 місяців тому
Join My Data Engineering Course Here: datavidhya.com/courses In this video, I have shared my high-paying data engineering projects where I made more than $100,000 in 1-2 years by working on these projects, these are projects that I worked with big companies/clients Get 250 End-To-End Projects (ProjectPro) - bit.ly/3GEuFgU Timestamps 0:00 My High-Paying Freelance Projects 0:37 250 HandsOn Projec...
Microsoft Azure Data Engineering Complete Roadmap 2024 (Top 10 Services To Focus)
Переглядів 104 тис.6 місяців тому
Join My Data Engineering Course Here: datavidhya.com/courses In this video, we will talk about how to become an Azure Data Engineer, the complete roadmap, top services to learn, and how to prepare for a certificate, I hope you find it useful. 🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟 Resources mentioned in the video - datawithdarshil.notion.site/Azure-Data-Engineering-R...
Recession Proof Data Roles To Focus in 2024 | Simply Explained
Переглядів 21 тис.7 місяців тому
Recession Proof Data Roles To Focus in 2024 | Simply Explained
These Data Engineering Projects Give You An Unfair Advantage
Переглядів 59 тис.7 місяців тому
These Data Engineering Projects Give You An Unfair Advantage
Asking Senior Engineer Manager about AI killing Coding Jobs? | Vlog 1
Переглядів 37 тис.8 місяців тому
Asking Senior Engineer Manager about AI killing Coding Jobs? | Vlog 1
High Paying Technologies You Should Learn In 2024
Переглядів 45 тис.8 місяців тому
High Paying Technologies You Should Learn In 2024
Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers
Переглядів 132 тис.8 місяців тому
Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers
Complete Freelance Guide | How I Earned $50000 in 6 Months as Freelance Data Engineer
Переглядів 144 тис.8 місяців тому
Complete Freelance Guide | How I Earned $50000 in 6 Months as Freelance Data Engineer
High-Paying Skill for Data Engineers | Master Data Warehouse with 4 Projects
Переглядів 18 тис.9 місяців тому
High-Paying Skill for Data Engineers | Master Data Warehouse with 4 Projects
Olympic Data Analytics | Azure End-To-End Data Engineering Project
Переглядів 285 тис.10 місяців тому
Olympic Data Analytics | Azure End-To-End Data Engineering Project
Freelance Data Engineer Shares His High Paying Projects
Переглядів 30 тис.10 місяців тому
Freelance Data Engineer Shares His High Paying Projects
Learn Apache Spark in 10 Minutes | Step by Step Guide
Переглядів 259 тис.11 місяців тому
Learn Apache Spark in 10 Minutes | Step by Step Guide
Best Data Engineering Books for Beginners
Переглядів 15 тис.11 місяців тому
Best Data Engineering Books for Beginners
How Dream 11 Process 7TB of Data Daily?
Переглядів 9 тис.Рік тому
How Dream 11 Process 7TB of Data Daily?
Azure Data Engineering Certification Step by Step Guide
Переглядів 125 тис.Рік тому
Azure Data Engineering Certification Step by Step Guide
Stock Market Data Pipeline Project on Google Cloud Platform
Переглядів 25 тис.Рік тому
Stock Market Data Pipeline Project on Google Cloud Platform
Master SQL in 15 Day for Data Engineering with 4 Projects
Переглядів 44 тис.Рік тому
Master SQL in 15 Day for Data Engineering with 4 Projects

КОМЕНТАРІ

  • @rishisurange8037
    @rishisurange8037 День тому

    This is what end to end means. Its a masterclass.

  • @Bhuvaneshscout
    @Bhuvaneshscout День тому

    This book means a lot

  • @clouderabypraful3686
    @clouderabypraful3686 День тому

    Bhai kaise samjha rahe ho kuch bhi nahi samjh raha title kya explanation kya

    • @DarshilParmar
      @DarshilParmar День тому

      Thoda basic knowledge required hai toh samje ga, comments padho sabko samajh raha hai

  • @nomalware2811
    @nomalware2811 День тому

    Hi I liked and commenting, Now Give me the book man !

  • @vaibhavgupta006
    @vaibhavgupta006 2 дні тому

    I think most users face problem in installing things, infact that acts as deterrent to follow the tutorials . Can you provide docker image of the setup

  • @dineshwijesuriya4899
    @dineshwijesuriya4899 2 дні тому

    I'm a totally a beginner. Completed the part 1 successfully. Truely thankful to you. ❤

  • @FirstNameLastName-fv4eu
    @FirstNameLastName-fv4eu 2 дні тому

    Poor guy does not even know he is getting scammed by AWS. this is what happens when you do things "unnecessarily" to make sure his/her CV has I know "AWS blah blah" :)

  • @vipinvivek5714
    @vipinvivek5714 3 дні тому

    Good job on explaining concepts clearly, but you need to not self promote your content too much, you can link it in the discription and just mention it once.

  • @likitan4076
    @likitan4076 3 дні тому

    Even i too did so much work better than u but still did not end up in a job even after having 8 years of exp

  • @hobbyofbobby2973
    @hobbyofbobby2973 3 дні тому

    Good job Darshil. Appreciate the work.

  • @sanskarjain1890
    @sanskarjain1890 3 дні тому

    for me wget command isnt working in ec2 cli can some one help me with it

  • @yashwanth4435
    @yashwanth4435 3 дні тому

    Hi Darshil, Need a favour I’ve recently taken a course but unable to access. Please help me out.

  • @beyyugi92
    @beyyugi92 4 дні тому

    So I might be interested but had a question- you mention that we build on projects from previous courses, but if we already know other basics, will these projects be started again?

    • @DarshilParmar
      @DarshilParmar 3 дні тому

      I included some parts of those course project that are important into current one

  • @user-lv4jm5vm7f
    @user-lv4jm5vm7f 4 дні тому

    i am encountering some error when i try to get text i.e. the productTitle is it because of captcha, please help

  • @Reactionz.3402
    @Reactionz.3402 4 дні тому

    I watched this video till the end.🇳🇬

  • @Reactionz.3402
    @Reactionz.3402 4 дні тому

    I'm going to become by Data Engineer before the end of 2024. The journey just began.

  • @BA-kh9ei
    @BA-kh9ei 4 дні тому

    It is such a great explanation.

  • @beyyugi92
    @beyyugi92 4 дні тому

    Can someone explain why we would use kafka vs simply using aws kinesis?

  • @labiqaekram4491
    @labiqaekram4491 4 дні тому

    when i am making storage account, it is not giving me free trial, it gives me azure subscription1. is it ok?

  • @user-zl4es9jc9w
    @user-zl4es9jc9w 4 дні тому

    Very helpful

  • @rahul-wz7rn
    @rahul-wz7rn 5 днів тому

    Cost is High

  • @shubhambhadra8196
    @shubhambhadra8196 5 днів тому

    "errorMessage": "Unable to import module 'lambda_function': No module named 'awssdkpandas'", "errorType": "Runtime.ImportModuleError", "stackTrace": [] getting this error even after adding layer "AWSSDKPandas-Python38" pls provide solution

  • @Moon01-ru5my
    @Moon01-ru5my 5 днів тому

    If youve done Sql very well then all you need here is just learning the few syntax differences in spark and Sql.

  • @ChandrakanthBoga
    @ChandrakanthBoga 5 днів тому

    Great work Darshil.. Thanks for sharing all your knowledge..

  • @vishalcrazy5121
    @vishalcrazy5121 5 днів тому

    Really good content .

  • @shrutigawande4168
    @shrutigawande4168 6 днів тому

    Hi Darshil, Thank you so much for this amazing video. I have started following your channel recently and I like all the videos. I am stuck in Lambda function. Since AWSDataWrangler has been updated to AWSSDKPandas-Python312 . I have added AWSSDKPandas-Python312 in layer and also updated code with 'import awssdkpandas as wr' but I still get error saying "Unable to import module 'lambda_function': No module named 'awssdkpandas'". Can you please explain this?

  • @HiraOmarAbbas
    @HiraOmarAbbas 6 днів тому

    Watched this video till the end

  • @marouanezaid4066
    @marouanezaid4066 6 днів тому

    Hi Darshil, Thank you for this amazing project When I am at the creation of the service account phase, I can't create the service access to "BigQuery Admin", there is no field for choosing this role, and only what I found is this message : "You do not have permission to set policy on the project. The permission 'resourcemanager.projects.setIamPolicy' is required to grant project-level permissions to the service account." PS : I use only a sandbox account that is going to disappear after 4 hours

  • @shivadev9558
    @shivadev9558 6 днів тому

    will u teach full course in one 10 working days.

  • @VipinKumar-yy9ez
    @VipinKumar-yy9ez 6 днів тому

    Darshil, I appreciate the effort you put into creating such helpful content-it's clear why this video deserves all the appreciation it gets! Looking forward to putting your advice into practice and achieving success in my job search

  • @Srikanth-v8j
    @Srikanth-v8j 7 днів тому

    I WILL CLEAR THIS EXAM IN 30 DAYS

  • @RajeshwarBusa
    @RajeshwarBusa 7 днів тому

    I watched the video till the end. Nicely explained

  • @anuragsrigyan677
    @anuragsrigyan677 7 днів тому

    Bro i am getting S3 connection issue. i checked the ~.aws/configuration file.. access key id ad secret is defined there but still not connecting..ClientError: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records. SO I AM SAVING IT IN MY LOCAL.

  • @ritupriyasingh916
    @ritupriyasingh916 7 днів тому

    great resource

  • @RiyaSingh-zd3yl
    @RiyaSingh-zd3yl 7 днів тому

    whenever i am clicking raw button on githhub ,an excel file of the data is downloaded and i am not able to get the raw data url....can anybody eplain ,how to fix it

  • @anuragsrigyan677
    @anuragsrigyan677 7 днів тому

    Bro I know some amount of Java but I know Pandas also..what do I go with Python or Java for DSA?

  • @amritmanash7950
    @amritmanash7950 8 днів тому

    Thank you so much this helped a lot to understand the data pipeline flow can you please help create a fully industry level azure data engineering project that contains the data from multiple sources and we are ingesting it using ADF and then we have 3 layers of Databricks notebook for cleaning and transformation and then also we are creating the Email event as soon as our pipeline is success or failure it should sent an email using Logic app and then finally the transformed data is loaded on to the azure synapse analytics

  • @cricbolly3252
    @cricbolly3252 8 днів тому

    I will became a Great Data Engineer

  • @MALIK_AR7
    @MALIK_AR7 8 днів тому

    hello darshal, I hope you are well i have decided to become a GCP data engineer. what advice would u give and what tech stack should I focus on. regards.

  • @harshitdtu7479
    @harshitdtu7479 8 днів тому

    Response { "statusCode": 200, "body": "\"Hello from Lambda!\"" } what does this means

  • @harshitdtu7479
    @harshitdtu7479 8 днів тому

    why did'nt we use ETL job instead of AWS Lambda?

  • @vasundras4746
    @vasundras4746 8 днів тому

    This is the best video on UA-cam right now 💯 we need more of this energy !!