Deploy Airflow on Heroku in Windows 10
This tutorial will walk you through on how to deploy airflow using Git, VSCode and Heroku CLI.
Prerequisites:
· Windows 10 operating system with stable internet.
· Must have a basic knowledge in Linux and Git bash command.
· Must have a Heroku account.
· Must have a basic knowledge in Python scripts.
· Must have a basic knowledge in Airflow and Dag’s file.
All instruction were easy to follow just download the script from my GitHub links and used it accordingly. If the instructions are not clear feel free to asked or watch the YouTube video for this topic: https://www.youtube.com/watch?v=Xa4Rw-cbKIQ .
What is Heroku?
Heroku is a cloud platform that let companies build, deliver, monitor and scale apps. Similar to GitHub and AWS or Microsoft Azure, you can deploy yours apps using your favorite programming language you use.
What is Airflow?
Airflow is an open-source platform written in Flask that monitors your workflows and with the help of DAGs that execute your tasks at a given time.
Shall we start?…
Step 1.
Enabling Windows Subsystem for Linux using Settings.
For this first step will walk you through on how to activate the WSL in Windows 10 and setup your Linux settings.
To install WSL using Setting on Windows 10, use these steps:
1. Open Settings.
2. Click on Apps.
3. Under the “Related settings” section, click the Programs and Features option.
4. Click the Turn Windows features on or off option from the left pane.
5. Check the Windows Subsystem for Linux option.
6. Click the OK button.
7. Click the Restart now button.
Installing Linux distros using Microsoft Store.
To install a distribution of Linux on Windows 10, use these steps:
1. Open Microsoft Store.
2. Search for the Linux distribution that you want to install.
Some of the distros available include:
Ubuntu.
OpenSUSE Leap 15.
Kali Linux.
Debian.
Alpine WSL.
Suse Linux Enterprise 12.
Select the distro of Linux to install on your device.
3. Click the Get (or Install) button.
4. Click the Launch button.
5. Create a username for the Linux distro and press Enter.
6. Specify a password for the distro and press Enter.
7. Repeat the password and press Enter to confirm.
8. Check if wsl alread install.
open command prompt and type wsl .
Step 2.
The Workspace.
In this step will setup the workspace using Visual Studio Code and install the necessary requirements for airflow with the help of python Pip and Git.
1. Update your Linux distro.
sudo apt update && sudo apt upgrade
2. Enter this script in the wsl.conf.
type sudo nano /etc/wsl.conf
[automount]
root = /
options = “metadata”
3. Install pip3.
sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get install python3-pip
pip3 -V
4. Create a Workspace Folder.
airflow_tutorial
5. Open WF in VSCode then create a Virtual Environment.
sudo apt-get install python3-venv
python3 -m venv .venv
source. venv/bin/activate
6. Copy the requirement text file and save inside the folder airflow tutorial then install.
pip3 install -r requirements.txt
alternative:
- install apache airflow
pip3 — default-timeout=100 install “apache-airflow[postgres, password]”
- save pip requirements to text file
pip3 freeze | grep -v “pkg-resources” > requirements.txt
7. create Procfile.
web: airflow db init
8. create .gitignore.
.venv/
__init__.py
__pycache__/
9. Git Setup.
git init
git add .
git commit -m “initial commit”
Step 3.
I n this procedure will install Heroku using curl, as normal Heroku cli wont work in WSL environment , also will setup the requirements for the airflow-app in Heroku including the database.
1. Install heroku.
curl https://cli-assets.heroku.com/install.sh | sh
you’ll need to include -a “your-app-name” at the end of every command you run.
2. Login to Heroku Website.
heroku login
3. Create your app.
heroku create heroku-airflow
4. Create Heroku Postgresql database.
heroku addons:create heroku-postgresql:hobby-dev -a heroku-airflow
5. Setup Heroku Config.
heroku config -a heroku-airflow
heroku config:set AIRFLOW__CORE__SQL_ALCHEMY_CONN=”postgresql://”
heroku config:set AIRFLOW__CORE__LOAD_EXAMPLES=False -a heroku-airflow
heroku config:set AIRFLOW_HOME=/app -a heroku-airflow
Run this line in Python.
>>>from cryptography.fernet import Fernet;
>>>print(Fernet.generate_key().decode())
heroku config:set AIRFLOW__CORE__FERNET_KEY=<secret_key> -a heroku-airflow
6. Push to Heroku Master.
git push heroku master
7. Check if no error.
heroku logs — tail -a heroku-airflow
8. Modify Procfile.
web: airflow webserver — port $PORT
9. Setup Additional Heroku Config.
heroku config:set AIRFLOW__WEBSERVER__AUTHENTICATE=True -a heroku-airflow
heroku config:set AIRFLOW__WEBSERVER__AUTH_BACKEND=airflow.contrib.auth.backends.password_auth -a heroku-airflow
10. git update.
git add .
git commit -m “change Procfile to start the webserver”
git push heroku master
11 Check if no error.
heroku logs — tail -a heroku-airflow
12 Preview.
Login to Heroku and goto app then click “open app”
Step 4.
I n this Steps we successfully uploaded our first airflow app in Heroku, from here were going to create the user Admin and upload our dags file for testing.
1. create a user account using ssh into the Heroku instance of our app:
heroku run bash -a heroku-airflow
airflow users create -r Admin -u admin -e <email address> -f <firstname> -l <lastname> -p <password>
2. exit bash.
exit
3. Login using the admin account using the information created earlier.
4. Create a sample dag file inside a dags folder.
Create folder name “dags” then create a dag file name “hellodags.py”.
“hellodags.py”
from datetime import datetime
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
def print_hello():
return ‘Hello world!’
dag = DAG(‘hello_world’, description=’Simple tutorial DAG’,
schedule_interval=’0 12 * * *’,
start_date=datetime(2017, 3, 20), catchup=False)
dummy_operator = DummyOperator(task_id=’dummy_task’, retries=3, dag=dag)
hello_operator = PythonOperator(task_id=’hello_task’, python_callable=print_hello, dag=dag)
dummy_operator >> hello_operator
5. Update dags in git.
git add dags
git commit -m “create hello dag”
6. update Procfile by setting up the auto webserver and scheduler.
web: airflow webserver — port $PORT — daemon & airflow scheduler
7. update the changes using git.
git add Procfile
git commit -m “change Procfile to run the webserver as daemon and scheduler”
8. Finally update changes to heroku master.
git push heroku master
9. Login again to airflow for checking.
Note:
I f there are no dags file display in the list then run again this line settings below then check again.
heroku config:set AIRFLOW_HOME=/app -a heroku-airflow
Then reload the airflow website to see changes.
That’s it! …..
There might be a lot of easy steps for you to perform this installation, as for this tutorial you can use it at your own risk. Hope you enjoy and feel free to asked. Happy coding!..
