KHUGPT airflow Engine

Data workflow for KHUGPT using airflow.




Report Issues · Pull Requests

(back to top)

• python v3.8+
• docker v18+

I recommend building your environment on top of a virtual environment or docker container. (e.g., venv, anaconda, etc.)

• Running Airflow locally

  #start airflow
  make start-airflow
  
  #stop airflow
  make stop-airflow

• Linting & Testing

  # Linting
  make lint
  
  # Testing
  make test

• Cleaning up your local environment

  make reset-airflow

(back to top)

• Providing collectors for data collection
• Loading into NoSQL (e.g., MongoDB)
• Trasform and load for loading into Vector Store (e.g., Pinecone)

(back to top)

Please refer to CONTRIBUTING.md for Contribution.

For issues, new functions and requests to modify please follow the following procedure. 🥰

1. Fork the Project
2. Create a Issue when you have new feature or bug, just not Typo fix
3. Create your Feature Branch from dev Branch (git checkout -b feat/Newfeature)
4. Commit your Changes (git commit -m 'feat: add new feature')
5. Push to the Branch (git push origin feat/Newfeature)
6. Open a Pull Request to dev branch with Issues

(back to top)

Please refer to LICENSE for LICENSE.

(back to top)

Yongwoo Song

(back to top)