tyrin/GaiaDBInterface
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Никита Тырин 2024-09-20 11:42:55 +03:00
parent e9c38ab778
commit c1e56539b7
1 changed files with 33 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
README.md
View File

@ -1,7 +1,36 @@
# GaiaDBInterface
A Django application for interacting with Gaia DR3 via ORM interface.
A Django application for interacting with Gaia DR3 via Django's ORM interface.
## Installation
- Install Django using pip. Creating a virtual environment (venv) is recommended. All python packages this application requires are specified in the `requirements.txt` file.
- Create a Django project;
- `cd` into the project's directory;
- Clone this repository: `git clone http://heagit.cosmos.ru/tyrin/GaiaDBInterface.git`;
- Add the application to the INSTALLED_APPS section in the settings.py of the project, also connect your database;
- Perform django migrations (`python manage.py makemigrations`, `python manage.py migrate`).
- Create URLs in the urls.py of the project, if you plan to use the API functionality.
## manage.py commands:
- `ingester` - ingests a directory of csv files into the database. Beware memory leaks.
- `indexer` - indexes the catalog in the database using healpix.
- `indexer-multithread` - a multithreaded version of indexer.
Because of the size of each file in GAIA DR3, ingesting and then indexing every file takes a long time, on the order of several minutes.
- `ingester` - ingests a directory of csv files into the database. Tuned to work with the csv distribution of GAIA DR3 in the ecsv format.
**Beware of memory leaks when ingesting a big number of files**; `monitor.py` script can be used to restart the script every hour, mitigating the issue.
Requests the folder containing the csv files at the start, with a 60 second timeout. Previous default choice is stored in `config.json`. `models.py` of the application contains models for both the database of ingested files, and the catalog sources. As for sources, most fields are disabled to save disk space; un-comment them to enable.
- `indexer` - indexes the catalog in the database using healpix. Requests the NSIDE parameter of the HEALPix grid at the start, remembers the previous value as default. Fills out columns in the source database with both the _ring_ and _nested_ HEALPix orders.
- `indexer-multithread` - a multithreaded version of indexer. **Is considerably faster, and should be used over the regular `indexer`**. Utilizes the `multiprocessing` module to divide each chunk between 8 processes, reference: https://rednafi.com/python/faster_bulk_update_in_django/ .
## API functionality:
This application also includes a cone search API utilizing the healpix indexation. It can be accessed using the GaiaAPIClient module. With django running on a server (or a development server, accessible via `python manage.py runserver`), a swagger documentation page is accessible at (with the default server) http://localhost:8000/api/schema/swagger-ui/#/GAIA .