GaiaDBInterface

A Django application for interacting with Gaia DR3 via Django's ORM interface.

Installation

• Install Django using pip. Creating a virtual environment (venv) is recommended. All python packages this application requires are specified in the requirements.txt file.

• Create a Django project;

• cd into the project's directory;

• Clone this repository: git clone http://heagit.cosmos.ru/tyrin/GaiaDBInterface.git;

• Add the application to the INSTALLED_APPS section in the settings.py of the project, also connect your database;

• Perform django migrations (python manage.py makemigrations, python manage.py migrate).

• Create URLs in the urls.py of the project, if you plan to use the API functionality.

manage.py commands:

Because of the size of each file in GAIA DR3, ingesting and then indexing every file takes a long time, on the order of several minutes.

• ingester - ingests a directory of csv files into the database. Tuned to work with the csv distribution of GAIA DR3 in the ecsv format. Beware of memory leaks when ingesting a big number of files; monitor.py script can be used to restart the script every hour, mitigating the issue. Requests the folder containing the csv files at the start, with a 60 second timeout. Previous default choice is stored in config.json. models.py of the application contains models for both the database of ingested files, and the catalog sources. As for sources, most fields are disabled to save disk space; un-comment them to enable.

• indexer - indexes the catalog in the database using healpix. Requests the NSIDE parameter of the HEALPix grid at the start, remembers the previous value as default. Fills out columns in the source database with both the ring and nested HEALPix orders.

• indexer-multithread - a multithreaded version of indexer. Is considerably faster, and should be used over the regular indexer. Utilizes the multiprocessing module to divide each chunk between 8 processes, reference: https://rednafi.com/python/faster_bulk_update_in_django/ .

API functionality:

This application also includes a cone search API utilizing the healpix indexation. It can be accessed using the GaiaAPIClient module. With django running on a server (or a development server, accessible via python manage.py runserver), a swagger documentation page is accessible at (with the default server) http://localhost:8000/api/schema/swagger-ui/#/GAIA .