User Manual
============

In this manual the general functionality of EODIE is described. In the end you can also find a usage example :ref:`example` (no external data needed).
For more examples with explanations, please check out :ref:`tutorial`.

Inputs 
^^^^^^^

The following sections describe EODIEs command line arguments and the configuration files. All of the following only matters when using EODIE as command line tool. 

Command line arguments
++++++++++++++++++++++

The following parameters and flags can be used in the commandline (more information on each parameter and flag below):

``python eodie_process.py --platform <> (--rasterdir <> / --rasterfile <> / --use_zarr) --download_files --vector <> --out <> --id <>  --gpkg_layer <> --epsg_for_csv <> --statistics_out --geotiff_out --array_out --database_out --statistics <> --index <> --start <> --end <> --tiles <> --tifbands <> --nc_vars <> --nc_coords <> --epsg_for_nc <> --exclude_border --external_cloudmask <> --no_cloudmask --maxcloudcover <> --resampling_method <> --delete_invalid_geometries --point_geoms --verbose --test --delayed_chunk_size <> --nodata <> --epsg_for_output <> --redownload --db_indices <>``

Note that some parameters have options, some have defaults and some are optional, all flags are optional. See :ref:`nec_input` for inputs that need to be given.

| ``--platform``
| Which platform does the data come from? 
| **type:** String
| **options:** s2, ls8, tif, nc
| The platform information needs to be given in order for EODIE to be able to read the data the right way, eg find tile and date information in the filename. It is possible to extent EODIE to be able to process other datasources than given in options. Please refer to :ref:`extending_eodie` for further information.

| ``--rasterdir``
| The directory where the data to be processed is stored as absolute path. 
| **type:** String
| EODIE can either be given a directory to process data from or a single file (use `--rasterfile` parameter instead). If a directory contains other data than what matches with `--platform`, `--startdate`/`--enddate`, (`maxcloudcover` in config) and the area of interest given as vectorfile, EODIE finds the fitting data based on these inputs. Avoid having dates in the format of YYYYMMDD and tilenames in format 00XXX in the path!

| ``--rasterfile``
| If only one file shall be processed use ``--rasterfile`` instead of ``--rasterdir``. Cannot be used together with ``--rasterdir``.
| **type:** String

| ``--use_zarrs``
| Flag to indicate if zarr files from eopf stac should be used instead of old .SAFE files. Cannot be used together with ``--rasterdir`` or ``rasterfile``.
| **type:** flag
| Either `--rasterdir`, `--rasterfile` or `--use_zarrs` needs to be given by user. Zarr files "live" on the cloud from where they are automatically accessed without the need to download anything. Currently processing of locally stored zarr files is not supported.

| ``--download_files``
| Flag to indicate whether imagery should be automatically downloaded. Sentinel-2 images are downloaded from CDSE and Landsat-8 images from Microsoft Planetary Computer. Downloaded files ar stored in ``--rasterdir``. 
| **type:** flag
| Downloading Sentinel-2 images requires CDSE credentials, which should be accessible as environment variables. Downloading Landsat-8 images requires no credentials.

| ``--vector``
| Absolute path to the vector file to be used for processing.
| **type:** String
| The given vector defines the area of interest. Internally, EODIE reads the vector file into a geopandas GeoDataFrame.

| ``--out``
| Absolute path to the directory where the results shall be stored. Will be created if it does not exist.
| **type:** String
| **default:** ``./results``

| ``--id``
| Name of the unique ID-field of the vector file provided at ``--vector``.
| **type:** String
| **example:** ``--id id``
| Not all vector files use `id` as the fieldname for the ID field, it can be `ID`, `PlotID`,`FieldID`,`plotnumber`, etc. The possibilities are endless. Therefore EODIE cannot find the right field automatically and it has to be given by the user. If your vector file is not a GeoPackage, you may examine available fieldnames with the auxiliary script `examine_vectorfile.py`. With GeoPackage, you may use `examine_geopackage.py` (see also :ref:`auxfiles`).

| ``--gpkg_layer``
| If the input vectorfile is a GeoPackage (.gpkg), insert the name of the layer in GeoPackage to be processed if there are more than one layer. With one layer only this parameter is not needed.
| **type:** String
| **default:** None

| ``--epsg_for_csv``
| If the input vectorfile is a Comma-Separated Value (.csv), a spatial reference system needs to be defined separately for a successful reading into a GeoDataframe.
| **type:** String
| **default:** None

| ``--statistics_out``
| set flag if statistics (see below) shall be calculated per polygon and saved as csv
| **type:** flag

| ``--geotiff_out``
| set flag if geotiff shall be extracted per polygon and saved as geotiff
| **type:** flag

| ``--array_out``
| set flag if arrays of all polygons shall be extracted and saved as pickle
| **type:** flag
| If none of the three above is given, only --statistics_out is set to true

| ``--database_out``
| set flag if statistics (see below) shall be calculated ber polygon and saved into a sqlite3 .db file. Please be aware that with several simultaneous processes, the output database file can get locked and prevent writing results. 
| **type:** flag

| ``--statistics``
| If --statistics_out or --database_out is given, specify here which statistics shall be calculated per polygon separated by a space
| **type:** list of Strings
| **options:** one or more of: sum, std, median, mean, majority, minority, unique, range, percentile_x (with x from 0 to 100)
| **default:** ``count`` (always included)
| **example:** ``--statistics sum median percentile_10 percentile_90``

| ``--index``
| Which vegetation index or band shall be extracted per polygon separated by a space
| **type:** list of Strings
| **options:** one or more of ndvi, rvi, savi, nbr, kndvi, ndmi, mndwi, evi, evi2, dvi, cvi, mcari, ndi45, tctb, tctg, tctw, ndwi, plus bands as named in platform filenames (e.g. for Sentinel-2: B02, B03, B04, B05, B06, B07, B08, B8A, B11, B12)
| **example:** ``--index ndvi evi2 B04 B8A``

| ``--start``
| Give the startdate of the timeframe of interest
| **type:** integer YYYYMMDD
| **default:** ``20160101``

| ``--end``
| Give the enddate of the timeframe of interest
| **type:** integer YYYYMMDD
| **default:** todays date

| ``--tiles``
| Sentinel-2 Tile or tiles to be processed in format XX*** where X are numbers and * are letters
| **type:** list of Strings

| ``--tifbands``
| Bands of tif to be processed
| **type:**  list of integers
| **default:** [1]

| ``--nc_vars``
| Variables of netcdf file to be processed
| **type:** list of String

| ``--nc_coords``
| Coordinate names of the dimensions of variables specified in ``--nc_vars``. First possible time dimension, then latitude and longitude
| **type:** list of String
| **example:** ``--nc_coords time lat lon``

| ``--epsg_for_nc``
| EPSG code for crs of netcdf.
| **type:** integer

| ``--delete_invalid_geometries``
| Flag to indicate that invalid geometries should be excluded from further processing. Does not necessarily work on all 
| **type**: flag

| ``--exclude_border``
| Flag to indicate that border pixels (within the polygon) should be excluded from statistics calculations / array extraction
| **type:** flag

| ``--external_cloudmask``
| [optional] Absolute path and name of external cloudmask (without tile and date and extension), if available
| **type:** String

| ``--no_cloudmask``
| Flag to indicate that no cloudmasking should be performed
| **type:** flag

| ``--verbose``
| For getting information and warnings in the terminal as well as the log file
| **type:** flag

| ``--test``
| For testing some datatypes are set to smaller, in general not needed by user 
| **type:** flag

| ``--maxcloudcover``
| A value restricting the processing of imagery with too high cloud coverage. Currently only working with Sentinel-2 imagery.
| **type:** integer
| **default**: 99

| ``--resampling_method``
|  If bands are not available directly in the given pixelsize, they need to be resampled. Here the resampling method for up- and downsampling can be changed.
| **Options:** Available resampling methods and a short description can be found here: https://rasterio.readthedocs.io/en/latest/api/rasterio.enums.html#rasterio.enums.Resampling. If zarr files are used, only options "nearest", "linear", "bilinear" and "cubic" are available.
| **Example:** ``resampling_method: 'bilinear'`` will use bilinear resampling for all necessary resampling of the rasterdata

| ``--delayed_chunk_size``
| Maximum number of delayed processes to run at once. Running fewer simultaneous processes reduces memory usage. Applied only to the data extraction step, as it is the most memory consuming step.
| **type:** integer 

| ``--nodata``
| Value denoting no data in raster
| **type:** integer

| ``--point_geoms``
| Flag to indicate that geometries in vectorfile are points. If set, EODIE uses faster method for extracting values at point locations. 
| **type:** flag
| If set, ``--statistics`` is overwritten with single ``value`` field and only csv and database outputs are produced. 
| If not set and vectorfile contains point geometries, slower rasterstats-based extraction process will be executed. If set and vectorfile contains other than point geometries, non-point geometries are excluded from processing.
| If your areas of interest are small compared to raster pixelsize so that most areas intersect with only one pixel, it might be wise to precompute area centroids and use point location extraction for better performance.

| ``epsg_for_output``
| EPSG code of the desired output geotiff crs. If not given, output crs will be source raster crs.
| **type:** integer

| ``--redownload``
| Flag to indicate if already existing s2 or ls8 files should be redownloaded. Only used if ``--download_files`` is set.
| **type:** flag

| ``--db_indices``
| List of indexes to create to output database for faster queries. Either single column index or multicolumn index with column names separated by comma.
| **type:** list of string
| **default:** [id]
| **example:** ``--db_indices id id,date``

Configuration files
+++++++++++++++++++

[TODO: image of Sentinel-2 tiles over Finland]

EODIE includes platform-specific configuration files called config_x.yml with x being some satellite platform name, tif or nc. Generally, these configuration files do not need to be touched or changed, but there are few exceptions. These are:

- If a 'red edge' band should be used in indices instead of the nir band, that could be changed in the platform specific configuration files. See more about this and about the possibility of extending EODIE to work with other platforms in ref:`platform_spec`.

- If you wish to change the pixelsize for the outputs (for geotiffs and arrays), the value can be changed in these configuration files. Only available for s2 and ls8.

- For downloading Landsat-8 images from Microsoft Planetary Computer, file config_ls8.yml includes a list of landsat assets to download. The default option includes surface reflectance bands (B01-B07) and metadata files.

| ``pixelsize`` 
| Enter the pixelsize that you want your results to be in. Bands are then resampled to match the given pixelsize. This has most influence on geotiff or array outputs.
| **Type:** Integer
| **Example:** ``pixelsize: 10`` will use bands that are available in 10 m as is and resample bands that are only available in larger pixelsizes to 10m before extracting statistics/array/geotiff

| ``landsat_assets``
| Enter landsat assets to download, For options see https://planetarycomputer.microsoft.com/dataset/landsat-c2-l2 for item-level assets and their STAC keys.
| **Type:** list
| **Example:** ``landsatassets:
  - ang          # ANG.txt
  - mtl.txt      # MTL.txt
  - mtl.xml      # MTL.xml
  - qa_pixel     # QA_PIXEL.TIF
  - qa_radsat    # QA_RADSAT.TIF
  - qa_aerosol   # SR_QA_AEROSOL.TIF
  - coastal      # SR_B1.TIF
  - blue         # SR_B2.TIF
  - green        # SR_B3.TIF
  - red          # SR_B4.TIF
  - nir08        # SR_B5.TIF
  - swir16       # SR_B6.TIF
  - swir22       # SR_B7.TIF``

Environment variables
+++++++++++++++++++++

In order to automatically download Sentinel-2 imagery from Copernicus Data Space Ecosystem (CDSE), EODIE needs to have access to your CDSE credentials. These credentials should be stored as environment variables.

.. code-block:: bash
  
  CDSE_USERNAME='your_username'
  CDSE_PASSWORD='your_password'

.. _nec_input:

Necessary inputs
^^^^^^^^^^^^^^^^^

| ``--platform --rasterdir/--rasterfile --vector --out --id`` and at least one of  ``--statistics_out/--geotiff_out/--array_out/--database_out``
| ``--index`` also needs to be given, unless ``--platform tif``
| if ``--platform nc``, ``--nc_vars``, ``nc_coords`` and ``epsg_for_nc`` need to be given


Outputs
^^^^^^^^

* A logfile: rasterdir_YYYY-MM-DD.log if --rasterdir was given; rasterfile_YYYY-MM-DD_HH-mm-ss if --rasterfile was given.

| ``--statistics_out``

* One csv per tile, band/vegetation index and date with polygon identifiers in the first column and statistics is the following columns.

| ``--array_out``

* One pickeled numpy array per tile, band/vegetation and date with all polygons

| ``--geotiff_out``

* One geotiff with georeferenced raster per tile, band/vegetation index and polygon

| ``--database_out``

* One SQLite3 database file (.db) that contains results for given indices or bands in tables. Structurally content is similar to statistics.


Usage of external cloudmask
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If a cloudmask for each file to be processed is available from an external source, make sure the cloudmask fulfills the following requirements:

* date (YYYYMMDD, eg 20210603) and tilename (NNCCC , eg 34VFN) in end of filenames: xxx_date_tile.xx
* supported raster file format (.tif, .jp2, and other formats supported by rasterio)
* binary rastervalues (1,0; with 1 representing clouds/invalid pixels)
* pixelsize == output pixelsize (given in config_x.yml)

The latter two criteria can be achieved by using the auxiliary script create_binary_cloudmask.py (but be aware of issue https://gitlab.com/eetun-tiimi/EODIE/-/issues/62)