A command-line interface (CLI) for working with fiboa files.

• Getting Started
• Documentation / Commands
• Development

In order to make working with fiboa easier we have developed command-line interface (CLI) tools such as inspection, validation and file format conversions.

This project uses Pixi for dependency management. Install Pixi first, then:

# Clone the repository and navigate to it
git clone https://github.com/fiboa/cli.git
cd cli

# Install all dependencies
pixi install

# Run the CLI
pixi run fiboa

Alternatively, you can install from PyPI with Python 3.10 or any later version:

pip install fiboa-cli

After the installation you should be able to run the following command: fiboa (or pixi run fiboa if using Pixi)

You should see usage instructions and available commands for the CLI.

fiboa CLI supports various commands to work with the files:

• fiboa CLI
  • Getting Started
    • Installation
      • Using Pixi (Recommended)
      • Using pip
    • Execute a command
  • Commands
    • Validation
    • Create fiboa GeoParquet from GeoJSON
    • Create fiboa GeoJSON from GeoParquet
    • Inspect fiboa GeoParquet file
    • Merge fiboa GeoParquet files
    • Create JSON Schema from fiboa Schema
    • Validate a Vecorel Schema
    • Improve a fiboa Parquet file
    • Update an extension template with new names
    • Converter for existing datasets
  • Development
    • Implement a converter

To validate a fiboa GeoParquet or GeoJSON file, you can for example run:

• GeoJSON: fiboa validate example.json --collection collection.json
• GeoParquet: fiboa validate example.parquet --data

Check fiboa validate --help for more details.

The validator also supports remote files.

• http:// or https://: no further configuration is needed.
• s3://: With Pixi, run pixi install -e s3 or with pip, run pip install fiboa-cli[s3] and you may need to set environment variables. Refer to the s3fs credentials documentation for how to define credentials.
• gs://: With Pixi, run pixi install -e gcs or with pip, run pip install fiboa-cli[gcs]. By default, gcsfs will attempt to use your default gcloud credentials or, attempt to get credentials from the google metadata service, or fall back to anonymous access.

To create a fiboa-compliant GeoParquet for a fiboa-compliant set of GeoJSON files containing Features or FeatureCollections, you can for example run:

• fiboa create-geoparquet geojson/example.json -o example.parquet -c geojson/collection.json

Check fiboa create-geoparquet --help for more details.

To create one or multiple fiboa-compliant GeoJSON file(s) for a fiboa-compliant GeoParquet file, you can for example run:

• GeoJSON FeatureCollection: fiboa create-geojson example.parquet -o dest-folder

• GeoJSON Features (with indentation and max. 100 features): fiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -f

  fiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -f

Check fiboa create-geojson --help for more details.

To look into a fiboa GeoParquet file to get a rough understanding of the content, the following can be executed:

• fiboa describe example.parquet

Check fiboa describe --help for more details.

Merges multiple fiboa datasets to a combined fiboa dataset:

• fiboa merge ec_ee.parquet ec_lv.parquet -o merged.parquet -e https://fiboa.org/hcat-extension/v0.1.0/schema.yaml -i ec:hcat_name -i ec:hcat_code -i ec:translated_name

Check fiboa merge --help for more details.

To create a JSON Schema for a fiboa Schema YAML file, you can for example run:

• fiboa jsonschema example.json --id=https://vecorel.org/specification/v0.3.0/geojson/schema.json -o schema.json

Check fiboa jsonschema --help for more details.

To validate a Vecorel Schema YAML file, you can for example run:

• fiboa validate-schema schema/schema.yaml

Check fiboa validate-schema --help for more details.

Various "improvements" can be applied to a fiboa GeoParquet file. The commands allows to

• detect fiboa-0.2 files and convert them to fiboa-0.3
• change the CRS (--crs)
• change the GeoParquet version (-gp1) and compression (-pc)
• add/fill missing perimeter/area values (-sz)
• fix invalid geometries (-g)
• rename columns (-r)
• add HCAT columns (--hcat=mapping.csv) based on the input crop:code (from crop-extension) and a hcat-mapping csv file

Example:

• fiboa improve file.parquet -o file2.parquet -g -sz -r old=new -pc zstd

Check fiboa improve --help for more details.

Once you've created and git cloned a new extension, you can use the CLI to update all template placeholders with proper names.

For example, if your extension is meant to have

• the title "Administrative Division Extension",
• the prefix admin (e.g. field admin:country_code or admin:subdivision_code),
• is hosted at https://github.io/vecorel/administrative-division-extension (organization: vecorel, repository /administrative-division-extension),
• and you run Vecorel in the folder of the extension.

Then the following command could be used:

• fiboa rename-extension . -t "Administrative Division" -p admin -s administrative-division-extension -o vecorel

Check fiboa rename-extension --help for more details.

The CLI ships various converters for existing datasets.

To get a list of available converters/datasets with title, license, etc. run:

• fiboa converters

Use any of the IDs from the list to convert an existing dataset to fiboa:

• fiboa convert de_nrw

See Implement a converter for details about how to

fiboa publish <dataset> -o <target>

The publish converts and publishes a fiboa dataset to source coop or your own s3 repository. The target directory will be filled with the following files:

<target>/
  <dataset>.parquet
  <dataset>.pmtiles      # requires working ogr2ogr and tippecanoe
  stac/collection.json
  README.md              # generated if --generate-meta/-gm flag is present
  LICENSE.txt            # generated if --generate-meta/-gm flag is present

This directory is synchronized to the s3 repository (default source.coop/fiboa/data).

Requirements: Requires the aws CLI to be installed, and AWS_ACCESS_KEY_ID with AWS_SECRET_ACCESS_KEY environment variables. Also, for generating the pmtiles file, it requires ogr2ogr and tippecanoe.

The command executes the following steps:

• fiboa convert to generate a fiboa parquet dataset. All convert parameters are passed to the converter.
• fiboa validate to validate the fiboa dataset
• creates a .pmtiles from the parquet file. Uses ogr2ogr and tippecanoe
• fiboa create-stac-collection to create a STAC collection
• fiboa publish to publish the fiboa dataset to a source coop or your own s3 repository

Examples:

• fiboa publish at_crop -o data/at_crop
• fiboa publish -c /tmp/cache -gm br_conab -o data/br_conab

Relevant parameters:

• --generate-meta/-gm Generatse the README.md and LICENSE.txt files if absent, based on data-survey and converter properties.
• --data-url The URL to the data repository, used when generating the README
• --s3-upload-path The aws s3 sync target. Defaults to s3://source.coop/fiboa/data . Uploading requires the aws CLI, and AWS_ACCESS_KEY_ID with AWS_SECRET_ACCESS_KEY environment variables.

Check fiboa publish --help for more details.

This project uses Pixi for dependency management and development workflows.

# Install all dependencies including development tools
pixi install -e dev

# Install the package in editable mode
pixi run install-dev

# Run tests
pixi run test

# Format and lint code
pixi run format
pixi run lint

# Run all checks (lint, format, test)
pixi run check

# Install and run pre-commit
pixi run pre-commit-install
pixi run pre-commit-run

The following high-level description gives an idea how to implement a converter in fiboa CLI:

1. Create a new file in fiboa_cli/datasets based on the template.py
2. Fill in the required variables / test it / run it
3. Add missing dependencies into the appropriate feature group in pixi.toml (or setup.py for pip users)
4. Add the converter to the list above
5. Create a PR to submit your converter for review

There's a Dockerfile based on a modern Ubuntu+GDAL image. With the Dockerfile, you can run the CLI in a container. This can be handy if you lack the required dependencies on you local machine (e.g. a modern GDAL capable of reading GeoParquet files).

Example usage (mounting /tmp/fiboa as a volume)

docker build . -t fiboa
docker run -it --rm -v /tmp/fiboa:/fiboa fiboa bash

fiboa convert de_nrw -o /fiboa/de_nrw.parquet

If you want to temporarily work on a specific branch of fiboa-cli, you can use the following in the container. This works because the fiboa-cli is pip-installed with -e (in-place):

cd fiboa && git checkout <branch>