Date   

Handling file extensions and drivers

hannes-fiona.groups.io@...
 

For a Fiona-based command line tool I am wondering how to best to handle user-selected output file formats and drivers.

Is there a ready-to-use way to select the driver based on the file extension? Maybe via a mapping of driver -> file extension?

E.g. if a user specified "out.geojson", the driver should be GeoJSON, if "out.shp" then use "ESRI Shapefile", etc.


Re: Handling file extensions and drivers

Sean Gillies
 

Hi Hannes,

We don't have such a mapping in Fiona. GDAL/OGR has one tucked away inside the metadata of format drivers. See https://github.com/OSGeo/gdal/blob/master/gdal/ogr/ogrsf_frmts/geojson/ogrgeojsonseqdriver.cpp#L849 for example. It might make sense to expose GDAL's mapping in some way. A consideration: Fiona would only be able to map extensions to drivers for currently enabled drivers at runtime. One might scrape the GDAL code to make a mapping for *all* possible supported drivers, but then some of these wouldn't be available to users at runtime, depending on the GDAL distribution they've installed.

In case you hadn't seen, Python has a mimetypes module already. Its how I think things should work in a perfect world where GDAL understood mimetypes.

>>> import mimetypes
>>> mimetypes.types_map['.json']
'application/json'
>>> mimetypes.types_map['.tif']
'image/tiff'

but it doesn't have the geospatial formats because we "geo-web" people haven't bothered to register them.

>>> mimetypes.types_map['.shp']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: '.shp'
>>> mimetypes.types_map['.gpkg']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: '.gpkg'

GeoJSON *is* registered, but I think the mimetypes data needs an update.

>>> mimetypes.types_map['.geojson'] == 'application/geo+json'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: '.geojson'

On Wed, Apr 3, 2019, 11:08 AM <hannes-fiona.groups.io@...> wrote:
For a Fiona-based command line tool I am wondering how to best to handle user-selected output file formats and drivers.

Is there a ready-to-use way to select the driver based on the file extension? Maybe via a mapping of driver -> file extension?

E.g. if a user specified "out.geojson", the driver should be GeoJSON, if "out.shp" then use "ESRI Shapefile", etc.
_._,_._,_


Re: Handling file extensions and drivers

hannes-fiona.groups.io@...
 

Thanks!

I hadn't thought of mime types at all. That would be perfect but as you say, not suitable because all the formats would need to be there.

I ended up writing a small dictionary of mappings between some popular driver names and possible file extensions (as decided by me):

```
driver_extensions = {
    "ESRI Shapefile": [".shp"],
    "GeoJSON": [".json", ".geojson"],
    "GPKG": [".gpkg"],
    "GML": [".gml", ".xml"],
}
```

Filtering the list of supported drivers to just those known ones can now be done with a fancy dict comprehension like:

```
drivers = {
    driver: modes
    for driver, modes in fiona.supported_drivers.items()
    if driver_extensions.get(driver)
}
```

And if needed a flat list of the extensions is a double list comprehension like:

```
extensions = [
    ext
    for sublist in extensions.values()
    for ext in sublist
]
```
-> `['.shp', '.json', '.geojson', '.gpkg', '.gml', '.xml']`


Rasterio-style GDAL bundling

michaelm@...
 

Hi all,

I'm looking to use Fiona in a project to provide Python geopackage support, but would like to avoid users having to independently install GDAL to make it work, i.e. I'd like to make self-contained binary wheels. There's a nice approach in rasterio where it bundles its own GDAL. Would something like that be possible in Fiona? I'd be happy just for some pointers on how to develop that functionality, whether it makes it into the main branch or not.

Thanks and regards,
Michael


Re: Rasterio-style GDAL bundling

Sean Gillies
 

Hi Michael,

The FIona and Rasterio wheels on the Python package index bundle GDAL in the same way. The wheel building infrastructure runs on Travis CI and is maintained in this repo: https://github.com/sgillies/fiona-wheels. It's based on the software that is used to create wheels for Numpy and SciPy. We only provide wheels for OS X and manylinux (version 1) at this time.

Yours,


On Tue, Jun 4, 2019 at 7:25 AM <michaelm@...> wrote:
Hi all,

I'm looking to use Fiona in a project to provide Python geopackage support, but would like to avoid users having to independently install GDAL to make it work, i.e. I'd like to make self-contained binary wheels. There's a nice approach in rasterio where it bundles its own GDAL. Would something like that be possible in Fiona? I'd be happy just for some pointers on how to develop that functionality, whether it makes it into the main branch or not.

Thanks and regards,
Michael



--
Sean Gillies


Re: Rasterio-style GDAL bundling

michaelm@...
 

Hi Sean,

Sorry, I misunderstood the instructions at https://fiona.readthedocs.io/en/latest/README.html#unix-like-systems to mean that I had to have my own GDAL already. It's great that it's already packaged this way :-). I really appreciate it.

Cheers,
Michael


Re: Rasterio-style GDAL bundling

Loïc DUTRIEUX
 

A quick install on a fresh docker ubuntu container confirms that fiona
installation does not require a system gdal.

docker run -it ubuntu /bin/bash
####
apt update
apt install --no-install-recommends python3-dev python3-pip
pip3 install fiona
python3 -c "import fiona; print(fiona.supported_drivers)"

{'AeronavFAA': 'r', 'ARCGEN': 'r', 'BNA': 'raw', 'DXF': 'raw', 'CSV':
'raw', 'OpenFileGDB': 'r', 'ESRIJSON': 'r', 'ESRI Shapefile': 'raw',
'GeoJSON': 'rw', 'GPKG': 'rw', 'GML': 'raw', 'GPX': 'raw',
'GPSTrackMaker': 'raw', 'Idrisi': 'r', 'MapInfo File': 'raw', 'DGN':
'raw', 'S57': 'r', 'SEGY': 'r', 'SUA': 'r', 'TopoJSON': 'r'}

Kind regards,
Loïc

On 6/4/19 3:38 PM, Sean Gillies wrote:
Hi Michael,

The FIona and Rasterio wheels on the Python package index bundle GDAL in
the same way. The wheel building infrastructure runs on Travis CI and is
maintained in this repo: https://github.com/sgillies/fiona-wheels. It's
based on the software that is used to create wheels for Numpy and SciPy.
We only provide wheels for OS X and manylinux (version 1) at this time.

Yours,

On Tue, Jun 4, 2019 at 7:25 AM <michaelm@aerobotics.com
<mailto:michaelm@aerobotics.com>> wrote:

Hi all,

I'm looking to use Fiona in a project to provide Python geopackage
support, but would like to avoid users having to independently
install GDAL to make it work, i.e. I'd like to make self-contained
binary wheels. There's a nice approach in rasterio where it bundles
its own GDAL. Would something like that be possible in Fiona? I'd be
happy just for some pointers on how to develop that functionality,
whether it makes it into the main branch or not.

Thanks and regards,
Michael



--
Sean Gillies


How to use fiona.open to overwrite existing GPKG file?

joe.huang@...
 

I'm trying to overwrite a temporary file with fiona,open, but it throws an error. Is there a way to get around this?


Re: How to use fiona.open to overwrite existing GPKG file?

Sean Gillies
 

Because a geopackage file can contain multiple layers, opening a layer in "w" mode does not overwrite the entire .gpkg file. If you want to overwrite a layer, you need to pass "overwrite=True" to fiona.open(). For example:

>>> with fiona.open("/tmp/coutwildrnp.gpkg", "w", driver="GPKG", crs=crs, schema=schema, layer="coutwildrnp", overwrite=True) as col:
...     col.write(feat)
...
>>> with fiona.open("/tmp/coutwildrnp.gpkg", layer="coutwildrnp") as col:
...     len(col)
...
1



On Thu, Jun 6, 2019 at 7:05 AM <joe.huang@...> wrote:
I'm trying to overwrite a temporary file with fiona,open, but it throws an error. Is there a way to get around this?



--
Sean Gillies


In fiona.open(), can I pass a namedtuple into schema, rather than a dict?

joe.huang@...
 

Named schemas are a little more lightweight and seem well-suited to the purpose of the schema dict.


Re: In fiona.open(), can I pass a namedtuple into schema, rather than a dict?

Sean Gillies
 

No, namedtuples aren't supported. fiona.open() only accepts a dict or ordered dict. We're switching from dicts to custom classes in several other places in Fiona, but haven't made plans to do so for the schema.

On Thu, Jun 13, 2019 at 2:32 PM <joe.huang@...> wrote:
Named schemas are a little more lightweight and seem well-suited to the purpose of the schema dict.



--
Sean Gillies


How do I use fiona.open() to write to a file-like object like StringIO or BytesIO?

joe.huang@...
 

I try passing the object directly into fiona.open(), but it throws a type error: string argument expected, got 'bytes'


Re: How do I use fiona.open() to write to a file-like object like StringIO or BytesIO?

Sean Gillies
 

Hi,

The project has an example for you in the test suite: https://github.com/Toblerity/Fiona/blob/master/tests/test_memoryfile.py#L69-L71.

Are you using the latest version of fiona? Can you show us the full exception traceback so we can see what line it's coming from?


On Tue, Jun 18, 2019 at 4:23 PM <joe.huang@...> wrote:
I try passing the object directly into fiona.open(), but it throws a type error: string argument expected, got 'bytes'



--
Sean Gillies


Re: How do I use fiona.open() to write to a file-like object like StringIO or BytesIO?

joe.huang@...
 

Your example did the trick. Thank you so much!

By the way, thank you so much for being so quick and responsive! You are an awesome member of the open source community.


When Fiona opens a directory of shapefiles, does it detect if a file is missing? If so, how do I catch the error?

joe.huang@...
 

My application requires that uploaded shapefiles require .shp, .dbf, .shx, and .prj. Is there a way that fiona.open() checks for that?


Re: When Fiona opens a directory of shapefiles, does it detect if a file is missing? If so, how do I catch the error?

Sean Gillies
 

GDAL, the library that fiona uses, can succeed in the absence of some shapefile members and won't necessarily report their absence. You'll need to validate the data using other methods.

On Wed, Jun 19, 2019 at 7:39 PM <joe.huang@...> wrote:
My application requires that uploaded shapefiles require .shp, .dbf, .shx, and .prj. Is there a way that fiona.open() checks for that?



--
Sean Gillies


How would I use fiona to detect if a GeoPackage layer is a raster?

joe.huang@...
 

Specifically, once I have opened the file and am accessing the data with fiona.Collection


Re: How would I use fiona to detect if a GeoPackage layer is a raster?

Sean Gillies
 

Hi,

On Tue, Jul 2, 2019 at 6:25 PM <joe.huang@...> wrote:
Specifically, once I have opened the file and am accessing the data with fiona.Collection

Good question. To be honest, I don't know. I rarely see geopackage files at work and never see ones that combine vectors and rasters, so I haven't explored the situation. What happens if you try to open a raster layer with fiona?

--
Sean Gillies


Fiona 1.8.8

Sean Gillies
 

Hi all,

Fiona 1.8.8 is on PyPI now. Please see the change log for notes about the bug fixes and future changes in 1.9.0 and 2.0.0.


This is likely to be one of the the very last 1.8.x releases.

--
Sean Gillies


Re: Fiona 1.8.8

Sean Gillies
 

Oops, I forgot to mention that the binary wheels on PyPI for 1.8.8 now include GDAL 2.4.2 and are lighter. We're down to about 85 MB extracted size, including GDAL and PROJ data files.

On Wed, Sep 25, 2019 at 10:06 AM Sean Gillies via Groups.Io <sean.gillies=gmail.com@groups.io> wrote:
Hi all,

Fiona 1.8.8 is on PyPI now. Please see the change log for notes about the bug fixes and future changes in 1.9.0 and 2.0.0.


This is likely to be one of the the very last 1.8.x releases.

--
Sean Gillies



--
Sean Gillies

1 - 20 of 104