Date   

Re: Using Fiona to consume ESRI FeatureServer service

Chander Ganesan
 

FYI, I've also tried to use fiona.Env(FEATURE_SERVER_PAGING='true') as a context processor - that doesn't work either.


Using Fiona to consume ESRI FeatureServer service

Chander Ganesan
 

Hello,

I'm attempting to use fiona to consume an ESRI FeatureServer service - I've pased an example call below..  The issue is that OGR's documentation indicates that so long as I omit the resultOffset parameter for the service, it will handle paging for me.  Case in point - OGRInfo returns the proper, expected count of 2000+ records.

However, when I use Fiona (or, in my case geopandas, which uses Fiona) I find that the paging doesn't happen - I get 1000 records and that's it.

Is there an option/argument to get fiona to handle paging of results with the service, as OGR does?

Thanks! 


Re: fiona.transform.transform - different behavior based on GDAL version

calebrob6@...
 

Thanks! For completeness, I just opened it here https://github.com/Toblerity/Fiona/issues/919.

Best,
Caleb


Re: fiona.transform.transform - different behavior based on GDAL version

Alan Snow
 

I am betting it has to do with axis order changes with PROJ 6+ and GDAL 3+:
You can see a similar issue raised for pyproj here: https://github.com/pyproj4/pyproj/issues/538


Re: fiona.transform.transform - different behavior based on GDAL version

Joshua Arnott
 

Hi Caleb,

I can reproduce your issue with the master branch on Travis by adding a test based on the code you provided.


Please can you raise an issue on GitHub for this? Include your examples. https://github.com/Toblerity/Fiona/issues

There are a few places this problem could be coming from as it involves the interaction with Fiona, GDAL and PROJ.

Josh


On Tue, 14 Jul 2020 at 20:32, <calebrob6@...> wrote:
Hi all,

I'm observing different behavior from fiona.transform.transform based on the underlying gdal version. 

Steps to reproduce
```
conda create --name test python gdal fiona # this installs gdal 3.0.2 and fiona 1.8.13.post1
conda activate test
python
>>> import fiona.transform
>>> fiona.transform.transform("epsg:4326", "epsg:3857", [-75.71], [38.06])
([4236819.819591993], [-13244913.18166699])
```

If I switch the lat/lon position then I get the correct result (http://epsg.io/transform#s_srs=4326&t_srs=3857&x=-75.7169053&y=38.0624956):
```
>>> fiona.transform.transform("epsg:4326", "epsg:3857", [38.06], [-75.71])
([-8427998.647958742], [4587905.271362515])
```

If I instead do the conda install with a gdal version less than 3 I get the expected result:
```
conda create --name test2 python "gdal<3" fiona # this installs gdal 2.3.3 and fiona 1.8.4
conda activate test2
python
>>> fiona.transform.transform("epsg:4326", "epsg:3857", [-75.71], [38.06])
([-8427998.647958742], [4587905.27136252])
```

Is this known / should I open an Issue on github?

Thanks,
Caleb


fiona.transform.transform - different behavior based on GDAL version

calebrob6@...
 

Hi all,

I'm observing different behavior from fiona.transform.transform based on the underlying gdal version. 

Steps to reproduce
```
conda create --name test python gdal fiona # this installs gdal 3.0.2 and fiona 1.8.13.post1
conda activate test
python
>>> import fiona.transform
>>> fiona.transform.transform("epsg:4326", "epsg:3857", [-75.71], [38.06])
([4236819.819591993], [-13244913.18166699])
```

If I switch the lat/lon position then I get the correct result (http://epsg.io/transform#s_srs=4326&t_srs=3857&x=-75.7169053&y=38.0624956):
```
>>> fiona.transform.transform("epsg:4326", "epsg:3857", [38.06], [-75.71])
([-8427998.647958742], [4587905.271362515])
```

If I instead do the conda install with a gdal version less than 3 I get the expected result:
```
conda create --name test2 python "gdal<3" fiona # this installs gdal 2.3.3 and fiona 1.8.4
conda activate test2
python
>>> fiona.transform.transform("epsg:4326", "epsg:3857", [-75.71], [38.06])
([-8427998.647958742], [4587905.27136252])
```

Is this known / should I open an Issue on github?

Thanks,
Caleb


RFC: Roadmap for Shapely 2.0

Joris Van den Bossche
 

Dear list,

This is not directly a Fiona-related message, but I assume there might be some of you that are also Shapely users. For those, I want to get across a message about a proposal for Shapely 2.0 (and if not, you can skip the rest of the message ;)).

There is a proposal for some significant changes in Shapely: "RFC 1: Roadmap for Shapely 2.0" (https://github.com/shapely/shapely-rfc/pull/1).

Background: we have been working on a new set of Python bindings to GEOS in the PyGEOS package (https://github.com/pygeos/pygeos/). The initial focus was to make it easier and more performant to work with arrays of geometries (where Shapely is now focused on scalar geometries). See https://caspervdw.github.io/Introducing-Pygeos/ for some more details. While this separate package made it easier to experiment (and which we are still doing), on the long term, it's not ideal to have two incompatible GEOS wrappers in the Python ecosystem. Therefore, the proposal linked above is about upstreaming this work into Shapely, along with some API changes.

Given that this are some profound changes to Shapely, feedback from Shapely users would be very much appreciated! 

Best,
Joris



Fiona 1.9 Release Date

Alan Snow
 

Is there a tentative release date for Fiona 1.9? (This question is purely out of curiosity and not out of need.)


Re: pip install succeeds, ImportError on import

gavinswanson@...
 

I went through a long process of doing that yesterday. Same result.
I also noticed while doing it that appveyor is pulling the latest dev build of gdal, rather than the latest release, but not for the libs. Is this intentional?

Link to latest dev:

https://github.com/Toblerity/Fiona/blob/71f34cfa914d4ca218c395657366042823b8969b/appveyor.yml#L77

As in here: http://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-mapserver.zip

Link to stable daily libs:

https://github.com/Toblerity/Fiona/blob/71f34cfa914d4ca218c395657366042823b8969b/appveyor.yml#L78

as in here: http://www.gisinternals.com/query.html?content=filelist&file=release-1911-x64-gdal-3-0-mapserver-7-4.zip

In the interest of getting the actual project moving I did break down and start from scratch on an Anaconda env, which ended up working for me. I was having issues initially, that's why I went down the path of pip install/build for everything.


Understood that currently pre-built binaries are the way to go... But shouldn't the goal be at least pre-built wheels?

I've got some time to kick this around a little if I can get some assistance.


Re: pip install succeeds, ImportError on import

René Buffat
 

I'm not really familiar with installing Fiona on Windows.  I could imagine that potential issues could be the wrong settings of environment variables, missing Visual C++ installations, wrong location of files or the Python 3.8 dll loading issue.  
I would recommend you to try to stay as close as to our appveyor setup as possible: https://github.com/Toblerity/Fiona/blob/master/appveyor.yml
as this is proven to work. 

As long as you don't intend to modify Fiona itself, there is no need to take the "hard" route. The easy route would be to use binary packages provided by either conda or https://www.lfd.uci.edu/~gohlke/pythonlibs/


pip install succeeds, ImportError on import

gavinswanson@...
 

Trying to work out some issues importing fiona.

Python 3.8.2
GDAL: http://download.gisinternals.com/sdk/downloads/release-1911-x64-gdal-3-0-4-mapserver-7-4-3.zip and associated libs
Everything installed via pip
Doing the `add_ddl_directory` call (already a pending PR for python 3.8) manually gets past the failure to find the module.
Now it fails to find a procedure, and I'm at a complete loss for how to hunt down the issue further.
Dependency walker run against `ogrext.cp38-win_amd64.pyd` shows issues, but I'm at a loss for how to solve them.

Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.add_dll_directory('c:\program files\gdal')
<AddedDllDirectory('c:\\program files\\gdal')>
>>> import fiona
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\gavin\covid_env\lib\site-packages\fiona\__init__.py", line 84, in <module>
    from fiona.collection import BytesCollection, Collection
  File "C:\Users\gavin\covid_env\lib\site-packages\fiona\collection.py", line 9, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: DLL load failed while importing ogrext: The specified procedure could not be found.




Re: KML SimpleData

Bonzo
 

Thanks Sean, much appreciated!  (thanks for the coding correction too - I was getting deprecation errors with mine!)

Regards, Bonzo.


Re: KML SimpleData

Sean Gillies
 

Hi Bonzo,

On Wed, Feb 26, 2020 at 9:41 AM Bonzo <bteluk@...> wrote:
Hi, is it possible to to retrieve SimpleData attributes from a KML document?

I've opened a collection, and tried both KML and KMZ documents - and whilst I can see the "normal" KML attributes - I cannot see the "SimpleData" attributes listed in the feature properties...

I tried to see if I could pass an OGR environment variable... but not sure if I got this right:

with fiona.drivers(LIBKML_USE_SCHEMADATA=True):
  cll = fiona.open('/vsizip/' + fnSrc + '/doc.kml',mode='r',layer='QFESCurrentIncidents')
  f1 = cll.next()
  f1['properties'].keys()
 

I'm using:
Python 2.7.9
Fiona 1.8.13
OS: Linux/Debian 8

Regards, Bonzo.

The modern usage, for fiona 1.8.x, is

import fiona

fiona.drvsupport.supported_drivers["LIBKML"] = "raw"

with fiona.Env(LIBKML_USE_SCHEMADATA=True):
    with fiona.open("file.kmz", layer="QFESCurrentIncidents") as collection:
        first = next(collection)

However, if you're using the fiona binary wheels installed from the Python package index using pip, you're out of luck. OGR's LIBKML driver, documented at https://gdal.org/drivers/vector/libkml.html, is not a default format driver and is not included in the binary wheels. This situation might explain why you don't see the SimpleData attributes.

If that's the case, I recommend installing the latest gdal-dev package from DebianGIS (I think this is very likely to entail libkml), and then installing fiona from a non-binary distribution like `pip install -I --no-binary fiona fiona`. Or get the python-fiona Debian package https://packages.debian.org/search?searchon=names&keywords=python-fiona.

I'm sorry about the lack of a LIBKML driver, but the binary wheels would be too big if we included every possible format.

--
Sean Gillies


KML SimpleData

Bonzo
 

Hi, is it possible to to retrieve SimpleData attributes from a KML document?

I've opened a collection, and tried both KML and KMZ documents - and whilst I can see the "normal" KML attributes - I cannot see the "SimpleData" attributes listed in the feature properties...

I tried to see if I could pass an OGR environment variable... but not sure if I got this right:

with fiona.drivers(LIBKML_USE_SCHEMADATA=True):
  cll = fiona.open('/vsizip/' + fnSrc + '/doc.kml',mode='r',layer='QFESCurrentIncidents')
  f1 = cll.next()
  f1['properties'].keys()
 

I'm using:
Python 2.7.9
Fiona 1.8.13
OS: Linux/Debian 8

Regards, Bonzo.


supplying SHAPE_RESTORE_SHX=YES to fiona.open

Ricky Teachey
 

Hello: I hope this is the right place to ask a question. I also hope I am not creating this topic twice; I tried emailing main@fiona.groups.io but it did not work.

I am trying to figure out how to open a .shp while without all the other accompanying files. I THINK want to do this because I believe I don't care about the data in those other files. However I'm also a total beginner in usage of shape files so please let me know if I am wrong about what it is I believe.

My belief is that the geometric information- basically, the XY coordinates- of the shapes I intend to plot (using plotly.express.choropleth) and turn into a little app for my own use is contained in the .shp file. These .shp files are being created for me in Autocad by a coworker; they are not actually maps of any kind, just a bunch of shapes, and .shp is a much more convenient way to save them than .dxf.

I understand that most of the time, you will pass shape files around with other files, such as .shx, and Autocad does create these files. However it would be easier to not worry about these other files for my purposes.

It appears it is possible to do what I want to do by supplying the option of SHAPE_RESTORE_SHX=YES to GDAL. However, I'm trying to do this using a python script with fiona, and not using the CLI.

Assuming what I am wanting to do makes some kind of sense, is there a way to supply the SHAPE_RESTORE_SHX=YES directive to fiona.open so that I can open a .shp file without the other files? I tried just doing this:

fiona.open(pathlib.Path(shp), SHAPE_RESTORE_SHX="YES")

But it doesn't seem to work.

details: Python 3.8.1 Windows 10 Fiona 1.8.13 GDAL 3.0.4 Installed using the wheels at Christoph Gohlke's web page


Re: Unable to open .zip in Docker container

David Johansen <davejohansen@...>
 

On Fri, Feb 21, 2020 at 1:56 PM René Buffat <buffat@...> wrote:
Under linux, you can view shared libraries linked to a binary using ldd.

E.g. ldd /usr/bin/gdalinfo

It looks like gdal is using minizip ( https://packages.debian.org/buster/minizip ) in the debian image and I wonder if it doesn't support the format of the file I'm using:
# ldd /usr/lib/libgdal.so | grep -i z
libsz.so.2 => /usr/lib/x86_64-linux-gnu/libsz.so.2 (0x00007fc51c3bb000)
libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fc51b8aa000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc51b882000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc51a963000)
libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007fc51921c000)

And on Alpine it looks like it's using libz:
# ldd /usr/lib/libgdal.so | grep -i z
libz.so.1 => /lib/libz.so.1 (0x7ff3ca10f000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x7ff3c9999000)
libbz2.so.1 => /usr/lib/libbz2.so.1 (0x7ff3c998a000)


Re: Unable to open .zip in Docker container

René Buffat
 

Under linux, you can view shared libraries linked to a binary using ldd.

E.g. ldd /usr/bin/gdalinfo


Re: Unable to open .zip in Docker container

David Johansen <davejohansen@...>
 

On Wed, Feb 19, 2020 at 7:05 AM David Johansen via Groups.Io <davejohansen=gmail.com@groups.io> wrote:
I'm using fiona 1.8.13 in both case, and for GDAL, on my Mac, it's 2.4.2, in the debian container it's 2.4.0 and in the Alpine container it's 3.0.3.

On Wed, Feb 19, 2020 at 12:18 AM <hannes-fiona.groups.io@...> wrote:
Hi Dave,

are you using the same versions of Fiona and GDAL?

Cheers, Hannes

On Tue, 18 Feb 2020 16:25:57 -0800
davejohansen@... wrote:

> I'm trying to load a .gdb.zip file in a Docker container using
> `fiona.listlayers()` but it says that it's not a supported format
> with the following exception:
>
> ```
>
> File \"fiona/_shim.pyx\", line 83, in fiona._shim.gdal_open_vector
>
> File \"fiona/_err.pyx\", line 270, in fiona._err.exc_wrap_pointer
>
> fiona._err.CPLE_OpenFailedError: '/tmp/test.gdb.zip' not recognized
> as a supported file format.
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>
> File \"/usr/src/app/numetric/connectrunner/shp_file/shp_file.py\",
> line 34, in _listlayers
>
> for layer in fiona.listlayers(filename):
>
> File
> \"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/env.py\",
> line 407, in wrapper
>
> return f(*args, **kwargs)
>
> File
> \"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/__init__.py\",
> line 344, in listlayers
>
> return _listlayers(vsi_path(pobj))
>
> File \"fiona/ogrext.pyx\", line 1512, in fiona.ogrext._listlayers
>
> File \"fiona/_shim.pyx\", line 90, in fiona._shim.gdal_open_vector
> ```
>
> The same code works fine on my Mac with the same file, but doesn't in
> the `3.8.1-slim-buster` and `3.8.1-alpine3.11` images. Any ideas on
> why and what I can do to fix this?
>
> Thanks,
>
> Dave
>
>
>

From playing around with this a bit more, I believe it's from the GDAL not being able to open the .zip file. Is there a way for me to check what compression libraries are available to GDAL? 


Re: Writing to GeoPackage fails when two attributes have the same name (ignoring case)

Sean Gillies
 

Hi Olmo,

On Fri, Feb 21, 2020 at 7:55 AM <olmo.nietosilleras@...> wrote:
Hi,

Fiona throws a "Failed to write record" RuntimeError when trying to write records to a GeoPackage if two or more attributes have equal names ignoring the case. I understand this is because GeoPackage is a SQLite container and that one should in principle not have two columns with the same name (ignoring case). But would it be possible to include a check and output a more specific error message in this particular case to facilitate debugging?

In my case I ran into this problem while performing a merge with geopandas on two keys that differed only by their case, so that the resulting GeoDataFrame contained two columns with the same name (ignoring the case). A simple solution is clearly to drop one of them or to modify the case before merging the two dataframes. But it would be quite useful if the error message directly pointed one to this problem.

Here is the error message I get with geopandas/fiona (the duplicate attributes are "CODE_OBJ" and "code_obj"):

Traceback (most recent call last):
  File "C:\Continuum\miniconda3\envs\geo\lib\site-packages\geopandas\io\file.py", line 130, in to_file
    colxn.writerecords(df.iterfeatures())
  File "C:\Continuum\miniconda3\envs\geo\lib\site-packages\fiona\collection.py", line 342, in writerecords
    self.session.writerecs(records, self)
  File "fiona/ogrext.pyx", line 1198, in fiona.ogrext.WritingSession.writerecs
RuntimeError: Failed to write record: {'id': '0', 'type': 'Feature', 'properties': {'CODE_OBJ': '000028064CC805B1', 'GWSCOD_H': 202, 'REF_ID': 912884578, 'code_obj': '000028064CC805B1'}, 'geometry': {'type': 'Polygon', 'coordinates': (((247632.21, 160821.25), (247683.44, 160754.28), (247728.75, 160791.5), (247726.5, 160796.37), (247813.0, 160869.0), (247843.21, 160827.61000000002), (247768.44, 160770.21), (247854.5, 160678.0), (247856.6, 160675.93), (247791.64, 160617.36000000002), (247782.0, 160609.0), (247780.41, 160607.24), (247772.44, 160601.25), (247615.69, 160806.28), (247632.21, 160821.25)),)}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'failed to prepare SQL: SELECT "fid", ST_MinX("geom"), ST_MaxX("geom"), ST_MinY("geom"), ST_MaxY("geom") FROM "layer0" WHERE "geom" NOT NULL AND NOT ST_IsEmpty("geom")'
Exception ignored in: 'fiona._shim.gdal_flush_cache'
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'failed to prepare SQL: SELECT "fid", ST_MinX("geom"), ST_MaxX("geom"), ST_MinY("geom"), ST_MaxY("geom") FROM "layer0" WHERE "geom" NOT NULL AND NOT ST_IsEmpty("geom")'
(This problem does not arise when writing to a shapefile or to GeoJSON.)

As a point of comparison, QGIS for example outputs the following error when trying to add a second column with the same name (ignoring case)

OGR error creating field CODE_OBJ: sqlite3_exec(ALTER TABLE "layer0" ADD COLUMN "CODE_OBJ" TEXT(80)) failed: duplicate column name: CODE_OBJ
Thanks,
Olmo

A better error message would be good, indeed. I can see in fiona where we may be hiding it and will put this on the list of issues to address. Thanks!

--
Sean Gillies


Writing to GeoPackage fails when two attributes have the same name (ignoring case)

olmo.nietosilleras@...
 

Hi,

Fiona throws a "Failed to write record" RuntimeError when trying to write records to a GeoPackage if two or more attributes have equal names ignoring the case. I understand this is because GeoPackage is a SQLite container and that one should in principle not have two columns with the same name (ignoring case). But would it be possible to include a check and output a more specific error message in this particular case to facilitate debugging?

In my case I ran into this problem while performing a merge with geopandas on two keys that differed only by their case, so that the resulting GeoDataFrame contained two columns with the same name (ignoring the case). A simple solution is clearly to drop one of them or to modify the case before merging the two dataframes. But it would be quite useful if the error message directly pointed one to this problem.

Here is the error message I get with geopandas/fiona (the duplicate attributes are "CODE_OBJ" and "code_obj"):

Traceback (most recent call last):
  File "C:\Continuum\miniconda3\envs\geo\lib\site-packages\geopandas\io\file.py", line 130, in to_file
    colxn.writerecords(df.iterfeatures())
  File "C:\Continuum\miniconda3\envs\geo\lib\site-packages\fiona\collection.py", line 342, in writerecords
    self.session.writerecs(records, self)
  File "fiona/ogrext.pyx", line 1198, in fiona.ogrext.WritingSession.writerecs
RuntimeError: Failed to write record: {'id': '0', 'type': 'Feature', 'properties': {'CODE_OBJ': '000028064CC805B1', 'GWSCOD_H': 202, 'REF_ID': 912884578, 'code_obj': '000028064CC805B1'}, 'geometry': {'type': 'Polygon', 'coordinates': (((247632.21, 160821.25), (247683.44, 160754.28), (247728.75, 160791.5), (247726.5, 160796.37), (247813.0, 160869.0), (247843.21, 160827.61000000002), (247768.44, 160770.21), (247854.5, 160678.0), (247856.6, 160675.93), (247791.64, 160617.36000000002), (247782.0, 160609.0), (247780.41, 160607.24), (247772.44, 160601.25), (247615.69, 160806.28), (247632.21, 160821.25)),)}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'failed to prepare SQL: SELECT "fid", ST_MinX("geom"), ST_MaxX("geom"), ST_MinY("geom"), ST_MaxY("geom") FROM "layer0" WHERE "geom" NOT NULL AND NOT ST_IsEmpty("geom")'
Exception ignored in: 'fiona._shim.gdal_flush_cache'
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'failed to prepare SQL: SELECT "fid", ST_MinX("geom"), ST_MaxX("geom"), ST_MinY("geom"), ST_MaxY("geom") FROM "layer0" WHERE "geom" NOT NULL AND NOT ST_IsEmpty("geom")'
(This problem does not arise when writing to a shapefile or to GeoJSON.)

As a point of comparison, QGIS for example outputs the following error when trying to add a second column with the same name (ignoring case)

OGR error creating field CODE_OBJ: sqlite3_exec(ALTER TABLE "layer0" ADD COLUMN "CODE_OBJ" TEXT(80)) failed: duplicate column name: CODE_OBJ
Thanks,
Olmo

41 - 60 of 104